EUSIPCO'2002 banner

Paper data
Multimodal signal analysis of prosody and hand motion: temporal correlation of speech and gestures

Valbonesi Lucia, University of Illinois at Chicago
Ansari Rashid, University of Illinois at Chicago
McNeill David, University of Chicago
Quek Francis, Wright State University
Duncan Susan, University of Chicago
McCullough Karl-Eric, University of Chicago

Page numbers in the proceedings:
Volume I pp 75-78

Multimedia Signal Processing: Communication and Data Processing

Paper abstract
This paper is concerned with the processing and analysis of signals pertaining to two communicative modalities, speech and gestures, and investigating the nature of their temporal relationship. Two hypotheses have been proposed about the activation of the gestural system during speech production: the inhibitory hypothesis and the excitatory hypothesis. The validation of either one of these hypotheses necessitates the processing of large amounts of data by experts. The work described here is an effort to develop valuable signal processing tools to facilitate the multi-modal analysis, at least partially, in an automated manner. In this work algorithms are developed to determine points of emphasis in each of the modalities of speech and gesture data using prosody and hand motion traces obtained from two experiments. The results agree with the excitatory hypothesis: if we compare the temporal locations of the speech focal points with the temporal locations of the gesture focal points, they co-occur in more than 90% of the locations.

A PDF version is available here