14th International Congress of Phonetic Sciences (ICPhS-14)

San Francisco, CA, USA
August 1-7, 1999

Prosodic Cues in Multimodal Speech Perception

Björn Granström, David House, Magnus Lundeberg

Centre for Speech Technology, Dept. of Speech, Music and Hearing, KTH, Stockholm, Sweden

Potential visual prosodic cues for prominence and phrasing comprising eyebrow movements were manipulated using a system for audio-visual text-to-speech synthesis which has been implemented based on the KTH rule-based synthesis. Two functions of prosody (prominence and phrasing) were tested in two separate experiments. A test sentence, ambiguous in terms of an internal phrase boundary, was used for both experiments. Acoustic cues and lower face visual cues were held constant for all stimuli. Upper face cues were eyebrow movement where the eyebrows were raised on successive words in the sentence. The results indicate a general coupling between eyebrow raising and perceived prominence while suggesting a more complicated relationship between eyebrow movement and phrasing. The results also point against a tight perceptual connection between F0 and eyebrow movement indicating that selective eyebrow movement can be effective as an independent prosodic cue to prominence.

Full Paper

Bibliographic reference.  Granström, Björn / House, David / Lundeberg, Magnus (1999): "Prosodic cues in multimodal speech perception", In ICPhS-14, 655-658.