14th International Congress of Phonetic Sciences (ICPhS-14)

San Francisco, CA, USA
August 1-7, 1999

Evidence of Correlation between Acoustic and Visual Features of Speech

J. P. Barker, Frédéric Berthommier

Institut de la Communication Parlée, UPRESA CNRS No 5009, INPG, Grenoble, France

This paper examines the degree of correlation between lip and jaw configuration and speech acoustics. The lip and jaw positions are characterised by a system of measurements taken from video images of the speaker's face and profile, and the acoustics are represented using line spectral pair parameters and a measure of RMS energy. A correlation is found between the measured acoustic parameters and a linear estimate of the acoustics recovered from the visual data. This correlation exists despite the simplicity of the visual representation and is in rough agreement with correlations measured in earlier work by Yehia et al. using different techniques. However, analysis of the estimation errors suggests that the visual information, as parameterised in our experiment, offers only a weak constraint on the acoustics. Results are discussed from the perspective of models of early audio-visual integration.

Full Paper

Bibliographic reference.  Barker, J. P. / Berthommier, Frédéric (1999): "Evidence of correlation between acoustic and visual features of speech", In ICPhS-14, 199-202.