The natural statistics of audiovisual speech

PLoS Computational Biology

Volumn 5, Issue 7, 2009, Pages

  Chandrasekaran, Chandramouli ^a   Trubanova, Andrea ^a   Stillittano, Sébastien ^b   Caplier, Alice ^b   Ghazanfar, Asif A ^a  

^a PRINCETON UNIVERSITY   (United States)

^b GIPSA LAB   (France)

Author keywords

[No Author keywords available]

Indexed keywords

SIGNAL RECEIVERS;

AUDIO-VISUAL SPEECH; COMPLEX INPUTS; DYNAMIC TIME; EXPOSED TO; HUMAN LIKE; INPUT SPACE; MULTI-MODAL; MULTISENSORY; SENSORY SYSTEM; TIME VARYING;

SPEECH COMMUNICATION;

ACOUSTICS; ARTICLE; AUDITORY DISCRIMINATION; CONSONANT; CORRELATION ANALYSIS; DATA BASE; HUMAN; INFORMATION; INTERPERSONAL COMMUNICATION; MOUTH; OSCILLATION; PERCEPTION; SENSORY EVALUATION; SIGNAL PROCESSING; SIGNAL TRANSDUCTION; SPECTROSCOPY; SPEECH; STATISTICS; STIMULUS; THEORY; TIME PERCEPTION; VISUAL DISCRIMINATION; VOICE; X RAY; FACTUAL DATABASE; FOURIER ANALYSIS; IMAGE PROCESSING; METHODOLOGY; PHYSIOLOGY; SPEECH PERCEPTION; STATISTICAL MODEL; TIME; VISION;

ANIMALIA;

DATABASES, FACTUAL; FOURIER ANALYSIS; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MODELS, STATISTICAL; MOUTH; SIGNAL PROCESSING, COMPUTER-ASSISTED; SPEECH; SPEECH PERCEPTION; TIME FACTORS; VISUAL PERCEPTION; VOICE;

EID: 68249124778     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1000436     Document Type: Article

References (82)

1. Pollack GS (2001) Analysis of temporal patterns of communication signals. Curr Opin Neurobiol 11: 734-738.
2. Kiebel SJ, Daunizeau J, Friston KJ (2008) A hierarchy of time-scales and the brain. PLoS Comput Biol 4: e1000209.
3. Barlow H (1961) Possible principles underlying the transformation of sensory messages. In: Rosenblith W, ed. Sensory Communication. Cambridge, MA: MIT Press. pp 217-234.
4. Lungarella M, Sporns O (2006) Mapping Information Flow in Sensorimotor Networks. PLoS Comput Biol 2: e144. (Pubitemid 44656523)
5. Smith L, Gasser M (2005) The development of embodied cognition: six lessons from babies. Artificial Life 11: 13-29. (Pubitemid 40394094)
6. Rosenblum LD (2005) Primacy of Multimodal Speech Perception. In: Pisoni DB, Remez RE, eds. The Handbook of Speech Perception: Blackwell publishing.
7. Campbell R (2008) The processing of audio-visual speech: empirical and neural bases. Philos Trans R Soc Lond B Biol Sci 363: 1001-1010. (Pubitemid 351158927)
8. Simoncelli EP, Olshausen BA (2001) Natural Image Statistics and Neural Representation. Annu Rev Neurosci 24: 1193-1216.
9. Singh NC, Theunissen FE (2003) Modulation spectra of natural sounds and ethological theories of auditory processing. J Acoust Soc Am 114: 3394-3411.
10. Mitra PP, Pesaran B (1999) Analysis of Dynamic Brain Imaging Data. Biophys J 76: 691-708.
11. Poeppel D (2003) The analysis of speech in different temporal integration windows: cerebral lateralization as 'asymmetric sampling in time'. Speech Comm 41: 245-255.
12. Schroeder CE, Lakatos P, Kajikawa Y, Partan S, Puce A (2008) Neuronal oscillations and visual amplification of speech. Trends Cogn Sci 12: 106-113.
13. Summerfield Q (1992) Lipreading and Audio-Visual Speech Perception. Philos Trans R Soc Lond B Biol Sci 335: 71-78.
14. Greenberg S, Carvey H, Hitchcock L, Chang S (2003) Temporal properties of spontaneous speech - a syllable-centric perspective. J Phon 31: 465-485. (Pubitemid 37495920)
15. Rosenblum LD (2008) Speech Perception as a Multimodal Phenomenon. Curr Dir Psychol Sci 17: 405-409.
16. Cooke M, Barker J, Cunningham S, Shao X (2006) An audio-visual corpus for speech perception and automatic speech recognition. J Acoust Soc Am 120: 2421-2424. (Pubitemid 44631681)
17. Garofolo JS, Lamel LF, Fisher WM, Fiscus JG, Pallett DS, et al. (1993) The DARPA TIMIT Acoustic Phonetic Continuous Speech Corpus CDROM. NTIS order number PB91-100354 ed.
18. Westbury J (1994) X Ray Microbeam Speech Production Database. Madison, WI: University of Wisconsin.
19. Sodoyer D, Rivet B, Girin L, Savariaux C, Schwartz J-L, et al. (2009) A study of lip movements during spontaneous dialog and its application to voice activity detection. J Acoust Soc Am 125: 1184-1196.
20. Eveno N, Caplier A, Coulon YP (2004) Accurate and Quasi-Automatic Lip Tracking. IEEE T Circ Syst Vid 14: 706-715.
21. Stillittano S, Caplier A (2008) Inner Lip Segmentation by Combining Active Contours and Parametric Models. VISAPP 2008 - International Conference on Computer Vision Theory and Applications. Madeira, Portugal.
22. Lallouache T. Un poste visage-parole. Acquisition et traitement des contours labiaux (A device for the capture and processing of lip contours); 1990; Montreal, Canada, 282-286.
23. Smith ZM, Delgutte B, Oxenham AJ (2002) Chimaeric sounds reveal dichotomies in auditory perception. Nature 416: 87-90. (Pubitemid 34215405)
24. Drullman R (1995) Temporal envelope and fine structure cues for speech intelligibility. J Acoust Soc Am 97: 585-592.
25. Chandrasekaran C, Ghazanfar AA (2009) Different Neural Frequency Bands Integrate Faces and Voices Differently in the Superior Temporal Sulcus. J Neurophysiol 101: 773-788.
26. Lakatos P, Chen C-M, O'Connell MN, Mills A, Schroeder CE (2007) Neuronal Oscillations and Multisensory Interaction in Primary Auditory Cortex. Neuron 53: 279-292. (Pubitemid 46074398)
27. Maier JX, Chandrasekaran C, Ghazanfar AA (2008) Integration of Bimodal Looming Signals through Neuronal Coherence in the Temporal Lobe. Curr Biol 18: 963-968.
28. Maier JX, Ghazanfar AA (2007) Looming biases in monkey auditory cortex. J Neurosci 27: 4093-4100.
29. Jarvis MR, Mitra PP (2001) Sampling Properties of the Spectrum and Coherency of Sequences of Action Potentials. Neural Computation 13: 717-749.
30. Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK (2005) Multisensory Integration of Dynamic Faces and Voices in Rhesus Monkey Auditory Cortex. J Neurosci 25: 5004-5012.
31. van Wassenhove V, Grant KW, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci U S A 102: 1181-1186. (Pubitemid 40170746)
32. van Wassenhove V, Grant KW, Poeppel D (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45: 598-607. (Pubitemid 44894453)
33. Grant KW, Seitz P-F (2000) The use of visible speech cues for improving auditory detection of spoken sentences. J Acoust Soc Am 108: 1197-1208.
34. Rosenblum LD, Saldana HM (1998) Time-varying information for visual speech perception. In: Campbell R, Dodd B, Burnham D, eds. Hearing by Eye: Part 2, The Psychology of Speechreading and Audiovisual Speech. Hillsdale, New Jersey: Lawrence Earlbaum. pp 61-81.
35. Munhall K, Vatikiotis-Bateson E (1998) The moving face during speech communication. In: Campbell R, Dodd B, Burnham D, eds. Hearing by Eye II. Sussex: Taylor and Francis. pp 123-139.
36. Sams M, Aulanko R, Hamalainen M, Hari R, Lounasmaa OV, et al. (1991) Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci Lett 127: 141-145.
37. McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264: 746-748.
38. Ghazanfar AA, Nielsen K, Logothetis NK (2006) Eye movements of monkey observers viewing vocalizing conspecifics. Cognition 101: 515-529. (Pubitemid 44511927)
39. Lansing CR, McConkie GW (2003) Word identification and eye fixation locations in visual and visual-plus-auditory presentations of spoken sentences. Percept Psychophys 65: 536-552.
40. Kim J, Davis C (2004) Investigating the audio-visual speech detection advantage. Speech Comm 44: 19-30.
41. Rosenblum LD, Johnson JA, Saldana HM (1996) Point-Light Facial Displays Enhance Comprehension of Speech in Noise. J Speech Hear Res 39: 1159-1170. (Pubitemid 26406242)
42. Rosenblum LD, Saldana HM (1996) An audiovisual test of kinematic primitives for visual speech perception. J Exp Psychol Hum Percept Perform 22: 318-331.
43. Munhall K, Vatikiotis-Bateson E (2004) Spatial and Temporal Constraints on Audiovisual Speech Perception. In: Calvert GA, Spence C, Stein BE, eds. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press.
44. Sodoyer D, Schwartz J-L, Girin L, Klinkisch J, Jutten C (2002) Separation of audio-visual speech sources: a new approach exploiting the audio-visual coherence of speech stimuli. EURASIP J Appl Signal Process. pp 1165-1173.
45. Monaci G, Vandergheynst P (2006) Audiovisual Gestalts. New York: Computer Vision and Pattern Recognition Workshop.
46. Munhall K (2006) Speech Production: The Force of Your Words. Curr Biol 16: R922-R923.
47. Lieberman P, Blumstein SE (1988) Speech physiology, speech perception, and acoustic phonetics. Cambridge: Cambridge University Press. pp 1-249.
48. Yehia H, Rubin P, Vatikiotis-Bateson E (1998) Quantitative association of vocaltract and facial behavior. Speech Comm 26: 23-43. (Pubitemid 128381217)
49. Munhall KG, Jones JA, Callan DE, Kuratate T, Vatikiotis-Bateson E (2004) Visual prosody and speech intelligibility - Head movement improves auditory speech perception. Psych Sci 15: 133-137. (Pubitemid 38361306)
50. Yehia HC, Kuratate T, Vatikiotis-Bateson E (2002) Linking facial animation, head motion and speech acoustics. J Phon 30: 555-568.
51. Kamachi M, Hill H, Lander K, Vatikiotis-Bateson E (2003) "Putting the face to the voice": matching identity across modality. Curr Biol 13: 1709-1714.
52. von Kriegstein K, Giraud AL (2006) Implicit multisensory associations influence voice recognition. PLoS Biol 4: 1809-1820.
53. Summerfield Q (1987) Some preliminaries to a comprehensive account of audiovisual speech perception. In: Dodd B, Campbell R, eds. Hearing by Eye: The Psychology of Lipreading. Hillsdale, New Jersey: Lawrence Earlbaum. pp 3-51.
54. Drullman R, Festen JM, Plomp R (1994) Effect of reducing slow temporal modulations on speech reception. J Acoust Soc Am 95: 2670-2680. (Pubitemid 24152861)
55. Shannon RV, Zeng F-G, Kamath V, Wygonski J, Ekelid M (1995) Speech Recognition with Primarily Temporal Cues. Science 270: 303-304.
56. Ahissar E, Nagarajan S, Ahissar M, Protopapas A, Mahncke H, et al. (2001) Speech comprehension is correlated with temporal response patterns recorded from auditory cortex. Proc Natl Acad Sci U S A 98: 13367-13372. (Pubitemid 33051369)
57. Suppes P, Han B (2000) Brain-wave representation of words by superposition of a few sine waves. Proc Natl Acad Sci U S A 97: 8738-8743. (Pubitemid 30639743)
58. Luo H, Poeppel D (2007) Phase Patterns of Neuronal Responses Reliably Discriminate Speech in Human Auditory Cortex. Neuron 54: 1001-1010. (Pubitemid 46907508)
59. Ohala J (1975) Temporal Regulation of Speech. In: Fant G, Tatham MAA, eds. Auditory Analysis and Perception of Speech. London: Academic Press.
60. Vitkovitch M, Barber P (1994) Effect of Video Frame Rate on Subjects' Ability to Shadow One of Two Competing Verbal Passages. J Speech Hear Res 37: 1204-1210. (Pubitemid 24307885)
61. Vitkovitch M, Barber P (1996) Visible Speech as a Function of Image Quality: Effects of Display Parameters on Lipreading Ability. Appl Cogn Psychol 10: 121-140. (Pubitemid 126482118)
62. Ghazanfar AA (2009) The multisensory roles for auditory cortex in primate vocal communication. Hear Res;doi:10.1016/j.heares.2009.04.003.
63. Ghazanfar AA, Schroeder CE (2006) Is neocortex essentially multisensory? Trends Cogn Sci 10: 278-285.
64. Lakatos P, Shah AS, Knuth KH, Ulbert I, Karmos G, et al. (2005) An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. J Neurophysiol 94: 1904-1911. (Pubitemid 41192862)
65. Canolty RT, Edwards E, Dalal SS, Soltani M, Nagarajan SS, et al. (2006) High gamma power is phase-locked to theta oscillations in human neocortex. Science 313: 1626-1628. (Pubitemid 44414034)
66. Giraud AL, Kleinschmidt A, Poeppel D, Lund TE, Frackowiak RSJ, et al. (2007) Endogenous cortical rhythms determine cerebral specialization for speech perception and production. Neuron 56: 1127-1134. (Pubitemid 350251110)
67. Saberi K, Perrott DR (1999) Cognitive restoration of reversed speech. Nature 398: 760-1760
68. Dixon NF, Spitz L (1980) The Detection of Auditory Visual Desynchrony. Perception. pp 719-721.
69. Hirsh IJ, Sherrick CE (1961) Perceived order in different sense modalities. J Exp Psychol 62: 423-432.
70. Munhall KG, Gribble P, Sacco L, Ward M (1996) Temporal constraints on the McGurk effect. Percept Psychophys 58: 351-362.
71. Kayser C, Petkov CI, Logothetis NK (2008) Visual Modulation of Neurons in Auditory Cortex. Cereb Cortex 18: 1560-1574. (Pubitemid 351882035)
72. Vatikiotis-Bateson E, Eigsti IM, Yano S, Munhall KG (1998) Eye movement of perceivers during audiovisual speech perception. Percept Psychophys 60: 926-940. (Pubitemid 128462142)
73. Ghazanfar AA, Chandrasekaran CF (2007) Paving the Way Forward: Integrating the Senses through Phase-Resetting of Cortical Oscillations. Neuron 53: 162-164.
74. Werner-Reiss U, Kelly KA, Trause AS, Underhill AM, Groh JM (2003) Eye position affects activity in primary auditory cortex of primates. Curr Biol 13: 554-562. (Pubitemid 36391932)
75. Fu KMG, Shah AS, O'Connell MN, McGinnis T, Eckholdt H, et al. (2004) Timing and laminar profile of eye-position effects on auditory responses in primate auditory cortex. J Neurophysiol 92: 3522-3531. (Pubitemid 39506328)
76. Rajkai C, Lakatos P, Chen C-M, Pincze Z, Karmos G, et al. (2008) Transient Cortical Excitation at the Onset of Visual Fixation. Cereb Cortex 18: 200-209. (Pubitemid 350233066)
77. Abry C, Lallouache M-T, Cathiard M-A (1996) How can coarticulation models account for speech sensitivity in audio-visual desynchronization? In: Stork D, Henneke M, eds. Speechreading by humans and machines: models, systems and applications. Berlin: Springer-Verlag. pp 247-255.
78. Cosi P, Caldognetto EM (1995) Lips and Jaw Movements for Vowels and Consonants: Spatio-Temporal Characteristics and Bimodal Recognition Applications. In: Stork D, Henneke M, eds. Speechreading by humans and machines: models, systems and applications. Berlin: Springer-verlag. pp 291-313.
79. Pfeifer R, Scheier C (1999) Understanding intelligence. Cambridge, MA: MIT Press.
80. Ghazanfar AA, Chandrasekaran C, Logothetis NK (2008) Interactions between the Superior Temporal Sulcus and Auditory Cortex Mediate Dynamic Face/Voice Integration in Rhesus Monkeys. J Neurosci 28: 4457-4469.
81. von Kriegstein K, Dogan O, Gruter M, Giraud AL, Kell CA, et al. (2008) Simulation of talking faces in the human brain improves auditory speech recognition. Proc Natl Acad Sci U S A 105: 6747-6752. (Pubitemid 351754578)
82. Buzsaki G, Draguhn A (2004) Neuronal oscillations in cortical networks. Science 304: 1926-1929.