Simulation of talking faces in the human brain improves auditory speech recognition

Proceedings of the National Academy of Sciences of the United States of America

Volumn 105, Issue 18, 2008, Pages 6747-6752

  Von Kriegstein, Katharina ^a,b   Dogan, Özgür ^c   Grüter, Martina ^d   Giraud, Anne Lise ^e   Kell, Christian A ^c,e   Grüter, Thomas ^d   Kleinschmidt, Andreas ^f,g   Kiebel, Stefan J ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b NEWCASTLE UNIVERSITY   (United Kingdom)

^c GOETHE UNIVERSITY   (Germany)

^d UNIVERSITY OF VIENNA   (Austria)

^e ECOLE NORMALE SUPÉRIEURE   (France)

^f DIF   (France)

^g INSERM   (France)

Author keywords

fMRI; Multisensory; Predictive coding; Prosopagnosia

Indexed keywords

ADULT; ARTICLE; AUDITORY RESPONSE; BRAIN FUNCTION; BRAIN REGION; CONTROLLED STUDY; FACE; FEMALE; HUMAN; HUMAN EXPERIMENT; INTERPERSONAL COMMUNICATION; MALE; NEUROIMAGING; NORMAL HUMAN; PRIORITY JOURNAL; SIMULATION; SPEECH DISCRIMINATION; BEHAVIOR; BRAIN; COMPUTER SIMULATION; HEARING; MIDDLE AGED; NUCLEAR MAGNETIC RESONANCE IMAGING; PHYSIOLOGY; SPEECH PERCEPTION;

ADULT; AUDITORY PERCEPTION; BEHAVIOR; BRAIN; COMMUNICATION; COMPUTER SIMULATION; FACE; FEMALE; HUMANS; MAGNETIC RESONANCE IMAGING; MALE; MIDDLE AGED; SPEECH PERCEPTION;

EID: 44349141662     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.0710826105     Document Type: Article

References (43)

1. Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212-215.
2. van Wassenhove V, Grant KW, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA 102:1181-1186.
3. Schweinberger SR, Robertson D, Kaufmann JM (2007) Hearing facial identities. Q J Exp Psychol (Colchester) 60:1446-1456.
4. Belin P, Fecteau S, Bedard C (2004) Thinking the voice: Neural correlates of voice perception. Trends Cognit Sci 8:129-135.
5. Braida LD (1991) Crossmodal integration in the identification of consonant segments. Q J Exp Psychol A 43:647-677.
6. Sheffert SM, Olson E (2004) Audiovisual speech facilitates voice learning. Percept Psychophys 66:352-362.
7. von Kriegstein K, Kleinschmidt A, Giraud AL (2006) Voice recognition and cross-modal responses to familiar speakers' voices in prosopagnosia. Cereb Cortex 16:1314-1322.
8. von Kriegstein K, Giraud AL (2006) Implicit multisensory associations influence voice recognition. PLoS Biol 4:e326.
9. Deneve S, Duhamel JR, Pouget A (2007) Optimal sensorimotor integration in recurrent cortical networks: A neural implementation of Kalman filters. J Neurosci 27:5744-5756.
10. Friston K (2005) A theory of cortical responses. Philos Trans R Soc London Ser B 360:815-836.
11. Halle M (2002) From Memory to Speech and Back: Papers on Phonetics and Phonology, 1954-2002 (de Gruyter, Berlin).
12. Knill D, Kersten D, Yuille A (1998) in Perception as Bayesian Inference, eds Knill D, Richards W (Cambridge Univ Press, Cambridge, UK), pp 1-21.
13. Ellis HD, Jones DM, Mosdell N (1997) Intra- and inter-modal repetition priming of familiar faces and voices. Br J Psychol 88:143-156.
14. Hickok G, Poeppel D (2007) The cortical organization of speech processing. Nat Rev Neurosci 8:393-402.
15. Dayan P (2006) Images, frames, and connectionist hierarchies. Neural Comput 18:2293-2319.
16. Rao RP, Ballard DH (1999) Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nat Neurosci 2:79-87.
17. Fant G (1960) Acoustic Theory of Speech Production (Mouton, Paris).
18. Yehia H, Rubin P, Vatikiotis-Bateson E (1998) Quantitative association of vocal-tract and facial behavior. Speech Commun 26:23-43.
19. Lavner Y, Gath I, Rosenhouse J (2000) The effects of acoustic modifications on the identification of familiar voices speaking isolated vowels. Speech Commun 30:9-26.
20. Humphreys K, Avidan G, Behrmann M (2007) A detailed investigation of facial expression processing in congenital prosopagnosia as compared to acquired prosopagnosia. Exp Brain Res 176:356-373.
21. Calvert GA, et al. (1997) Activation of auditory cortex during silent lipreading. Science 276:593-596.
22. Puce A, Allison T, Bentin S, Gore JC, McCarthy G (1998) Temporal cortex activation in humans viewing eye and mouth movements. J Neurosci 18:2188-2199.
23. Eger E, Schyns PG, Kleinschmidt A (2004) Scale invariant adaptation in fusiform face-responsive regions. NeuroImage 22:232-242.
24. Loffler G, Yourganov G, Wilkinson F, Wilson HR (2005) fMRI evidence for the neural representation of faces. Nat Neurosci 8:1386-1390.
25. Rotshtein P, Henson RN, Treves A, Driver J, Dolan RJ (2005) Morphing Marilyn into Maggie dissociates physical and identity face representations in the brain. Nat Neurosci 8:107-113.
26. Haxby JV, Hoffman EA, Gobbini MI (2000) The distributed human neural system for face perception. Trends Cognit Sci 4:223-233.
27. Kanwisher N, Yovel G (2006) The fusiform face area: A cortical region specialized for the perception of faces. Philos Trans R Soc London Ser B 361:2109-2128.
28. Dupont S, Luettin J (2000) Audio-visual speech modeling for continous speech recognition. IEEE Trans Multimedia 2:141-151.
29. O'Toole AJ, Roark DA, Abdi H (2002) Recognizing moving faces: A psychological and neural synthesis. Trends Cognit Sci 6:261-266.
30. Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK (2005) Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci 25:5004-5012.
31. Lander K, Davies R (2007) Exploring the role of characteristic motion when learning new faces. Q J Exp Psychol (Colchester) 60:519-526.
32. Kamachi M, Hill H, Lander K, Vatikiotis-Bateson E (2003) Putting the face to the voice: Matching identity across modality. Curr Biol 13:1709-1714.
33. Siciliano C, Williams G, Beskow J, Faulkner A (2002) Evaluation of a multilingual synthetic talking face as a communication aid for the hearing-impaired. Speech Hear Lang Work Prog 14:51-61.
34. Ballard DH, Hinton GE, Sejnowski TJ (1983) Parallel visual computation. Nature 306:21-26.
35. Kawato M, Hayakawa H, Inui T (1983) A forward-inverse optics model of reciprocal connections between visual cortical areas. Network-Comput Neural Syst 4:415-422.
36. Bar M (2007) The proactive brain: Using analogies and associations to generate predictions. Trends Cognit Sci 11:280-289.
37. Wolpert DM, Ghahramani Z, Jordan MI (1995) An internal model for sensorimotor integration. Science 269:1880-1882.
38. de Gelder B, Pourtois G, Weiskrantz L (2002) Fear recognition in the voice is modulated by unconsciously recognized facial expressions but not by unconsciously recognized affective pictures. Proc Natl Acad Sci USA 99:4121-4126.
39. de Gelder B, Morris JS, Dolan RJ (2005) Unconscious fear influences emotional awareness of faces and voices. Proc Natl Acad Sci USA 102:18682-18687.
40. Amedi A, et al. (2007) Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nat Neurosci 10:687-689.
41. Gruter M, et al. (2007) Hereditary prosopagnosia: The first case series. Cortex 43:734-749.
42. Kennerknecht I, et al. (2006) First report of prevalence of non-syndromic hereditary prosopagnosia (HPA). Am J Med Genet A 140:1617-1622.
43. Carbon CC, Grueter T, Weber JE, Lueschow A (2007) Faces as objects of non-expertise: Processing of thatcherised faces in congenital prosopagnosia. Perception 36:1635-1645.