Multisensory recognition in vertebrates (especially primates)

Integrating Face and Voice in Person Perception

Volumn , Issue , 2013, Pages 3-27

  Kulahci, Ipek G ^a   Ghazanfar, Asif A ^a  

^a PRINCETON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84898648904     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-1-4614-3585-3_1     Document Type: Chapter

References (98)

1. Barnes CL, Pandya DN (1992) Efferent cortical connections of multimodal cortex of the superior temporal sulcus in the rhesus-monkey. The Journal of Comparative Neurology 318:222-244.
2. Barraclough NE, Xiao D, Baker CI, Oram MW, Perrett DI (2005) Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. Journal of Cognitive Neuroscience 17:377-391.
3. Benevento LA, Fallon J, Davis BJ, Rezak M (1977) Auditory-visual interactions in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey. Experimental Neurology 57:849-872.
4. Bernstein LE, Auer ET, Takayanagi S (2004) Auditory speech detection in noise enhanced by lipreading. Speech Communication 44:5-18.
5. Besle J, Fort A, Delpuech C, Giard MH (2004) Bimodal speech: early suppressive visual effects in human auditory cortex. The European Journal of Neuroscience 20:2225-2234.
6. Bizley JK, Nodal FR, Bajo VM, Nelken I, King AJ (2007) Physiological and anatomical evidence for multisensory interactions in auditory cortex. Cerebral Cortex 17:2172-2189.
7. Bradbury JW (1981) The evolution of leks. In: Alexander RD, Tinkle DW (eds) Natural selection and social behavior. Chiron Press, New York, NY.
8. Bro-Jorgensen J (2010) Dynamics of multiple signalling systems: animal communication in a world in flux. Trends in Ecology & Evolution 25:292-300.
9. Brown JL (1964) The evolution of diversity in avian territorial systems. The Wilson Bulletin 76: 160-169.
10. Bruce C, Desimone R, Gross CG (1981) Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. Journal of Neurophysiology 46:369-384.
11. Candolin U (2003) The use of multiple cues in mate choice. Biological Reviews 78:575-595.
12. Cappe C, Thut G, Romei V, Murray MM (2009) Selective integration of auditory-visual looming cues by humans. Neuropsycholgia 47:1045-1052.
13. Chandrasekaran, C., Lemus, L., & Ghazanfar, A. A. (2011) Dynamic faces speedup vocalprocessing in the auditory cortex of behaving monkeys. Washington, D.C: Society for Neuroscience.
14. Chandrasekaran C, Ghazanfar AA (2009) Different neural frequency bands integrate faces and voices differently in the superior temporal sulcus. Journal of Neurophysiology 101:773-788.
15. Chandrasekaran C, Lemus L, Trubanova A, Gondan M, Ghazanfar AA (2011b) Monkeys and humans share a common computation for face/voice integration. PLoS Computational Biology 7(9):e1002165.
16. Chandrasekaran C, Trubanova A, Stillittano S, Caplier A, Ghazanfar AA (2009) The natural statistics of audiovisual speech. PLoS Computational Biology 5:e1000436.
17. Cheney DL, Seyfarth RM (1982) How vervet monkeys perceive their gruntsfield playback experiments. Animal Behaviour 30:739-751.
18. Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodal influences on ‘sensory- specific’ brain regions, neural responses, and judgments. Neuron 57:11-23.
19. Ettlinger G, Wilson WA (1990) Cross-modal performance: behavioural processes, phylogenetic considerations and neural mechanisms. Behavioural Brain Research 40:169-192.
20. Evans TA, Howell S, Westergaard GC (2005) Auditory-visual cross-modal perception of communicative stimuli in tufted capuchin monkeys (Cebus apella). Journal of Experimental Psychology Animal Behavior Processes 31:399-406.
21. Fitch WT (1997) Vocal tract length and formant frequency dispersion correlate with body size in rhesus macaques. The Journal of the Acoustical Society of America 102:1213-1222.
22. Fu KMG, Shah AS, O'Connell MN, McGinnis T, Eckholdt H, Lakatos P et al (2004) Timing and laminar profile of eye-position effects on auditory responses in primate auditory cortex. Journal of Neurophysiology 92:3522-3531.
23. Fusani L, Hutchison RE, Hutchison JB (1997) Vocal-postural Co-ordination of a sexually dimorphic display in a monomorphic species: The Barbary dove. Behaviour 134:321-335.
24. Geschwind N (1965a) Disconnexion syndromes in animals and man, part I. Brain 88:237-294.
25. Geschwind N (1965b) Disconnexion syndromes in animals and man, part II. Brain 88:585-644.
26. Ghazanfar AA, Chandrasekaran CF (2007) Paving the way forward: integrating the senses through phase-resetting of cortical oscillations. Neuron 53:162-164.
27. Ghazanfar AA, Chandrasekaran C, Logothetis NK (2008) Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys. Journal of Neuroscience 28:4457-4469.
28. Ghazanfar AA, Logothetis NK (2003) Facial expressions linked to monkey calls. Nature 423: 937-938.
29. Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK (2005) Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. Journal of Neuroscience 25:5004-5012.
30. Ghazanfar AA, Nielsen K, Logothetis NK (2006) Eye movements of monkeys viewing vocalizing conspecifics. Cognition 101:515-529.
31. Ghazanfar AA, Rendall D (2008) Evolution of human vocal production. Current Biology 18: R457-R460.
32. Ghazanfar AA, Schroeder CE (2006) Is neocortex essentially multisensory? Trends in Cognitive Sciences 10:278-285.
33. Ghazanfar AA, Turesson HK, Maier JX, van Dinther R, Patterson RD, Logothetis NK (2007) Vocal tract resonances as indexical cues in rhesus monkeys. Current Biology 17:425-430.
34. Gordon MS, Rosenblum LD (2005) Effects of intrastimulus modality change on audiovisual time- to-arrival judgments. Perception & Psychophysics 67:580-594.
35. Gothard KM, Battaglia FP, Erickson CA, Spitler KM, Amaral DG (2007) Neural responses to facial expression and face identity in the monkey amygdala. Journal of Neurophysiology 97: 1671-1683.
36. Grafe TU, Wanger TC (2007) Multimodal signaling in male and female foot-flagging frogs Staurois guttatus (ranidae): An alerting function of calling. Ethology 113:772-781.
37. Grant BR, Grant PR (1989) Evolutionary dynamics of a natural population: the large cactus finch of the Galapagos. Chicago University Press, Chicago, IL.
38. Grant PR, Grant BR (2002) Unpredictable evolution in a 30-year study of Darwin’s finches. Science 296:707-711.
39. Guilford T, Dawkins MS (1991) Receiver psychology and the evolution of animal signals. Animal Behaviour 42:1-14.
40. Hackett TA, Stepniewska I, Kaas JH (1999) Prefrontal connections of the parabelt auditory cortex in macaque monkeys. Brain Research 817:45-58.
41. Harries MH, Perrett DI (1991) Visual processing of faces in temporal cortex - physiological evidence for a modular organization and possible anatomical correlates. Journal of Cognitive Neuroscience 3:9-24.
42. Hauser MD, Chomsky N, Fitch W (2002) The faculty of language: What is it, who has it, and how did it evolve? Science 298:1569-1579.
43. Hauser MD, Evans CS, Marler P (1993) The role of articulation in the production of rhesus- monkey, macaca-mulatta, vocalizations. Animal Behaviour 45:423-433.
44. Hauser MD, Ybarra MS (1994) The role of lip configuration in monkey vocalizations - experiments using xylocaine as a nerve block. Brain and Language 46:232-244.
45. Hebets EA (2005) Attention-altering signal interactions in the multimodal courtship display of the wolf spider Schizocosa uetzi. Behavioral Ecology 16:75-82.
46. Hebets EA, Papaj DR (2005) Complex signal function: developing a framework of testable hypotheses. Behavioral Ecology and Sociobiology 57:197-214.
47. Hibbitts T, Whiting M, Stuart-Fox D (2007) Shouting the odds: Vocalization signals status in a lizard. Behavioral Ecology and Sociobiology 61:1169-1176.
48. Hoy R (2005) Animal awareness: The (un)binding of multisensory cues in decision making by animals. Proceedings of the National Academy of Sciences of the United States of America 102:2267-2268.
49. Huber SK, Leon LFD, Hendry AP, Bermingham E, Podos J (2007) Reproductive isolation of sympatric morphs in a population of Darwin’s finches. Proceedings of the Royal Society B: Biological Sciences 274:1709-1714.
50. Huber SK, Podos J (2006) Beak morphology and song features covary in a population of Darwin’s finches (Geospiza Fortis). Biological Journal of the Linnean Society 88:489-498.
51. Izumi A, Kojima S (2004) Matching vocalizations to vocalizing faces in a chimpanzee (Pan troglodytes). Animal Cognition 7:179-184.
52. Jiang JT, Alwan A, Keating PA, Auer ET, Bernstein LE (2002) On the relationship between face movements, tongue movements, and speech acoustics. Eurasip Journal on Applied Signal Processing 2002:1174-1188.
53. Jordan KE, Brannon EM, Logothetis NK, Ghazanfar AA (2005) Monkeys match the number of voices they hear with the number of faces they see. Current Biology 15:1034-1038.
54. Kayser C, Logothetis NK (2009) Directed interactions between auditory and superior temporal cortices and their role in sensory integration. Frontiers in Integrative Neuroscience 3:7.
55. Kayser C, Petkov CI, Augath M, Logothetis NK (2007) Functional imaging reveals visual modulation of specific fields in auditory cortex. Journal of Neuroscience 27:1824-1835.
56. Kayser C, Petkov CI, Logothetis NK (2008) Visual modulation of neurons in auditory cortex. Cerebral Cortex 18:1560-1574.
57. Klin A, Jones W, Schultz R, Volkmar F, Cohen D (2002) Visual fixation patterns during viewing of naturalistic social situations as predictors of social competence in individuals with autism. Archives of General Psychiatry 59:809-816.
58. Kuhl PK, Williams KA, Meltzoff AN (1991) Cross-modal speech perception in adults and infants using nonspeech auditory stimuli. Journal of Experimental Psychology Human Perception and Performance 17:829-840.
59. Kuraoka K, Nakamura K (2007) Responses of single neurons in monkey amygdala to facial and vocal emotions. Journal of Neurophysiology 97:1379-1387.
60. Lakatos P, Chen C-M, O'Connell MN, Mills A, Schroeder CE (2007) Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53:279-292.
61. Lakatos P, Shah AS, Knuth KH, Ulbert I, Karmos G, Schroeder CE (2005) An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. Journal of Neurophysiology 94:1904-1911.
62. Lansing IR, McConkie GW (2003) Word identification and eye fixation locations in visual and visual-plus-auditory presentations of spoken sentences. Perception & Psychophysics 65: 536-552.
63. Lombardo S, Mackey E, Tang L, Smith B, Blumstein D (2008) Multimodal communication and spatial binding in pied currawongs (Strepera graculina). Animal Cognition 11:675-682.
64. Maier JX, Chandrasekaran C, Ghazanfar AA (2008) Integration of bimodal looming signals through neuronal coherence in the temporal lobe. Current Biology 18:963-968.
65. Maier JX, Neuhoff JG, Logothetis NK, Ghazanfar AA (2004) Multisensory integration of looming signals by rhesus monkeys. Neuron 43:177-181.
66. Narins PM, Grabul DS, Soma KK, Gaucher P, Hodl W (2005) Cross-modal integration in a dartpoison frog. Proceedings of the National Academy of Sciences of the United States of America 102:2425-2429.
67. Narins PM, Hodl W, Grabul DS (2003) Bimodal signal requisite for agonistic behavior in a dartpoison frog, epipedobatesfemoralis. Proceedings of the National Academy of Sciences of the United States of America 100:577-580.
68. Noesselt T, Rieger JW, Schoenfeld MA, Kanowski M, Hinrichs H, Heinze H-J et al (2007) Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices. Journal of Neuroscience 27:11431-11441.
69. Oram MW, Perrett DI (1994) Responses of anterior superior temporal polysensory (Stpa) neurons to biological motion stimuli. Journal of Cognitive Neuroscience 6:99-116.
70. Palombit RA, Cheney DL, Seyfarth RM (1999) Male grunts as mediators of social interaction with females in wild chacma baboons (papio cynocephalus ursinus). Behaviour 136:221-242.
71. Parr LA (2004) Perceptual biases for multimodal cues in chimpanzee (pan troglodytes) affect recognition. Animal Cognition 7:171-178.
72. Partan SR, Larco CP, Owens MJ (2009) Wild tree squirrels respond with multisensory enhancement to conspecific robot alarm behaviour. Animal Behaviour 77:1127-1135.
73. Partan S, Marler P (1999) Communication goes multimodal. Science 283:1272-1273.
74. Partan S, Yelda S, Price V, Shimizu T (2005) Female pigeons, Columba livia, respond to multisensory audio/video playbacks of male courtship behaviour. Animal Behaviour 70:957-966.
75. Pevsner J (2002) Leonardo da Vinci’s contributions to neuroscience. Trends in Neurosciences 25:217-220.
76. Podos J (2010) Acoustic discrimination of sympatric morphs in Darwin’s finches: a behavioural mechanism for assortative mating? Philosophical Transactions of the Royal Society B: Biological Sciences 365:1031-1039.
77. Roberts JA, Taylor PW, Uetz GW (2007) Consequences of complex signaling: Predator detection of multimodal cues. Behavioral Ecology 18:236-240.
78. Robinson DA, FUchs AF (1969) Eye movements evoked by stimulation of frontal eye fields. Journal of Neurophysiology 32:637-648.
79. Romanski LM, Averbeck BB, Diltz M (2005) Neural representation of vocalizations in the primate ventrolateral prefrontal cortex. Journal of Neurophysiology 93:734-747.
80. Romanski LM, Bates JF, Goldman-Rakic PS (1999) Auditory belt and parabelt projections to the prefrontal cortex in the rhesus monkey. The Journal of Comparative Neurology 403:141-157.
81. Rowe C (1999) Receiver psychology and the evolution of multicomponent signals. Animal Behaviour 58:921-931.
82. Schall JD, Morel A, King DJ, Bullier J (1995) Topography of visual cortex connections with frontal eye field in macaque: Convergence and segregation of processing streams. Journal of Neuroscience 15:4464-4487.
83. Schroeder CE, Foxe JJ (2002) The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Cognitive Brain Research 14:187-198.
84. Schroeder CE, Lakatos P, Kajikawa Y, Partan S, Puce A (2008) Neuronal oscillations and visual amplification of speech. Trends in Cognitive Sciences 12:106-113.
85. Schwartz J-L, Berthommier F, Savariaux C (2004) Seeing to hear better: Evidence for early audiovisual interactions in speech identification. Cognition 93:B69-B78.
86. Seltzer B, Pandya DN (1989) Frontal-lobe connections of the superior temporal sulcus in the rhesus- monkey. The Journal of Comparative Neurology 281:97-113.
87. Seltzer B, Pandya DN (1994) Parietal, temporal, and occipital projections to cortex of the superior temporal sulcus in the rhesus monkey: A retrograde tracer study. The Journal of Comparative Neurology 343:445-463.
88. Sherman PW, Reeve HK, Pfennig DW (1997) Recognition systems. In: Krebs JR, Davies NB (eds) Behavioural ecology: an evolutionary approach. Cambridge University Press, Cambridge, pp 69-96.
89. Sliwa J, Duhamel JR, Pascalis O, Wirth S (2011) Spontaneous voice-face identity matching by rhesus monkeys for familiar conspecifics and humans. Proceedings of the National Academy of Sciences of the United States of America 108:1735-1740.
90. Sugihara T, Diltz MD, Averbeck BB, Romanski LM (2006) Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. Journal of Neuroscience 26:11138-11147.
91. Taylor RC, Buchanan BW, Doherty JL (2007) Sexual selection in the squirrel treefrog Hyla squirella: the role of multimodal cue assessment in female choice. Animal Behaviour 74:1753-1763.
92. Taylor RC, Klein BA, Stein J, Ryan MJ (2008) Faux frogs: multimodal signalling and the value of robotics in animal behavior. Animal Behaviour 76:1089-1097.
93. van Wassenhove V, Grant KW, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences of the United States of America 102:1181-1186.
94. Werner-Reiss U, Kelly KA, Trause AS, Underhill AM, Groh JM (2003) Eye position affects activity in primary auditory cortex of primates. Current Biology 13:554-562.
95. Wilson EO (1975) Sociobiology the new synthesis. Harvard University Press, Cambridge.
96. Wollerman L (1999) Acoustic interference limits call detection in a neotropical frog, Hyla ebraccata. Animal Behaviour 57:529-536.
97. Yehia HC, Kuratate T, Vatikiotis-Bateson E (2002) Linking facial animation, head motion and speech acoustics. Journal of Phonetics 30:555-568.
98. Yehia H, Rubin P, Vatikiotis-Bateson E (1998) Quantitative association of vocal-tract and facial behavior. Speech Communication 26:23-43.