Multisensory Processes: A Balancing Act across the Lifespan

Trends in Neurosciences

Volumn 39, Issue 8, 2016, Pages 567-579

  Murray, Micah M ^a,b,c   Lewkowicz, David J ^d   Amedi, Amir ^e,f   Wallace, Mark T ^c,g,h  

^a UNIVERSITY OF LAUSANNE   (Switzerland)

^b CardioVascular Magnetic Resonance (CVMR) Research Centre Centre for BioMedical Imaging (CIBM)   (Switzerland)

^c VANDERBILT UNIVERSITY MEDICAL CENTER   (United States)

^d NORTHEASTERN UNIVERSITY   (United States)

^e HEBREW UNIVERSITY OF JERUSALEM   (Israel)

^f HEBREW UNIVERSITY OF JERUSALEM   (Israel)

^g VANDERBILT UNIVERSITY   (United States)

^h VANDERBILT UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

aging.; cross modal; development; learning; multisensory; plasticity

Indexed keywords

ASSOCIATION; BRAIN; COGNITION; HUMAN; LEARNING; MULTISENSORY PROCESSING; NONHUMAN; PRIORITY JOURNAL; REVIEW; SENSORY SYSTEM; STIMULUS; AGING; ANIMAL; PERCEPTION; PHYSIOLOGY;

AGING; ANIMALS; HUMANS; LEARNING; PERCEPTION;

EID: 84973137855     PISSN: 01662236     EISSN: 1878108X     Source Type: Journal    
DOI: 10.1016/j.tins.2016.05.003     Document Type: Review

References (93)

1. 1 Sumby, W.H., Pollack, I., Visual contribution to speech intelligibility in noise. J. Acoust. Soc. Am. 26 (1954), 212–215.
2. 2 Lewkowicz, D.J., Hansen-Tift, A.M., Infants deploy selective attention to the mouth of a talking face when learning speech. Proc. Natl Acad. Sci. U.S.A. 109 (2012), 1431–1436.
3. 3 Pons, F., et al. Bilingualism modulates infants’ selective attention to the mouth of a talking face. Psychol. Sci. 26 (2015), 490–498.
4. 4 Bavelier, D., Neville, H.J., Cross-modal plasticity: where and how?. Nat. Rev. Neurosci. 3 (2002), 443–452.
5. 5 Stein, B.E., (eds.) The New Handbook of Multisensory Processes, 2012, MIT Press.
6. 6 Stein, B.E., Meredith, M.A., The Merging of the Senses. 1993, MIT Press.
7. 7 Murray, M.M., Wallace, M.T., (eds.) The Neural Bases of Multisensory Processes, 2012, CRC Press.
8. 8 Murray, M.M., et al. The multisensory function of the human primary visual cortex. Neuropsychologia 83 (2016), 161–169.
9. 9 Ghazanfar, A.A., Schroeder, C.E., Is neocortex essentially multisensory?. Trends Cogn. Sci. 10 (2006), 278–285.
10. 10 Kayser, C., Shams, L., Multisensory causal inference in the brain. PLoS Biol., 13, 2015, e1002075.
11. 11 van Atteveldt, N., et al. Multisensory integration: flexible use of general operations. Neuron 81 (2014), 1240–1253.
12. 12 Cappe, C., et al. Looming signals reveal synergistic principles of multisensory integration. J. Neurosci. 32 (2012), 1171–1182.
13. 13 Krueger, J., et al. Spatial receptive field organization of multisensory neurons and its impact on multisensory interactions. Hear. Res. 258 (2009), 47–54.
14. 14 Royal, D.W., et al. Spatiotemporal architecture of cortical receptive fields and its impact on multisensory interactions. Exp. Brain Res. 198 (2009), 127–136.
15. 15 Carriere, B.N., et al. Spatial heterogeneity of cortical receptive fields and its impact on multisensory interactions. J. Neurophysiol. 99 (2008), 2357–2368.
16. 16 Talsma, D., et al. The multifaceted interplay between attention and multisensory integration. Trends Cogn. Sci. 14 (2010), 400–410.
17. 17 De Meo, R., et al. Top-down control and early multisensory processes: chicken vs. egg. Front. Integr. Neurosci. 9 (2015), 1–6.
18. 18 ten Oever, S., et al. The COGs (context, object, and goals) in multisensory processing. Exp. Brain Res. 234 (2016), 1307–1323.
19. 19 Sarmiento, B.R., et al. Contextual factors multiplex to control multisensory processes. Hum. Brain Mapp. 37 (2016), 273–288.
20. 20 Lewkowicz, D.J., Ghazanfar, A.A., The emergence of multisensory systems through perceptual narrowing. Trends Cogn. Sci. 13 (2009), 470–478.
21. 21 Heimler, B., et al. Origins of task-specific sensory-independent organization in the visual and auditory brain: neuroscience evidence, open questions and clinical implications. Curr. Opin. Neurobiol. 35 (2015), 169–177.
22. 22 Lewkowicz, D.J., Early experience and multisensory perceptual narrowing. Dev. Psychobiol. 56 (2014), 292–315.
23. 23 Lewkowicz, D.J., King, A.J., The developmental and evolutionary emergence of multisensory processing: from single cells to behavior. Stein, B.E., (eds.) In The New Handbook of Multisensory Processing, 2012, MIT Press, 559–572.
24. 24 Bremner, A.J., et al. Multisensory Development. 2012, Oxford University Press.
25. 25 Röder, B., Wallace, M., Development and plasticity of multisensory functions. Restor. Neurol. Neurosci. 28 (2010), 141–142.
26. 26 Polley, D.B., et al. Development and plasticity of intra- and intersensory information processing. J. Am. Acad. Audiol. 19 (2008), 780–798.
27. 27 Lewkowicz, D.J., Turkewitz, G., Cross-modal equivalence in early infancy: auditory–visual intensity matching. Dev. Psychol. 16 (1980), 597–607.
28. 28 Lewkowicz, D.J., et al. Intersensory perception at birth: newborns match nonhuman primate faces and voices. Infancy 15 (2010), 46–60.
29. 29 Bahrick, L.E., Lickliter, R., Intersensory redundancy guides attentional selectivity and perceptual learning in infancy. Dev. Psychol. 36 (2000), 190–201.
30. 30 Pons, F., et al. Narrowing of intersensory speech perception in infancy. Proc. Natl Acad. Sci. U.S.A. 106 (2009), 10598–10602.
31. 31 Lewkowicz, D.J., Ghazanfar, A.A., The decline of cross-species intersensory perception in human infants. Proc. Natl Acad. Sci. U.S.A. 103 (2006), 6771–6774.
32. 32 Lewkowicz, D.J., Infant perception of audio-visual speech synchrony. Dev. Psychol. 46 (2010), 66–77.
33. 33 Lewkowicz, D.J., Perception of auditory–visual temporal synchrony in human infants. J. Exp. Psychol. Hum. Percept. Perform. 22 (1996), 1094–1106.
34. 34 Scheier, C., et al. Sound induces perceptual reorganization of an ambiguous motion display in human infants. Dev. Sci. 6 (2003), 233–241.
35. 35 Kuhl, P.K., Meltzoff, A.N., The bimodal perception of speech in infancy. Science 218 (1982), 1138–1141.
36. 36 Patterson, M.L., Werker, J.F., Matching phonetic information in lips and voice is robust in 4.5-month-old infants. Infant Behav. Dev. 22 (1999), 237–247.
37. 37 Rosenblum, L.D., et al. The McGurk effect in infants. Percept. Psychophys. 59 (1997), 347–357.
38. 38 Hillairet de Boisferon, A., et al. Perception of multisensory gender coherence in 6- and 9-month-old infants. Infancy 20 (2015), 661–674.
39. 39 Walker-Andrews, A.S., Infants’ perception of expressive behaviors: differentiation of multimodal information. Psychol. Bull. 121 (1997), 437–456.
40. 40 Lewkowicz, D.J., et al. Perception of the multisensory coherence of fluent audiovisual speech in infancy: its emergence and the role of experience. J. Exp. Child Psychol. 130 (2015), 147–162.
41. 41 Lewkowicz, D.J., Pons, F., Recognition of amodal language identity emerges in infancy. Int. J. Behav. Dev. 37 (2013), 90–94.
42. 42 Durand, K., et al. Eye-catching odors: olfaction elicits sustained gazing to faces and eyes in 4-month-old infants. PLoS One, 8, 2013, e70677.
43. 43 Gottfried, A.W., et al. Cross-modal transfer in human infants. Child Dev. 48 (1977), 118–123.
44. 44 Colonius, H., Diederich, A., Multisensory interaction in saccadic reaction time: a time-window-of-integration model. J. Cogn. Neurosci. 16 (2004), 1000–1009.
45. 45 Stevenson, R.a., Wallace, M.T., Multisensory temporal integration: task and stimulus dependencies. Exp. Brain Res. 227 (2013), 249–261.
46. 46 Stevenson, R.A., et al. Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance. Exp. Brain Res. 219 (2012), 121–137.
47. 47 Musacchia, G., Schroeder, C.E., Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex. Hear. Res. 258 (2009), 72–79.
48. 48 Hillock-Dunn, A., Wallace, M.T., Developmental changes in the multisensory temporal binding window persist into adolescence. Dev. Sci. 15 (2012), 688–696.
49. 49 Hillock-Dunn, A., et al. The temporal binding window for audiovisual speech: children are like little adults. Neuropsychologia, 2016 Published online February 23, 2016 http://dx.doi.org/10.1016/j.neuropsychologia.2016.02.017.
50. 50 Powers, A.R., et al. Perceptual training narrows the temporal window of multisensory binding. J. Neurosci. 29 (2009), 12265–12274.
51. 51 Powers, A.R., et al. Neural correlates of multisensory perceptual learning. J. Neurosci. 32 (2012), 6263–6274.
52. 52 Wallace, M.T., Stevenson, R.A., The construct of the multisensory temporal binding window and its dysregulation in developmental disabilities. Neuropsychologia 64C (2014), 105–123.
53. 53 Fiehler, K., Rösler, F., Plasticity of multisensory dorsal stream functions: evidence from congenitally blind and sighted adults. Restor. Neurol. Neurosci. 28 (2010), 193–205.
54. 54 Ricciardi, E., et al. Mind the blind brain to understand the sighted one! Is there a supramodal cortical functional architecture?. Neurosci. Biobehav. Rev. 41 (2014), 64–77.
55. 55 Murray, M.M., et al. Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. Neuroimage 21 (2004), 125–135.
56. 56 Murray, M.M., et al. The brain uses single-trial multisensory memories to discriminate without awareness. Neuroimage 27 (2005), 473–478.
57. 57 Lehmann, S., Murray, M.M., The role of multisensory memories in unisensory object discrimination. Brain Res. Cogn. Brain Res. 24 (2005), 326–334.
58. 58 Thelen, A., et al. Electrical neuroimaging of memory discrimination based on single-trial multisensory learning. Neuroimage 62 (2012), 1478–1488.
59. 59 Thelen, A., et al. Multisensory context portends object memory. Curr. Biol. 24 (2014), R734–R735.
60. 60 Thelen, A., et al. Single-trial multisensory memories affect later auditory and visual object discrimination. Cognition 138 (2015), 148–160.
61. 61 Thelen, A., Murray, M.M., The efficacy of single-trial multisensory memories. Multisens. Res. 26 (2013), 483–502.
62. 62 Schall, S., et al. Early auditory sensory processing of voices is facilitated by visual mechanisms. Neuroimage 77 (2013), 237–245.
63. 63 Matusz, P.J., et al. The role of auditory cortices in the retrieval of single-trial auditory–visual object memories. Eur. J. Neurosci. 41 (2015), 699–708.
64. 64 Baddeley, A., et al. Memory. 2009, Psychology Press.
65. 65 Johansson, B.B., Multisensory stimulation in stroke rehabilitation. Front. Hum. Neurosci., 6, 2012, 60.
66. 66 Reich, L., et al. The brain as a flexible task machine: implications for visual rehabilitation using noninvasive vs. invasive approaches. Curr. Opin. Neurol. 25 (2012), 86–95.
67. 67 Amedi, A., et al. Visuo-haptic object-related activation in the ventral visual pathway. Nat. Neurosci. 4 (2001), 324–330.
68. 68 Amedi, A., et al. Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nat. Neurosci. 10 (2007), 687–689.
69. 69 Merabet, L.B., Pascual-Leone, A., Neural reorganization following sensory loss: the opportunity of change. Nat. Rev. Neurosci. 11 (2010), 44–52.
70. 70 Reich, L., et al. A ventral visual stream reading center independent of visual experience. Curr. Biol. 21 (2011), 363–368.
71. 71 Striem-Amit, E., et al. Functional connectivity of visual cortex in the blind follows retinotopic organization principles. Brain 138 (2015), 1679–1695.
72. 72 Wang, X., et al. How visual is the visual cortex? Comparing connectional and functional fingerprints between congenitally blind and sighted individuals. J. Neurosci. 35 (2015), 12545–12559.
73. 73 Bock, A.S., et al. Resting-state retinotopic organization in the absence of retinal input and visual experience. J. Neurosci. 35 (2015), 12366–12382.
74. 74 Striem-Amit, E., et al. Reading with sounds: sensory substitution selectively activates the visual word form area in the blind. Neuron 76 (2012), 640–652.
75. 75 Striem-Amit, E., Amedi, A., Visual cortex extrastriate body-selective area activation in congenitally blind people ‘seeing’ by using sounds. Curr. Biol. 24 (2014), 687–692.
76. 76 Kayser, C., Logothetis, N.K., Do early sensory cortices integrate cross-modal information?. Brain Struct. Funct. 212 (2007), 121–132.
77. 77 Meredith, M.A., Allman, B.L., Subthreshold multisensory processing in cat auditory cortex. Neuroreport 20 (2009), 126–131.
78. 78 Amedi, A., et al. Cortical activity during tactile exploration of objects in blind and sighted humans. Restor. Neurol. Neurosci. 28 (2010), 143–156.
79. 79 Tal, N., Amedi, A., Multisensory visual-tactile object related network in humans: insights gained using a novel crossmodal adaptation approach. Exp. Brain Res. 198 (2009), 165–182.
80. 80 Tal, Z., et al. The origins of metamodality in visual object area LO: bodily topographical biases and increased functional connectivity to S1. Neuroimage 127 (2015), 363–375.
81. 81 Hertz, U., Amedi, A., Flexibility and stability in sensory processing revealed using visual-to-auditory sensory substitution. Cereb. Cortex 25 (2015), 2049–2064.
82. 82 Murray, M.M., et al. Neuroplasticity: unexpected consequences of early blindness. Curr. Biol. 25 (2015), R998–R1001.
83. 83 Falchier, A., et al. Anatomical evidence of multimodal integration in primate striate cortex. J. Neurosci. 22 (2002), 5749–5759.
84. 84 Falchier, A., et al. Projection from visual areas V2 and prostriata to caudal auditory cortex in the monkey. Cereb. Cortex 20 (2010), 1529–1538.
85. 85 Rockland, K.S., Ojima, H., Multisensory convergence in calcarine visual areas in macaque monkey. Int. J. Psychophysiol. 50 (2003), 19–26.
86. 86 Beer, A.L., et al. Diffusion tensor imaging shows white matter tracts between human auditory and visual cortex. Exp. Brain Res. 213 (2011), 299–308.
87. 87 Beer, A.L., et al. Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal–occipital network involved in auditory–visual object processing. Front. Integr. Neurosci., 7, 2013, 5.
88. 88 Cappe, C., et al. Selective integration of auditory–visual looming cues by humans. Neuropsychologia 47 (2009), 1045–1052.
89. 89 Wallace, M.T., et al. Visual experience is necessary for the development of multisensory integration. J. Neurosci. 24 (2004), 9580–9584.
90. 90 Wallace, M.T., et al. The development of cortical multisensory integration. J. Neurosci. 26 (2006), 11844–11849.
91. 91 Wallace, M.T., Stein, B.E., Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17 (1997), 2429–2444.
92. 92 Stevenson, R.A., et al. Identifying and quantifying multisensory integration: a tutorial review. Brain Topogr. 27 (2014), 707–730.
93. 93 Panzeri, S., et al. Neural population coding: combining insights from microscopic and mass signals. Trends Cogn. Sci. 19 (2015), 162–172.