How prior expectations shape multisensory perception

NeuroImage

Volumn 124, Issue , 2016, Pages 876-886

  Gau, Remi ^a   Noppeney, Uta ^a  

^a UNIVERSITY OF BIRMINGHAM   (United Kingdom)

Author keywords

Causal inference; Congruency; Crossmodal integration; FMRI; Learning; McGurk; Multisensory perception; Perceptual illusion; Prefrontal cortex

Indexed keywords

ADULT; ARTICLE; AUDIOVISUAL EQUIPMENT; AUDITORY STIMULATION; BRAIN FUNCTION; BRAIN REGION; COGNITION; CONTROLLED STUDY; EXPECTATION; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; LEFT INFERIOR FRONTAL SULCUS; MALE; MULTISENSORY INTEGRATION; MULTISENSORY PERCEPTION; MULTISENSORY SEGREGATION; NORMAL HUMAN; PERCEPTION; PHONETICS; PRIORITY JOURNAL; STIMULUS RESPONSE; VISUAL STIMULATION; ASSOCIATION; HEARING; ILLUSION; LEARNING; MIDDLE AGED; NOISE; NUCLEAR MAGNETIC RESONANCE IMAGING; PHOTOSTIMULATION; PHYSIOLOGY; PREFRONTAL CORTEX; PSYCHOMETRY; REACTION TIME; SPEECH PERCEPTION; VISION; YOUNG ADULT;

ACOUSTIC STIMULATION; ADULT; AUDITORY PERCEPTION; CUES; FEMALE; HUMANS; ILLUSIONS; MAGNETIC RESONANCE IMAGING; MALE; MIDDLE AGED; NOISE; PERCEPTION; PHOTIC STIMULATION; PREFRONTAL CORTEX; PSYCHOMETRICS; REACTION TIME; SET (PSYCHOLOGY); SPEECH PERCEPTION; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84944111522     PISSN: 10538119     EISSN: 10959572     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2015.09.045     Document Type: Article

References (59)

1. Adam R., Noppeney U. Prior auditory information shapes visual category-selectivity in ventral occipito-temporal cortex. NeuroImage 2010, 52:1592-1602. 10.1016/j.neuroimage.2010.05.002.
2. Alsius A., Navarra J., Campbell R., Soto-Faraco S. Audiovisual integration of speech falters under high attention demands. Curr. Biol. 2005, 15:839-843. 10.1016/j.cub.2005.03.046.
3. Beauchamp M.S., Nath A.R., Pasalar S. fMRI-Guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect. J. Neurosci. 2010, 30:2414-2417. 10.1523/JNEUROSCI.4865-09.2010.
4. Benoit M.M., Raij T., Lin F.-H., Jääskeläinen I.P., Stufflebeam S. Primary and multisensory cortical activity is correlated with audiovisual percepts. Hum. Brain Mapp. 2010, 31:526-538. 10.1002/hbm.20884.
5. Bonath B., Noesselt T., Krauel K., Tyll S., Tempelmann C., Hillyard S.a. Audio-visual synchrony modulates the ventriloquist illusion and its neural/spatial representation in the auditory cortex. NeuroImage 2014, 98:425-434. 10.1016/j.neuroimage.2014.04.077.
6. Brainard D.H. The psychophysics toolbox. Spat. Vis. 1997, 10:433-436.
7. Brown J.W., Braver T.S. Learned predictions of error likelihood in the anterior cingulate cortex. Science 2005, 80(307):1118-1121. 10.1126/science.1105783.
8. Cappe C., Thelen A., Romei V., Thut G., Murray M.M. Looming signals reveal synergistic principles of multisensory integration. J. Neurosci. 2012, 32:1171-1182. 10.1523/JNEUROSCI.5517-11.2012.
9. Duncan J., Owen a M. Common regions of the human frontal lobe recruited by diverse cognitive demands. Trends Neurosci. 2000, 23:475-483.
10. Ernst M.O., Banks M.S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 2002, 415:429-433. 10.1038/415429a.
11. Ernst M.O., Buelthoff H.H. Merging the senses into a robust percept. Trends Cogn. Sci. 2004, 8:162-169. 10.1016/j.tics.2004.02.002.
12. Foxe J.J., Molholm S., Del Bene V.a, Frey H.-P., Russo N.N., Blanco D., Saint-Amour D., Ross L.a Severe multisensory speech integration deficits in high-functioning school-aged children with autism spectrum disorder (ASD) and their resolution during early adolescence. Cereb. Cortex 2013, 1-15. 10.1093/cercor/bht213.
13. Green K.P., Gerdeman A. Cross-modal discrepancies in coarticulation and the integration of speech information: the McGurk effect with mismatched vowels. J. Exp. Psychol. Hum. Percept. Perform. 1995, 21:1409-1426.
14. Hasson U., Skipper J.I., Nusbaum H.C., Small S.L. Abstract coding of audiovisual speech: beyond sensory representation. Neuron 2007, 56:1116-1126. 10.1016/j.neuron.2007.09.037.
15. Hein G., Doehrmann O., Müller N.G., Kaiser J., Muckli L., Naumer M.J. Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J. Neurosci. 2007, 27:7881-7887. 10.1523/JNEUROSCI.1740-07.2007.
16. Helbig H.B., Ernst M.O., Ricciardi E., Pietrini P., Thielscher A., Mayer K.M., Schultz J., Noppeney U. The neural mechanisms of reliability weighted integration of shape information from vision and touch. NeuroImage 2012, 60:1063-1072. 10.1016/j.neuroimage.2011.09.072.
17. Keil J., Müller N., Ihssen N., Weisz N. On the variability of the McGurk effect: audiovisual integration depends on prestimulus brain states. Cereb. Cortex 2012, 22:221-231. 10.1093/cercor/bhr125.
18. Kerns J.G., Cohen J.D., MacDonald A.W., Cho R.Y., Stenger V.A., Carter C.S. Anterior cingulate conflict monitoring and adjustments in control. Science 2004, 80(303):1023-1026. 10.1126/science.1089910.
19. Kleiner M., Brainard D., Pelli D., Ingling A. What's new in Psychtoolbox-3. Perception 2007.
20. Körding K.P., Beierholm U., Ma W.J., Quartz S., Tenenbaum J.B., Shams L., Kording K.P. Causal inference in multisensory perception. PLoS One 2007, 2. 10.1371/journal.pone.0000943.
21. Laurienti P.J., Kraft R.a, Maldjian J.a, Burdette J.H., Wallace M.T. Semantic congruence is a critical factor in multisensory behavioral performance. Exp. Brain Res. 2004, 158:405-414. 10.1007/s00221-004-1913-2.
22. Lee H., Noppeney U. Long-term music training tunes how the brain temporally binds signals from multiple senses. Proc. Natl. Acad. Sci. U. S. A. 2011, 108:E1441-E1450. 10.1073/pnas.1115267108.
23. Lee H., Noppeney U. Temporal prediction errors in visual and auditory cortices. Curr. Biol. 2014, 24:R309-R310. 10.1016/j.cub.2014.02.007.
24. Lewis R., Noppeney U. Audiovisual synchrony improves motion discrimination via enhanced connectivity between early visual and auditory areas. J. Neurosci. 2010, 30:12329-12339. 10.1523/JNEUROSCI.5745-09.2010.
25. Magnotti J.F., Ma W.J., Beauchamp M.S. Causal inference of asynchronous audiovisual speech. Front. Psychol. 2013, 4:798. 10.3389/fpsyg.2013.00798.
26. Mazaika P.K., Hoeft F., Glover G.H., Reiss A.L. Methods and software for fMRI analysis of clinical subjects. NeuroImage 2009, 47:S58. 10.1016/S1053-8119(09)70238-1.
27. McGurk H., MacDonald J. Hearing lips and seeing voices. Nature 1976, 264:746-748.
28. Miller L.M., Esposito M.D. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J. Neurosci. 2005, 25:5884-5893. 10.1523/JNEUROSCI.0896-05.2005.
29. Munhall K.G., Gribble P., Sacco L., Ward M. Temporal constraints on the McGurk effect. Percept. Psychophys. 1996, 58:351-362.
30. Nahorna O., Berthommier F., Schwartz J.-L. Binding and unbinding the auditory and visual streams in the McGurk effect. J. Acoust. Soc. Am. 2012, 132:1061-1077. 10.1121/1.4728187.
31. Nahorna O., Berthommier F., Schwartz J. Audio-visual speech scene analysis: characterization of the dynamics of unbinding and rebinding the McGurk effect. J. Acoust. Soc. Am. 2015, 137:362-377. 10.1121/1.4904536.
32. Nath A.R., Beauchamp M.S. A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. NeuroImage 2012, 59:781-787. 10.1016/j.neuroimage.2011.07.024.
33. Naumer M.J., Doehrmann O., Müller N.G., Muckli L., Kaiser J., Hein G. Cortical plasticity of audio-visual object representations. Cereb. Cortex 2009, 19:1641-1653. 10.1093/cercor/bhn200.
34. Noppeney U., Josephs O., Hocking J., Price C.J., Friston K.J. The effect of prior visual information on recognition of speech and sounds. Cereb. Cortex 2008, 18:598-609. 10.1093/cercor/bhm091.
35. Noppeney U., Ostwald D., Werner S. Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex. J. Neurosci. 2010, 30:7434-7446. 10.1523/JNEUROSCI.0455-10.2010.
36. Ojanen V., Möttönen R., Pekkola J., Jääskeläinen I.P., Joensuu R., Autti T., Sams M. Processing of audiovisual speech in Broca's area. NeuroImage 2005, 25:333-338. 10.1016/j.neuroimage.2004.12.001.
37. Olivetti Belardinelli M., Sestieri C., Matteo R., Delogu F., Gratta C., Ferretti A., Caulo M., Tartaro A., Romani G. Audio-visual crossmodal interactions in environmental perception: an fMRI investigation. Cogn. Process. 2004, 5:167-174. 10.1007/s10339-004-0024-0.
38. Orr J.M., Weissman D.H. Anterior cingulate cortex makes 2 contributions to minimizing distraction. Cereb. Cortex 2009, 19:703-711. 10.1093/cercor/bhn119.
39. Otten L.J., Henson R.N.a., Rugg M.D. State-related and item-related neural correlates of successful memory encoding. Nat. Neurosci. 2002, 5:1339-1344. 10.1038/nn967.
40. Pekkola J., Laasonen M., Ojanen V., Autti T., Jääskeläinen I.P., Kujala T., Sams M. Perception of matching and conflicting audiovisual speech in dyslexic and fluent readers: an fMRI study at 3T. NeuroImage 2006, 29:797-807. 10.1016/j.neuroimage.2005.09.069.
41. Roach N.W., Heron J., McGraw P.V. Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration. Proc. Biol. Sci. 2006, 273:2159-2168. 10.1098/rspb.2006.3578.
42. Rohe T., Noppeney U. Cortical hierarchies perform Bayesian causal inference in multisensory perception. PLoS Biol. 2015, 13. 10.1371/journal.pbio.1002073.
43. Rohe T., Noppeney U. Sensory reliability shapes perceptual inference via two mechanisms. J. Vis. 2015, 15:22. 10.1167/15.5.22.
44. Ross L.a, Saint-Amour D., Leavitt V.M., Javitt D.C., Foxe J.J. Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb. Cortex 2007, 17:1147-1153. 10.1093/cercor/bhl024.
45. Rouger J., Fraysse B., Deguine O., Barone P. McGurk effects in cochlear-implanted deaf subjects. Brain Res. 2008, 1188:87-99. 10.1016/j.brainres.2007.10.049.
46. Rushworth M.F.S., Walton M.E., Kennerley S.W., Bannerman D.M. Action sets and decisions in the medial frontal cortex. Trends Cogn. Sci. 2004, 8:410-417. 10.1016/j.tics.2004.07.009.
47. Schwartz J.-L. A reanalysis of McGurk data suggests that audiovisual fusion in speech perception is subject-dependent. J. Acoust. Soc. Am. 2010, 127:1584-1594. 10.1121/1.3293001.
48. Senkowski D., Saint-Amour D., Gruber T., Foxe J.J. Look who's talking: the deployment of visuo-spatial attention during multisensory speech processing under noisy environmental conditions. NeuroImage 2008, 43:379-387. 10.1016/j.neuroimage.2008.06.046.
49. Shams L., Beierholm U. Causal inference in perception. Trends Cogn. Sci. 2010, 14:425-432. 10.1016/j.tics.2010.07.001.
50. Shenhav A., Botvinick M., Cohen J. The expected value of control: an integrative theory of anterior cingulate cortex function. Neuron 2013, 79:217-240. 10.1016/j.neuron.2013.07.007.
51. Skipper J.I., Wassenhove V.Van, Nusbaum H.C., Small S.L. Hearing lips and seeing voices: how cortical areas supporting speech production mediate audiovisual speech perception. Cereb. Cortex 2007, 17:2387-2399. 10.1093/cercor/bhl147.
52. Stevenson R.a, Fister J.K., Barnett Z.P., Nidiffer A.R., Wallace M.T. Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance. Exp. Brain Res. 2012, 219:121-137. 10.1007/s00221-012-3072-1.
53. Szycik G.R., Stadler J., Tempelmann C., Münte T.F. Examining the McGurk illusion using high-field 7Tesla functional MRI. Front. Hum. Neurosci. 2012, 6:95. 10.3389/fnhum.2012.00095.
54. Talsma D., Doty T.J., Woldorff M.G. Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration?. Cereb. Cortex 2007, 17:679-690. 10.1093/cercor/bhk016.
55. Van Atteveldt N.M., Formisano E., Goebel R., Blomert L. Top-down task effects overrule automatic multisensory responses to letter-sound pairs in auditory association cortex. NeuroImage 2007, 36:1345-1360. 10.1016/j.neuroimage.2007.03.065.
56. Van Wassenhove V. Speech through ears and eyes: interfacing the senses with the supramodal brain. Front. Psychol. 2013, 4:1-17. 10.3389/fpsyg.2013.00388.
57. Van Wassenhove V., Grant K.W., Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proc. Natl. Acad. Sci. U. S. A. 2005, 102:1181-1186. 10.1073/pnas.0408949102.
58. Wassenhove V.Van, Grant K., Poeppel D. Temporal window of integration in auditory-visual speech perception. Neuropsychologia 2007, 45:598-607. 10.1016/j.neuropsychologia.2006.01.001.
59. Zilber N., Ciuciu P., Gramfort A., Azizi L., van Wassenhove V. Supramodal processing optimizes visual perceptual learning and plasticity. NeuroImage 2014, 93(Pt 1):32-46. 10.1016/j.neuroimage.2014.02.017.