Seeing and Hearing a Word: Combining Eye and Ear Is More Efficient than Combining the Parts of a Word

PLoS ONE

Volumn 8, Issue 5, 2013, Pages

  Dubois, Matthieu ^a,b,c   Poeppel, David ^a   Pelli, Denis G ^a  

^a NEW YORK UNIVERSITY   (United States)

^b UNIVERSITÉ LIBRE DE BRUXELLES   (Belgium)

^c FNRS   (Belgium)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ALGORITHM; ARTICLE; ASSOCIATION; BRAIN FUNCTION; CONTROLLED STUDY; EAR; EYE; HEARING; HUMAN; HUMAN EXPERIMENT; IMAGERY; NORMAL HUMAN; OBSERVER VARIATION; PERCEPTIVE DISCRIMINATION; SENSITIVITY ANALYSIS; SOUND; TASK PERFORMANCE; VISION; WORD RECOGNITION;

EID: 84878427805     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0064803     Document Type: Article

References (92)

1. Fechner GT (1860/1966) Elemente der Psychophysik (Breitkopf & Hartel, Leipzig); Elements of psychophysics. Trans Adler HE (Holt, Rinehart, & Winston, New York).
2. Walraven J, Enroth-Cugell C, Hood DC, MacLeod DIA, Schnapf JL (1990) The control of visual sensitivity: Receptoral and postreceptoral processes. In: Spillmann L, Werner JS, editors. Visual perception: The neurophysiological foundations. San Diego, CA, US: Academic Press. 53-101.
3. von der Malsburg C, (1995) Binding in models of perception and brain function. Curr Opin Neurobiol 5: 520-526.
4. Treisman A, (1996) The binding problem. Curr Opin Neurobiol 6: 171-178.
5. Ghose GM, Maunsell J, (1999) Specialized representations in visual cortex: A role for binding? Neuron 24: 79-85.
6. Wolfe JM, Cave KR, (1999) The psychophysical evidence for a binding problem in human vision. Neuron 24: 11-17.
7. Ernst MO, Banks MS, (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415: 429-433.
8. Alais D, Burr DC, (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14: 257-262.
9. McGurk H, MacDonald J, (1976) Hearing lips and seeing voices. Nature 264: 746-748.
10. Shams L, Kamitani Y, Shimojo S, (2000) Illusions. What you see is what you hear. Nature 408: 788.
11. Gick B, Derrick D, (2009) Aero-tactile integration in speech perception. Nature 462: 502-504.
12. Ho Y-X, Serwe S, Trommershäuser J, Maloney LT, Landy MS, (2009) The role of visuohaptic experience in visually perceived depth. J Neurophysiol 101: 2789-2801.
13. Stein BE, Stanford TR, (2008) Multisensory integration: Current issues from the perspective of the single neuron. Nat Rev Neurosci 9: 255-266.
14. Musacchia G, Schroeder CE, (2009) Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex. Hear Res 258: 72-79.
15. Luo H, Liu Z, Poeppel D, (2010) Auditory cortex tracks both auditory and visual stimulus dynamics using low-frequency neuronal phase modulation. PLoS Biol 8: e1000445.
16. Ramachandran VS, Altschuler EL, (2009) The use of visual feedback, in particular mirror visual feedback, in restoring brain function. Brain 132: 1693-1710.
17. Ball EW, Blachman BA, (1991) Does phoneme awareness training in Kindergarten make a difference in early word recognition and developmental spelling? Read Res Q 26: 1-19.
18. Hulme C, Goetz K, Gooch D, Adams J, Snowling MJ, (2007) Paired-associate learning, phoneme awareness, and learning to read. J Exp Child Psychol 96: 150-166.
19. Trommershäuser J, Körding KP, Landy MS (2011) Sensory cue integration. New York, NY: Oxford University Press. 464 p.
20. Tanner WP, Birdsall TG, (1958) Definitions of d′ and η as psychophysical measures. J Acoust Soc Am 30: 922-928.
21. Peterson WW, Birdsall TG, Fox WC, (1954) Theory of signal detectability. IEEE Trans Inf Theory 4: 171-212.
22. Green DM, Swets JA (1966) Signal detection theory and psychophysics. New York: John Wiley & Sons. Integrating model (Sec. 9.2) and energy-limited performance (p. 455, Eq. 6-37).
23. Geisler WS, (1989) Sequential ideal-observer analysis of visual discriminations. Psychol Rev 96: 267-314.
24. Pelli DG, Farell B, Moore DC, (2003) The remarkable inefficiency of word recognition. Nature 423: 752-756.
25. Pelli DG, Burns CW, Farell B, Moore-Page DC, (2006) Feature detection and letter identification. Vision Res 46: 4646-4674.
26. Nandy AS, Tjan BS, (2008) Efficient integration across spatial frequencies for letter identification in foveal and peripheral vision. J Vis 8: 3.1-20.
27. Creelman CD, (1961) Detection of complex signals as a function of signal bandwidth and duration. J Acoust Soc Am 33: 89-94.
28. Kersten D, (1984) Spatial summation in visual noise. Vision Res 24: 1977-1990.
29. Robson JG, Graham N (1981) Probability summation and regional variation in contrast sensitivity across the visual field. Vision Res 21: 409-418. Eq. 4.
30. Allen JB (1994) How do humans process and recognize speech? IEEE Trans Speech Audio Process 2: 567-577. Eq. 17.
31. Graham N (1989) Visual pattern analysers. New York: Oxford University Press. 646 p.
32. Fletcher H (1953) Speech and hearing in communication. Oxford, England: Van Nostrand. 461 p. Eq. 15-1.
33. Graham N, Robson JG, Nachmias J, (1978) Grating summation in fovea and periphery. Vision Res 18: 815-825.
34. Fletcher H, Galt R (1950) The perception of speech and its relation to telephony. J Acoust Soc Am 22: 89-151. Eq. 1.
35. Potter MC (1984) Rapid serial visual presentation (RSVP): A method for studying language processing. In: Kieras D, Just M, editors. New methods in reading comprehension research. Hillsdale, NJ: Erlbaum. 91-118.
36. Braida LD, (1991) Crossmodal integration in the identification of consonant segments. Q J Exp Psychol 43: 647-677.
37. Barlow HB, (1953) Summation and inhibition in the frog's retina. J Physiol (Lond) 119: 69-88.
38. Pelli DG, (1985) Uncertainty explains many aspects of visual contrast detection and discrimination. J Opt Soc Am A 2: 1508-1532.
39. Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK, (2005) Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci 25: 5004-5012.
40. Sumby W, Pollack I, (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26: 212-215.
41. Grant KW, Tufts JB, Greenberg S, (2007) Integration efficiency for speech perception within and across sensory modalities by normal-hearing and hearing-impaired individuals. J Acoust Soc Am 121: 1164-1176.
42. Mitterer H, McQueen JM, (2009) Foreign subtitles help but native-language subtitles harm foreign speech perception. PLoS ONE 4: e7785.
43. Miller GA (1951) Language and communication. New York, NY, US: McGraw-Hill. 298 p.
44. Elliott LL, (1995) Verbal auditory closure and the speech perception in noise (SPIN) Test. J Speech Hear Res 38: 1363-1376.
45. Alais D, Burr DC, (2004) No direction-specific bimodal facilitation for audiovisual motion detection. Cogn Brain Res 19: 185-194.
46. Pelli DG, Tillman KA, (2007) Parts, wholes, and context in reading: A triple dissociation. PLoS ONE 2: e680.
47. Miller GA, Heise GA, Lichten W, (1951) The intelligibility of speech as a function of the context of the test materials. J Exp Psychol Hum Percept Perform 41: 329-335.
48. Hirsh IJ, Reynolds EG, Joseph M, (1954) Intelligibility of different speech materials. J Acoust Soc Am 26: 530-538.
49. Rubenstein H, Decker L, Pollack I, (1959) Word length and intelligibility. Lang Speech 2: 175-178.
50. Burgess AE, Wagner RF, Jennings RJ, Barlow HB, (1981) Efficiency of human visual signal discrimination. Science 214: 93-94.
51. Gepshtein S, Burge J, Ernst MO, Banks MS, (2005) The combination of vision and touch depends on spatial proximity. J Vis 5: 1013-1023.
52. Gepshtein S, Banks MS, (2003) Viewing geometry determines how vision and haptics combine in size perception. Curr Biol 13: 483-488.
53. Arrighi R, Marini F, Burr D, (2009) Meaningful auditory information enhances perception of visual biological motion. J Vis 9: 25.21-27.
54. Meyer GF, Wuerger SM, Röhrbein F, Zetzsche C, (2005) Low-level integration of auditory and visual motion signals requires spatial co-localisation. Exp Brain Res 166: 538-547.
55. Gori M, Mazzilli G, Sandini G, Burr DC, (2011) Cross-sensory facilitation reveals neural interactions between visual and tactile motion in humans. Front Psychol 2: 1-9.
56. Green DM, McKay MJ, Licklider JCR, (1959) Detection of a pulsed sinusoid in noise as a function of frequency. J Acoust Soc Am 31: 1446-1452.
57. Näsänen R, Kukkonen H, Rovamo J, (1993) Spatial integration of band-pass filtered patterns in noise. Vision Res 33: 903-911.
58. Watson AB, (1979) Probability summation over time. Vision Res 19: 515-522.
59. Rovamo J, Kukkonen H, Raninen A, Donner K, (2003) Efficiency of temporal integration of sinusoidal flicker. Invest Ophthalmol Vis Sci 44: 5049-5055.
60. Watson AB, Thompson PG, Murphy BJ, Nachmias J, (1980) Summation and discrimination of gratings moving in opposite directions. Vision Res 20: 341-347.
61. Knill DC, Saunders JA, (2003) Do humans optimally integrate stereo and texture information for judgments of surface slant? Vision Res 43: 2539-2558.
62. Hillis JM, Watt SJ, Landy MS, Banks MS, (2004) Slant from texture and disparity cues: Optimal cue combination. J Vis 4: 967-992.
63. Oruç I, Maloney LT, Landy MS, (2003) Weighted linear cue combination with possibly correlated error. Vision Res 43: 2451-2468.
64. Rivest J, Cavanagh P, (1996) Localizing contours defined by more than one attribute. Vision Res 36: 53-66.
65. Landy MS, Kojima H, (2001) Ideal cue combination for localizing texture-defined edges. J Opt Soc Am A 18: 2307-2320.
66. Rosch E, Mervis CB, Gray WD, Johnson DM, Boyes-Braem P, (1976) Basic objects in natural categories. Cogn Psychol 8: 382-439.
67. Suchow JW, Pelli DG, (2013) Learning to detect and combine the features of an object. Proc Natl Acad Sci U S A 110: 785-790.
68. Holcombe AO, (2009) Seeing slow and seeing fast: Two limits on perception. Trends Cogn Sci 13: 216-221.
69. Wallace MT, Roberson GE, Hairston WD, Stein BE, Vaughan JW, et al. (2004) Unifying multisensory signals across time and space. Exp Brain Res 158: 252-258.
70. Welch RB, Warren DH, (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88: 638-667.
71. Sato Y, Toyoizumi T, Aihara K, (2007) Bayesian inference explains perception of unity and ventriloquism aftereffect: identification of common sources of audiovisual stimuli. Neural Comput 19: 3335-3355.
72. Körding KP, Beierholm U, Ma WJ, Quartz S, Tenenbaum JB, et al. (2007) Causal inference in multisensory perception. PLoS ONE 2: e943.
73. Brainard D, (1997) The Psychophysics Toolbox. Spat Vis 10: 433-436.
74. Pelli DG, (1997) The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spat Vis 10: 437-442.
75. Kleiner M, Brainard DH, Pelli DG (2007) What's new in Psychtoolbox-3? Perception 36: ECVP Abstract supplement.
76. Kwon M, Legge GE, Dubbels BR, (2007) Developmental changes in the visual span for reading. Vision Res 47: 2889-2900.
77. Kučera H, Francis WN (1967) Computational analysis of present-day American English. Providence, RI: Brown University Press. 424 p.
78. Tillman TW, Carhart R (1966) An expanded test for speech discrimination utilizing CNC monosyllabic words. Northwestern University Auditory Test No. 6. Aerospace Medical Division (AFSC). Brooks Air Force Base, Texas: USAF School of Aerospace Medicine. 1-12.
79. Boersma P, Weenink D (2012) "Praat: Doing phonetics by computer". [Computer program]. Version 5.1.43, retrieved from http://www.praat.org/.
80. Moulines E, Charpentier F, (1990) Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones. Speech Commun 9: 453-467.
81. van Wassenhove V, Grant KW, Poeppel D, (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45: 598-607.
82. Wilson RH, Coley KE, Haenel JL, Browning KM, (1976) Northwestern University Auditory Test No. 6: Normative and comparative intelligibility functions. J Am Audiol Soc 1: 221-228.
83. Watson AB, Pelli DG, (1983) QUEST: A Bayesian adaptive psychometric method. Percept Psychophys 33: 113-120.
84. Pelli DG, (1987) On the relation between summation and facilitation. Vision Res 27: 119-123.
85. Legge GE, Viemeister NF, (1988) Sensory analysis in vision and audition. Behav Brain Sci 11: 301-302.
86. Watson AB, Barlow HB, Robson JG, (1983) What does the eye see best? Nature 302: 419-422.
87. Barlow HB, (1980) The absolute efficiency of perceptual decisions. Philos Trans R Soc Lond B Biol Sci 290: 71-82.
88. Kersten D, (1987) Statistical efficiency for the detection of visual noise. Vision Res 27: 1029-1040.
89. van Meeteren A, Barlow HB, (1981) The statistical efficiency for detecting sinusoidal modulation of average dot density in random figures. Vision Res 21: 765-777.
90. Nachmias J, Sansbury RV, (1974) Grating contrast: Discrimination may be better than detection. Vision Res 14: 1039-1042.
91. Parish DH, Sperling G, (1991) Object spatial frequencies, retinal spatial frequencies, noise, and the efficiency of letter discrimination. Vision Res 31: 1399-1415.
92. Pelli DG, Farell B, (1999) Why use noise? J Opt Soc Am A 16: 647-653.