Audiovisual integration in the human perception of materials

Journal of Vision

Volumn 14, Issue 4, 2014, Pages

  Fujisaki, Waka ^a   Goda, Naokazu ^b   Motoyoshi, Isamu ^c,d   Komatsu, Hidehiko ^a   Nishida, Shin'ya ^c  

^a NATIONAL INSTITUTE OF ADVANCED INDUSTRIAL SCIENCE AND TECHNOLOGY AIST   (Japan)

^b NATIONAL INSTITUTE FOR PHYSIOLOGICAL SCIENCES   (Japan)

^c NTT CORPORATION   (Japan)

^d UNIVERSITY OF TOKYO   (Japan)

Author keywords

Audio visual integration; Bayesian integration; Impact sound; Material perception; Surface texture

Indexed keywords

ADULT; ARTICLE; AUDIO-VISUAL INTEGRATION; BAYES THEOREM; BAYESIAN INTEGRATION; FEMALE; HEARING; HUMAN; IMPACT SOUND; MALE; MATERIAL PERCEPTION; METHODOLOGY; PATTERN RECOGNITION; PHOTOSTIMULATION; PHYSIOLOGY; QUESTIONNAIRE; SOUND; SURFACE TEXTURE; VISION; YOUNG ADULT;

AUDIO-VISUAL INTEGRATION; BAYESIAN INTEGRATION; IMPACT SOUND; MATERIAL PERCEPTION; SURFACE TEXTURE;

ADULT; AUDITORY PERCEPTION; BAYES THEOREM; FEMALE; FORM PERCEPTION; HUMANS; MALE; PHOTIC STIMULATION; QUESTIONNAIRES; SOUND; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84899085326     PISSN: None     EISSN: 15347362     Source Type: Journal    
DOI: 10.1167/14.4.12     Document Type: Article

References (47)

1. Adelson, E. H. (2001). On seeing stuff: The perception of materials by humans and machines. Proceedings of the SPIE, Vol. 4299, Human Vision and Electronic Imaging VI, San Jose, CA.
2. Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14(3), 257-262.
3. Anderson, B. L. (2011). Visual perception of materials and surfaces. Curr Biol, 21(24), R978-R983.
4. Aramaki, M., Besson, M., Kronland-Martinet, R.,& Ystad, S. (2011). Controlling the perceived material in an impact sound synthesizer. IEEE Transactions on Audio, Speech, and Language Processing, 19(2), 301-314.
5. Arnott, S. R., Cant, J. S., Dutton, G. N., & Goodale, M. A. (2008). Crinkling and crumpling: An auditory fMRI study of material properties. NeuroImage, 43(2), 368-378.
6. Cant, J. S., & Goodale, M. A. (2007). Attention to form or surface properties modulates different regions of human occipitotemporal cortex. Cerebral Cortex, 17(3), 713-731.
7. Cant, J. S., & Goodale, M. A. (2011). Scratching beneath the surface: New insights into the functional properties of the lateral occipital area and parahippocampal place area. The Journal of Neuroscience, 31(22), 8248-8258.
8. Cavina-Pratesi, C., Kentridge, R. W., Heywood, C. A., & Milner, A. D. (2010a). Separate channels for processing form, texture, and color: Evidence from fMRI adaptation and visual object agnosia. Cerebral Cortex, 20(10), 2319-2332.
9. Cavina-Pratesi, C., Kentridge, R. W., Heywood, C. A., & Milner, A. D. (2010b). Separate processing of texture and form in the ventral stream: Evidence from fMRI and visual agnosia. Cerebral Cortex, 20(2), 433-446.
10. Cochran, W. (1937). Problems arising in the analysis of a series of similar experiments. Journal of the Royal Statistical Society (Supplement), 4(1), 102-118.
11. Cunningham, D., Wallraven, C., Fleming, R.,& Strasser, W. (2007). Perceptual reparameterization of material properties. Proceedings of the Computational Aesthetics in Graphics, Visualization, and Imaging, Banff, Alberta, Canada.
12. Doerschner, K., Boyaci, H., & Maloney, L. (2010). Estimating the glossiness transfer function induced by illumination change. Journal of Vision, 10(4):8, 1-9, http://www.journalofvision.org/content/10/4/8, doi:10.1167/10.4.8.
13. Ernst, M. O. (2006). A Bayesian view on multimodal cue integration. In G. Knoblich, I. M. Thornton, M. Grosjean, & M. Shiffrar (Eds.), Human body perception from the inside out (pp. 105-131). Oxford, UK: Oxford University Press.
14. Ernst, M. O. (2012). Optimal multisensory integration: Assumptions and limits. In B. E. Stein (Ed.), The new handbook of multisensory processes (pp. 1084-1124). Cambridge, MA: MIT Press.
15. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415(6870), 429-433.
16. Fleming, R. W. (2014). Visual perception of materials and their properties. Vision Research, 94, 62-75. Fleming, R. W., Wiebel, C., & Gegenfurtner, K. (2013).
17. Fleming, R. W., Wiebel, C., & Gegenfurtner, K. (2013). Perceptual qualities and material classes. Journal of Vision, 13(8):9, 1-20, http://www.journalofvision. org/content/13/8/9, doi:10.1167/13.8.9.
18. Fujisawa, N., Iwamiya, S., & Takada, M. (2004). Auditory imagery associated with Japanese onomatopoeic representation. Journal of Physiological Anthropology and Applied Human Science, 23(6), 351-355.
19. Gabrielsson, A., & Sjogren, H. (1979). Perceived sound quality of sound-reproducing systems. Journal of the Acoustical Society of America, 65(4), 1019-1033.
20. Giordano, B. L., & McAdams, S. (2006). Material identification of real impact sounds: Effects of size variation in steel, glass, wood, and plexiglass plates. Journal of the Acoustical Society of America, 119(2), 1171-1181.
21. Goda, N., Tachibana, A., Okazawa, G., & Komatsu, H. (2014). Representation of the material properties of objects in the visual cortex of non-human primates. Journal of Neuroscience, 34(7), 2660-2673.
22. Hiramatsu, C., Goda, N., & Komatsu, H. (2011). Transformation from image-based to perceptual representation of materials along the human ventral visual pathway. Neuroimage, 57(2), 482-494.
23. Kim, J., Marlow, P. J., & Anderson, B. L. (2012). The dark side of gloss. Nature Neuroscience, 15(11), 1590-1595.
24. Klatzky, R., Pai, D., & Krotkov, E. (2000). Perception of material from contact sounds. Presence: Teleoperators & Virtual Environments, 9(4), 399-410.
25. Klatzky, R. L., & Lederman, S. J. (2010). Multisensory texture perception. In J. Kaiser & M. J. Naumer (Eds.), Multisensory object perception in the primate brain (pp. 211-230). New York: Springer.
26. Landy, M. S., Maloney, L. T., Johnston, E. B.,& Young, M. (1995). Measurement and modeling of depth cue combination: In defense of weak fusion. Vision Research, 35(3), 389-412.
27. Lederman, S. J., Thorne, G., & Jones, B. (1986). Perception of texture by vision and touch: Multidimensionality and intersensory integration. Journal of Experimental Psychology-Human Perception and Performance, 12(2), 169-180.
28. Lemaitre, G., & Heller, L. M. (2012). Auditory perception of material is fragile while action is strikingly robust. Journal of the Acoustical Society of America, 131(2), 1337-1348.
29. Lutfi, R. A., & Oh, E. L. (1997). Auditory discrimination of material changes in a struck-clamped bar. Journal of the Acoustical Society of America, 102(6), 3647-3656.
30. Maloney, L. T., & Brainard, D. H. (2010). Color and material perception: Achievements and challenges. Journal of Vision, 10(9):19, 1-6, http://www.journalofvision.org/content/10/9/19, doi:10.1167/10.9.19.
31. Massaro, D. (1987). Speech perception by ear and eye: A paradigm for psychological inquiry. Hillsdale, NJ: Erlbaum.
32. Massaro, D. (2004). From multisensory integration to talking heads and language learning. In G. Calvert, C. Spence, & B. E. Stein (Eds.), Handbook of multisensory processes (pp. 153-176). Cambridge, MA: MIT Press.
33. Massaro, D., & Stork, D. (1998). Speech recognition and sensory integration: A 240-year-old theorem helps explain how people and machines can integrate auditory and visual information to understand speech. American Scientist, 86(3), 236-244.
34. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746-748. Motoyoshi, I., Nishida, S., Sharan, L., & Adelson, E. H. (2007). Image statistics and the perception of surface qualities. Nature, 447(7141), 206-209.
35. Motoyoshi, I., Nishida, S., Sharan, L., & Adelson, E. H. (2007). Image statistics and the perception of surface qualities. Nature, 447(7141), 206-209.
36. Nishida, S., & Shinya, M. (1998). Use of image-based information in judgments of surface-reflectance properties. The Journal of the Optical Society of America A, Optics, Image Science, and Vision, 15(12), 2951-2965.
37. Nishio, A., Goda, N., & Komatsu, H. (2012). Neural selectivity and representation of gloss in the monkey inferior temporal cortex. The Journal of Neuroscience, 32(31), 10780-10793.
38. Osgood, C. E., & Anderson, L. (1957). Certain relations among experienced contingencies, associative structure, and contingencies in encoded messages. American Journal of Psychology, 70(3), 411-420.
39. Sharan, L., Liu, C., Rosenholtz, R., & Adelson, E. (2013). Recognizing materials using perceptually inspired features. International Journal of Computer Vision, 103, 348-371.
40. Sharan, L., Rosenholtz, R., & Adelson, E. H. (2009). Material perception: What can you see in a brief glance? Journal of Vision, 9(8): 784, http://www.journalofvision.org/content/9/8/784, doi:10.1167/9. 8.784. [Abstract] .
41. Solomon, L. (1958). Semantic approach to the perception of complex sounds. The Journal of the Acoustical Society of America, 30(5), 421-425.
42. von Bismarck, G. (1974a). Sharpness as an attribute of the timbre of steady sounds. Acustica, 30, 159-172.
43. von Bismarck, G. (1974b). Timbre of steady sounds: A factorial investigation of its verbal attributes. Acustica, 30, 146-159.
44. Wildes, R. P., & Richards, W. A. (1988). Recovering material properties from sound. In W. A. Richards (Ed.), Natural computation (pp. 356-363). Cambridge, MA: The MIT Press.
45. Wijntjes, M. W. A., & Pont, S. C. (2010). Illusory gloss on Lambertian surfaces. Journal of Vision, 10(9):13, 1-12, http://www.journalofvision.org/content/10/9/13, doi:10.1167/10.9.13.
46. Yuille, A., & Bülthoff, H. (1996). Bayesian theory and psychophysics. In D. Knill & W. Richards (Eds.), Perception as Bayesian inference (pp. 123-161). New York: Cambridge University Press.
47. Zaidi, Q. (2011). Visual inferences of material changes: Color as clue and distraction. Wiley Interdisciplinary Reviews: Cognitive Science, 2(6), 686-700.