Integrating information from vision and touch: A neural network modeling study

IEEE Transactions on Information Technology in Biomedicine

Volumn 14, Issue 3, 2010, Pages 598-612

  Magosso, Elisa ^a  

^a UNIVERSITY OF BOLOGNA   (Italy)

Author keywords

Inverse effectiveness; Multisensory conflict; Multisensory perception; Tactile processing; Visual tactile enhancement

Indexed keywords

CLINICAL PRACTICES; CONFLICT SITUATION; CROSS-MODAL; FEEDFORWARD CONNECTIONS; INTEGRATING INFORMATION; LOW-INTENSITY; MULTISENSORY; MULTISENSORY PERCEPTION; NEURAL NETWORK MODEL; NEURAL NETWORK MODELING; SENSORY INFORMATION; SPATIAL LOCALIZATION; SPATIAL RESOLUTION; TACTILE INFORMATION; TACTILE INTERACTION; TACTILE STIMULI; TECHNOLOGICAL APPLICATIONS; UNIMODAL; VISUAL INFORMATION;

INTEGRATION; SENSORY PERCEPTION; VISUAL COMMUNICATION;

NEURAL NETWORKS;

ARTICLE; ARTIFICIAL NEURAL NETWORK; BIOLOGICAL MODEL; COMPUTER SIMULATION; FEEDBACK SYSTEM; HUMAN; PHYSIOLOGY; SIGNAL PROCESSING; SYNAPTIC POTENTIAL; TOUCH; VISION;

COMPUTER SIMULATION; FEEDBACK; HUMANS; MODELS, NEUROLOGICAL; NEURAL NETWORKS (COMPUTER); SIGNAL PROCESSING, COMPUTER-ASSISTED; SYNAPTIC POTENTIALS; TOUCH PERCEPTION; VISUAL PERCEPTION;

EID: 77953141188     PISSN: 10897771     EISSN: None     Source Type: Journal    
DOI: 10.1109/TITB.2010.2040750     Document Type: Article

References (66)

1. G. A. Calvert, C. Spence, and B. E. Stein, The Handbook of Multisensory Processes. Cambridge, MA: MIT Press, 2004.
2. B. E. Stein andM. A.Meredith, TheMerging of the Senses. Cambridge, MA: MIT Press, 1993.
3. J. R. Duhamel, C. L. Colby, and M. E. Goldberg, "Ventral intraparietal area of the macaque: Congruent visual and somatic response properties," J. Neurophysiol., vol.79, no.1, pp. 126-136, Jan. 1998.
4. E. Ladavas and A. Farǹe, "Visuo-tactile representation of near-the-body space," J. Physiol., Paris, vol.98, no.1-3, pp. 161-170, 2004.
5. E. Macaluso and J. Driver, "Spatial attention and the crossmodal interactions between vision and touch," Neuropsychologia, vol.39, no.12, pp. 1304-1316, 2001. (Pubitemid 32826274)
6. A. Maravita, C. Spence, and J. Driver, "Multisensory integration and the body schema: Close to hand and within reach," Curr. Biol., vol.13, no.13, pp. R531-R539, Jul. 2003.
7. P.W. Halligan, J. C. Marshall, M. Hunt, and D. T.Wade, "Somatosensory assessment: Can seeing produce feeling?," J. Neurol., vol.244, no.3, pp. 199-203, Mar. 1997.
8. E. L̀adavas,A. Farǹe, G. Zeloni, and G. di Pellegrino, "Seeing or not seeing where your hands are," Exp. Brain Res., vol.131, no.4, pp. 458-467, Apr. 2000.
9. T. J. Perrault, J. W. Vaughan, B. E. Stein, and M. T. Wallace, "Neuronspecific response characteristics predict the magnitude of multisensory integration," J. Neurophysiol., vol.90, no.6, pp. 4022-4026, Dec. 2003.
10. T. R. Stanford and B. E. Stein, "Superadditivity in multisensory integration: Putting the computation in context," Neuroreport, vol.18, no.8, pp. 787-792, May 2007.
11. L. Fogassi, V. Gallese, L. Fadiga, G. Luppino, M. Matelli, and G. Rizzolatti, "Coding of peripersonal space in inferior premotor cortex (area F4)," J. Neurophysiol., vol.76, no.1, pp. 141-157, Jul. 1996.
12. M. S. Graziano, G. S. Yap, and C. G. Gross, "Coding of visual space by premotor neurons," Science, vol.266, no.5187, pp. 1054-1057, Nov. 1994.
13. J. Driver and C. Spence, "Multisensory perception: Beyond modularity and convergence," Curr. Biol., vol.10, no.20, pp. R731-R735, Oct. 2000.
14. E. Macaluso and J. Driver, "Multisensory spatial interactions: A window onto functional integration in the human brain," Trends Neurosci., vol.28, no.5, pp. 264-271, May 2005.
15. S. Kennett, M. Taylor-Clarke, and P. Haggard, "Noninformative vision improves the spatial resolution of touch in humans," Curr. Biol., vol.11, no.15, pp. 1188-1191, Aug. 2001.
16. C. Press, M. Taylor-Clarke, S. Kennett, and P. Haggard, "Visual enhancement of touch in spatial body representation," Exp. Brain Res., vol.154, no.2, pp. 238-245, Jan. 2004.
17. A. Serino, A. Farǹe, M. L. Rinaldesi, P. Haggard, and E. L̀adavas, "Can vision of the body ameliorate impaired somatosensory function?" Neuropsychologia, vol.45, no.5, pp. 1101-1107, Mar. 2007.
18. M. Taylor-Clarke, S. Kennett, and P. Haggard, "Vision modulates somatosensory cortical processing," Curr. Biol., vol.12, no.3, pp. 233-236, Feb. 2002.
19. P. Haggard, A. Christakou, and A. Serino, "Viewing the body modulates tactile receptive fields," Exp. Brain Res., vol.180, no.1, pp. 187-193, Jun. 2007.
20. A. A. Ghazanfar and C. E. Schroeder, "Is neocortex essentially multisensory?" Trends Cogn. Sci., vol.10, no.6, pp. 278-285, Jun. 2006.
21. C. Kayser andN.K. Logothetis, "Do early sensory cortices integrate crossmodal information," Brain Struct. Funct., vol.212, no.2, pp. 121-132, Sep. 2007.
22. R. Martuzzi, M. M. Murray, C. M. Michel, J. P. Thiran, P. P. Maeder, S. Clarke, and R. A. Meuli, "Multisensory interactions within human primary cortices revealed by BOLD dynamics," Cereb. Cortex, vol.17, no.7, pp. 1672-1679, Jul. 2007.
23. C. E. Schroeder and J. Foxe, "Multisensory contributions to low-level, 'unisensory' processing," Curr. Opin. Neurobiol., vol.15, no.4, pp. 454-458, Aug. 2005.
24. J. Driver and T. Noesselt, "Multisensory interplay reveals cross-modal influences on 'sensory-specific' brain regions, neural responses and judgments," Neuron, vol.57, pp. 11-23, Jan. 2008.
25. E. R. Kandel, J. H. Schwartz, and T. M. Jessell, "The bodily senses," in Principles of Neural Science, 4th ed. New York: McGraw-Hill, 2000, pp. 430-450.
26. P. Dayan and L. F. Abbott, "Model neurons I: Neuroelectronics," in Theoretical Neuroscience. Cambridge,MA: MIT Press, 2001, pp. 153-194.
27. M. S. Graziano, X. T. Hu, and C. G. Gross, "Visuospatial properties of ventral premotor cortex," J. Neurophysiol., vol.77, no.5, pp. 2268-2292, May 1997.
28. G. Rizzolatti, C. Scandolara, M. Matelli, and M. Gentilucci, "Afferent properties of periarcuate neurons in macaque monkeys. II. Visual responses," Behav. Brain Res., vol.2, no.2, pp. 147-163, Mar. 1981. (Pubitemid 11186553)
29. M. Avillac, H. S. Ben, and J. R. Duhamel, "Multisensory integration in the ventral intraparietal area of the macaque monkey," J. Neurosci., vol.27, no.8, pp. 1922-1932, Feb. 2007.
30. T. R. Stanford, S. Quessy, and B. E. Stein, "Evaluating the operations underlying multisensory integration in the cat superior colliculus," J. Neurosci., vol.25, no.28, pp. 6499-6508, Jul. 2005.
31. M. O. Ernst and H. H. B̈ulthoff, "Merging the senses into a robust percept," Trends Cogn. Sci., vol.8, no.4, pp. 162-169, Apr. 2004.
32. N. W. Roach, J. Heron, and P. V. McGraw, "Resolving multisensory conflict: A strategy for balancing the costs and benefits of audio-visual integration," Proc. R. Soc. B, vol.273, no.1598, pp. 2159-2168, Sep. 2006.
33. T. J. Anastasio, P. E. Patton, and K. Belkacem-Boussaid, "Using Bayes' rule to model multisensory enhancement in the superior colliculus," Neural Comput., vol.12, no.5, pp. 1165-1187, May 2000.
34. P. Patton, K. Belkacem-Boussaid, and T. J. Anastasio, "Multimodality in the superior colliculus: An information theoretic analysis," Brain Res. Cogn. Brain Res., vol.14, no.1, pp. 10-19, Jun. 2002.
35. T. J. Anastasio and P. E. Patton, "A two-stage unsupervised learning algorithm reproduces multisensory enhancement in a neural network model of the corticotectal system," J. Neurosci., vol.23, no.17, pp. 6713-6727, Jul. 2003.
36. J. G. Martin,M. A.Meredith, and K. Ahmad, "Modeling multisensory enhancement with self-organizing maps," Front. Comput. Neurosci., vol.3, pp. 1-10, Jun. 2009.
37. J. C. Alvarado, B. A. Rowland, T. R. Stanford, and B. E. Stein, "A neural network model of multisensory integration also accounts for unisensory integration in superior colliculus," Brain Res., vol.1242, pp. 13-23, Nov. 2008.
38. B. A. Rowland, T. R. Stanford, and B. E. Stein, "A model of the neural mechanisms underlying multisensory integration in the superior colliculus," Perception, vol.36, no.10, pp. 1431-1443, 2007.
39. E. Magosso, C. Cuppini, A. Serino, G. di Pellegrino, and M. Ursino, "A theoretical study of multisensory integration in the superior colliculus by a neural network model," Neural Netw., vol.21, no.6, pp. 817-829, Aug. 2008.
40. M. Ursino, C. Cuppini, E. Magosso, A. Serino, and G. di Pellegrino, "Multisensory integration in the superior colliculus: A neural network model," J. Comput. Neurosci., vol.26, no.1, pp. 55-73, Feb. 2009.
41. A. Pouget and J. T. Sejnowski, "Spatial transformations in the parietal cortex using basis functions," J. Cogn. Neurosci., vol.9, pp. 222-237, 1997.
42. J. Xing and R. Andersen, "Models of the posterior parietal cortex which perform multimodal integration and represent space in several coordinate frames," J. Cogn. Neurosci., vol.12, no.4, pp. 601-614, Jul. 2000.
43. M. Avillac, S. Deǹeve, E. Olivier, A. Pouget, and J. R. Duhamel, "Reference frames for representing visual and tactile locations in parietal cortex," Nat. Neurosci., vol.8, no.7, pp. 941-949, Jul. 2005.
44. A. Pouget, S. Deneve, and J. R. Duhamel, "A computational perspective on the neural basis of multisensory spatial representations," Nat. Rev. Neurosci., vol.3, no.9, pp. 741-747, Sep. 2002.
45. E. T. Rolls and G. Deco, Computational Neuroscience of Vision. New York: Oxford Univ. Press, 2002.
46. J. P. Thivierge and G. F. Marcus, "The topographic brain: From neural connectivity to cognition," Trends Neurosci., vol.30, no.6, pp. 251-259, Jun. 2007.
47. F. Bremmer, A. Schlack, N. J. Shah, O. Zafiris, M. Kubischik, K. Hoffmann, K. Zilles, and G. R. Fink, "Polymodal motion processing in posterior parietal and premotor cortex: A human fMRI study strongly implies equivalencies between humans and monkeys," Neuron, vol.29, no.1, pp. 287-296, Jan. 2001.
48. E. Macaluso, C. D. Frith, and J. Driver, "Modulation of human visual cortex by crossmodal spatial attention," Science, vol.289, no.5482, pp. 1206-1208, Aug. 2000.
49. T. Ro, R. Wallace, J. Hagedorn, A. Farǹe, and E. Pienkos, "Visual enhancing of tactile perception in the posterior parietal cortex," J. Cogn. Neurosci., vol.16, no.1, pp. 24-30, 2004.
50. C. Cappe, E. M. Rouiller, and P. Barone, "Multisensory anatomical pathways," Hear. Res., vol. 258, no. 1/2, pp. 28-36, May 2009.
51. A. J. Hall and S. G. Lomber, "Auditory cortex projections target the peripheral field representation of primary visual cortex," Exp. Brain Res., vol.190, no.4, pp. 413-430, Oct. 2008.
52. R. W. Dykes, "Mechanisms controlling neuronal plasticity in somatosensory cortex," Can. J. Physiol. Pharmacol., vol.75, no.5, pp. 535-545, May 1997.
53. H. Ozeki, I. M. Finn, E. S. Schaffer, K. D. Miller, and D. Ferster, "Inhibitory stabilization of the cortical network underlies visual surround suppression," Neuron, vol.62, no.4, pp. 578-592, May 2009.
54. S. Dehaene and L. Naccache, "Towards a cognitive neuroscience of consciousness: Basic evidence and a workspace framework," Cognition, vol. 79, no. 1/2, pp. 1-37, Apr. 2001
55. M. Sarri, F. Blankenburg, and J. Driver, "Neural correlates of crossmodal visual-tactile extinction and of tactile awareness revealed by fMRI in a right-hemisphere stroke patient," Neuropsychologia, vol.44, no.12, pp. 2398-2410, 2006.
56. R. P. Power, "The dominance of touch by vision: occurs with familiar objects," Perception, vol.10, no.1, pp. 29-33, 1981.
57. I. Rock and J. Victor, "Vision and touch: An experimentally created conflict between the two senses," Science, vol.143, pp. 594-596, Feb. 1964.
58. M. O. Ernst and M. S. Banks, "Humans integrate visual and haptic information in a statistically optimal fashion," Nature, vol.415, no.6870, pp. 429-433, Jan. 2002.
59. H. B. Helbig and M. O. Ernst, "Optimal integration of shape information from vision and touch," Exp. Brain Res., vol.179, no.4, pp. 595-606, Jun. 2007.
60. D. Alais and D. Burr, "The ventriloquist effect results from near-optimal bimodal integration," Curr. Biol., vol.14, pp. 257-262, Feb. 2004.
61. E. Ladavas, "Multisensory-based approach to the recovery of unisensory deficit," Ann. N. Y. Acad. Sci., vol.1124, pp. 98-110, Mar. 2008.
62. J. S. Feigenson, F. H. McDowell, P. Meese, M. L. McCharty, and S. D. Greenberg, "Factors influencing outcome and length of stay in a stroke rehabilitation unit. Part 1. Analysis of 248 unscreened patients-Medical and functional prognostic indicators," Stroke, vol.8, no.6, pp. 651-656, 1977.
63. J. S. Feigenson, M. L. McCharty, S. D. Greenberg, and W. D. Feigenson, "Factors influencing outcome and length of stay in a stroke rehabilitation unit. Part 2. Comparison of 318 screened and 248 unscreened patients," Stroke, vol.8, no.6, pp. 657-662, 1977.
64. C. Ho, H. Z. Tan, and C. Spence, "The differential effect of vibrotactile and auditory cues on visual spatial attention," Ergonomics, vol.49, no.7, pp. 724-738, Jun. 2006.
65. C. Ho, N. Reed, and C. Spence, "Multisensory in-car warning signals for collision avoidance," Hum. Factors, vol.49, no.6, pp. 1107-1114, Dec. 2007.
66. J. B. F. Van Erp and H. A. H. C. Van Veen, "Vibrotactile in-vehicle navigation system," Transp. Res. F, Traffic Psychol. Behav., vol.7, pp. 247-256, 2004.