Emergence of multimodal action representations from neural network self-organization

Cognitive Systems Research

Volumn 43, Issue , 2017, Pages 208-221

  Parisi, German I ^a   Tani, Jun ^b   Weber, Cornelius ^a   Wermter, Stefan ^a  

^a UNIVERSITY OF HAMBURG   (Germany)

^b Korea Advanced Institute of Science and Technology (KAIST)   (South Korea)

Author keywords

Human action recognition; multimodal integration; self organizing neural networks

Indexed keywords

CLASSIFICATION (OF INFORMATION); IMAGE RECOGNITION; NETWORK ARCHITECTURE; NEURAL NETWORKS;

AUTOMATIC SPEECH RECOGNITION; CLASSIFICATION PERFORMANCE; HIERARCHICAL ARCHITECTURES; HUMAN-ACTION RECOGNITION; MULTIMODAL INTEGRATION; SELF-ORGANIZING NEURAL NETWORK; SEMI-SUPERVISED ALGORITHM; SPATIO-TEMPORAL DYNAMICS;

SPEECH RECOGNITION;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; AUDITORY SYSTEM; BRAIN CORTEX; CONCEPT FORMATION; GROWTH RATE; HUMAN; MOTOR SYSTEM; NERVOUS SYSTEM DEVELOPMENT; NEUROBIOLOGY; NEUROPHYSIOLOGY; PHONEME; PREMOTOR CORTEX; ROBOTICS; SOMATOSENSORY CORTEX; STATE DEPENDENT LEARNING; STIMULUS RESPONSE; VISUAL MEMORY;

EID: 85016777125     PISSN: None     EISSN: 13890417     Source Type: Journal    
DOI: 10.1016/j.cogsys.2016.08.002     Document Type: Article

References (64)

1. Adolphs, R., Cognitive neuroscience of human social behaviour. Nature Reviews Neuroscience 4 (2003), 165–178, 10.1038/nrn1056.
2. Allison, T., Puce, A., McCarthy, G., Social perception from visual cues: Role of the STS region. Trends in Cognitive Sciences 4 (2000), 267–278, 10.1016/S1364-6613(00)01501-1.
3. Barraclough, N.E., Xiao, D., Baker, C.I., Oram, M.W., Perrett, D.I., Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. Journal of Cognitive Neuroscience 17 (2005), 377–391, 10.1162/0898929053279586.
4. Bauer, J., Magg, S., Wermter, S., Attention modeled as information in learning multisensory integration. Neural Networks 65 (2015), 44–52, 10.1016/j.neunet.2015.01.004.
5. Beauchamp, M.S., See me, hear me, touch me: Multisensory integration in lateral occipital-temporal cortex. Current Opinion in Neurobiology 15 (2005), 145–153, 10.1016/j.conb.2005.03.011 Cognitive Neuroscience.
6. Beauchamp, M.S., Lee, K.E., Argall, B.D., Martin, A., Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41 (2004), 809–823, 10.1016/S0896-6273(04)00070-4.
7. Beauchamp, M.S., Yasar, N.E., Frye, R.E., Ro, T., Touch, sound and vision in human superior temporal sulcus. NeuroImage 41 (2008), 1011–1020, 10.1016/j.neuroimage.2008.03.015.
8. Belin, P., Zatorre, R.J., Ahad, P., Human temporal-lobe response to vocal sounds. Cognitive Brain Research 13 (2002), 17–26, 10.1016/S0926-6410(01)00084-2.
9. Belin, P., Zatorre, R.J., Lafaille, P., Ahad, P., Pike, B., Voice-selective areas in human auditory cortex. Nature 403 (2000), 309–312.
10. Beyer, O., Cimiano, P., Online labelling strategies for growing neural gas. Yin, H., Wang, W., Rayward-Smith, V., (eds.) Proceedings of the international conference on intelligent data engineering and automated learning (IDEAL), 2011, Springer, Berlin, Heidelberg, 76–83, 10.1007/978-3-642-23878-9_10.
11. Bloom, L., The transition from infancy to language: Acquiring the power of expression. 1993, Cambridge University Press, New York.
12. Bushara, K., Grafman, J., Hallett, M., Neural correlates of auditory-visual stimulus onset asynchrony detection. Journal of Neuroscience 21 (2001), 300–304.
13. Calvert, G.A., Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex 11 (2001), 1110–1123, 10.1093/cercor/11.12.1110.
14. Calvert, G.A., Campbell, R., Brammer, M.J., Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology 10 (2000), 649–657, 10.1016/S0960-9822(00)00513-3.
15. Eriksson, P.S., Perfilieva, E., Bjork-Eriksson, T., Alborn, A.M., Nordborg, C., Peterson, D.A., Gage, F.H., Neurogenesis in the adult human hippocampus. Nature Medicine 4 (1998), 1313–1317, 10.1038/3305.
16. Ernst, M.O., Bulthoff, H., Merging the senses into a robust percept. Trends in Cognitive Sciences 8 (2004), 162–169, 10.1016/j.tics.2004.02.002.
17. Fiebelkorn, I.C., Foxe, J.J., Molholm, S., Dual mechanisms for the cross-sensory spread of attention: How much do learned associations matter?. Cerebral Cortex 20 (2009), 109–120, 10.1093/cercor/bhp083.
18. Fonlupt, P., Perception and judgement of physical causality involve different brain structures. Cognitive Brain Research 17 (2003), 248–254, 10.1016/S0926-6410(03)00112-5.
19. Foxe, J.J., Morocz, I.A., Murray, M.M., Higgins, B.A., Javitt, D.C., Schroeder, C.E., Multisensory auditory–somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Cognitive Brain Research 10 (2000), 77–83, 10.1016/S0926-6410(00)00024-0.
20. Fritzke, B., A growing neural gas network learns topologies. Advances in neural information processing systems, Vol. 7, 1995, MIT Press, 625–632.
21. Fritzke, B., A self-organizing network that can follow non-stationary distributions. ICANN’97: International conference on artificial neural networks, 1997, Springer, 613–618.
22. Garcia, J.O., Grossman, E.D., Necessary but not sufficient: Motion perception is required for perceiving biological motion. Vision Research 48 (2008), 1144–1149, 10.1016/j.visres.2008.01.027.
23. Gentner, D., Why nouns are learned before verbs: Linguistic relativity versus natural partitioning. Language Development: Language, Thought, and Culture 2 (1982), 301–334.
24. Giese, M.A., Poggio, T., Neural mechanisms for the recognition of biological movements. Nature Reviews Neuroscience 4 (2003), 179–192, 10.1038/nrn1057.
25. Giese, M.A., Rizzolatti, G., Neural and computational mechanisms of action processing: Interaction between visual and motor representations. Neuron 88 (2015), 167–180, 10.1016/675 j.neuron.2015.09.040.
26. Gould, E., How widespread is adult neurogenesis in mammals?. Nature Reviews Neuroscience 8 (2007), 481–488, 10.1038/nrn2147.
27. Hasson, U., Yang, E., Vallines, I., Heeger, D.J., Rubin, N., A hierarchy of temporal receptive windows in human cortex. The Journal of Neuroscience 28 (2008), 2539–2550, 10.1523/jneurosci.5487-07.2008.
28. Hirsch-Pasek, K., Golinkoff, R., Hollich, G., An emergentist coalition model for word learning: Mapping words to objects is a product of the interaction of multiple cues. 2000, Oxford University Press, 136–165.
29. Jung, M., Hwang, J., Tani, J., Self-organization of spatio-temporal hierarchy via learning of dynamic visual image patterns on action sequences. PLoS One, 10, 2015, e0131214, 10.1371/journal.pone.0131214.
30. Kachouie, R., Sedighadeli, S., Khosla, R., Chu, M., Socially assistive robots in elderly care: A mixed-method systematic literature review. International Journal of Human-Computer Interaction 30 (2014), 369–393, 10.1080/10447318.2013.873278.
31. Kohonen, T., Self-organized formation of topologically correct feature maps. Biological Cybernetics 43 (1988), 59–69.
32. Layher, G., Giese, M.A., Neumann, H., Learning representations of animated motion sequences? A neural model. Topics in Cognitive Science 6 (2014), 170–182, 10.1111/tops.12075.
33. Lerner, Y., Honey, C.J., Silbert, L.J., Hasson, U., Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. The Journal of Neuroscience 31 (2011), 2906–2915, 10.1523/jneurosci.3684-10.2011.
34. Macaluso, E., George, N., Dolan, R., Spence, C., Driver, J., Spatial and temporal factors during processing of audiovisual speech: A {PET} study. NeuroImage 21 (2004), 725–732, 10.1016/j.neuroimage.2003.09.049.
35. Marsland, S., Nehmzow, U., Shapiro, J., On-line novelty detection for autonomous mobile robots. Robotics and Autonomous Systems 51 (2005), 191–206.
36. Marsland, S., Shapiro, J., Nehmzow, U., A self-organising network that grows when required. Neural Networks 15 (2002), 1041–1058.
37. Martinetz, T.M., Competitive Hebbian learning rule forms perfectly topology preserving maps. ICANN’93: International conference on artificial neural networks, 1993, Springer, Amsterdam, 427–434.
38. Miikkulainen, R., Bednar, J.A., Choe, Y., Sirosh, J., Computational maps in the visual cortex. 2005, Springer, 10.1007/0-387-28806-6.
39. Miller, L.E., Saygin, A.P., Individual differences in the perception of biological motion: Links to social cognition and motor imagery. Cognition 128 (2013), 140–148, 10.1016/j.cognition.2013.03.013.
40. Mineiro, P., Zipser, D., Analysis of direction selectivity arising from recurrent cortical interactions. Neural Computation 10 (1998), 353–371, 10.1162/089976698300017791.
41. Ming, G.l., Song, H., Adult neurogenesis in the mammalian brain: Significant answers and significant questions. Neuron 70 (2011), 687–702, 10.1016/j.neuron.2011.05.001 http://dx.doi.org/10.1038/nrn2147.
42. Morse, A.F., Benitez, V.L., Belpaeme, T., Cangelosi, A., Smith, L.B., Posture affects how robots and infants map words to objects. PLoS One, 10, 2015, e0116012, 10.1371/journal.pone.0116012.
43. Nelson, C.A., Neural plasticity and human development: The role of early experience in sculpting memory systems. Developmental Science 3 (2000), 115–136.
44. Noda, K., Arie, H., Suga, Y., Ogata, T., Multimodal integration learning of robot behavior using deep neural networks. Robotics and Autonomous Systems 62 (2014), 721–736, 10.1016/j.robot.2014.03.003.
45. Papadopoulos, G.T., Axenopoulos, A., Daras, P., Real-time skeleton-tracking-based human action recognition using kinect data. MultiMedia modeling, 2014, Springer, 473–483, 10.1007/978-3-319-04114-8_40.
46. Parisi, G.I., Weber, C., Wermter, S., Human action recognition with hierarchical growing neural gas learning. Artificial neural networks and machine learning (ICANN), 2014, 89–96, 10.1007/978-3-319-11179-7_12.
47. Parisi, G.I., Weber, C., Wermter, S., Self-organizing neural integration of pose-motion features for human action recognition. Frontiers in Neurorobotics, 9, 2015, 10.3389/fnbot.735 2015.00003.
48. Pulvermüller, F., Brain mechanisms linking language and action. Nature Reviews Neuroscience 6 (2005), 576–582, 10.1038/nrn1706.
49. Raij, T., Uutela, K., Hari, R., Audiovisual integration of letters in the human brain. Neuron 28 (2000), 617–625, 10.1016/S0896-6273(00)00138-0.
50. Senkowski, D., Saint-Amour, D., Hfle, M., Foxe, J.J., Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness. NeuroImage 56 (2011), 2200–2208, 10.1016/j.neuroimage.2011.03.075.
51. Smith, L., Yu, C., Infants rapidly learn word-referent mappings via cross-situational statistics. Cognition 106 (2008), 1558–1568, 10.1016/j.cognition.2007.06.010.
52. Stanley, J.C., Computer simulation of a model of habituation. Nature 261 (1976), 146–148, 10.1038/261146a0.
53. Stein, B.E., Meredith, M.A., The merging of the senses. 1993, The MIT Press, Cambridge, MA, US.
54. Stein, B.E., Stanford, T.R., Rowland, B.A., The neural basis of multisensory integration in the midbrain: Its organization and maturation. Hearing Research 258 (2009), 4–15, 10.1016/j.heares.2009.03.012.
55. Stevenson, R.A., James, T.W., Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition. NeuroImage 44 (2009), 1210–1223, 10.1016/j.neuroimage.2008.09.034.
56. Strickert, M., Hammer, B., Merge SOM for temporal data. Neurocomputing, 64, 2005, 10.1016/j.neucom.2004.11.014.
57. Taylor, P., Hobbs, J.N., Burroni, J., Siegelmann, H.T., The global landscape of cognition: Hierarchical aggregation as an organizational principle of human cortical networks and functions. Scientific Reports, 5, 2015, 10.1038/srep18112.
58. Thirkettle, M., Benton, C.P., Scott-Samuel, N.E., Contributions of form, motion and task to biological motion perception. Journal of Vision, 9, 2009, 28, 10.1167/9.3.28.
59. Twiefel, J., Baumann, T., Heinrich, S., Wermter, S., Improving domain-independent cloud-based speech recognition with domain-dependent phonetic post-processing. AAAI conference on artificial intelligence, 2014, 1529–1536.
60. Ursino, M., Cuppini, C., Magosso, E., Neurocomputational approaches to modelling multisensory integration in the brain: A review. Neural Networks 60 (2014), 141–165, 10.1016/j.neunet.2014.08.003.
61. Vangeneugden, J., Pollick, F., Vogels, R., Neural and computational mechanisms of action processing: Interaction between visual and motor representations. Cerebral Cortex 19 (2009), 593–611, 10.1093/cercor/bhn109.
62. Vavrečka, M., Farkaš, I., A multimodal connectionist architecture for unsupervised grounding of spatial language. Cognitive Computation 6 (2014), 101–112, 10.1007/s12559-013-9212-5.
63. Willshaw, D.J., von der Malsburg, C., How patterned neural connections can be set up by self-organization. Proceedings of the Royal Society of London B: Biological Sciences 194 (1976), 431–445.
64. Wright, T.M., Pelphrey, K.A., Allison, T., Mckeown, M.J., Mccarthy, G., Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cerebral Cortex 13 (2003), 1034–1043, 10.1093/cercor/13.10.1034.