Lifelong learning of human actions with deep neural network self-organization

Neural Networks

Volumn 96, Issue , 2017, Pages 137-149

  Parisi, German I ^a   Tani, Jun ^b   Weber, Cornelius ^a   Wermter, Stefan ^a  

^a UNIVERSITY OF HAMBURG   (Germany)

^b OKINAWA INSTITUTE OF SCIENCE AND TECHNOLOGY GRADUATE UNIVERSITY   (Japan)

Author keywords

Action recognition; Lifelong learning; Self organizing neural networks; Unsupervised deep learning

Indexed keywords

CLASSIFICATION (OF INFORMATION); DEEP NEURAL NETWORKS; IMAGE RECOGNITION; NETWORK ARCHITECTURE; NEURAL NETWORKS; RECURRENT NEURAL NETWORKS;

ACTION RECOGNITION; ACTION REPRESENTATIONS; ADDITIONAL EXPERIMENTS; GROWING SELF-ORGANIZING NETWORKS; LIFE LONG LEARNING; SELF-ORGANIZING NEURAL NETWORK; SPATIOTEMPORAL RECEPTIVE FIELD; STATE-OF-THE-ART METHODS;

DEEP LEARNING;

ARTICLE; ARTIFICIAL NEURAL NETWORK; CLASSIFICATION ALGORITHM; HUMAN ACTIVITIES; MEASUREMENT ACCURACY; PREDICTION; PRIORITY JOURNAL; UNSUPERVISED MACHINE LEARNING; VIDEORECORDING; HUMAN; MACHINE LEARNING; NERVE CELL; PATTERN RECOGNITION; PHYSIOLOGY; PROCEDURES; ROBOTICS; TRENDS;

HUMANS; MACHINE LEARNING; NEURAL NETWORKS (COMPUTER); NEURONS; PATTERN RECOGNITION, VISUAL; ROBOTICS;

EID: 85030761514     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2017.09.001     Document Type: Article

References (82)

1. Andreakis, A., von Hoyningen-Huene, N., Beetz, M., Incremental unsupervised time series analysis using merge growing neural gas. WSOM Lecture Notes in Computer Science, vol. 5629, 2009, Springer, 10–18.
2. Aswolinskiy, W., Pipa, G., RM-SORN: a reward-modulated self-organizing recurrent neural network. Frontiers in Computational Neuroscience, 9, 2015, 36.
3. Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., Baskurt, A., Sequential deep learning for human action recognition. Human behavior understanding: Second international workshop, HBU 2011, Amsterdam, the Netherlands, November 16, 2011. Proceedings, 2011, Springer Berlin Heidelberg, Berlin, Heidelberg, 29–39, 10.1007/978-3-642-25446-8_4.
4. Beyer, O., Cimiano, P., Online labelling strategies for growing neural gas. Yin, H., Wang, W., Rayward-Smith, V., (eds.) Proceedings of the international conference on intelligent data engineering and automated learning, (IDEAL) LNCS, vol. 6936, 2011, Springer Berlin Heidelberg, 76–83, 10.1007/978-3-642-23878-9_10.
5. Blakemore, C., Cooper, G.F., Development of the brain depends on the visual environment. Nature 228:5270 (1970), 477–478.
6. Blakemore, C., Van Sluyters, R.C., Innate and environmental factors in the development of the kitten's visual cortex. Journal Physiology 248:3 (1975), 663–716.
7. Blank, M., Gorelick, L., Shechtman, E., Irani, M., Basri, R., Actions as space-time shapes. Transactions on Pattern Analysis and Machine Intelligence 29:12 (2007), 2247–2253.
8. Buonomano, D., Maass, W., State-dependent computations: Spatiotemporal processing in cortical networks. Nature Reviews Neuroscience 10:2 (2009), 113–125.
9. Chappell, G.J., Taylor, J.G., The temporal Kohonen map. Neural Networks 6:3 (1993), 441–445, 10.1016/0893-6080(93)90011-K.
10. Chen, M.-y., & Hauptmann, A. (2009). MoSIFT: Reocgnizing human actions in surveillance videos. CMU-CS-09-161. Carnegie Mellon University.
11. de Vries, B., Príncipe, J.C., The gamma model–A new neural model for temporal processing. Neural Networks 5:4 (1992), 565–576.
12. Ditzler, G., Roveri, M., Alippi, C., Polikar, R., Learning in nonstationary environments: A survey. IEEE Computational Intelligence Magazine 10:4 (2015), 12–25, 10.1109/MCI.2015.2471196.
13. Eriksson, P.S., Perfilieva, E., Bjork-Eriksson, T., Alborn, A.-M., Nordborg, C., Peterson, D.A., Gage, F.H., Neurogenesis in the adult human hippocampus. Nature Medicine 4:11 (1998), 1313–1317, 10.1038/3305.
14. Estévez, P. A., & Hernández, R. (2011). Gamma-filter self-organizing neural networks for time series analysis. In WSOM, (pp. 151–159).
15. Estévez, P. A., & Vergara, J. R. (2012). Nonlinear time series analysis by using gamma growing neural gas. In WSOM, (pp. 205–214).
16. Felleman, D., Van Essen, D., Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex 1:1 (1991), 1–47.
17. Fritzke, B., A growing neural gas network learns topologies. Advances in neural information processing systems, Vol. 7, 1995, MIT Press, 625–632.
18. Gao, Z., Chen, M.-y., Hauptmann, A.G., Cai, A., Comparing evaluation protocols on the KTH dataset. Human behavior understanding: First international workshop, HBU 2010, Istanbul, Turkey, August 22, 2010. Proceedings, 2010, Springer Berlin Heidelberg, Berlin, Heidelberg, 88–100.
19. Giese, M.A., Poggio, T., Neural mechanisms for the recognition of biological movements. Nature Reviews Neuroscience 4:3 (2003), 179–192, 10.1038/nrn1057.
20. Giese, M.A., Rizzolatti, G., Neural and computational mechanisms of action processing: Interaction between visual and motor representations. Neuron 88:1 (2015), 167–180, 10.1016/j.neuron.2015.09.040.
21. Gorelick, L., Blank, M., Shechtman, E., Irani, M., & Basri, R. (2005). Actions as space-time shapes. In ICCV, (pp. 1395–1402).
22. Grossberg, S., Competitive learning: From interactive activation to adaptive resonance. Cognitive Science 11:1 (1987), 23–63, 10.1016/S0364-0213(87)80025-3.
23. Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., Lew, M.S., Deep learning for visual understanding: A review. Neurocomputing 187 (2016), 27–48, 10.1016/j.neucom.2015.09.116 Recent Developments on Deep Big Vision.
24. Hasson, U., Yang, E., Vallines, I., Heeger, D.J., Rubin, N., A hierarchy of temporal receptive windows in human cortex. The Journal of Neuroscience 28:10 (2008), 2539–2550, 10.1523/jneurosci.5487-07.2008.
25. Hirsch, H., Spinelli, D., Visual experience modifies distribution of horizontally and vertically oriented receptive fields in cats. Science 168:3933 (1970), 869–871.
26. Hochreiter, S., Schmidhuber, J., Long short-term memory. Neural Computation, 8(9), 1997, 10.1162/neco.1997.9.8.1735.
27. Huang, Y., Rao, R.P.N., Predictive coding. Wiley Interdisciplinary Reviews: Cognitive Science 2:5 (2011), 580–593.
28. Hubel, D.H., Wiesel, T.H., Receptive fields, binocular and functional architecture in the cat's visual cortex. Journal Physiology 160 (1962), 106–154.
29. Ivanov, I., Liu, X., Clerkin, S., Schulz, K., Friston, K., Newcorn, J.H., Fan, J., Effects of motivation on reward and attentional networks: an fMRI study. Brain and Behavior 2:6 (2012), 741–753, 10.1002/brb3.80.
30. Jain, A., Tompson, J., LeCun, Y., Bregler, C., MoDeep: A deep learning framework using motion features for human pose estimation. Computer vision –ACCV 2014: 12th Asian conference on computer vision, Singapore, Singapore, November 1-5, 2014, Revised selected papers, part II, 2015, Springer International Publishing, Cham, 302–315, 10.1007/978-3-319-16808-1_21.
31. Jhuang, H., Serre, T., Wolf, L., & Poggio, T. (2007). A biologically inspired system for action recognition. In International conference on computer vision, ICCV.
32. Ji, S., Xu, W., Yang, M., Yu, K., 3D convolutional neural networks for human action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 35:1 (2013), 221–231.
33. Johnson, J., Newport, E., Critical period effects on universal properties of language: the status of subjacency in the acquisition of a second language. Cognition 39:3 (1991), 215–258.
34. Jung, M., Hwang, J., Tani, J., Self-organization of spatio-temporal hierarchy via learning of dynamic visual image patterns on action sequences. PLoS One, 10(7), 2015, e0131214, 10.1371/journal.pone.0131214.
35. Kasabov, N.K., NeuCube: A spiking neural network architecture for mapping, learning and understanding of spatio-temporal brain data. Neural Networks 52 (2014), 62–76.
36. Kasabov, N.K., Doborjeh, M.G., Doborjeh, Z.G., Mapping, learning, visualization, classification, and understanding of fMRI data in the NeuCube evolving spatiotemporal data machine of spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems 28:4 (2017), 887–899.
37. Kohonen, T., The self-organizing map. Proceedings of the IEEE 78 (1990), 1464–1480.
38. Lee, J., Cumulative learning. Encyclopedia of the sciences of learning, 2012, Springer US, Boston, MA, 887–893 (Chapter).
39. Lerner, Y., Honey, C.J., Silbert, L.J., Hasson, U., Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. The Journal of Neuroscience 31:8 (2011), 2906–2915, 10.1523/jneurosci.3684-10.2011.
40. Marsland, S., Nehmzow, U., Shapiro, J., On-line novelty detection for autonomous mobile robots. Robotics and Autonomous Systems 51:2–3 (2005), 191–206.
41. Marsland, S., Shapiro, J., Nehmzow, U., A self-organising network that grows when required. Neural Networks 15:8–9 (2002), 1041–1058.
42. Martinetz, T.M., Competitive Hebbian learning rule forms perfectly topology preserving maps. ICANN’93: International conference on artificial neural networks, 1993, Springer, Amsterdam, 427–434.
43. McClelland, J.L., McNaughton, B.L., O'Reilly, R.C., Why There are Complementary Learning Systems in the Hippocampus and Neocobelrtex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review 102 (1995), 419–457.
44. Ming, G.-l., Song, H., Adult neurogenesis in the mammalian brain: Significant answers and significant questions. Neuron 70:4 (2011), 687–702, 10.1016/j.neuron.2011.05.001 URL http://dx.doi.org/10.1038/nrn2147.
45. Mozer, M., A focused backpropagation algorithm for temporal pattern recognition. Hillsdale, 1995, Lawrence Erlbaum Associates, NJ, 137–169.
46. Nelson, C.A., Neural plasticity and human development: the role of early experience in sculpting memory systems. Developmental Science 3:2 (2000), 115–136.
47. Niebles, J.C., Wang, H., Fei-Fei, L., Unsupervised learning of human action categories using spatial-temporal words. International Journal of Computer Vision 79:3 (2008), 299–318.
48. Parisi, G. I., Magg, S., & Wermter, S. (2016). Human motion assessment in real time using recurrent self-organization. In Proc. of the IEEE international symposium on robot and human interactive communication, RO-MAN (pp. 71–76). New York, US.
49. Parisi, G.I., Tani, J., Weber, C., Wermter, S., Emergence of multimodal action representations from neural network self-organization. Cognitive Systems Research, 2016, 10.1016/j.cogsys.2016.08.002.
50. Parisi, G.I., Weber, C., Wermter, S., Self-Organizing neural integration of pose-motion features for human action recognition. Frontiers in Neurorobotics, 9(3), 2015, 10.3389/fnbot.2015.00003.
51. Perrett, D., Rolls, E., Caan, W., Visual neurons responsive to faces in the monkey temporal cortex. Experimental Brain Research 47 (1982), 329–342.
52. Poggio, T., Smale, S., The mathematics of learning: Dealing with data. Notices of the American Mathematical Society, 50, 2003, 2003.
53. Rao, R., Ballard, D., Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience 2 (1999), 79–87.
54. Ravanbakhsh, M., Mousavi, H., Rastegari, M., Murino, V., & Davis, L. S. (2015). Action recognition with image based CNN features. CoRR abs/1512.03980.
55. Richardson, F.M., Thomas, M.S.C., Critical periods and catastrophic interference effects in the development of self-organising feature maps. Developmental Science 11:3 (2008), 371–389.
56. Riesenhuber, M., Poggio, T., Hierarchical models of object recognition in cortex. Nature Neuroscience 2 (1999), 1019–1025.
57. Sato, T., Interactions of visual stimuli in the receptive fields of inferior temporal neurons in macaque. Experimental Brain Research 77 (1989), 23–30.
58. Scherer, D., Müller, A., Behnke, S., Evaluation of pooling operations in convolutional architectures for object recognition. Proceedings of the 20th international conference on artificial neural networks: Part III, ICANN’10, 2010, Springer-Verlag, Berlin, Heidelberg, 92–101.
59. Schindler, K., Van Gool, L.J., Action snippets: How many frames does human action recognition require?. CVPR, 2008, IEEE Computer Society.
60. Schmidhuber, J., Learning complex, extended sequences using the principle of history compression. Neural Computation 4 (1992), 234–242.
61. Schuldt, C., Laptev, I., Caputo, B., Recognizing human actions: A local SVM approach. Proceedings of the pattern recognition, 17th international conference on (ICPR’04) Volume 3 - Volume 03, ICPR ’04, 2004, IEEE Computer Society, Washington, DC, USA, 32–36.
62. Senghas, A., Kita, S., Ozyurek, A., Children creating core properties of language: Evidence from an emerging sign language in nicaragua. Science 305:5691 (2004), 1779–1782, 10.1126/science.1100199.
63. Sengpiel, F., Stawinski, P.T.B., Influence of experience on orientation maps in cat visual cortex. Nature Neuroscience 2:8 (1999), 727–732.
64. Soomro, K., Zamir, A. R., & Shah, M. (2012). UCF101: A dataset of 101 human action classes from videos in the wild. In CRCV-TR.
65. Stanley, J.C., Computer simulation of a model of habituation. Nature 261 (1976), 146–148, 10.1038/261146a0.
66. Stevens, J.-L.R., Law, J.S., Antolík, J., Bednar, J.A., Mechanisms for stable, robust, and adaptive development of orientation maps in the primary visual cortex. Journal of Neuroscience 33:40 (2013), 15747–15766.
67. Strickert, M., Hammer, B., Merge SOM for temporal data. Neurocomputing, 64, 2005, 10.1016/j.neucom.2004.11.014.
68. Tan, C., Singer, J. M., Serre, T., Sheinberg, D., & Poggio, T. A. (2013). Neural representation of action sequences: how far can a simple snippet-matching model take us? In Advances in neural information processing systems 26: 27th Annual conference on neural information processing systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States. (pp. 593–601).
69. Tani, J., Learning to generate articulated behavior through the bottom-up and the top-down interaction processes. Neural Networks 16:1 (2003), 11–23.
70. Tani, J., Nolfi, S., Learning to perceive the world as articulated: an approach for hierarchical learning in sensory-motor systems. Neural Networks 12:7–8 (1999), 1131–1141.
71. Taylor, P., Hobbs, J.N., Burroni, J., Siegelmann, H.T., The global landscape of cognition: hierarchical aggregation as an organizational principle of human cortical networks and functions. Scientific Reports, 5(18112), 2015, 10.1038/srep18112.
72. Thurau, C., Hlavác, V., Pose primitive based human action recognition in videos or still images. CVPR, 2008, IEEE Computer Society.
73. Ungerleider, L., Mishkin, M., Two cortical visual systems. Analysis of visual behavior, 1982, MIT Press, Cambridge, 549–586.
74. Vangeneugden, J., Pollick, F., Vogels, R., Neural and computational mechanisms of action processing: Interaction between visual and motor representations. Cerebral Cortex 19:3 (2009), 593–611, 10.1093/cercor/bhn109.
75. Wang, H., Klaser, A., Schmid, C., Liu, C.-L., Action recognition by dense trajectories. Proceedings of the 2011 IEEE conference on computer vision and pattern recognition, 2011, IEEE Computer Society, 3169–3176.
76. Willshaw, D.J., von der Malsburg, C., How Patterned Neural Connections Can Be Set Up by Self-Organization. Proceedings of the Royal Society of London B: Biological Sciences 194:1117 (1976), 431–445.
77. Yamins, D.L.K., DiCarlo, J.J., Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience 19:3 (2016), 356–365.
78. Zenke, F., Agnes, E.J., Gerstner, W., Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks. Nature Communications, 6(6922), 2015.
79. Zenke, F., Gerstner, W., Hebbian plasticity requires compensatory processes on multiple timescales. Philosophical Transactions of the Royal Society, Series B (Biological Sciences, 372:20160259, 2017.
80. Zenke, F., Gerstner, W., Ganguli, S., The temporal paradox of Hebbian learning and homeostatic plasticity. Current Opinion in Neurobiology 43 (2017), 166–176.
81. Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016). Understanding deep learning requires rethinking generalization. CoRR abs/1611.03530.
82. Zhou, H.H., CSM: A computational model of cumulative learning. Machine Learning 5:4 (1990), 383–406.