Self-organizing neural integration of pose-motion features for human action recognition

Frontiers in Neurorobotics

Volumn 9, Issue JUN, 2015, Pages

  Parisi, German I ^a   Weber, Cornelius ^a   Wermter, Stefan ^a  

^a UNIVERSITY OF HAMBURG   (Germany)

Author keywords

Action recognition; Depth information; Neural networks; Robot perception; Self organizing learning; Visual processing

Indexed keywords

ARTICLE; ARTIFICIAL NEURAL NETWORK; CLASSIFICATION ALGORITHM; DATA BASE; MEASUREMENT ACCURACY; MOTION ANALYSIS SYSTEM; NOISE; VIDEORECORDING;

EID: 84949190281     PISSN: None     EISSN: 16625218     Source Type: Journal    
DOI: 10.3389/fnbot.2015.00003     Document Type: Article

References (74)

1. Andreakis, N., Hoyningen-Huene, N. v., and Beetz, M. (2009). "Incremental unsupervised time series analysis using merge growing neural gas," in Advances in Self-Organizing Maps, eds J. C. Principe and R. Miikkulainen (St. Augustine, FL: Springer), 10-18.
2. Bülthoff, I., Bülthoff, H., and Sinha, P. (1998). Top-down influences on stereoscopic depth perception. Nat. Neurosci. 1, 254-257. doi: 10.1038/699
3. Barros, P., Parisi, G. I., Jirak, D., and Wermter, S. (2014). "Real-time gesture recognition using a humanoid robot with a deep neural Architecture," in 2014 14th IEEE-RAS International Conference on Humanoid Robots (Humanoids) (Madrid: IEEE-RAS), 83-88.
4. Beauchamp, M. S., Lee, K. E., Haxby, J. V., and Martin, A. (2003) FMRI responses to video and point-light displays of moving humans and manipulable objects. J. Cogn. Neurosci. 15, 991-1001. doi: 10.1162/089892903770007380
5. Bertenthal, B. I., and Pinto, J. (1993). "Complementary processes in the perception and production of human movement," in Dynamic Approaches to Development 2, eds E. Thelen, and L. Smith (Cambridge: MIT Press), 209-239.
6. Beyer, O., and Cimiano, P. (2011) "Online labelling strategies for growing neural gas," in IDEAL11, eds H. Yin, W. Wang, and V. Rayward-Smith (Norwich: Springer Berlin Heidelberg), 76-83.
7. Blake, R., and Shiffrar, M. (2007). Perception of human motion. Annu. Rev. Psychol. 58, 47-73. doi: 10.1146/annurev.psych.57.102904.190152
8. Ellis, C., Masood, S. Z., Tappen, M. F., LaViola, J. J. Jr., and Sukthankar, R. (2013). Exploring the trade-offbetween accuracy and observational latency in action recognition. Int. J. Comput. Vis. 101, 420-436. doi: 10.1007/s11263-012-0550-7
9. Fanello, S. R., Gori, I., Metta, G., and Odone, F. (2013) Keep it simple and sparse: real-time action recognition. J. Mach. Learn. Res. 14, 2617-2640.
10. Faria, D. R., Premebida, C., and Nunes, U. (2014). "A probabilistic approach for human everyday activities recognition using body motion from RGB-D images," in 2014 RO-MAN: the 23rd IEEE International Symposium Robot and Human Interactive Communication (Edinburgh: IEEE).
11. Fritzke, B. (1995). "A growing neural gas network learns topologies," in Advances in Neural Information Processing Systems, Vol. 7, eds G. Tesauro, D. S. Touretzky, and T. K. Leen (Cambridge, MA: MIT Press), 625-632.
12. Gaglio, S., Lo Re, M., and Morana, M. (2014) Human activity recognition process using 3-D posture data. IEEE Trans. Hum. Mach. Syst. 99, 1-12. doi: 10.1109/THMS.2014.2377111
13. Garcia, J. O., and Grossman, E. D. (2008). Necessary but not sufficient: motion perception is required for perceiving biological motion. Vis. Res. 48, 1144-1149. doi: 10.1016/j.visres.2008.01.027
14. Giese, M. A., and Poggio, T. (2003). Neural mechanisms for the recognition of biological movements. Nat. Rev. Neurosci. 4, 179-192. doi: 10.1038/nrn1057
15. Giese, M. A., Thornton, I., and Edelman, S. (2008). Metrics of the perception of body movement. J. Vis. 8, 1-18. doi: 10.1167/8.9.13
16. Gupta, R., Chia, A. Y.-S., and Rajan, D. (2013). "Human activities recognition using depth images," in ACM International Conference on Multimedia (New York, NY: ACM), 283-292.
17. Han, J., Shao, L., Xu, D., and Shotton, J. (2013) Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans. Cybern. 43, 1318-1334. doi: 10.1109/TCYB.2013.2265378
18. Hebb, D. O. (1949). The Organization of Behavior. New York, NY: Wiley.
19. Hiris, E. (2007) Detection of biological and nonbiological motion. J. Vis. 7, 1-16. doi: 10.1167/7.12.4
20. Hogg, D. (1983). Model-based vision: a program to see a walking person. Image Vis. Comp. 1, 5-19. doi: 10.1016/0262-8856(83)90003-3
21. Hubel, D. H., and Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol. 160, 106-154. doi: 10.1113/jphysiol.1962.sp006837
22. Hubel, D. H., and Wiesel, T. N. (1977). Ferrier lecture. Functional architecture of macaque monkey visual cortex. Proc. R. Soc. Lond. B Biol. Sci. 198, 1-59. doi: 10.1098/rspb.1977.0085
23. Jastorff, J., Kourtzi, Z., and Giese, M. A. (2006). Learning to discriminate complex movements: biological versus artificial trajectories. J. Vis. 6, 791-804. doi: 10.1167/6.8.3
24. Jiang, Z., Lin, Z., and Davis, L. S. (2012). Recognizing human actions by learning and matching shape-motion prototype trees. IEEE Trans. Pattern Anal. Mach. Intell. 34, 533-547. doi: 10.1109/TPAMI.2011.147
25. Kachouie, R., Sedighadeli, S., Khosla, R., and Chu, M-T. (2014). Socially assistive robots in elderly care: a mixed-method systematic literature review. Int. J. Hum. Comput. Interact. 30, 369-393. doi: 10.1080/10447318.2013.873278
26. Kohonen, T. (1993). Self-Organization and Associative Memory, 3rd Edn. Berlin: Springer.
27. Kohonen, T. (1995). "Self-organizing maps," in Series in Information Science 30, eds T. S. Huang, T. Kohonen, and M. R. Schroeder (Heidelberg: Springer), 502.
28. Koppula, H. S., Gupta, R., and Saxena, A. (2013). Learning human activities and object affordances from RGB-D videos. Int. J. Robot. Res. 32, 951-970. doi: 10.1177/0278364913478446
29. Lange, J., and Lappe, M. (2006). A model of biological motion perception from configural form cues. J. Neurosci. 26, 2894-2906. doi: 10.1523/JNEUROSCI.4915-05.2006
30. Layher, G., Giese, M. A., and Neumann, H. (2012). "Learning representations for animated motion sequence and implied motion recognition," in ICANN12 (Helderberg: Springer), 427-434.
31. Marsland, S., Shapiro, J., and Nehmzow, U. (2002). A self-organising network that grows when required. Neural Netw. 15, 1041-1058. doi: 10.1016/S0893-6080(02)00078-3
32. Martinetz, T., and Schluten, K. (1991). "A "neural-gas" network learns topologies," in Artificial Neural Networks, eds T. Kohonen, K. Makisara, O. Simula, and J. Kangas (North Holland: Elsevier Science Publishers B.V.), 397-402.
33. Martinetz, T. (1993). "Competitive Hebbian learning rule forms perfectly topology preserving maps," in ICANN93 (Helderberg: Springer), 427-434.
34. Mastorakis, G., and Makris, D. (2012). Fall detection system using Kinect's infrared sensor. J. Real Time Image Process 9, 635-646.
35. Meltzoff, A. N., and Moore, M. K. (1977). Imitation of facial and manual gestures by human neonates. Science 198, 74-78. doi: 10.1126/science.897687
36. Miikkulainen, R., Bednar, J. A., Choe, Y., and Sirosh, J. (2005) Computational Maps in the Visual Cortex. New York, NY: Springer.
37. Neri, P., Concetta Morrone, M., and Burr, D. C. (1998). Seeing biological motion. Nature 395, 894-896. doi: 10.1038/27661
38. Ni, B., Pei, Y., Moulin, P., and Yan, S. (2013). Multilevel depth and image fusion for human activity detection. IEEE Trans. Cybern. 43, 1383-1394. doi: 10.1109/TCYB.2013.2276433
39. Oram, M. W., and Perrett, D. I. (1996). Integration of form and motion in the anterior superior temporal polysensory area (STPa) of the macaque monkey. J. Neurophysiol. 76, 109-129.
40. Papadopoulos, G., Th. Axenopoulos, A., and Daras, P. (2014). "Real-time skeleton-tracking-based human action recognition using kinect data," in MultiMedia Modeling '14 eds C. Gurrin, F. Hopfgartner, W. Hurst, H. Johansen, H. Lee, and N. O'Connor (Dublin: Springer International Publishing), 473-483.
41. Parisi, G. I., and Wermter, S. (2013). "Hierarchical SOM-based detection of novel behavior for 3D human tracking," in The 2013 International Joint Conference on Neural Networks (Dallas, TX: IEEE), 1380-1387.
42. Parisi, G. I., Barros, P., and Wermter, S. (2014a). "FINGeR: framework for interactive neural-based gesture recognition," in ESANN14 443-447.
43. Parisi, G. I., Jirak, D., and Wermter, S. (2014b). "HandSOM - neural clustering of hand motion for gesture recognition in real time," in Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN '14) (Edinburgh: IEEE), 981-986.
44. Parisi, G. I., Weber, C., and Wermter, S. (2014c). "Human action recognition with hierarchical growing neural gas learning," in ICANN14 (Helderberg: Springer), 89-96.
45. Parisi, G. I., and Wermter, S. (2015). Neurocognitive assistive robot for robust fall detection. Smart Environ.
46. Pascual-Leone, A., Freitas, C., Oberman, L., Horvath, J. C., Halko, M., Eldaief, M., et al. (2011). Characterizing brain cortical plasticity and network dynamics across the age-span in health and disease with TMS-EEG and TMS-fMRI. Brain Topogr. 24, 302-315. doi: 10.1007/s10548-011-0196-8
47. Pavlova, M., and Sokolov, S. (2000). Orientation specificity in biological motion perception. Percept. Psychophys. 62, 889-899. doi: 10.3758/BF03212075
48. Pavlova, M., Krageloh-Mann, I., Sokolov, A., and Birbaumer, N. (2001). Recognition of point-light biological motion displays by young children. Perception 30, 925-933. doi: 10.1068/p3157
49. Perrett, D. I., Rolls, E. T., and Caan, W. (1982). Visual neurons responsive to faces in the monkey temporal cortex. Exp. Brain Res. 47, 329-342. doi: 10.1007/BF00239352
50. Rougier, C., Auvinet, E., Rousseau, J., Mignotte, M., and Meunier, J. (2011). "Fall detection from depth map video sequences," in Toward Useful Services for Elderly and People with Disabilities, eds B. Abdulrazak, S. Giroux, B. Bouchard, H. Pigot, and M. Mokhtari (Montreal, QC: Springer Berlin - Heidelberg), 121-128.
51. Shan, J., and Akella, S. (2014). "3D Human action segmentation and recognition using pose kinetic energy," in Workshop on Advanced Robotics and its Social Impacts (ARSO14) (Evanston, IL: IEEE), 69-75.
52. Shiffrar, M., and Freyd, J. J. (1990). Apparent motion of the human body. Psychol. Sci. 1, 257-264. doi: 10.1111/j.1467-9280.1990.tb00210.x
53. Singer, J. M., and Sheinberg, D. L. (2010). Temporal cortex neurons encode articulated actions as slow sequences of integrated poses. J. Neurosci. 30, 3133-3145. doi: 10.1523/JNEUROSCI.3211-09.2010
54. Sofatzis, R. J., Afshar, S., and Hamilton, T. J. (2014). "Rotationally invariant vision recognition with neuromorphic transformation and learning networks," in 2014 IEEE International Symposium on Circuits and Systems (Melbourne, VIC: IEEE), 469-472.
55. Soltoggio, A., Lemme, A., Reinhart, F., and Steil, J. (2013a). Rare neural correlations implement robotic conditioning with delayed rewards and disturbances. Front. Neurorobot. 7:6. doi: 10.3389/fnbot.2013.00006
56. Soltoggio, A., Reinhart, F., Lemme, A., and Steil, J. (2013b). "Learning the rules of a game: neural conditioning in human-robot interaction with delayed rewards," in IEEE International Conference of Development and Learning and on Epigenetic Robotics (Osaka).
57. Stanley, J. C. (1976). Computer simulation of a model of habituation. Nature 261, 146-148. doi: 10.1038/261146a0
58. Strickert, M., and Hammer, B. (2005). Merge SOM for temporal data. Neurocomputing 64, 39-71. doi: 10.1016/j.neucom.2004.11.014
59. Suarez, J., and Murphy, R. (2012). "Hand gesture recognition with depth images: a review," in RO-MAN14 (Edinburgh, UK: IEEE), 411-417.
60. Sumi, S. (1984). Upside-down presentation of the Johansson moving light-spot pattern. Perception 13, 283-302. doi: 10.1068/p130283
61. Sung, J., Ponce, C., Selman, B., and Saxena, A. (2012). "Unstructured human activity detection from RGBD images," in 2012 IEEE International Conference on Robotics and Automation (Saint Paul, MN: IEEE), 842-849.
62. Tan, C., Singer, J. M., Serre, T., Sheinberg, D., and Poggio, T. A. (2013). "Neural representation of action sequences: how far can a simple snippet-matching model take us?" in Advances in Neural Information Processing Systems, NIPS13 (Lake Tahoe, CA: Harrahs and Harveys), 1-9.
63. Thirkettle, M., Benton, C. P., and Scott-Samuel, N. E. (2009). Contributions of form, motion and task to biological motion perception. J. Vis. 9, 1-11. doi: 10.1167/9.3.28
64. Thompson, R. F., and Spencer, W. A. (1966). Habituation: a model phenomenon for the study of neuronal substrates of behavior. Psychol. Rev. 73, 16-43. doi: 10.1037/h0022681
65. Thornton, I. M., Rensink, R. A., and Shiffrar, M. (2002). Active versus passive processing of biological motion. Perception 31, 837-853. doi: 10.1068/p3072
66. Troje, N. F. (2002). Decomposing biological motion: a framework for analysis and synthesis of human gait patterns. J. Vis. 2, 371-387. doi: 10.1167/2.5.2
67. Tyler, S. C., and Grossman, E. D. (2011). Feature-based attention promotes biological motion recognition. J. Vis. 11, 1-6. doi: 10.1167/11.10.11
68. Vangeneugden, J., Pollick, F., and Vogels, R. (2009). Functional differentiation of macaque visual temporal cortical neurons using a parametric action space. Cereb. Cortex 19, 593-611. doi: 10.1093/cercor/bhn109
69. von der Malsburg, C. (1973). Self-organization of orientation sensitive cells in the striate cortex. Kybernetik 14, 85-100. doi: 10.1007/BF00288907
70. Weinland, D., Ronfard, R., and Boyer, R. (2011). A survey of vision-based methods for action representation, segmentation and recognition. Comput. Vis. Image Underst. 115, 224-241. doi: 10.1016/j.cviu.2010.10.002
71. Xu, R., Agarwal, P., Kumar, S., Krovi, V. N., and Corso, J. J. (2012). "Combining skeletal pose with local motion for human activity recognition," in Articulated Motion and Deformable Objects (LNCS 7378), eds F. J. Perales, R. B. Fisher, and T. B. Moeslund (Mallorca: Springer Berlin Heidelberg) 114-123.
72. Yanik, P. M., Manganelli, J., Merino, J., Threatt, A. L., Brooks, J. O., Evan Green, K., et al. (2014) A gesture learning interface for simulated robot path shaping with a human teacher. IEEE Trans. Hum. Mach. Syst. 44, 44-51. doi: 10.1109/TSMC.2013.2291714
73. Zhang, C., and Y. Tian. (2012). RGB-D camera-based daily living activity recognition. J. Comput. Vis. Image Process. 2, 1-7.
74. Zhu, Y., Chen, W., and Guo, G. (2014). Evaluating spatiotemporal interest point features for depth-based action recognition. Image Vis. Comput. 32, 453-464. doi: 10.1016/j.imavis.2014.04.005