A self-organizing neural network architecture for learning human-object interactions

Neurocomputing

Volumn 307, Issue , 2018, Pages 14-24

  Mici, Luiza ^a   Parisi, German I ^a   Wermter, Stefan ^a  

^a UNIVERSITY OF HAMBURG   (Germany)

Author keywords

Action recognition; Hierarchical learning; Human object interaction; Object recognition; Self organization

Indexed keywords

CLASSIFICATION (OF INFORMATION); NEURAL NETWORKS; OBJECT RECOGNITION; SEMANTICS;

ACTION RECOGNITION; CLASSIFICATION RESULTS; HIERARCHICAL LEARNING; HUMAN-OBJECT INTERACTION; MANIPULATED OBJECTS; NATURAL ENVIRONMENTS; SELF ORGANIZATIONS; SELF-ORGANIZING NEURAL NETWORK;

NETWORK ARCHITECTURE;

ARTICLE; ARTIFICIAL NEURAL NETWORK; BODY MOVEMENT; CONTROLLED STUDY; FEMALE; GROW WHEN REQUIRED ALGORITHM; HUMAN; HUMAN COMPUTER INTERACTION; HUMAN EXPERIMENT; LEARNING ALGORITHM; MALE; MEASUREMENT ACCURACY; MEASUREMENT PRECISION; PRIORITY JOURNAL; QUANTITATIVE ANALYSIS; SELF ORGANIZING NEURAL NETWORK; TASK PERFORMANCE;

EID: 85047092718     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2018.04.015     Document Type: Article

References (67)

1. Fleischer, F., Caggiano, V., Thier, P., Giese, M.A., Physiologically inspired model for the visual recognition of transitive hand actions. J. Neurosci. 33:15 (2013), 6563–6580, 10.1523/JNEUROSCI.4129-12.2013.
2. Nelissen, K., Vanduffel, W., Orban, G.A., Charting the lower superior temporal region, a new motion-sensitive region in monkey superior temporal sulcus. J. Neurosci. 26:22 (2006), 5929–5947, 10.1523/JNEUROSCI.0824-06.2006.
3. Prevete, R., Tessitore, G., Santoro, M., Catanzariti, E., A connectionist architecture for view-independent grip-aperture computation. Brain Res. 1225 (2008), 133–145, 10.1016/j.brainres.2008.04.076.
4. Tessitore, G., Prevete, R., Catanzariti, E., Tamburrini, G., From motor to sensory processing in mirror neuron computational modelling. Biol. Cybern. 103:6 (2010), 471–485, 10.1007/s00422-010-0415-5.
5. Beauchamp, M.S., Lee, K.E., Haxby, J.V., Martin, A., Parallel visual motion processing streams for manipulable objects and human movements. Neuron 34:1 (2002), 149–159, 10.1016/S0896-6273(02)00642-6.
6. Downing, P.E., Peelen, M.V., The role of occipitotemporal body-selective regions in person perception. Cogn. Neurosci. 2:3–4 (2011), 186–203, 10.1080/17588928.2011.582945.
7. Grill-Spector, K., Representation of objects. The Oxford Handbook of Cognitive Neuroscience, Volume 2: The Cutting Edges, 2, 2013.
8. Hubel, D.H., Wiesel, T., Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol. 160:1 (1962), 106–154.
9. Grossman, E.D., Blake, R., Brain areas active during visual perception of biological motion. Neuron 35:6 (2002), 1167–1175, 10.1016/S0896-6273(02)00897-8.
10. Giese, M.A., Poggio, T., Neural mechanisms for the recognition of biological movements. Nat. Rev. Neurosci. 4:3 (2003), 179–192, 10.1038/nrn1057.
11. Saxe, R., Carey, S., Kanwisher, N., Understanding other minds: linking developmental psychology and functional neuroimaging. Annu. Rev. Psychol. 55 (2004), 87–124, 10.1146/annurev.psych.55.090902.142044.
12. Gallese, V., Fadiga, L., Fogassi, L., Rizzolatti, G., Action recognition in the premotor cortex. Brain 119:2 (1996), 593–609, 10.1093/brain/119.2.593.
13. Nelissen, K., Luppino, G., Vanduffel, W., Rizzolatti, G., Orban, G.A., Observing others: multiple action representation in the frontal lobe. Science 310:5746 (2005), 332–336, 10.1126/science.1115593.
14. Yoon, E.Y., Humphreys, G.W., Kumar, S., Rotshtein, P., The neural selection and integration of actions and objects: an fMRI study. J. Cogn. Neurosci. 24:11 (2012), 2268–2279, 10.1162/jocn_a_00256.
15. Miikkulainen, R., Bednar, J.A., Choe, Y., Sirosh, J., Computational Maps in the Visual Cortex, 2006, Springer Science & Business Media.
16. Rosch, E., Mervis, C.B., Family resemblances: Studies in the internal structure of categories. Cogn. Psychol. 7:4 (1975), 573–605, 10.1016/0010-0285(75)90024-9.
17. Marsland, S., Shapiro, J., Nehmzow, U., A self-organising network that grows when required. Neural Netw. 15:8 (2002), 1041–1058, 10.1016/S0893-6080(02)00078-3.
18. Parisi, G.I., Weber, C., Wermter, S., Self-organizing neural integration of pose-motion features for human action recognition. Front. Neurorobotics, 9, 2015, 10.3389/fnbot.2015.00003.
19. Parisi, G.I., Weber, C., Wermter, S., Human action recognition with hierarchical growing neural gas learning. Proceedings of the International Conference on Artificial Neural Networks (ICANN), 2014, Springer, 89–96, 10.1007/978-3-319-11179-7_12.
20. Donatti, G.S., Lomp, O., Würtz, R.P., Evolutionary optimization of growing neural gas parameters for object categorization and recognition. Proceedings of the International Joint Conference on Neural Networks (IJCNN), 2010, IEEE, 1–8, 10.1109/IJCNN.2010.5596682.
21. Mici, L., Parisi, G.I., Wermter, S., Recognition of transitive actions with hierarchical neural network learning. Proceedings of the Artificial Neural Networks and Machine Learning (ICANN), 2016, Springer International Publishing, 472–479, 10.1007/978-3-319-44781-0_56.
22. Kohonen, T., Essentials of the self-organizing map. Neural Netw. 37 (2013), 52–65, 10.1016/j.neunet.2012.09.018.
23. Fritzke, B., Kohonen feature maps and growing cell structures-a performance comparison. Proceedings of the Advances in Neural Information Processing Systems 5 (NIPS), 1993, Morgan Kaufmann, 123–130.
24. Koppula, H.S., Gupta, R., Saxena, A., Learning human activities and object affordances from RGB-D videos. Int. J. Robot. Res. 32:8 (2013), 951–970, 10.1177/0278364913478446.
25. Aggarwal, J.K., Ryoo, M.S., Human activity analysis: a review. ACM Comput. Surv. (CSUR), 43(3), 2011, 10.1145/1922649.1922653.
26. Ziaeefard, M., Bergevin, R., Semantic human activity recognition: a literature review. Pattern Recognit. 48:8 (2015), 2329–2345, 10.1016/j.patcog.2015.03.006.
27. Aggarwal, J.K., Xia, L., Human activity recognition from 3D data: a review. Pattern Recognit. Lett. 48 (2014), 70–80, 10.1016/j.patrec.2014.04.011.
28. Cippitelli, E., Gasparrini, S., Gambi, E., Spinsante, S., A human activity recognition system using skeleton data from RGBD sensors. Comput. Intell. Neurosci., 2016, 2016, 10.1155/2016/4351435.
29. Yang, X., Tian, Y., Effective 3D action recognition using eigenjoints. J. Vis. Commun. Image Represent. 25:1 (2014), 2–11, 10.1016/j.jvcir.2013.03.001.
30. Sung, J., Ponce, C., Selman, B., Saxena, A., Unstructured human activity detection from RGBD images. Proceedings of the International Conference on Robotics and Automation, (ICRA), 2012, IEEE, 842–849, 10.1109/ICRA.2012.6224591.
31. Wu, J., Osuntogun, A., Choudhury, T., Philipose, M., Rehg, J.M., A scalable approach to activity recognition based on object use. Proceedings of the International Conference on Computer Vision, (ICCV), 2007, IEEE, 1–8.
32. Yang, Y., Li, Y., Fermüller, C., Aloimonos, Y., Robot learning manipulation action plans by “watching” unconstrained videos from the world wide web. Proceedings of the Association for the Advancement of Artificial Intelligence, (AAAI), 2015, 3686–3693.
33. Pieropan, A., Salvi, G., Pauwels, K., Kjellström, H., Audio-visual classification and detection of human manipulation actions. Proceedings of the IEEE International Conference On Intelligent Robots and Systems, (IROS), 2014, IEEE, 3045–3052.
34. Gupta, A., Davis, L.S., Objects in action: an approach for combining action understanding and object perception. Proceedings of the Computer Vision and Pattern Recognition, (CVPR), 2007, IEEE, 1–8, 10.1109/CVPR.2007.383331.
35. Gupta, A., Kembhavi, A., Davis, L.S., Observing human-object interactions: using spatial and functional compatibility for recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31:10 (2009), 1775–1789, 10.1109/TPAMI.2009.83.
36. Ryoo, M.S., Aggarwal, J., Hierarchical recognition of human activities interacting with objects. Proceedings of the Computer Vision and Pattern Recognition, (CVPR), 2007, IEEE, 1–8, 10.1109/CVPR.2007.383487.
37. Yao, B., Fei-Fei, L., Recognizing human-object interactions in still images by modeling the mutual context of objects and human poses. IEEE Trans. Pattern Anal. Mach. Intell. 34:9 (2012), 1691–1703, 10.1109/TPAMI.2012.67.
38. Kjellström, H., Romero, J., Kragić D., Visual object-action recognition: inferring object affordances from human demonstration. Comput. Vis. Image Understanding 115:1 (2011), 81–90, 10.1016/j.cviu.2010.08.002.
39. Yao, B., Fei-Fei, L., Grouplet: a structured image representation for recognizing human and object interactions. Proceedings of the Computer Vision and Pattern Recognition, (CVPR), 2010, IEEE, 9–16, 10.1109/CVPR.2010.5540234.
40. Lowe, D.G., Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60:2 (2004), 91–110, 10.1023/B:VISI.0000029664.99615.94.
41. Aksoy, E.E., Abramov, A., Dörr, J., Ning, K., Dellen, B., Wörgötter, F., Learning the semantics of object–action relations by observation. Int. J. Robot. Res., 2011, 1229–1249, 10.1177/0278364911410459.
42. Shimozaki, M., Kuniyoshi, Y., Integration of spatial and temporal contexts for action recognition by self organizing neural networks. Proceedings of the IEEE International Conference On Intelligent Robots and Systems, (IROS), 3, 2003, IEEE, 2385–2391.
43. Lea, C., Reiter, A., Vidal, R., Hager, G.D., Segmental spatiotemporal CNNS for fine-grained action segmentation. Proceedings of the European Conference on Computer Vision, (ECCV), 2016, Springer, 36–52.
44. CY. Ma, A. Kadav, I. Melvin, Z. Kira, G. AlRegib, H.P. Graf, Attend and Interact: Higher-Order Object Interactions for Video Understanding, arXiv preprint arXiv: 1711.06330 (2017).
45. Han, J., Shao, L., Xu, D., Shotton, J., Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans. Cybern. 43:5 (2013), 1318–1334, 10.1109/TCYB.2013.2265378.
46. Wang, J., Liu, Z., Wu, Y., Learning actionlet ensemble for 3D human action recognition. Proceedings of the Human Action Recognition with Depth Cameras, 2014, Springer International Publishing, 11–40, 10.1109/TPAMI.2013.198.
47. Poggio, T., Edelman, S., A network that learns to recognize three-dimensional objects. Nature 343:6255 (1990), 263–266, 10.1038/343263a0.
48. Perrett, D., View-dependent coding in the ventral stream and its consequences for recognition. Vision Movement Mechanisms in the Cerebral Cortex, 1996, 142–151.
49. Martinetz, T., Schulten, K., A “neural-gas” network learns topologies. Proceedings of the Artificial Neural Networks, 1991, Elsevier Science Publisher B.V., 397–402.
50. Fritzke, B., A growing neural gas network learns topologies. Adv. Neural Inf. Process. Syst. 7 (1995), 625–632.
51. Aggarwal, C.C., Hinneburg, A., Keim, D.A., On the surprising behavior of distance metrics in high dimensional space. Proceedings of the International Conference on Database Theory, (ICDT), 2001, Springer, 420–434, 10.1007/3-540-44503-X_27.
52. Strickert, M., Hammer, B., Merge SOM for temporal data. Neurocomputing 64 (2005), 39–71, 10.1016/j.neucom.2004.11.014.
53. Evangelidis, G., Singh, G., Horaud, R., Skeletal quads: human action recognition using joint quadruples. Proceedings of the International Conference on Pattern Recognition (ICPR), 2014, 4513–4518, 10.1109/ICPR.2014.772.
54. Vatanen, T., Osmala, M., Raiko, T., Lagus, K., Sysi-Aho, M., Orešič M., Honkela, T., Lähdesmäki, H., Self-organization and missing values in SOM and GTM. Neurocomputing 147 (2015), 60–70.
55. Tuytelaars, T., Lampert, C.H., Blaschko, M.B., Buntine, W., Unsupervised object discovery: a comparison. Int. J. Comput. Vis. 88:2 (2010), 284–302, 10.1007/s11263-009-0271-8.
56. Kinnunen, T., Kamarainen, J.-K., Lensu, L., Kälviäinen, H., Unsupervised object discovery via self-organisation. Pattern Recognit. Lett. 33:16 (2012), 2102–2112, 10.1016/j.patrec.2012.07.013.
57. Jegou, H., Perronnin, F., Douze, M., Sánchez, J., Perez, P., Schmid, C., Aggregating local image descriptors into compact codes. IEEE Trans. Pattern Anal. Mach. Intell. 34:9 (2012), 1704–1716, 10.1109/TPAMI.2011.235.
58. Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A., The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88:2 (2010), 303–338, 10.1007/s11263-009-0275-4.
59. Szeliski, R., Computer vision: algorithms and applications. 2010, Springer Science & Business Media.
60. Sokolova, M., Lapalme, G., A systematic analysis of performance measures for classification tasks. Inf. Process. Manag. 45:4 (2009), 427–437, 10.1016/j.ipm.2009.03.002.
61. Hu, N., Englebienne, G., Lou, Z., Kröse, B., Learning latent structure for activity recognition. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2014, IEEE, 1048–1053.
62. Taha, A., Zayed, H.H., Khalifa, M., El-Horbaty, E.-S.M., Skeleton-based human activity recognition for video surveillance. Int. J. Sci. Eng. Res. 6:1 (2015), 993–1004.
63. Rybok, L., Schauerte, B., Al-Halah, Z., Stiefelhagen, R., “Important stuff, everywhere!” Activity recognition with salient proto-objects as context. Proceedings of the IEEE Winter Conference on Applications of Computer Vision, (WACV), 2014, IEEE, 646–651, 10.1109/WACV.2014.6836041.
64. Koppula, H.S., Saxena, A., Learning spatio-temporal structure from RGB-D videos for human activity detection and anticipation. Proceedings of the IEEE International Conference on Machine Learning (ICML), 2013, 792–800.
65. Tayyub, A. JawadTavanai, Gatsoulis, Y., Cohn, A.G., Hogg, D.C., Qualitative and quantitative spatio-temporal relations in daily living activity recognition. Proceedings of the IEEE Computer Vision–ACCV 2014, 2015, Springer International Publishing, Cham, 115–130.
66. Martinetz, T., Competitive Hebbian learning rule forms perfectly topology preserving maps. Proceedings of the IEEE International Conference on Artificial Nerual networks, (ICANN), 1993, Springer, 427–434, 10.1007/978-1-4471-2063-6_104.
67. Parisi, G.I., Tani, J., Weber, C., Wermter, S., Lifelong learning of human actions with deep neural network self-organization. Neural Netw. 96 (2017), 137–149.