Human action recognition using genetic algorithms and convolutional neural networks

Pattern Recognition

Volumn 59, Issue , 2016, Pages 199-212

  Ijjina, Earnest Paul ^a   Chalavadi, Krishna Mohan ^a  

^a INDIAN INSTITUTE OF TECHNOLOGY HYDERABAD   (India)

Author keywords

Action bank features; Convolutional neural network (CNN); Genetic algorithms (GA); Human action recognition

Indexed keywords

ALGORITHMS; CLASSIFICATION (OF INFORMATION); CONVOLUTION; GESTURE RECOGNITION; MOTION ESTIMATION; NEURAL NETWORKS;

ACTION BANK FEATURES; CLASSIFICATION ERRORS; CLASSIFICATION SYSTEM; CONVOLUTIONAL NEURAL NETWORK; FITNESS EVALUATIONS; GLOBAL SEARCH CAPABILITY; GRADIENT DESCENT ALGORITHMS; HUMAN-ACTION RECOGNITION;

GENETIC ALGORITHMS;

EID: 84956860695     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2016.01.012     Document Type: Article

References (79)

1. [1] Bengio, Y., Simard, P., Frasconi, P., Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5:2 (1994), 157–166.
2. [2] Y. Lecun, C. Cortes, The MNIST database of handwritten digits. URL 〈 http://yann.lecun.com/exdb/mnist/ 〉.
3. [3] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M.S. Bernstein, A.C. Berg, L. Fei-Fei, Imagenet large scale visual recognition challenge, CoRR abs/1409.0575.
4. [4] S.E. Kahou, C. Pal, X. Bouthillier, P. Froumenty, c. Gülçehre, R. Memisevic, P. Vincent, A. Courville, Y. Bengio, R.C. Ferrari, M. Mirza, S. Jean, P.-L. Carrier, Y. Dauphin, N. Boulanger-Lewandowski, A. Aggarwal, J. Zumer, P. Lamblin, J.-P. Raymond, G. Desjardins, R. Pascanu, D. Warde-Farley, A. Torabi, A. Sharma, E. Bengio, M. Côté, K. R. Konda, Z. Wu, Combining modality specific deep neural networks for emotion recognition in video, in: Proceedings of the 15th ACM International Conference on Multimodal Interaction, ICMI ׳13, ACM, New York, NY, USA, 2013, pp. 543–550. http://dx.doi.org/10.1145/2522848.2531745.
5. [5] Aggarwal, J., Ryoo, M., Human activity analysis: a review. ACM Comput. Surv. 43:3 (2011), 1–43, 10.1145/1922649.1922653.
6. [6] Weinland, D., Ronfard, R., Boyer, E., A survey of vision-based methods for action representation, segmentation and recognition. Comput. Vis. Image Underst. 115:2 (2011), 224–241, 10.1016/j.cviu.2010.10.002.
7. [7] M.B. Holte, C. Tran, M.M. Trivedi, T.B. Moeslund, Human action recognition using multiple views: a comparative perspective on recent developments, in: Proceedings of the 2011 Joint ACM Workshop on Human Gesture and Behavior Understanding, J-HGBU ׳11, ACM, New York, NY, USA, 2011, pp. 47–52. http://dx.doi.org/10.1145/2072572.2072588.
8. [8] Chaquet, J.M., Carmona, E.J., Fernández-Caballero, A., A survey of video datasets for human action and activity recognition. Comput. Vis. Image Underst. 117:6 (2013), 633–659.
9. [9] P. Foggia, G. Percannella, A. Saggese, M. Vento, Recognizing human actions by a bag of visual words, in: Proceedings of the 2013 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2013, pp. 2910–2915. http://dx.doi.org/10.1109/SMC.2013.496.
10. [10] I. Laptev, M. Marszalek, C. Schmid, B. Rozenfeld, Learning realistic human actions from movies, In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1–8.
11. [11] N. Dalal, B. Triggs, C. Schmid, Human detection using oriented histograms of flow and appearance, In: Proceedings of the 9th European Conference on Computer Vision - Volume Part II, ECCV 06, Springer-Verlag, Berlin, Heidelberg, 2006, pp. 428–441.
12. [12] S. Sadanand, J. J. Corso, Action bank: A high-level representation of activity in video, in: Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, pp. 1234–1241.
13. [13] H. Wang, A. Klaser, C. Schmid, C.-L. Liu, Action recognition by dense trajectories, in: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011, pp. 3169–3176.
14. [14] X. Liang, L. Lin, L. Cao, Learning latent spatio-temporal compositional model for human action recognition, In: ACM International Conference on Multimedia (ACM MM), 2013, pp. 263–272.
15. [15] Bengio, Y., Courville, A., Vincent, P., Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35:8 (2013), 1798–1828, 10.1109/TPAMI.2013.50.
16. [16] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, In: Advances in Neural Information Processing Systems (NIPS 2012), 2012, pp. 1097–1105.
17. [17] R.B. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, CoRR abs/1311.2524.
18. [18] Lawrence, S., Giles, C., Tsoi, A.C., Back, A., Face recognition: a convolutional neural-network approach. IEEE Trans. Neural Netw. 8:1 (1997), 98–113, 10.1109/72.554195.
19. [19] Matsugu, M., Mori, K., Mitari, Y., Kaneda, Y., Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Netw. 16:5–6 (2003), 555–559.
20. [20] Le Callet, P., Viard-Gaudin, C., Barba, D., A convolutional neural network approach for objective video quality assessment. IEEE Trans. Neural Netw. 17:5 (2006), 1316–1327, 10.1109/TNN.2006.879766.
21. [21] Ji, S., Xu, W., Yang, M., Yu, K., 3d convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 35:1 (2013), 221–231.
22. [22] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, L. Fei-Fei, Large-scale video classification with convolutional neural networks, In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 1725–1732. http://dx.doi.org/10.1109/CVPR.2014.223.
23. [23] D. Tran, L. D. Bourdev, R. Fergus, L. Torresani, M. Paluri, C3D: generic features for video analysis, CoRR abs/1412.0767.
24. [24] Y. LeCun, K. Kavukcuoglu, C. Farabet, Convolutional networks and applications in vision, in: Proceedings of 2010 IEEE International Symposium on Circuits and Systems (ISCAS), 2010, pp. 253–256. http://dx.doi.org/10.1109/ISCAS.2010.5537907.
25. [25] Lecun, Y., Bottou, L., Bengio, Y., Haffner, P., Gradient-based learning applied to document recognition. Proc. IEEE 86:11 (1998), 2278–2324.
26. [26] M. Baccouche, F. Mamalet, C. Wolf, C. Garcia, A. Baskurt, Sequential deep learning for human action recognition, In: Proceedings of the Second International Conference on Human Behavior Unterstanding, HBU׳11, Springer-Verlag, Berlin, Heidelberg, 2011, pp. 29–39.
27. [27] K. Wang, X. Wang, L. Lin, M. Wang, W. Zuo, 3d human activity recognition with reconfigurable convolutional neural networks, in: Proceedings of the ACM International Conference on Multimedia, MM ׳14, ACM, New York, NY, USA, 2014, pp. 97–106. http://dx.doi.org/10.1145/2647868.2654912.
28. [28] L. Lin, K. Wang, W. Zuo, M. Wang, J. Luo, L. Zhang, A deep structured model with radius-margin bound for 3d human activity recognition, Int. J. Comput. Vis., 2015, pp. 1–18. http://dx.doi.org/10.1007/s11263-015-0876-z.
29. [29] G. Chéron, I. Laptev, C. Schmid, P-CNN: pose-based CNN features for action recognition, CoRR abs/1506.03607 URL arxiv.org/abs/1506.03607.
30. [30] V. Veeriah, N. Zhuang, G. Qi, Differential recurrent neural networks for action recognition, CoRR abs/1504.06678. URL arxiv.org/abs/1504.06678.
31. [31] K. Simonyan, A. Zisserman, Two-stream convolutional networks for action recognition in videos, CoRR abs/1406.2199. URL arxiv.org/abs/1406.2199.
32. [32] Y. Bengio, O. Delalleau, On the expressive power of deep architectures, in: Proceedings of the 22nd International Conference on Algorithmic Learning Theory, ALT׳11, Springer-Verlag, Berlin, Heidelberg, 2011, pp. 18–36.
33. [33] Bengio, Y., Learning deep architectures for ai. Found. Trends Mach. Learn. 2:1 (2009), 1–127, 10.1561/2200000006.
34. [34] Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle, U.D. Montral, M. Québec, Greedy layer-wise training of deep networks, In: In NIPS, MIT Press, 2007.
35. [35] Hinton, G.E., Osindero, S., Teh, Y.-W., A fast learning algorithm for deep belief nets. Neural Comput. 18:7 (2006), 1527–1554.
36. [36] Larochelle, H., Bengio, Y., Louradour, J., Lamblin, P., Exploring strategies for training deep neural networks. J. Mach. Learn. Res. 10 (2009), 1–40.
37. [37] X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in: Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS׳10). Society for Artificial Intelligence and Statistics, 2010.
38. [38] L. Prechelt, Early stopping – but when? In: Neural Networks: Tricks of the Trade, Lecture Notes in Computer Science, vol. 1524, Springer-Verlag, 1997, pp. 55–69, Chapter 2.
39. [39] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15:1 (2014), 1929–1958.
40. [40] G.E. Dahl, T.N. Sainath, G.E. Hinton, Improving deep neural networks for lvcsr using rectified linear units and dropout, in: Proceedings of the 2013 International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2013, pp. 8609–8613.
41. [41] Erhan, D., Bengio, Y., Courville, A., Manzagol, P.-A., Vincent, P., Bengio, S., Why does unsupervised pre-training help deep learning?. J. Mach. Learn. Res. 11 (2010), 625–660.
42. [42] I. Sutskever, J. Martens, G.E. Dahl, G.E. Hinton, On the importance of initialization and momentum in deep learning, In: Proceedings of the 30th International Conference on Machine Learning (ICML-13), JMLR Proceedings, vol. 28, JMLR.org, 2013, pp. 1139–1147.
43. [43] Bergstra, J., Bengio, Y., Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13 (2012), 281–305.
44. [44] Chang, E.I., Lippmann, R., Using genetic algorithms to improve pattern classification performance. Lippmann, R., Moody, J.E., Touretzky, D.S., (eds.) Proceedings of Advances in Neural Information Processing Systems (NIPS), November 26–29, 1990, Morgan Kaufmann, Denver, Colorado, USA, 797–803.
45. [45] D. Decker, J. Hintz, A genetic algorithm and neural network hybrid classification scheme, in: Proceedings of 9th AIAA Computers in Aerospace Conference, AIAA, 1993, pp. 473–475.
46. [46] S.A. Harp, T. Samad, A. Guha, Towards the genetic synthesis of neural network, in: Proceedings of the Third International Conference on Genetic Algorithms, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1989, pp. 360–369.
47. [47] Schaffer, J., Caruana, R.A., Eshelman, L.J., Using genetic search to exploit the emergent behavior of neural networks. Phys. D: Nonlinear Phenom. 42:1–3 (1990), 244–248 http://dx.doi.org/10.1016/0167-2789(90)90078-4.
48. [48] D.J. Montana, L. Davis, Training feedforward neural networks using genetic algorithms, in: Proceedings of the 11th International Joint Conference on Artificial Intelligence (IJCAI׳89), vol. 1, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1989, pp. 762–767.
49. [49] Ding, S., Li, H., Su, C., Yu, J., Jin, F., Evolutionary artificial neural networks: a review. Artif. Intell. Rev. 39:3 (2013), 251–260, 10.1007/s10462-011-9270-6.
50. [50] J.R. Koza, J.P. Rice, Genetic generation of both the weights and architecture for a neural network, in: International Joint Conference on Neural Networks (IJCNN-91), vol. ii, 1991, pp. 397–404 vol.2. http://dx.doi.org/10.1109/IJCNN.1991.155366.
51. [51] F. Gruau, Genetic synthesis of boolean neural networks with a cell rewriting developmental process, in: International Workshop on Combinations of Genetic Algorithms and Neural Networks (COGANN-92), 1992, pp. 55–74. http://dx.doi.org/10.1109/COGANN.1992.273948.
52. [52] R.J. Collins, D.R. Jefferson, An artificial neural network representation for artificial organisms, in: Parallel Problem Solving from Nature, Springer-Verlag, 1990, pp. 259–263.
53. [53] Bornholdt, S., Graudenz, D., General asymmetric neural networks and structure design by genetic algorithms. Neural Netw. 5:2 (1992), 327–334 http://dx.doi.org/10.1016/S0893-6080(05)80030-9.
54. [54] J. Schaffer, D. Whitley, L. Eshelman, Combinations of genetic algorithms and neural networks: a survey of the state of the art, in: International Workshop on Combinations of Genetic Algorithms and Neural Networks (COGANN-92), 1992, pp. 1–37. http://dx.doi.org/10.1109/COGANN.1992.273950.
55. [55] Yao, X., Evolving artificial neural networks. Proc. IEEE 87:9 (1999), 1423–1447, 10.1109/5.784219.
56. [56] Yao, X., A review of evolutionary artificial neural networks. Int. J. Intell. Syst. 4 (1993), 539–567.
57. [57] O.E. David, I. Greental, Genetic algorithms for evolving deep neural networks, in: Proceedings of the 2014 Conference Companion on Genetic and Evolutionary Computation Companion, GECCO Comp ׳14, ACM, New York, NY, USA, 2014, pp. 1451–1452. http://dx.doi.org/10.1145/2598394.2602287.
58. [58] R. Oullette, M. Browne, K. Hirasawa, Genetic algorithm optimization of a convolutional neural network for autonomous crack detection, in: Congress on Evolutionary Computation (CEC2004), vol. 1, 2004, pp. 516–521.
59. [59] L.-O. Fedorovici, R.-E. Precup, F. Dragan, C. Purcaru, Evolutionary optimization-based training of convolutional neural networks for ocr applications, in: 17th International Conference on System Theory, Control and Computing (ICSTCC), 2013, pp. 207–212.
60. [60] Rashedi, E., Nezamabadi-pour, H., Saryazdi, S., Gsa: a gravitational search algorithm. Inf. Sci. 179:13 (2009), 2232–2248.
61. [61] J. Kennedy, R.C. Eberhart, Particle swarm optimization, in: IEEE International Conference on Neural Networks, vol. 4, Perth, Australia, IEEE Service Center, Piscataway, NJ, 1995, pp. 1942–1948.
62. [62] Koutník, J., Schmidhuber, J., Gomez, F., Lecture Notes in Computer Science. del Pobil, A., Chinellato, E., Martinez-Martin, E., Hallam, J., Cervera, E., Morales, A., (eds.) From Animals to Animats 13, Lecture Notes in Computer Science, vol. 8575, 2014, Springer International Publishing, 260–269.
63. [63] Gomez, F., Schmidhuber, J., Miikkulainen, R., Accelerated neural evolution through cooperatively coevolved synapses. J. Mach. Learn. Res. 9 (2009), 937–965.
64. [64] J. Holland, Adaptation in Natural and Artificial Systems, University of Michigan Press, Ann Arbor, MI, USA, 1975. URL 〈 http://books.google.com/books?id=YE5RAAAAMAAJ〉.
65. [65] Bascom, J., Darwin׳s theory of the origin of species. Am. Theol. Rev. 3 (1871), 349–379.
66. [66] Goldberg, D.E., Genetic algorithms. 2006, Pearson Education, India.
67. [67] Zhou, Z.-H., Ensemble Methods: Foundations and Algorithms. 1st Ed., 2012, Chapman & Hall/CRC.
68. [68] R.B. Palm, Prediction as a candidate for learning deep hierarchical models of data (Master׳s thesis). Technical University of Denmark, Asmussens Alle, Denmark, 2012.
69. [69] Reddy, K.K., Shah, M., Recognizing 50 human action categories of web videos. Mach. Vis. Appl. 24:5 (2012), 971–981, 10.1007/s00138-012-0450-4.
70. [70] Action bank: a high-level representation of activity in video, URL 〈 http://www.cse.buffalo.edu/~jcorso/r/actionbank/〉, accessed on: 2015-08-08.
71. [71] Huang, G.-B., Zhu, Q.-Y., Siew, C.K., Extreme learning machine: theory and applications. Neurocomputing 70:1–3 (2006), 489–501.
72. [72] O. Kliper-Gross, Y. Gurovich, T. Hassner, L. Wolf, Motion interchange patterns for action recognition in unconstrained videos, In: Proceedings of the 12th European Conference on Computer Vision(ECCV) – volume Part VI, ECCV׳12, Springer-Verlag, Berlin, Heidelberg, 2012, pp. 256–269.
73. [73] F. Shi, E. Petriu, R. Laganiere, Sampling strategies for real-time action recognition, In: Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 2595–2602.
74. [74] L. Wang, Y. Qiao, X. Tang, Motionlets: Mid-level 3d parts for human motion recognition, in: Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 2674–2681.
75. [75] Q. Zhou, G. Wang, K. Jia, Q. Zhao, Learning to share latent tasks for action recognition, in: Proceedings of the 2013 IEEE International Conference on Computer Vision (ICCV), 2013, pp. 2264 –2271.
76. [76] E.P. Ijjina, C. Mohan, Human action recognition based on recognition of linear patterns in action bank features using convolutional neural networks, In: Proceedings of the 13th International Conference on Machine Learning and Applications (ICMLA), 2014, pp. 178–182. http://dx.doi.org/10.1109/ICMLA.2014.33.
77. [77] N. Ballas, Y. Yang, Z.-Z. Lan, B. Delezoide, F. Preteux, A. Hauptmann, Space-time robust representation for action recognition, In: The IEEE International Conference on Computer Vision (ICCV), 2013.
78. [78] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, E. Shelhamer, cudnn: Efficient primitives for deep learning, CoRR abs/1410.0759. URL arxiv.org/abs/1410.0759.
79. [79] D. Tran, L. Torresani, EXMOVES: classifier-based features for scalable action recognition, CoRR abs/1312.5785.