Human Action Recognition in Unconstrained Videos by Explicit Motion Modeling

IEEE Transactions on Image Processing

Volumn 24, Issue 11, 2015, Pages 3781-3795

  Jiang, Yu Gang ^a   Dai, Qi ^a   Liu, Wei ^b   Xue, Xiangyang ^a   Ngo, Chong Wah ^c  

^a FUDAN UNIVERSITY   (China)

^b IBM T J WATSON RESEARCH CENTER   (United States)

^c CITY UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

camera motion; Human action recognition; motion representation; reference points; trajectory

Indexed keywords

BENCHMARKING; CAMERAS; FACE RECOGNITION; GESTURE RECOGNITION; MOTION ESTIMATION; TRAJECTORIES;

BACKGROUND SEPARATION; CAMERA MOTIONS; COMPETITIVE PERFORMANCE; EXPERIMENTAL EVALUATION; HUMAN-ACTION RECOGNITION; MOTION REPRESENTATION; REFERENCE POINTS; STATE-OF-THE-ART APPROACH;

MOTION ANALYSIS;

ALGORITHM; AUTOMATED PATTERN RECOGNITION; HUMAN; IMAGE PROCESSING; MOVEMENT (PHYSIOLOGY); PROCEDURES; VIDEORECORDING;

ALGORITHMS; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MOVEMENT; PATTERN RECOGNITION, AUTOMATED; VIDEO RECORDING;

EID: 84938629614     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2015.2456412     Document Type: Article

References (68)

1. I. Laptev, "On space-time interest points, " Int. J. Comput. Vis., vol. 64, pp. 107-123, Sep. 2005.
2. I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, "Learning realistic human actions from movies, " in Proc. IEEE Conf. CVPR, Jun. 2008, pp. 1-8.
3. H. Wang, A. Klaser, C. Schmid, and C.-L. Liu, "Action recognition by dense trajectories, " in Proc. IEEE Conf. CVPR, Jun. 2011, pp. 3169-3176.
4. H. Wang, A. Klaser, C. Schmid, and C.-L. Liu, "Dense trajectories and motion boundary descriptors for action recognition, " Int. J. Comput. Vis., vol. 103, no. 1, pp. 60-79, 2013.
5. J. Liu, J. Luo, and M. Shah, "Recognizing realistic actions from videos 'in the wild, "' in Proc. IEEE Conf. CVPR, Jun. 2009, pp. 1996-2003.
6. F. Perronnin, J. Sánchez, and T. Mensink, "Improving the Fisher kernel for large-scale image classification, " in Proc. 11th ECCV, 2010, pp. 143-156.
7. H. Jegou, M. Douze, C. Schmid, and P. Pérez, "Aggregating local descriptors into a compact image representation, " in Proc. IEEE Conf. CVPR, Jun. 2010, pp. 3304-3311.
8. X. Zhou, K. Yu, T. Zhang, and T. S. Huang, "Image classification using super-vector coding of local image descriptors, " in Proc. 11th ECCV, 2010, pp. 141-154.
9. P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie, "Behavior recognition via sparse spatio-temporal features, " in Proc. 2nd Joint IEEE Int. Workshop VS-PETS, Oct. 2005, pp. 65-72.
10. I. N. Junejo, E. Dexter, I. Laptev, and P. Pérez, "Cross-view action recognition from temporal self-similarities, " in Proc. 10th ECCV, 2008, pp. 293-306.
11. P. Scovanner, S. Ali, and M. Shah, "A 3-dimensional SIFT descriptor and its application to action recognition, " in Proc. 15th Int. Conf. MM, 2007, pp. 357-360.
12. G. Willems, T. Tuytelaars, and L. Van Gool, "An efficient dense and scale-invariant spatio-temporal interest point detector, " in Proc. 10th ECCV, 2008, pp. 650-663.
13. J. Knopp, M. Prasad, G. Willems, R. Timofte, and L. Van Gool, "Hough transform and 3D SURF for robust three dimensional classification, " in Proc. 11th ECCV, 2010, pp. 589-602.
14. G. W. Taylor, R. Fergus, Y. LeCun, and C. Bregler, "Convolutional learning of spatio-temporal features, " in Proc. 11th ECCV, 2010, pp. 140-153.
15. Q. V. Le, W. Y. Zou, S. Y. Yeung, and A. Y. Ng, "Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis, " in Proc. IEEE Conf. CVPR, Jun. 2011, pp. 3361-3368.
16. A. Kovashka and K. Grauman, "Learning a hierarchy of discriminative space-time neighborhood features for human action recognition, " in Proc. IEEE Conf. CVPR, Jun. 2010, pp. 2046-2053.
17. E. Vig, M. Dorr, and D. Cox, "Space-variant descriptor sampling for action recognition based on saliency and eye movements, " in Proc. 12th ECCV, 2012, pp. 84-97.
18. F. Shi, E. Petriu, and R. Laganiere, "Sampling strategies for realtime action recognition, " in Proc. IEEE Conf. CVPR, Jun. 2013, pp. 2595-2602.
19. J. Liu, B. Kuipers, and S. Savarese, "Recognizing human actions by attributes, " in Proc. IEEE Conf. CVPR, Jun. 2011, pp. 3337-3344.
20. S. Sadanand and J. J. Corso, "Action bank: A high-level representation of activity in video, " in Proc. IEEE Conf. CVPR, Jun. 2012, pp. 1234-1241.
21. H. Uemura, S. Ishikawa, and K. Mikolajczyk, "Feature tracking and motion compensation for action recognition, " in Proc. BMVC, 2008, pp. 1-10.
22. R. Messing, C. Pal, and H. Kautz, "Activity recognition using the velocity histories of tracked keypoints, " in Proc. IEEE 12th ICCV, Sep./Oct. 2009, pp. 104-111.
23. F. Wang, Y.-G. Jiang, and C.-W. Ngo, "Video event detection using motion relativity and visual relatedness, " in Proc. 16th Int. Conf. MM, 2008, pp. 239-248.
24. M. Raptis and S. Soatto, "Tracklet descriptors for action modeling and video analysis, " in Proc. 11th ECCV, 2010, pp. 577-590.
25. P. Matikainen, M. Hebert, and R. Sukthankar, "Representing pairwise spatial and temporal relations for action recognition, " in Proc. 11th ECCV, 2010, pp. 508-521.
26. J. Sun, X. Wu, S. Yan, L.-F. Cheong, T.-S. Chua, and J. Li, "Hierarchical spatio-temporal context modeling for action recognition, " in Proc. IEEE Conf. CVPR, Jun. 2009, pp. 2004-2011.
27. P. Turaga, R. Chellappa, V. S. Subrahmanian, and O. Udrea, "Machine recognition of human activities: A survey, " IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 11, pp. 1473-1488, Nov. 2008.
28. R. Poppe, "A survey on vision-based human action recognition, " Image Vis. Comput., vol. 28, no. 6, pp. 976-990, 2010.
29. Y.-G. Jiang, S. Bhattacharya, S.-F. Chang, and M. Shah, "High-level event recognition in unconstrained videos, " Int. J. Multimedia Inf. Retr., vol. 2, no. 2, pp. 73-101, 2013.
30. B. D. Lucas and T. Kanade, "An iterative image registration technique with an application to stereo vision, " in Proc. 7th IJCAI, 1981, pp. 674-679.
31. A. Gaidon, Z. Harchaoui, and C. Schmid, "Recognizing activities with cluster-trees of tracklets, " in Proc. BMVC, 2012, pp. 30.1-30.13.
32. M. Raptis, I. Kokkinos, and S. Soatto, "Discovering discriminative action parts from mid-level video representations, " in Proc. IEEE Conf. CVPR, Jun. 2012, pp. 1242-1249.
33. O. Kliper-Gross, Y. Gurovich, T. Hassner, and L. Wolf, "Motion interchange patterns for action recognition in unconstrained videos, " in Proc. 12th ECCV, 2012, pp. 256-269.
34. X. Wang, K. T. Ma, G.-W. Ng, and W. E. L. Grimson, "Trajectory analysis and semantic region modeling using nonparametric hierarchical Bayesian models, " Int. J. Comput. Vis., vol. 95, no. 3, pp. 287-312, 2011.
35. S. Wu, O. Oreifej, and M. Shah, "Action recognition in videos acquired by a moving camera using motion decomposition of Lagrangian particle trajectories, " in Proc. ICCV, 2011, pp. 1419-1426.
36. F. I. Bashir, A. A. Khokhar, and D. Schonfeld, "Object trajectory-based activity classification and recognition using hidden Markov models, " IEEE Trans. Image Process., vol. 16, no. 7, pp. 1912-1919, Jul. 2007.
37. A. Hervieu, P. Bouthemy, and J.-P. Le Cadre, "A statistical video content recognition method using invariant features on object trajectories, " IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 11, pp. 1533-1543, Nov. 2008.
38. H. Wang and C. Schmid, "Action recognition with improved trajectories, " in Proc. IEEE ICCV, Dec. 2013, pp. 3551-3558.
39. N. Dalal, B. Triggs, and C. Schmid, "Human detection using oriented histograms of flow and appearance, " in Proc. 9th ECCV, 2006, pp. 428-441.
40. M. Jain, H. Jegou, and P. Bouthemy, "Better exploiting motion for better action recognition, " in Proc. IEEE Conf. CVPR, Jun. 2013, pp. 2555-2562.
41. C. R. Jung, L. Hennemann, and S. R. Musse, "Event detection using trajectory clustering and 4D histograms, " IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 11, pp. 1565-1575, Nov. 2008.
42. G. Piriou, P. Bouthemy, and J.-F. Yao, "Recognition of dynamic video contents with global probabilistic models of visual motion, " IEEE Trans. Image Process., vol. 15, no. 11, pp. 3417-3430, Nov. 2006.
43. Y.-G. Jiang, Q. Dai, X. Xue, W. Liu, and C.-W. Ngo, "Trajectory-based modeling of human actions with motion reference points, " in Proc. 12th ECCV, 2012, pp. 425-438.
44. G. Farnebäck, "Two-frame motion estimation based on polynomial expansion, " in Proc. 13th Scandin. Conf. Image Anal., 2003, pp. 363-370.
45. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection, " in Proc. IEEE Conf. CVPR, Jun. 2005, pp. 886-893.
46. H. Wang, M. M. Ullah, A. Klaser, I. Laptev, and C. Schmid, "Evaluation of local spatio-temporal features for action recognition, " in Proc. BMVC, 2008, pp. 124.1-124.11.
47. H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, "HMDB: A large video database for human motion recognition, " in Proc. IEEE ICCV, Nov. 2011, pp. 2556-2563.
48. J. Sivic and A. Zisserman, "Video Google: A text retrieval approach to object matching in videos, " in Proc. 9th IEEE ICCV, Oct. 2003, pp. 1470-1477.
49. J. Yuan, Y. Wu, and M. Yang, "Discovery of collocation patterns: From visual words to visual phrases, " in Proc. IEEE Conf. CVPR, Jun. 2007, pp. 1-8.
50. A. Gilbert, J. Illingworth, and R. Bowden, "Action recognition using mined hierarchical compound features, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 5, pp. 883-897, May 2011.
51. H. Jegou, M. Douze, and C. Schmid, "Product quantization for nearest neighbor search, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 1, pp. 117-128, Jan. 2011.
52. S. Roweis, "EM algorithms for PCA and SPCA, " in Proc. NIPS, 1998, pp. 626-632.
53. M. Marszalek, I. Laptev, and C. Schmid, "Actions in context, " in Proc. IEEE Conf. CVPR, Jun. 2009, pp. 2929-2936.
54. J. C. Niebles, C.-W. Chen, and L. Fei-Fei, "Modeling temporal structure of decomposable motion segments for activity classification, " in Proc. 11th ECCV, 2010, pp. 392-405.
55. K. Soomro, A. R. Zamir, and M. Shah. (2012). "UCF101: A dataset of 101 human actions classes from videos in the wild." [Online]. Available: http://arxiv.org/abs/1212.0402
56. M. Brown and D. G. Lowe, "Automatic panoramic image stitching using invariant features, " Int. J. Comput. Vis., vol. 74, no. 1, pp. 59-73, 2007.
57. Y.-G. Jiang et al. (2013). THUMOS Challenge: Action Recognition With a Large Number of Classes. [Online]. Available: http://crcv.ucf.edu/ICCV13-Action-Workshop/
58. Z. Cai, L. Wang, X. Peng, and Y. Qiao, "Multi-view super vector for action recognition, " in Proc. IEEE Conf. CVPR, Jun. 2014, pp. 596-603.
59. M. M. Ullah, S. N. Parizi, and I. Laptev, "Improving bag-offeatures action recognition with non-local cues, " in Proc. BMVC, 2010, pp. 95.1-95.11.
60. J. Wu, Y. Zhang, and W. Lin, "Towards good practices for action video encoding, " in Proc. IEEE Conf. CVPR, Jun. 2014, pp. 2577-2584.
61. W. Brendel and S. Todorovic, "Learning spatiotemporal graphs of human activities, " in Proc. IEEE ICCV, Nov. 2011, pp. 778-785.
62. R. Gopalan, "Joint sparsity-based representation and analysis of unconstrained activities, " in Proc. IEEE Conf. CVPR, Jun. 2013, pp. 2738-2745.
63. V. Kantorov and I. Laptev, "Efficient feature extraction, encoding, and classification for action recognition, " in Proc. IEEE Conf. CVPR, Jun. 2014, pp. 2593-2600.
64. W. Li, Q. Yu, H. Sawhney, and N. Vasconcelos, "Recognizing activities via bag of words for attribute dynamics, " in Proc. IEEE Conf. CVPR, Jun. 2013, pp. 2587-2594.
65. L. Wang, Y. Qiao, and X. Tang, "Mining motion atoms and phrases for complex action recognition, " in Proc. IEEE ICCV, Dec. 2013, pp. 2680-2687.
66. S. Narayan and K. R. Ramakrishnan, "A cause and effect analysis of motion trajectories for modeling actions, " in Proc. IEEE Conf. CVPR, Jun. 2014, pp. 2633-2640.
67. X. Peng, C. Zou, Y. Qiao, and Q. Peng, "Action recognition with stacked fisher vectors, " in Proc. 13th ECCV, 2014, pp. 581-595.
68. T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, and T. Poggio, "Robust object recognition with cortex-like mechanisms, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 3, pp. 411-426, Mar. 2007.