Action recognition from video using feature covariance matrices

IEEE Transactions on Image Processing

Volumn 22, Issue 6, 2013, Pages 2479-2494

  Guo, Kai ^a   Ishwar, Prakash ^a   Konrad, Janusz ^a  

^a Boston University   (United States)

Author keywords

Action recognition; feature covariance matrix; nearest neighbor (NN) classifier; optical flow; Riemannian metric; silhouette tunnel; sparse linear approximation (SLA); video analysis

Indexed keywords

ACTION RECOGNITION; FEATURE COVARIANCE MATRIXES; LINEAR APPROXIMATIONS; NEAREST-NEIGHBORS; RIEMANNIAN METRICS; VIDEO ANALYSIS;

CLASSIFICATION (OF INFORMATION); CONES; LINEAR TRANSFORMATIONS; OPTICAL FLOWS; REAL TIME CONTROL;

COVARIANCE MATRIX;

EID: 84877775685     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2013.2252622     Document Type: Article

References (66)

1. J. Aggarwal and M. Ryoo, "Human activity analysis: A review," ACM Comput. Surv., vol. 43, no. 3, pp. 1-43, Apr. 2011.
2. M. Ahmad, I. Parvin, and S. W. Lee, "Silhouette history and energy image information for human movement recognition," J. Multimedia, vol. 5, no. 1, pp. 12-21, Feb. 2010.
3. A. Ali and J. Aggarwal, "Segmentation and recognition of continuous human activity," in Proc. IEEE Workshop Detect. Recognit. Events Video, Jul. 2001, pp. 28-35.
4. S. Ali and M. Shah, "Human action recognition in videos using kinematic features and multiple instance learning," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 2, pp. 288-303, Feb. 2010.
5. V. Arsigny, P. Pennec, and X. Ayache, "Log-Euclidean metrics for fast and simple calculus on diffusion tensors," Magn. Resonance Med., vol. 56, no. 2, pp. 411-421, Aug. 2006. (Pubitemid 44162789)
6. A. Bobick and J. Davis, "The recognition of human movement using temporal templates," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 3, pp. 257-267, Mar. 2001. (Pubitemid 32286632)
7. C. C. Chen, M. S. Ryoo, and J. K. Aggarwal. (2010, Aug. 19). UT-Tower Dataset: Aerial View Activity Classification Challenge [Online]. Available: http://cvrc.ece.utexas.edu/SDHA2010/Aerial-View-Activity.html
8. Y. Chen, Q. Wu, and X. He, "Human action recognition by Radon transform," in Proc. IEEE Int. Conf. Data Mining Workshops, Dec. 2008, pp. 862-868.
9. O. Chomat and J. Crowley, "Probabilistic recognition of activity using local appearance," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 1999, pp. 104-109.
10. D. Cunado, M. S. Nixon, and J. N. Carter, "Automatic extraction and description of human gait models for recognition purposes," Comput. Vis. Image Understand., vol. 90, no. 1, pp. 1-41, Apr. 2003.
11. S. Danafar and N. Gheissari, "Action recognition for surveillance applications using optic flow and SVM," in Proc. Asian Conf. Comput. Vis., 2007, pp. 457-466.
12. K. Derpanis, M. Sizintsev, K. Cannons, and R. Wildes, "Efficient action spotting based on a spacetime oriented structure representation," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2010, pp. 1990-1997.
13. P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie, "Behavior recognition via sparse spatio-temporal features," in Proc. 2nd IEEE Int. Workshop Vis. Surveill. Perform. Evaluation Tracking Surveill., Oct. 2005, pp. 65-72. (Pubitemid 46178025)
14. J. Donelan, R. Kram, and A. Kuo, "Mechanical work for step-to-step transitions is a major determinant of the metabolic cost of human walking," J. Experim. Biol., vol. 205, pp. 3717-3727, Dec. 2002. (Pubitemid 36010759)
15. R. Duda, P. Hart, and D. Stork, Pattern Classification. New York, NY, USA: Wiley, 2001.
16. E. Candès, J. Romberg, and T. Tao, "Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information," IEEE Trans. Inform. Theory, vol. 52, no. 2, pp. 489-509, Feb. 2006. (Pubitemid 43174093)
17. A. Elgammal, R. Duraiswami, D. Harwood, and L. Davis, "Background and foreground modeling using nonparametric kernel density for visual surveillance," Proc. IEEE, vol. 90, no. 7, pp. 1151-1163, Feb. 2002.
18. A. Fathi and G. Mori, "Action recognition by learning mid-level motion features," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2008, pp. 1-8.
19. W. Förstner and B. Moonen, "A metric for covariance matrices," in Festschrift for Erik W. Grafarend on the Occasion of His 60th Birthday, F. Krumm and V. S. Schwarze, Eds. Stuttgart, Germany: Geodätisches Institut der Universität Stuttgart, 1999.
20. L. Goncalves, E. D. Bernardo, E. Ursella, and P. Perona, "Monocular tracking of the human arm in 3-D," in Proc. IEEE Int. Conf. Comput. Vis., Jun. 1995, pp. 764-770.
21. L. Gorelick, M. Blank, E. Shechtman, M. Irani, and R. Basri, "Actions as space-time shapes," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 12, pp. 2247-2253, Dec. 2007. (Pubitemid 350124072)
22. K. Guo, "Action recognition using log-covariance matrices of silhouette and optical-flow features," Ph.D. dissertation, Department of Electrical and Computer Engineering, Boston Univ., Boston, MA, USA, Sep. 2011.
23. K. Guo, P. Ishwar, and J. Konrad, "Action recognition from video by covariance matching of silhouette tunnels," in Proc. Brazilian Symp. Comput. Graph. Image, Oct. 2009, pp. 299-306.
24. K. Guo, P. Ishwar, and J. Konrad, "Action change detection in video by covariance matching of silhouette tunnels," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Mar. 2010, pp. 1110-1113.
25. K. Guo, P. Ishwar, and J. Konrad, "Action recognition in video by sparse representation on covariance manifolds of silhouette tunnels," in Proc. Int. Conf. Pattern Recognit. Semantic Descript. Human Act. Contest, Aug. 2010, pp. 294-305.
26. K. Guo, P. Ishwar, and J. Konrad, "Action recognition using sparse representation on covariance manifolds of optical flow," in Proc. IEEE Int. Conf. Adv. Video Signal Based Surveill., Aug. 2010, pp. 88-195.
27. H. Jhuang, T. Serre, L. Wolf, and T. Poggio, "A biologically inspired system for action recognition," in Proc. IEEE Int. Conf. Comput. Vis., Oct. 2007, pp. 1-8.
28. R. Hill and S. Waters, "On the cone of positive semidefinite matrices," Linear Algebra Appl., vol. 90, pp. 81-88, Oct. 1987.
29. N. Ikizler and P. Duygulu, "Human action recognition using distribution of oriented rectangular patches," in Proc. Human Motion, Understand. Model. Capture Animation, 2007, pp. 271-284.
30. N. Ikizler-Cinbis and S. Sclaroff, "Object, scene and actions: Combining multiple features for human action recognition," in Proc. Eur. Conf. Comput. Vis., 2010, pp. 494-507.
31. A. Kale, A. Sundaresan, A. N. Rajagopalan, N. P. Cuntoor, A. K. Roy-Chowdhury, V. Kruger, and R. Chellappa, "Identification of humans using gait," IEEE Trans. Image Process., vol. 13, no. 9, pp. 1163-1173, Sep. 2004.
32. Y. Ke, R. Sukthankar, and M. Hebert, "Efficient visual event detection using volumetric features," in Proc. IEEE Int. Conf. Comput. Vis., vol. 1. Oct. 2005, pp. 166-173. (Pubitemid 44054955)
33. T. Kim and R. Cipolla, "Canonical correlation analysis of video volume tensors for action categorization and detection," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 8, pp. 1415-1428, Aug. 2009.
34. A. Kovashka and K. Grauman, "Learning a hierarchy of discriminative space-time neighborhood features," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2010, pp. 2046-2053.
35. K. Lai, J. Konrad, and P. Ishwar, "A gesture-driven computer interface using Kinect camera," in Proc. IEEE Southwest Symp. Image Anal. Int., Apr. 2012, pp. 185-188.
36. I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, "Learing realistic human actions from movies," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2008, pp. 1-8.
37. Q. Le, W. Zou, S. Yeung, and A. Ng, "Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 3361-3368.
38. J. Liu, S. Ali, and M. Shah, "Recognizing human actions using multiple features," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2008, pp. 1-8.
39. J. Liu, J. Luo, and M. Shah, "Recognizing realistic actions from videos in the wild," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2009, pp. 1996-2003.
40. P. Matikainen, R. Sukthankar, M. Hebert, and Y. Ke, "Fast motion consistency through matrix quantization," in Proc. Brit.Mach. Vis. Conf., Sep. 2008, pp. 1055-1064.
41. J. Niebles, H. Wang, and L. Fei-Fei, "Unsupervised learning of human action categories using spatial-temporal words," Int. J. Comput. Vis., vol. 79, no. 3, pp. 299-318, Sep. 2008.
42. A. Noguchi and K. Yanai, "A SURF-based spatio-temporal feature for feature-fusion-based action recognition," in Proc. Eur. Conf. Comput. Vis. Workshop Human Motion, Understand. Model. Capture Animation, 2010, pp. 1-15.
43. M. Ristivojevíc and J. Konrad, "Space-time image sequence analysis: Object tunnels and occlusion volumes," IEEE Trans. Image Process., vol. 15, no. 2, pp. 364-376, Feb. 2006. (Pubitemid 43126161)
44. K. Rohr, "Toward model-based recognition of human movements in image sequences," CVGIP, Image Understand., vol. 59, no. 1, pp. 94-115, Jan. 1994.
45. Y. Rui and P. Anandan, "Segmenting visual actions based on spatiotemporal motion patterns," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2000, pp. 111-118.
46. C. Schuldt, I. Laptev, and B. Caputo, "Recognizing human actions: A local SVM approach," in Proc. Int. Conf. Pattern Recognit., vol. 3. Aug. 2004, pp. 32-36.
47. P. Scovanner, S. Ali, and M. Shah, "A 3-D SIFT descriptor and its application to action recognition," in Proc. Int. Conf. Multimedia, Sep. 2007, pp. 357-360.
48. H. J. Seo and P. Milanfar, "Action recognition from one example," IEEE Trans. Pattern Anal. Mach. Intell., vol. 5, no. 5, pp. 867-882, May 2011.
49. P. Smith, N. D. Vitoria Lobo, and M. Shah, "TemporalBoost for event recognition," in Proc. IEEE Int. Conf. Comput. Vis., vol. 1. Oct. 2005, pp. 733-740. (Pubitemid 44055028)
50. T. Starner and A. Pentland, "Visual recognition of American sign language using hidden Markov model," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., Jan. 1995, pp. 1-52.
51. O. Tuzel, F. Porikli, and P. Meer, "Region covariance: A fast descriptor for detection and classification," in Proc. Eur. Conf. Comput. Vis., May 2006, pp. 589-600. (Pubitemid 44029757)
52. O. Tuzel, F. Porikli, and P. Meer, "Pedestrian detection via classification on Riemannian manifolds," IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 10, pp. 1713-1727, Oct. 2008.
53. H. Wang, A. Klaser, C. Schmid, and C. Liu, "Action recognition by dense trajectories," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 3169-3176.
54. L. Wang, H. Ning, T. Tan, and W. Hu, "Fusion of static and dynamic body biometrics for gait recognition," IEEE Trans. Circuits Syst. Video Technol., vol. 14, no. 2, pp. 149-158, Feb. 2004.
55. Y. Wang, K. Huang, and T. Tan, "Human activity recognition based on R transform," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Oct. 2007, pp. 1-8.
56. S. F. Wong and R. Cipolla, "Extracting spatio-temporal interest points using global information," in Proc. IEEE Int. Conf. Comput. Vis., Oct. 2007, pp. 1-8.
57. J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, "Robust face recognition via sparse representation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 2, pp. 210-227, Feb. 2009.
58. X. Wu, D. Xu, L. Duan, and J. Luo, "Action recognition using context and appearance distribution features," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 489-496.
59. J. Yamato, J. Ohya, and K. Ishii, "Recognizing human action in time sequential image using hidden markov model," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 1992, pp. 379-385.
60. A. Yilmaz and M. Shah, "Action sketch: A novel action representation," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., vol. 1. Jun. 2005, pp. 984-989.
61. C. Zach, T. Pock, and H. Bischof, "A duality based approach for realtime TV-L1 optical flow," in Proc. 29th DAGM Conf. Pattern Recognit., 2007, pp. 214-223.
62. T. Zhang, J. Liu, Y. Ouyang, and H. Lu, "Boosted exemplar learning for human action recognition," in Proc. IEEE 12th Int. Conf. Comput. Vis., Sep. 2009, pp. 538-545.
63. TREC Video Retrieval Evaluation: TRECVID. (2013) [Online]. Available: http://trecvid.nist.gov
64. VIRAT Video Dataset. (2013) [Online]. Available: http://www.viratdata.org
65. Action Recognition. (2013) [Online]. Available: http://vip.bu.edu/ projects/vsns/action-recognition
66. Next-Generation Human-Computer Interfaces. (2013) [Online]. Available: http://vip.bu.edu/projects/hcis