Structured time series analysis for human action segmentation and recognition

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 36, Issue 7, 2014, Pages 1414-1427

  Gong, Dian ^a   Medioni, Gérard ^a   Zhao, Xuemei ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

Computer vision; Machine learning

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER VISION; GESTURE RECOGNITION; IMAGE SEGMENTATION; LEARNING SYSTEMS; TIME SERIES ANALYSIS;

ALIGNMENT ALGORITHMS; CHANGE POINT DETECTION; HUMAN ACTION SEGMENTATIONS; HUMAN MOTION CAPTURE DATA; HUMAN-ACTION RECOGNITION; MULTIVARIATE TIME SERIES; REAL-TIME SEGMENTATION; TEMPORAL SEGMENTATIONS;

MOTION ESTIMATION;

ACTIMETRY; ALGORITHM; AUTOMATED PATTERN RECOGNITION; COMPUTER ASSISTED DIAGNOSIS; HUMAN; IMAGE ENHANCEMENT; IMAGE SUBTRACTION; MOTOR ACTIVITY; MOVEMENT (PHYSIOLOGY); PHYSIOLOGY; PROCEDURES; REPRODUCIBILITY; SENSITIVITY AND SPECIFICITY; THREE DIMENSIONAL IMAGING; VIDEORECORDING; WHOLE BODY IMAGING;

ACTIGRAPHY; ALGORITHMS; HUMANS; IMAGE ENHANCEMENT; IMAGE INTERPRETATION, COMPUTER-ASSISTED; IMAGING, THREE-DIMENSIONAL; MOTOR ACTIVITY; MOVEMENT; PATTERN RECOGNITION, AUTOMATED; REPRODUCIBILITY OF RESULTS; SENSITIVITY AND SPECIFICITY; SUBTRACTION TECHNIQUE; VIDEO RECORDING; WHOLE BODY IMAGING;

EID: 84903175597     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2013.244     Document Type: Article

References (69)

1. I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, "Learning realistic human actions from movies," in Proc. CVPR, Anchorage, AK, USA, 2008
2. J. Nielbes, H. Wang, and L. Fei-Fei, "Unsupervised learning of human action categories using spatial-temporal words," IJCV, vol. 79, no. 3, pp. 299-318, 2008
3. P. Matikainen, M. Hebert, and R. Sukthankar, "Representing pairwise spatial and temporal relations for action recognition," in Proc. 11th ECCV, vol. 6311. Berlin, Germany, 2010, pp. 508-521
4. P. Natarajan and R. Nevatia, "View and scale invariant action recognition using multiview shape-flow models," in Proc. CVPR, Anchorage, AK, USA, 2008
5. P. Yan, S. M. Khan, and M. Shah, "Learning 4D action feature models for arbitary view action recognition," in Proc. CVPR, Anchorage, AK, USA, 2008
6. H. Ning, W. Xu, Y. Gong, and T. Huang, "Latent pose estimator for continuous action recognition," in Proc. ECCV, vol. 5303. Marseille, France, 2008, pp. 419-433
7. J. C. Niebles, C.-W. Chen, and L. Fei-Fei, "Modeling temporal structure of decomposable motion segments for activity classification," in Proc. 12th ECCV, Crete, Greece, Sept. 2010
8. T. Hofmann, B. Scholkopf, and A. J. Smola, "Kernel methods in machine learning," Ann. Statist., vol. 36, no. 3, pp. 1171-1220, 2008
9. A. Smola, A. Gretton, L. Song, and B. Schoelkopf, "A Hilbert space embedding for distributions," in Algorithmic Learning Theory: 18th International Conference. Berlin, Germany: Springer-Verlag, 2007, pp. 13-31
10. A. Gretton, K. Borgwardt, M. Rasch, B. Scholkopf, and A. Smola, "A kernel method for the two-sample-problem," in Advances in Neural Information Processing Systems. Vancouver, BC, Canada: MIT Press, 2007, pp. 513-520
11. A. Gretton, K. Fukumizu, Z. Harchaoui, and B. Sriperumbudur, "A fast, consistent kernel two-sample test," in Proc. NIPS, vol. 19. 2009, pp. 673-681
12. J. B. Tenenbaum, V. de Silva, and J. C. Langford, "A global geometric framework for nonlinear dimensionality reduction," Sci., vol. 290, pp. 2319-2323, Dec. 2000 (Pubitemid 32041577)
13. L. K. Saul and S. T. Roweis, "Think globally, fit locally: Unsupervised learning of low dimensional manifolds," J. Mach. Learn. Res., vol. 4, pp. 119-155, Jun. 2003
14. O. C. Jenkins and M. J. Mataric, "A spatio-temporal extension to isomap nonlinear dimension reduction," in Proc. ICML, New York, NY, USA, 2004
15. W. Liao and G. Medioni, "3D face tracking and expression inference from a 2D sequence using manifold learning," in Proc. CVPR, vol. 2. Anchorage, AK, USA, 2008, pp. 416-423
16. P. Mordohai and G. Medioni, "Dimensionality estimation, manifold learning and function approximation using tensor voting," JMLR, vol. 11, pp. 411-450, Jan. 2010
17. N. Lawrence, "Probabilistic non-linear principal component analysis with Gaussian process latent variable models," JMLR, vol. 6, pp. 1783-1816, Nov. 2005
18. J. M. Wang, D. J. Fleet, and A. Hertzmann, "Gaussian process dynamical models for human motion," IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 2, pp. 283-298, Feb. 2008
19. R. Urtasun, D. J. Fleet, A. Geiger, J. Popovic, T. Darrell, and N. D. Lawrence., "Topologically-constrained latent variable models," in Proc. ICML, New York, NY, USA, 2008, pp. 1080-1087
20. R. Urtasun, D. Fleet, and P. Fua, "3D people tracking with Gaussian process dynamical models," in Proc. CVPR, vol. 1. 2006, pp. 238-245 (Pubitemid 44931336)
21. C.-S. Lee, "Modeling human motion using manifold learning and factorized generative models," Ph.D. dissertation, Rutgers University, 2007
22. A. Elgammal and C.-S. Lee, "The role of manifold learning in human motion analysis," Comput. Imaging Vis., vol. 36, pp. 25-56, 2008
23. N. C. Tang, C.-T. Hsu, T.-Y. Lin, and H.-Y. M. Liao, "Examplebased human motion extrapolation based on manifold learning," in Proc. ACM MM, New York, NY, USA, 2011
24. J. Blackburn and E. Ribeiro, "Human motion recognition using isomap and dynamic time warping," in Proc. ICCV Workshop, Rio de Janeiro, Brazil, 2007
25. J. Chen and A. Gupta, Parametric Statistical Change-Point Analysis. Birkhauser, 2000
26. X. Xuan and K. Murphy, "Modeling changing dependency structure in multivariate time series," in Proc. 24th ICML, New York, NY, USA, 2007
27. R. P. Adams and D. J. MacKay, "Bayesian online changepoint detection," Univ. Cambridge, Tech. Rep., 2007
28. Y. Saatci, R. Turner, and C. Rasmussen, "Gaussian process change point models," in Proc. ICML, 2010
29. F. Desobry, M. Davy, and C. Doncarli, "An online kernel change detection algorithm," IEEE Trans. Signal Process., vol. 53, pp. 2961-2974, Aug. 2005 (Pubitemid 41372807)
30. Z. Harchaoui, F. Bach, and E. Moulines, "Kernel change-point analysis," in Proc. Adv. NIPS, 2009
31. A. Y. Ng, M. I. Jordan, and Y. Weiss, "On spectral clustering: Analysis and an algorithm," in Advances in Neural Information Processing Systems 14. Vancouver, BC, Canada: MIT Press, 2002, pp. 849-856
32. U. von Luxburg, "A tutorial on spectral clustering," Statist. Comput., vol. 17, no. 4, pp. 395-416, 2007
33. F. Zhou, F. De la Torre, and J. K. Hodgins, "Hierarchical aligned cluster analysis for temporal clustering of human motion," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 3, pp. 582-596, Mar. 2013
34. E. Fox, E. Sudderth, M. Jordan, and A. Willsky, "Nonparametric bayesian learning of switching linear dynamical systems," in Proc. NIPS, 2009, pp. 457-464
35. H. Zhong, J. Shi, and M. Visontai, "Detecting unusual activity in video," in Proc. CVPR, Washington, DC, USA, 2004, pp. 816-823
36. L. Zelnik-Manor and M. Irani, "Statistical analysis of dynamic actions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, pp. 1530-1535, Sept. 2006 (Pubitemid 44132659)
37. F. De la Torre, J. Campoy, Z. Ambadar, and J. F. Conn, "Temporal segmentation of facial behavior," in Proc. ICCV, Rio de Janeiro, Brazil, 2007
38. J. Barbic, A. Safonova, J.-Y. Pan, C. Faloutsos, J. K. Hodgins, and N. S. Pollard, "Segmenting motion capture data into distinct behaviors," in Proc. GI, Waterloo, ON, Canada, 2004, pp. 185-194
39. I. Junejo, E. Dexter, I. Laptev, and P. Perez, "View-independent action recognition from temporal self-similarities," IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 1, pp. 172-185, Jan. 2011
40. D. Weinland, E. Boyer, and R. Ronfard, "Action recognition from arbitrary views using 3D exemplars," in Proc. ICCV, Rio de Janeiro, Brazil, 2007
41. F. Lv and R. Nevatia, "Single view human action recognition using key pose matching and viterbi path searching," in Proc. CVPR, Minneapolis, MN, USA, 2007, pp. 1-8
42. R. Messing, C. Pal, and H. Kautz, "Activity recognition using the velocity histories of tracked keypoints," in Proc. ICCV, Kyoto, Japan, 2009, pp. 104-111
43. J. Sun, X. Wu, S. Yan, L. F. Cheong, T. S. Chua, and J. Li, "Hierarchical spatial-temporal context modeling for action recognition," in Proc. CVPR, Miami, FL, USA, 2009, pp. 2004-2011
44. R. Poppe, "A survey on vision-based human action recognition," Image Vis. Comput., vol. 28, no. 6, pp. 976-990, 2010
45. F. Lv and R. Nevatia, "Recognition and segmentation of 3-D human action using hmm and multi-calss adaboost," in Proc. ECCV, vol. 3954. Berlin, Germany, 2006, pp. 359-372 (Pubitemid 44029643)
46. D. Weinland, M. Ozuysal, and P. Fua, "Making action recognition robust to occlusions and viewpoint changes," in Proc. ECCV, vol. 6313. Heraklion, Greece, 2010, pp. 635-648
47. W. Li, Z. Zhang, and Z. Liu, "Action recognition based on a bag of 3d points," in Proc. Workshop CVPR Human Communicative Behavior Analysis, 2010, pp. 9-14
48. B. Ni, G. Wang, and P. Moulin, "Rgbd-hudaact: A color-depth video database for huamn daily activity recognition," in Proc. Workshop Consumer Depth Cameras Computer Vision, in Conjunction With ICCV, 2011
49. J. Wang, Z. Liu, Y. Wu, and J. Yuan, "Mining actionlet ensemble for action recognition with depth cameras," in Proc. CVPR, Providence, RI, USA, 2012, pp. 1290-1297
50. F. R. Bach and M. I. Jordan, "Kernel independent component analysis," JMLR, vol. 3, pp. 1-48, Jul. 2003
51. T. K. Kim and R. Cipolla, "Canonical correlation analysis of video volume tensors for action categorization and detection," IEEE Pattern Anal. Mach. Intell., vol. 31, no. 8, pp. 1415-1428, Aug. 2009
52. C. C. Loy, T. Xiang, and S. Gong, "Multi-camera activity correlation analysis," in Proc. CVPR, Miami, FL, USA, 2009, pp. 1988-1995
53. C. Rao, A. Gritaiand, M. Shah, and T. Syeda-Mahmood, "Viewinvariant alignment and matching of video sequences," in Proc. ICCV, Nice, France, 2003, pp. 939-945
54. M. Singh, I. Cheng, M. Mandal, and A. Basu, "Optimization of symmetric transfer error for sub-frame video synchronization," in Proc. ECCV, vol. 5303. Marseille, France, 2008, pp. 554-567
55. L. R. Rabiner and B. Juang, Fundamentals of Speech Recognition. Englewood Cliffs, NJ, USA:Prentice-Hall, Inc., 1993
56. Y. Ukainitz and M. Irani, "Aligning sequences and actions by maximizing space-time correlations," in Proc. ECCV, vol. 3953. 2006, pp. 538-550 (Pubitemid 44029704)
57. F. Padua, F. Carceroni, R. Santos, and G. Kutulakos, "Linear sequence-to-sequence alignment," IEEE Pattern Anal. Mach. Intell., vol. 32, no. 2, pp. 304-320, Feb. 2010
58. F. Zhou and F. De la Torre, "Canonical time warping for alignment of human behavior," in Proc. NIPS, vol. 22. 2009, pp. 2286-2294
59. D. Gong and G. Medioni, "Dynamic manifold warping for view invariant action recognition," in Proc. ICCV, Barcelona, Spain, 2011, pp. 571-578
60. M. Hoai, Z. Lan, and F. De la Torre, "Joint segmentation and classification of human actions in video," in Proc. CVPR, 2011
61. D. Gong, G. Medioni, S. Zhu, and X. Zhao, "Kernelized temporal cut for online tempoal segmentation and recognition," in Proc. ECCV, 2012
62. L. Faivishevsky and J. Goldberger, "A nonparametric information theoretic clustering algorithm," in Proc. ICML, Haifa, Israel, 2010, pp. 351-358
63. I. Laptev, S. Belongie, P. Perez, and J. Wills, "Periodic motion detection and segmentation via approximate sequence alignment," in Proc. ICCV. Berlin, Germany: Springer-Verlag, 2005, pp. 816-8231
64. Y. W. Teh and S. Roweis, "Automatic alignment of local representations," in Advances in Neural Information Processing Systems, M. Kearns, S. Solla, and D. Cohn, Eds. Cambridge, MA, USA: MIT Press, 2003, pp. 841-848
65. CMU Motion Capture Database [Online]. Available: http://mocap.cs.cmu.edu/
66. J. Listgarten, R. M. Neal, S. T. Roweis, and A. Emili, "Multiple alignment of continuous time series," in Proc. NIPS, vol. 17. 2005
67. L. Sigal, A. O. Balan, and M. J. Black, "HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion," IJCV, vol. 87, no. 1-2, pp. 4-27, Mar. 2010
68. OpenNI [Online]. Available: http://www.openni.org/Downloads/ OpenNIModules.aspx
69. B. Yao and L. Fei-Fei, "Action recognition with exemplar based 2.5D graph matching," in Proc. 12th ECCV, Florence, Italy, 2012, pp. 173-186.