Robust action recognition via borrowing information across video modalities

IEEE Transactions on Image Processing

Volumn 24, Issue 2, 2015, Pages 709-723

  Tang, Nick C ^a,b   Lin, Yen Yu ^c   Hua, Ju Hsuan ^d   Wei, Shih En ^d   Weng, Ming Fang ^e   Liao, Hong Yuan Mark ^a  

^a INSTITUTE OF INFORMATION SCIENCE   (Taiwan)

^b CHUNG YUAN CHRISTIAN UNIVERSITY   (Taiwan)

^c RESEARCH CENTER FOR INFORMATION TECHNOLOGY INNOVATION   (Taiwan)

^d CARNEGIE MELLON UNIVERSITY   (United States)

^e INSTITUTE FOR INFORMATION INDUSTRY   (Taiwan)

Author keywords

Action recognition; feature borrowing; next generation cameras; transfer learning

Indexed keywords

CAMERAS; IMAGE RECOGNITION; MOTION ESTIMATION; MUSCULOSKELETAL SYSTEM;

ACTION RECOGNITION; EFFECTIVE DISTANCE; FEATURE BORROWING; RECOGNITION ACCURACY; SURVEILLANCE SYSTEMS; TRANSFER LEARNING; VIDEO-CONTENT ANALYSIS; VISUAL KNOWLEDGE;

SECURITY SYSTEMS;

EID: 84921364725     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2014.2385591     Document Type: Article

References (59)

1. R. Poppe, "A survey on vision-based human action recognition," Image Vis. Comput., vol. 28, no. 6, pp. 976-990, 2010.
2. Microsoft Kinect. [Online]. Available: http://en.wikipedia.org/wiki/Kinect, accessed Oct. 2013.
3. FUJIFILM FinePix 3D. [Online]. Available: http://en.wikipedia.org/wiki/Fujifilm-FinePix-Real-3D, accessed Oct. 2013.
4. FLIR T620. [Online]. Available: http://www.flir.com, accessed Oct. 2013.
5. Lytro. [Online]. Available: http://en.wikipedia.org/wiki/Lytro, accessed Oct. 2013.
6. J. Shotton et al., "Real-time human pose recognition in parts from single depth images," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 1297-1304.
7. W. Shen, K. Deng, X. Bai, T. Leyvand, B. Guo, and Z. Tu, "Exemplarbased human action pose correction and tagging," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2012, pp. 1784-1791.
8. S. Z. Li, Markov Random Field Modeling in Image Analysis. New York, NY, USA: Springer-Verlag, 2009.
9. F. R. Bach and M. I. Jordan, "Kernel independent component analysis," J. Mach. Learn. Res., vol. 3, pp. 1-48, Jul. 2002.
10. A. F. Bobick and J. W. Davis, "The recognition of human movement using temporal templates," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 3, pp. 257-267, Mar. 2001.
11. L. Gorelick, M. Blank, E. Shechtman, M. Irani, and R. Basri, "Actions as space-time shapes," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 12, pp. 2247-2253, Dec. 2007.
12. I. Laptev and T. Linderberg, "Space-time interest points," in Proc. Int. Conf. Comput. Vis., 2003, pp. 432-439.
13. H. Wang, A. Kläser, C. Schmid, and C.-L. Liu, "Action recognition by dense trajectories," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 3169-3176.
14. P. Matikainen, M. Hebert, and R. Sukthankar, "Representing pairwise spatial and temporal relations for action recognition," in Proc. 11th Eur. Conf. Comput. Vis., 2010, pp. 508-521.
15. K. Prabhaka, S. Oh, P. Wang, G. D. Abowd, and J. M. Rehg, "Temporal causality for the analysis of visual events," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2010, pp. 1967-1974.
16. L. Wang and D. Suter, "Recognizing human activities from silhouettes: Motion subspace and factorial discriminative graphical model," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2007, pp. 1-8.
17. C.-C. Chen and J. K. Aggarwal, "Modeling human activities as speech," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 3425-3432.
18. R. Girshick, J. Shotton, P. Kohli, A. Criminisi, and A. Fitzgibbon, "Efficient regression of general-Activity human poses from depth images," in Proc. IEEE Int. Conf. Comput. Vis., Nov. 2011, pp. 415-422.
19. X. Yang, C. Zhang, and Y. Tian, "Recognizing actions using depth motion maps-based histograms of oriented gradients," in Proc. 20th ACM Int. Conf. Multimedia, 2012, pp. 1057-1060.
20. OpenNI Library. [Online]. Available: http://www.openni.org/, accessed Oct. 2013.
21. J. Wang, Z. Liu, Y. Wu, and J. Yuan, "Mining actionlet ensemble for action recognition with depth cameras," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2012, pp. 1290-1297.
22. C. Ellis, S. Z. Masood, M. F. Tappen, J. J. LaViola, Jr., and R. Sukthankar, "Exploring the trade-off between accuracy and observational latency in action recognition," Int. J. Comput. Vis., vol. 101, no. 3, pp. 420-436, 2013.
23. N. Ashraf, C. Sun, and H. Foroosh, "View invariant action recognition using projective depth," Comput. Vis. Image Understand., vol. 123, pp. 41-52, Jul. 2014.
24. N. C. Tang, Y.-Y. Lin, J.-H. Hua, M.-F. Weng, and H.-Y. M. Liao, "Human action recognition using associated depth and skeleton information," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., May 2014, pp. 4608-4612.
25. S. J. Pan and Q. Yang, "A survey on transfer learning," IEEE Trans. Knowl. Data Eng., vol. 22, no. 10, pp. 1345-1359, Oct. 2010.
26. L. Jie, T. Tommasi, and B. Caputo, "Multiclass transfer learning from unconstrained priors," in Proc. IEEE Int. Conf. Comput. Vis., Nov. 2011, pp. 1863-1870.
27. L. Cao, Z. Liu, and T. S. Huang, "Cross-dataset action detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2010, pp. 1998-2005.
28. Q. Yin, X. Tang, and J. Sun, "An associate-predict model for face recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 497-504.
29. L. Torresani, M. Szummer, and A. Fitzgibbon, "Efficient object category recognition using classemes," in Proc. 11th Eur. Conf. Comput. Vis., 2010, pp. 776-789.
30. M.-F. Weng and Y.-Y. Chuang, "Cross-domain multicue fusion for concept-based video indexing," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 10, pp. 1927-1941, Oct. 2012.
31. C. G. M. Snoek, M. Worring, and A. W. M. Smeulders, "Early versus late fusion in semantic video analysis," in Proc. 13th Annu. ACM Int. Conf. Multimedia, 2005, pp. 399-402.
32. G. R. G. Lanckriet, N. Cristianini, P. Bartlett, L. El Ghaoui, and M. I. Jordan, "Learning the kernel matrix with semidefinite programming," J. Mach. Learn. Res., vol. 5, pp. 27-72, Jan. 2004.
33. A. Rakotomamonjy, F. R. Bach, S. Canu, and Y. Grandvalet, "SimpleMKL," J. Mach. Learn. Res., vol. 9, pp. 2491-2521, Nov. 2008.
34. S. Sonnenburg, G. Rätsch, C. Schäfer, and B. Schölkopf, "Large scale multiple kernel learning," J. Mach. Learn. Res., vol. 7, pp. 1531-1565, Jul. 2006.
35. P. Gehler and S. Nowozin, "On feature combination for multiclass object classification," in Proc. IEEE 12th Int. Conf. Comput. Vis., Sep./Oct. 2009, pp. 221-228.
36. Y.-Y. Lin, T.-L. Liu, and C.-S. Fuh, "Multiple kernel learning for dimensionality reduction," IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 6, pp. 1147-1160, Jun. 2011.
37. Y. Boykov, O. Veksler, and R. Zabih, "Fast approximate energy minimization via graph cuts," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 11, pp. 1222-1239, Nov. 2001.
38. B. Schölkopf, A. Smola, and K.-R. Müller, "Nonlinear component analysis as a kernel eigenvalue problem," Neural Comput., vol. 10, no. 5, pp. 1299-1319, 1998.
39. B. Schölkopf and A. J. Smola, Learning With Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. Cambridge, MA, USA: MIT Press, 2002.
40. S. T. Roweis and L. K. Saul, "Nonlinear dimensionality reduction by locally linear embedding," Science, vol. 290, no. 5500, pp. 2323-2326, Dec. 2000.
41. N. C. Tang, C.-T. Hsu, C.-W. Su, T. K. Shih, and H.-Y. M. Liao, "Video inpainting on digitized vintage films via maintaining spatiotemporal continuity," IEEE Trans. Multimedia, vol. 13, no. 4, pp. 602-614, Aug. 2011.
42. O. Barnich and M. Van Droogenbroeck, "ViBe: A universal background subtraction algorithm for video sequences," IEEE Trans. Image Process., vol. 20, no. 6, pp. 1709-1724, Jun. 2011.
43. D. Weinland, M. Özuysal, and P. Fua, "Making action recognition robust to occlusions and viewpoint changes," in Proc. 11th Eur. Conf. Comput. Vis., 2010, pp. 635-648.
44. H. Wang, A. Kläser, C. Schmid, and C.-L. Liu, "Dense trajectories and motion boundary descriptors for action recognition," Int. J. Comput. Vis., vol. 103, no. 1, pp. 60-79, 2013.
45. G. Zhao and M. Pietikainen, "Dynamic texture recognition using local binary patterns with an application to facial expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 6, pp. 915-928, Jun. 2007.
46. D. Weinland, E. Boyer, and R. Ronfard, "Action recognition from arbitrary views using 3D exemplars," in Proc. IEEE 11th Int. Conf. Comput. Vis., Oct. 2007, pp. 1-7.
47. N. Gkalelis, H. Kim, A. Hilton, N. Nikolaidis, and I. Pitas, "The i3DPost multi-view and 3D human action/interaction database," in Proc. Conf. Vis. Media Prod., 2009, pp. 159-168.
48. D. Tran and A. Sorokin, "Human activity recognition with metric learning," in Proc. 10th Eur. Conf. Comput. Vis., 2008, pp. 548-561.
49. M. D. Rodriguez, J. Ahmed, and M. Shah, "Action MACH a spatiotemporal maximum average correlation height filter for action recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2008, pp. 1-8.
50. Y. Shen and H. Foroosh, "View-invariant recognition of body pose from space-time templates," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2008, pp. 1-6.
51. G. J. Burghouts and K. Schutte, "Spatio-temporal layout of human actions for improved bag-of-words action detection," Pattern Recognit. Lett., vol. 34, no. 15, pp. 1861-1869, 2013.
52. X. Wu, D. Xu, L. Duan, and J. Luo, "Action recognition using context and appearance distribution features," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 489-496.
53. M. Varma and B. R. Babu, "More generality in efficient multiple kernel learning," in Proc. 26th Int. Conf. Mach. Learn., 2009, pp. 1065-1072.
54. A. Iosifidis, A. Tefas, and I. Pitas, "View-invariant action recognition based on artificial neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 3, pp. 412-424, Mar. 2012.
55. A. Iosifidis, A. Tefas, and I. Pitas, "Multi-view action recognition based on action volumes, fuzzy distances and cluster discriminant analysis," Signal Process., vol. 93, no. 6, pp. 1445-1457, 2013.
56. J. Hernández, R. Cabido, A. S. Montemayor, and J. J. Pantrigo, "Human activity recognition based on kinematic features," Expert Syst., vol. 31, no. 4, pp. 345-353, Sep. 2014.
57. D. Cai, X. He, and J. Han, "Semi-supervised discriminant analysis," in Proc. IEEE 11th Int. Conf. Comput. Vis., Oct. 2007, pp. 1-7.
58. Y. Shen and H. Foroosh, "View-invariant action recognition using fundamental ratios," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2008, pp. 1-6.
59. Y. Shen and H. Foroosh, "View-invariant action recognition from point triplets," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 10, pp. 1898-1905, Oct. 2009.