Structure-preserving binary representations for RGB-D action recognition

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 38, Issue 8, 2016, Pages 1651-1664

  Yu, Mengyang ^a   Liu, Li ^a   Shao, Ling ^a  

^a NORTHUMBRIA UNIVERSITY   (United Kingdom)

Author keywords

binary; dimensionality reduction; flux; local feature; RGB D fusion; structure preserving

Indexed keywords

BINS; DATA FUSION; DATA HANDLING; FLUXES; VIDEO RECORDING;

ACTION RECOGNITION; BINARY; BINARY REPRESENTATIONS; DIMENSIONALITY REDUCTION; LOCAL DESCRIPTORS; LOCAL FEATURE; ORTHOGONAL PROJECTION MATRICES; STRUCTURE-PRESERVING;

MOTION ESTIMATION;

ALGORITHM; AUTOMATED PATTERN RECOGNITION; ORIENTATION; VIDEORECORDING;

ALGORITHMS; ORIENTATION; PATTERN RECOGNITION, AUTOMATED; VIDEO RECORDING;

EID: 84978764118     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2015.2491925     Document Type: Article

References (74)

1. J. Han, L. Shao, D. Xu, and J. Shotton, "Enhanced computer vision with microsoft kinect sensor: A review," IEEE Trans. Cybernetics, vol. 43, no. 5, pp. 1318-1334, 2013.
2. J. Wang, Z. Liu, Y. Wu, and J. Yuan, "Mining actionlet ensemble for action recognition with depth cameras," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 1290-1297.
3. M. Paton and J. Kosecka, "Adaptive rgb-d localization," in Proc. Conf. Comput. and Robot Vis., 2012, pp. 24-31.
4. L. Spinello and K. O. Arras, "People detection in rgb-d data," in Proc. IEEE/RSJ Int. Conf. Intell. Robots and Syst., 2011, pp. 3838-3843.
5. X. Ren, L. Bo, and D. Fox, "rgb-(d) scene labeling: Features and algorithms," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2759-2766.
6. L. Cruz, D. Lucio, and L. Velho, "Kinect and rgbd images: Challenges and applications," in Proc. SIBGRAPI Conf. Graphics, Patterns and Images - Tutorials, 2012, pp. 36-49.
7. P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, "rgb-d mapping: Using Kinect-style depth cameras for dense 3d modeling of indoor environments," The Int. J. Robotics Res., vol. 31, no. 5, pp. 647-663, 2012.
8. D. G. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Comput. Vis., vol. 60, no. 2, pp. 91-110, 2004.
9. A. Klaser, M. Marszalek, and C. Schmid, "A spatio-temporal descriptor based on 3d-gradients," in Proc. Brit. Mach. Vis. Conf., 2008, pp. 1-10.
10. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2005, pp. 886-893.
11. I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, "Learning realistic human actions from movies," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2008, pp. 1-8.
12. N. Dalal, B. Triggs, and C. Schmid, "Human detection using oriented histograms of flow and appearance," in Proc. Eur. Conf. Comput. Vis., 2006, pp. 428-441.
13. J. Sivic and A. Zisserman, "Video google: A text retrieval approach to object matching in videos," in Proc. IEEE Int. Conf. Comput. Vis., 2003, pp. 1470-1477.
14. H. Lee, A. Battle, R. Raina, and A. Y. Ng, "Efficient sparse coding algorithms," in Proc. Neural Inf. Process. Syst., 2006, pp. 801-808.
15. G. Arfken and H. Weber, Mathematical Methods for Physicists. San Diego, CA: Elsevier, 2005.
16. O. Boiman, E. Shechtman, and M. Irani, "In defense of nearestneighbor based image classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2008, pp. 1-8.
17. L. Liu, L. Shao, X. Li, and K. Lu, "Learning spatio-temporal representations for action recognition: A genetic programming approach," IEEE Trans. Cybernetics, 2015, Doi: 10.1109/TCYB. 2015.2399172.
18. L. Shao, X. Zhen, D. Tao, and X. Li, "Spatio-temporal Laplacian pyramid coding for action recognition," IEEE Trans. Cybernetics, vol. 44, no. 6, pp. 817-827, Jun. 2014.
19. F. Zhu and L. Shao, "Weakly-supervised cross-domain dictionary learning for visual recognition," Int. J. Comput. Vis., vol. 109, no. 1-2, pp. 42-59, 2014.
20. L. Liu, L. Shao, and P. Rockett, "Boosted key-frame selection and correlated pyramidal motion-feature representation for human action recognition," Pattern Recog., vol. 46, no. 7, pp. 1810-1818, 2013.
21. I. Laptev, "On space-time interest points," Int. J. Comput. Vis., vol. 64, no. 2-3, pp. 107-123, 2005.
22. P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie, "Behavior recognition via sparse spatio-temporal features," in Proc. Joint IEEE Int. Workshop Vis. Surveillance and Performance Evaluation of Tracking and Surveillance, 2005, pp. 65-72.
23. W. T. Freeman and M. Roth, "Orientation histograms for hand gesture recognition," in Proc. Int. Workshop Automat. Face and Gesture Recog., vol. 12, 1995, pp. 296-301.
24. K. Mikolajczyk and C. Schmid, "A performance evaluation of local descriptors," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 10, pp. 1615-1630, Oct. 2005.
25. H. Bay, T. Tuytelaars, and L. J. V. Gool, "Surf: Speeded up robust features," in Proc. Eur. Conf. Comput. Vis., 2006, pp. 404-417.
26. H. Bay, A. Ess, T. Tuytelaars, and L. J. V. Gool, "Speeded-up robust features (surf)," Comput. Vis. Image Understanding, vol. 110, no. 3, pp. 346-359, 2008.
27. P. Scovanner, S. Ali, and M. Shah, "A 3-dimensional sift descriptor and its application to action recognition," in Proc. ACM Int. Conf. Multimedia, 2007, pp. 357-360.
28. S. Hadfield, K. Lebeda, and R. Bowden, "Natural action recognition using invariant 3d motion encoding," in Proc. Eur. Conf. Comput. Vis., 2014, pp. 758-771.
29. M. B. Holte, T. B. Moeslund, N. Nikolaidis, and I. Pitas, "3d human action recognition for multi-view camera systems," in Proc. Int. Conf. 3D Imag., Model., Process., Vis. Transmiss., 2011, pp. 342-349.
30. A. Fathi and G. Mori, "Action recognition by learning mid-level motion features," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2008, pp. 1-8.
31. H. Wang, A. Klaser, C. Schmid, and C.-L. Liu, "Dense trajectories and motion boundary descriptors for action recognition," Int. J. Comput. Vis., vol. 103, no. 1, pp. 60-79, 2013.
32. R. Poppe, "A survey on vision-based human action recognition," Image and Vis. Comput., vol. 28, no. 6, pp. 976-990, 2010.
33. A. F. Bobick and J. W. Davis, "The recognition of human movement using temporal templates," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 3, pp. 257-267, Mar. 2001.
34. M. A. R. Ahad, J. K. Tan, H. Kim, and S. Ishikawa, "Motion history image: Its variants and applications," Mach. Vis. Appl., vol. 23, no. 2, pp. 255-281, 2012.
35. J. Shotton, A. W. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, "Real-time human pose recognition in parts from single depth images," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 1297-1304.
36. L. Spinello and K. O. Arras, "Leveraging rgb-d data: Adaptive fusion and domain adaptation for object detection," in Proc. IEEE Int. Conf. Robotics and Autom., 2012, pp. 4469-4474.
37. K. Lai, L. Bo, X. Ren, and D. Fox, "Sparse distance learning for object recognition combining rgb and depth information," in Proc. IEEE Int. Conf. Robotics and Autom., 2011, pp. 4007-4013.
38. B. Ni, G. Wang, and P. Moulin, "rgbd-hudaact: A Color-depth video database for human daily activity recognition," in Proc. IEEE Int. Conf. Comput. Vis. Workshops, 2011, pp. 1147-1153.
39. O. Oreifej and Z. Liu, "Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences," in Proc. IEEE Conf. Comput. Vis. and Pattern Recog., 2013, pp. 716-723.
40. D. J. Berndt and J. Clifford, "Using dynamic time warping to find patterns in time series," in Proc. KDD Workshop, vol. 10, no. 16, 1994, pp. 359-370.
41. S. Sempena, N. U. Maulidevi, and P. R. Aryan, "Human action recognition using dynamic time warping," in Proc. Int. Conf. Elect. Eng. Informat., 2011, pp. 1-5.
42. M. Muller and T. Roder, "Motion templates for automatic classification and retrieval of motion capture data," in Proc. ACM SIGGRAPH/Eurographics Symp. Comput. Animation, 2006, pp. 137-146.
43. C. Xu and J. L. Prince, "Snakes, shapes, and gradient vector flow," IEEE Trans. Image Process., vol. 7, no. 3, pp. 359-369, Mar. 1998.
44. D. Engel and C. Curio, "Scale-invariant medial features based on gradient vector flow fields," in Proc. Int. Conf. Pattern Recog., 2008, pp. 1-4.
45. S. A. A. Shah, M. Bennamoun, F. Boussaid, and A. A. El-Sallam, "A novel local surface description for automatic 3d object recognition in low resolution cluttered scenes," in Proc. IEEE Int. Conf. Comput. Vis. Workshops, 2013, pp. 638-643.
46. J. B. Tenenbaum, V. De Silva, and J. C. Langford, "A global geometric framework for nonlinear dimensionality reduction," Science, vol. 290, no. 5500, pp. 2319-2323, 2000.
47. M. Belkin and P. Niyogi, "Laplacian eigenmaps and spectral techniques for embedding and clustering," in Proc. Neural Inf. Process. Syst., 2001, pp. 585-591.
48. S. T. Roweis and L. K. Saul, "Nonlinear dimensionality reduction by locally linear embedding," Science, vol. 290, no. 5500, pp. 2323-2326, 2000.
49. Y. Bengio, J.-F. Paiement, P. Vincent, O. Delalleau, N. L. Roux, and M. Ouimet, "Out-of-sample extensions for lle, isomap, mds, eigenmaps, and spectral clustering," in Proc. Neural Inf. Process. Syst., 2003, pp. 177-184.
50. X. He and P. Niyogi, "Locality preserving projections," in Proc. Neural Inf. Process. Syst., 2003.
51. X. He, D. Cai, S. Yan, and H. Zhang, "Neighborhood preserving embedding," in Proc. IEEE Int. Conf. Comput. Vis., 2005, pp. 1208-1213.
52. D. Cai, H. Bao, and X. He, "Sparse concept coding for visual analysis," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 2905-2910.
53. G. Shakhnarovich, "Learning task-specific similarity," Ph.D. dissertation, Massachusetts Inst. Technol., US, 2005.
54. R. Caseiro, P. Martins, J. F. Henriques, F. S. Leite, and J. Batista, "Rolling Riemannian manifolds to solve the multi-class classification problem," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 41-48.
55. L. Liu, M. Yu, and L. Shao, "Multiview alignment hashing for efficient image search," IEEE Trans. Image Process., vol. 24, no. 3, pp. 956-966, Mar. 2015.
56. A. Gionis, P. Indyk, and R. Motwani, "Similarity search in high dimensions via hashing," in Proc. Int. Conf. Very Large Data Bases, 1999, pp. 518-529.
57. Y. Weiss, A. Torralba, and R. Fergus, "Spectral hashing," in Proc. Neural Inf. Process. Syst., 2008, pp. 1753-1760.
58. W. Liu, J. Wang, R. Ji, Y.-G. Jiang, and S.-F. Chang, "Supervised hashing with kernels," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2074-2081.
59. C. Strecha, A. M. Bronstein, M. M. Bronstein, and P. Fua, "Ldahash: Improved matching with smaller descriptors," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 1, pp. 66-78, Jan. 2012.
60. B. Kulis and T. Darrell, "Learning to hash with binary reconstructive embeddings," in Proc. Advances Neural Inf. Process. Syst., 2009, pp. 1042-1050.
61. M. Norouzi and D. J. Fleet, "Minimal loss hashing for compact binary codes," in Proc. Int. Conf. Mach. Learning, 2011, pp. 353-360.
62. H. Jegou, M. Douze, and C. Schmid, "Hamming embedding and weak geometric consistency for large scale image search," in Proc. Eur. Conf. Comput. Vis., 2008, pp. 304-317.
63. L. Liu, M. Yu, and L. Shao, "Local feature binary coding for approximate nearest neighbor search," in Proc. Brit. Mach. Vis. Conf., 2015, pp. 173.1-173.12.
64. G. H. Hardy, J. E. Littlewood, and G. Polya, Inequalities. Cambridge Univ. Press, 1952.
65. A. V. Aho, J. E. Hopcroft, and J. D. Ullman, The Design and Analysis of Computer Algorithms. Addison-Wesley, 1974.
66. L. Liu and L. Shao, "Learning discriminative representations from rgb-d video data," in Proc. 23rd Int. Joint Conf. Artif. Intell., 2013, pp. 1493-1500.
67. J. Sung, C. Ponce, B. Selman, and A. Saxena, "Unstructured human activity detection from rgbd images," in Proc. IEEE Int. Conf. Robotics Autom., 2012, pp. 842-849.
68. W. Liu, J. Wang, S. Kumar, and S.-F. Chang, "Hashing with graphs," in Proc. Int. Conf. Mach. Learning, 2011, pp. 1-8.
69. J. Wang, S. Kumar, and S.-F. Chang, "Semi-supervised hashing for large-scale search," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 12, pp. 2393-2406, Dec. 2012.
70. G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504-507, 2006.
71. F. Perronnin, J. Sanchez, and T. Mensink, "Improving the Fisher kernel for large-scale image classification," in Proc. Eur. Conf. Comput. Vis., 2010, pp. 143-156.
72. J. Liu, J. Luo, and M. Shah, "Recognizing realistic actions from videos in the wild," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 1996-2003.
73. H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, "Hmdb: A large video database for human motion recognition," in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 2556-2563.
74. H. Wang, A. Klaser, C. Schmid, and C.-L. Liu, "Action recognition by dense trajectories," in Proc. IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2011, pp. 3169-3176.