Enhanced computer vision with Microsoft Kinect sensor: A review

IEEE Transactions on Cybernetics

Volumn 43, Issue 5, 2013, Pages 1318-1334

  Han, Jungong ^a   Shao, Ling ^b,c   Xu, Dong ^d   Shotton, Jamie ^e  

^a Civolution   (Netherlands)

^b NANJING UNIVERSITY OF INFORMATION SCIENCE AND TECHNOLOGY   (China)

^c UNIVERSITY OF SHEFFIELD   (United Kingdom)

^d NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^e MICROSOFT RESEARCH   (United Kingdom)

Author keywords

Computer vision; Depth image; Information fusion; Kinect sensor

Indexed keywords

COMPUTER VISION ALGORITHMS; DEPTH IMAGE; HAND-GESTURE ANALYSIS; HIGH-RESOLUTION DEPTH; HUMAN ACTIVITY ANALYSIS; KINECT SENSORS; MICROSOFT KINECT SENSORS; VISUAL INFORMATION;

ALGORITHMS; COMPUTER VISION; INFORMATION FUSION; TRACKING (POSITION);

SENSORS;

ACTIMETRY; ALGORITHM; ARTIFICIAL INTELLIGENCE; AUTOMATED PATTERN RECOGNITION; COMPUTER; COMPUTER SIMULATION; DEVICES; IMAGE ENHANCEMENT; PROCEDURES; RECREATION; THREE DIMENSIONAL IMAGING; TRANSDUCER; WHOLE BODY IMAGING; EQUIPMENT; METHODOLOGY; REVIEW;

ACTIGRAPHY; ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER PERIPHERALS; COMPUTER SIMULATION; IMAGE ENHANCEMENT; IMAGING, THREE-DIMENSIONAL; PATTERN RECOGNITION, AUTOMATED; TRANSDUCERS; VIDEO GAMES; WHOLE BODY IMAGING;

EID: 84890399408     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2013.2265378     Document Type: Article

References (109)

1. Kinect camera [Online]. Available: http://www.xbox.com/en-US/kinect/ default.htm
2. Stereo camera [Online]. Available: http://en.wikipedia.org/wiki/Stereo camera
3. S. Gokturk, H. Yalcin, and C. Bamji, "A time-of-flight depth sensor system description, issues and solutions," in Proc. IEEE Conf. Comput. Vision Pattern Recognit. Workshops, 2004, pp. 35-45.
4. Z. Zhang, "Microsoft Kinect sensor and its effect," IEEE Multimedia Mag., vol. 19, no. 2, pp. 4-10, Feb. 2012.
5. I. Tashev, "Recent advances in human-machine interfaces for gaming and entertainment," Int. J. Inform. Technol. Security, vol. 3, no. 3, pp. 69-76, 2011.
6. K. Kumatani, T. Arakawa, K. Yamamoto, J. McDonough, B. Raj, R. Singh, and I. Tashev, "Microphone array processing for distant speech recognition: Towards real-world deployment," in Proc. APSIPA Annu. Summit Conf., 2012, pp. 1-10.
7. J. Geng, "Structured-light 3-D surface imaging: A tutorial," Adv. Optics Photonics, vol. 3, no. 2, pp. 128-160, 2011.
8. OpenNI [Online]. Available: http://www.openni.org/
9. Microsoft Kinect SDK [Online]. Available: http://www.microsoft.com/en-us/ kinectforwindows/.
10. OpenKinect [Online]. Available: https://github.com/OpenKinect/ libfreenect/.
11. J. Smisek, M. Jancosek, and T. Pajdla, "3-D with Kinect," in Proc. IEEE ICCV Workshops, 2011, pp. 1154-1160.
12. T. Stoyanov, A. Louloudi, H. Andreasson, and A. Lilienthal, "Comparative evaluation of range sensor accuracy in indoor environments," in Proc. Eur. Conf. Mobile Robots, 2011, pp. 19-24.
13. K. Khoshelham and S. Elberink, "Accuracy and resolution of kinect depth data for indoor mapping applications," Sensors, vol. 12, no. 2, pp. 1437-1454, 2012.
14. T. Dutta, "Evaluation of the Kinect sensor for 3-D kinematic measurement in the workplace," Appl. Ergonom., vol. 43, no. 4, pp. 645-649, Jul. 2012.
15. S. Obdrzalek, G. Kurillo, F. Ofli, R. Bajcsy, E. Seto, H. Jimison, and M. Pavel, "Accuracy and robustness of Kinect pose estimation in the context of coaching of elderly population," in Proc. IEEE EMBC, 2012, pp. 1188-1193.
16. C. Zhang and Z. Zhang, "Calibration between depth and color sensors for commodity depth cameras," in Proc. IEEE ICME, 2011, pp. 1-6.
17. D. Herrera, J. Kannala, and J. Heikkila, "Accurate and practical calibration of a depth and color camera pair," in Proc. CAIP, 2011, pp. 437-445.
18. Z. Zhang, "A flexible new technique for camera calibration," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 11, pp. 1330-1334, Nov. 2000.
19. D. Herrera, J. Kannala, and J. Heikkila, "Joint depth and color camera calibration with distortion correction," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 10, pp. 2058-2064, Oct. 2012.
20. D. Fiedler and H. Muller, "Impact of thermal and environmental conditions on the Kinect sensor," in Proc. Int. Workshop Depth Image Anal., 2012.
21. Y. Berdnikov and D. Vatolin, "Eal-time depth map occlusion filling and scene background restoration for projected-pattern-based depth camera," in Proc. Int. Conf. Comput. Graph. Vision, 2011, pp. 1-4.
22. S. Milani and G. Calvagno, "Joint denoising and interpolation of depth maps for MS Kinect sensors," in Proc. ICASSP, 2012, pp. 797-800.
23. M. Schmeing and X. Jiang, "Color Segmentation Based Depth Image Filtering," in Proc. Int. Workshop Depth Image Anal., 2012.
24. M. Camplani and L. Salgado, "Efficient spatio-temporal hole filling strategy for Kinect depth maps," in Proc. SPIE Int. Conf. 3-D Image Process. Appl., vol. 8290, 2012, pp. 1-10.
25. N. Silberman and R. Fergus, "Indoor scene segmentation using a structured light sensor," in Proc. Int. Conf. Comput. Vision Workshop 3-D Representation Recognit., 2011, pp. 601-608.
26. F. Qi, J. Han, P. Wang, G. Shi, and F. Li, "Structure guided fusion for depth map inpainting," Pattern Recognition Letters, vol. 34, no. 1, pp. 70-76, 2013.
27. L. Xia, C. Chen, and J. Aggarwal, "Human detection using depth information by Kinect," in Proc. Int. Workshop HAU3D, Jun. 2011, pp. 15-22.
28. C. Rougier, E. Auvinet, J. Rousseau, M. Mignotte, and J. Meunier, "Fall detection from depth map video sequences," in Proc. ICOST, 2011, pp. 121-128.
29. Z. Zhang, W. Liu, V. Metsis, and V. Athitsos, "A viewpoint-independent statistical method for fall detection," in Proc. ICPR, Nov. 2012, pp. 3626-3630.
30. Y. Shen, P. Wang, and S. Ma, "Performance study of feature descriptors for human detection on depthMap," in Proc. Int. Conf. Syst. Simulat. Sci. Comput., 2012.
31. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2005, pp. 886-893.
32. A. Bosch, A. Zisserman, and X. Munoz, "Representing shape with a spatial pyramid kernel," in Proc. Int. Conf. Image Video Retrieval, 2007, pp. 401-408.
33. T. Ojala, M. Pietikainen, and T. Maenpaa, "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 971-987, Jul. 2002.
34. X. Tan and B. Triggs, "Enhanced local texture feature sets for face recognition under difficult lighting conditions," IEEE Trans. Image Process., vol. 19, no. 6, pp. 1635-1650, Jun. 2010.
35. J. Wu and J. Rehg, "CENTRIST: A visual descriptor for scene categorization," IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 8, pp. 1489-1501, Aug. 2011.
36. J. Han, E. Pauwels, P. de Zeeuw, and P. de With, "Employing an RGBD sensor for real-time tracking of humans across multiple re-entries in a smart environment," IEEE Trans. Consumer Electron., vol. 58, no. 2, pp. 255-263, May 2012.
37. L. Spinello and K. Arras, "People detection in RGB-D data," in Proc. Int. Conf. IROS, 2011, pp. 3838-3843.
38. L. Spinello, C. Stachniss, and W. Burgard, "Scene in the loop: Towards adaptation-by-tracking in RGB-D data," in Proc. Workshop RGB-D, Adv. Reason. Depth Cameras, 2012.
39. M. Luber, L. Spinello, and K. Arras, "People tracking in RGB-D data with On-line boosted target models," in Proc. Int. Conf. IROS, 2011.
40. L. Spinello and K. Arras, "Leveraging RGB-D Data: Adaptive fusion and domain adaptation for object detection," in Proc. IEEE ICRA, 2012, pp. 4469-4474.
41. W. Liu, T. Xia, J. Wan, Y. Zhang, and J. Li, "RGB-D based multiattribute people search in intelligent visual surveillance," in Proc. Int. Conf. Adv. Multimedia Modeling, 2012, pp. 750-760.
42. W. Choi, C. Pantofaru, and S. Savarese, "Detecting and tracking people using an RGB-D camera via multiple detector fusion," in Proc. Workshop Challenges Opportunities Robot Perception, 2011, pp. 1076-1083.
43. K. Lai, L. Bo, X. Ren, and D. Fox, "A large-scale hierarchical multiview RGB-D object dataset," in Proc. IEEE Int. Conf. Robot. Autom., 2011, pp. 1817-1824.
44. N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, "Indoor segmentation and support inference from RGBD images," in Proc. Eur. Conf. Comput. Vision, 2012, pp. 746-760.
45. K. Lai, L. Bo, X. Ren, and D. Fox, "Sparse distance learning for object recognition combining RGB and depth information," in Proc. IEEE Int. Conf. Robot. Autom., 2011, pp. 4007-4013.
46. L. Bo, X. Ren, and D. Fox, "Unsupervised feature learning for RGBD based object recognition," in Proc. Int. Symp. Experimental Robot., 2012.
47. L. Bo, K. Lai, X. Ren, and D. Fox, "Object recognition with hierarchical kernel descriptors," in Proc. CVPR, 2011, pp. 1729-1736.
48. X. Ren, L. Bo, and D. Fox, "RGB-(D) scene labeling: Features and algorithms," in Proc. CVPR, 2012, pp. 2759-2766.
49. S. Tang, X. Wang, X. Lv, T. Han, J. Keller, Z. He, M. Skubic, and S. Lao, "Histogram of oriented normal vectors for object recognition with a depth sensor," in Proc. Asian Conf. Comput. Vision, 2012.
50. M. Blum, J. Springenberg, J. Wulfing, and M. Riedmiller, "A learned feature descriptor for object recognition in RGB-D data," in Proc. IEEE Int. Conf. Robot. Autom., 2012, pp. 1298-1303.
51. L. Bo, X. Ren, and D. Fox, "Depth kernel descriptors for object recognition," in Proc. Int. Conf. IROS, 2011, pp. 821-826.
52. J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, "Real-time human pose recognition in parts from a single depth image," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2011, pp. 1297-1304.
53. M. Budiu, J. Shotton, D. Murray, and M. Finocchio, "Parallelizing the training of the Kinect body parts labeling algorithm," in Proc. Workshop Big Learn., Algorithms, Syst. Tools Learn. Scale, 2011, pp. 6-12.
54. R. Girshick, J. Shotton, P. Kohli, A. Criminisi, and A. Fitzgibbon, "Efficient regression of general-activity human poses from depth images," in Proc. ICCV, 2011, pp. 415-422.
55. J. Taylor, J. Shotton, T. Sharp, and A. Fitzgibbon, "The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation," in Proc. CVPR, 2012, pp. 103-110.
56. M. Ye, X. Wang, R. Yang, L. Ren, and M. Pollefeys, "Accurate 3-D pose estimation from A single depth image," in Proc. ICCV, 2011, pp. 731-738.
57. W. Shen, K. Deng, X. Bai, T. Leyvand, B. Guo, and Z. Tu, "Exemplar-based human action pose correction and tagging," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2012, pp. 1784-1791.
58. X. Yang and Y. Tian, "EigenJoints-based action recognition using naive-bayes-nearest-neighbor," in Proc. IEEE Workshop CVPR Human Activity Understand. 3-D Data, 2012, pp. 14-19.
59. X. Yang and Y. Tian, "Effective 3-D action recognition using eigenJoints," J. Visual Commun. Image Represent., doi: 10.1016/j.jvcir. 2013.03.001.
60. L. Xia, C. Chen, and J. Aggarwal, "View invariant human action recognition using histograms of 3-D joints," in Proc. IEEE Workshop CVPR Human Activity Understand. 3-D Data, 2012, pp. 20-27.
61. A. Jalal, S. Lee, J. Kim, and T. Kim, "Human activity recognition via the features of labeled depth body parts," in Proc. Int. Conf. Smart Homes Health Telematics, 2012, pp. 246-249.
62. J. Sung, C. Ponce, B. Selman, and A. Saxena, "Human activity detection from RGBD images," in Proc. AAAI Workshop Plan, Activity, Intent Recognit., 2011, pp. 47-55.
63. J. Sung, C. Ponce, B. Selman, and A. Saxena, "Unstructured human activity detection from RGBD images," in Proc. IEEE ICRA, 2012, pp. 842-849.
64. M. Reyes, G. Dominguez, and S. Escalera, "Featureweighting in dynamic timewarping for gesture recognition in depth data," in Proc. IEEE ICCV Workshop, 2011, pp. 1182-1188.
65. J. Wang, Z. Liu, Y. Wu, and J. Yuan, "Mining actionlet ensemble for action recognition with depth cameras," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2012, pp. 1290-1297.
66. S. Nowozin and J. Shotton, "Action points: A representation for low-latency online human action recognition," Microsoft Research Cambridge, Tech. Rep. MSR-TR-2012-68, 2012.
67. B. Ni, G. Wang, and P. Moulin, "RGBD-HuDaAct: A color-depth video database for huamn daily activity recognition," in Proc. IEEE Workshop Consum. Depth Cameras Comput. Vision, 2011, pp. 1147-1153.
68. S. Fothergill, H. Mentis, S. Nowozin, and P. Kohli, "Instructing people for training gestural interactive systems," in Proc. ACM Conf. Human Factors Comput. Syst., 2012, pp. 1737-1746.
69. Z. Cheng, L. Qin, Y. Ye, Q. Huang, and Q. Tian, "Human daily action analysis with multiview and color-depth data," in Proc. ECCV Workshop Consum. Depth Cameras Comput. Vision, 2012, pp. 52-61.
70. T. Berg, D. Chattopadhyay, M. Schedel, and T. Vallier, "Interactive music: Human motion initiated music generation using skeletal tracking by Kinect," in Proc. Conf. Soc. Electro-Acoustic Music United States, 2012.
71. M. Raptis, D. Kirovski, and H. Hoppe, "Real-time classification of dance gestures from skeleton animation," in Proc. ACM SIGGRAPH/Eurographics Symp. Comput. Animation, 2011, pp. 147-156.
72. V. Escorcia, M. Golparvar-Fard, and J. Niebles, "Automated vision-based recognition of construction worker actions for building interior construction operations using RGBD cameras," in Proc. Construction Res. Congr., 2012, pp. 879-888.
73. R. Tara, P. Santosa, and T. Adji, "Hand segmentation from depth image using anthropometric approach in natural interface development," Int. J. Sci. Eng. Res., vol. 3, no. 5, pp. 1-4, May 2012.
74. U. Lee and J. Tanaka, "Hand controller: Image manipulation interface using fingertips and palm tracking with Kinect depth data," in Proc. Asia Pacific Conf. Comput. Human Interact., 2012, pp. 705-706.
75. H. Liang, J. Yuan, and D. Thalmann, "3-D fingertip and palm tracking in depth image sequences," in Proc. ACM Int. Conf. Multimedia, 2012, pp. 785-788.
76. G. Hackenberg, R. McCall, and W. Broll, "Lightweight palm and finger tracking for real-time 3-D gesture control," in Proc. IEEE Conf. Virtual Reality, 2011, pp. 19-26.
77. M. Caputo, K. Denker, B. Dums, and G. Umlauf, "3-D hand gesture recognition based on sensor fusion of commodity hardware," in Proc. Conf. Mensch Comput., 2012, pp. 293-302.
78. I. Oikonomidis, N. Kyriazis, and A. Argyros, "Efficient model-based 3-D tracking of hand articulations using Kinect," in Proc. Brit. Mach. Vision Conf., 2011, pp. 101.1-101.11.
79. I. Oikonomidis, N. Kyriazis, and A. Argyros, "Tracking the articulated motion of two strongly interacting hands," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2012, pp. 1862-1869.
80. P. Doliotis, V. Athitsos, D. Kosmopoulos, and S. Perantoni, "Hand shape and 3-D pose estimation using depth data from a single cluttered frame," in Proc. ISVC, 2012, pp. 148-158.
81. Z. Ren, J. Meng, J. Yuan, and Z. Zhang, "Robust hand gesture recognition with Kinect sensor," in Proc. ACM Int. Conf. Multimedia, 2011, pp. 759-760.
82. Y. Yao and Y. Fu, "Real-time hand pose estimation from RGB-D sensor," in Proc. IEEE Conf. Multimedia Expo, 2012, pp. 705-710.
83. C. Keskin, F. Kira, Y. Kara, and L. Akarun, "Real time hand pose estimation using depth sensors," in Proc. ICCV Workshop, 2011, pp. 1228-1234.
84. C. Keskin, F. Kira, Y. Kara, and L. Akarun, "Hand pose estimation and hand shape classification using multilayered randomized decision forests," in Proc. Eur. Conf. Comput. Vision, 2012, pp. 852-863.
85. M. Tang, "Recognizing hand gestures with microsoft's Kinect," Dept. Electrical Eng., Stanford Univ., CA, USA, Tech. Rep., 2011.
86. Y. Li, "Hand gesture recognition using Kinect," in Proc. Int. Conf. Software Eng. Service Sci., 2012, pp. 196-199.
87. P. Doliotis, A. Stefan, C. Mcmurrough, D. Eckhard, and V. Athitsos, "Comparing gesture recognition accuracy using color and depth information," in Proc. Conf. Pervasive Technol. Related Assistive Environments, 2011, article 20.
88. R. Mihail, N. Jacobs, and J. Goldsmith, "Static hand gesture recognition with 2 Kinect sensors," in Proc. Int. Conf. Image Process., Comput. Vision Pattern Recognit., 2012.
89. S. Thrun, W. Burgard, and D. Fox, "A real-time algorithm for mobile robot mapping with applications to multirobot and 3-D mapping," in Proc. IEEE Int. Conf. ICRA, 2000, pp. 321-328.
90. J. Biswas and M. Veloso, "Depth camera based indoor mobile robot localization and navigation," in Proc. IEEE Int. Conf. ICRA, 2012, pp. 1697-1702.
91. L. Paz, P. Pinies, J. Tardos, and J. Neira, "Large-scale 6-DOF SLAM with stereo-in-hand," IEEE Trans. Robot., vol. 24, no. 5, pp. 946-957, Oct. 2008.
92. P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, "RGB-D mapping: Using depth cameras for dense 3-D modeling of indoor environments," in Proc. Int. Symp. Experimental Robot., 2010.
93. P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, "RGB-D mapping: Using Kinect-style depth cameras for dense 3-D modeling of indoor environments," Int. J. Robot. Res., vol. 31, no. 5, pp. 647-663, Apr. 2012.
94. N. Engelhard, F. Endres, J. Hess, J. Sturm, and W. Burgard, "Realtime 3-D visual SLAM with A hand-held camera," in Proc. RGB-D Workshop 3-D Perception Robot. Eur. Robot. Forum, 2011.
95. D. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Comput. Vision, vol. 60, no. 2, pp. 91-110, 2004.
96. E. Rosten, R. Porter, and T. Drummond, "Faster and Better: A machine learning approach to corner detection," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 1, pp. 105-119, Jan. 2010.
97. H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, "SURF: Speeded up robust features," Comput. Vision Image Understanding, vol. 110, no. 3, pp. 346-359, 2008.
98. J. Stuckler and S. Behnke, "Integrating depth and color cues for dense multiresolution scene mapping using RGB-D cameras," in Proc. IEEE Int. Conf. Multisensor Fusion Inform. Integr., 2012.
99. Y. Zhang, C. Luo, and J. Liu, "Walk & Sketch: Create floor plans with an RGB-D camera," in Proc. ACM Int. Conf. Ubiquitous Comput., 2012.
100. R. Newcombe, A. Davison, S. Izadi, P. Kohli, O. Hilliges, J. Shotton, D. Molyneaux, S. Hodges, D. Kim, and A. Fitzgibbon, "KinectFusion: Real-time dense surface mapping and tracking," in Proc. IEEE Int. Symp. Mixed Augmented Reality, 2011, pp. 127-136.
101. S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, "KinectFusion: Real-time 3-D reconstruction and interaction using a moving depth camera," in Proc. ACM Symp. User Interface Software Technol., 2011, pp. 559-568.
102. S. Meister, S. Izadi, P. Kohli, M. Haemmerle, C. Rother, and D. Kondermann, "When can we use KinectFusion for ground truth acquisition?" in Proc. Workshop Color-Depth Camera Fusion Robot., 2012.
103. M. Meilland, A. Comport, and P. Rives, "Real-time dense visual tracking under large lighting variations," in Proc. Conf. Brit. Mach. Vision, pp. 1-11, 2011.
104. M. Meilland, A. Comport, and P. Rives, "Dense RGB-D mapping for real-time localisation and navigation," in Proc. Workshop Navigation Positiong Mapping, 2012.
105. H. Roth and M. Vona, "Moving volume KinectFusion," in Proc. Brit. Mach. Vision Conf., 2012, pp. 1-11.
106. T. Whelan, M. Kaess, M. Fallon, H. Johannsson, J. Leonard, and J. McDonald, "Kintinuous: Spatially extended KinectFusion," in Proc. Workshop RGB-D, Adv. Reason. Depth Cameras, 2012, article 4.
107. J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, "A benchmark for the evaluation of RGB-D SLAM systems," in Proc. Int. Conf. Intell. Robot Syst., 2012, pp. 573-580.
108. J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi, and A. Fitzgibbon, "Scene coordinate regression forests for camera relocalization in RGB-D images," in Proc. CVPR, accepted by CVPR.
109. L. Liu and L. Shao, "Learning discriminative representations from RGB-D video data," in Proc. IJCAI, accepted by IJCAI.