High-quality depth map upsampling and completion for RGB-D Cameras

IEEE Transactions on Image Processing

Volumn 23, Issue 12, 2014, Pages 5559-5572

  Park, Jaesik ^a   Kim, Hyeongwoo ^a   Tai, Yu Wing ^a   Brown, Michael S ^b   Kweon, In So ^a  

^a Korea Advanced Institute of Science and Technology (KAIST)   (South Korea)

^b NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

Author keywords

Depth Map Completion; Depth Map Upsampling; RGB D cameras

Indexed keywords

CAMERAS; REPAIR;

APPLICATION FRAMEWORKS; COMPLEMENTARY SYSTEMS; DEPTH MAP; DEPTH MAP UP SAMPLINGS; DEVICE CALIBRATION; EDGE INFORMATION; RGB-D CAMERAS; TEMPORAL COHERENCE;

SIGNAL SAMPLING;

EID: 84913588498     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2014.2361034     Document Type: Article

References (31)

1. SwissRanger SR4000 Data Sheet. [Online]. Available: http://www.mesa-imaging.ch/prodview4k.php, accessed 2011.
2. P. Bhat, C. L. Zitnick, M. Cohen, and B. Curless, "GradientShop: A gradient-Domain optimization framework for image and video filtering," ACM Trans. Graph., vol. 29, no. 2, p. 1-10, Mar. 2010.
3. D. Chan, H. Buisman, C. Theobalt, and S. Thrun, "A noise-aware filter for real-time depth upsampling," in Proc. ECCV Workshop Multi-Camera Multi-Modal Sensor Fus. Algorithms Appl., 2008.
4. J. Chen, C.-K. Tang, and J. Wang, "Noise brush: Interactive high quality image-noise separation," ACM Trans. Graph., vol. 28, no. 5, p. 146, Dec. 2009.
5. J. Diebel and S. Thrun, "An application of Markov random fields to range sensing," in Neural Information Processing System (NIPS). Cambridge, MA, USA: MIT Press, 2005.
6. J. Dolson, J. Baek, C. Plagemann, and S. Thrun, "Upsampling range data in dynamic environments," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2010, pp. 1141-1148.
7. E. Eisemann and F. Durand, "Flash photography enhancement via intrinsic relighting," ACM Trans. Graph., vol. 23, no. 3, pp. 673-678, 2004.
8. P. Favaro, "Recovering thin structures via nonlocal-means regularization with application to depth from defocus," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2010, pp. 1133-1140.
9. K. He, J. Sun, and X. Tang, "Guided image filtering," in Proc. Eur. Conf. Comput. Vis. (ECCV), 2010, pp. 1-14.
10. D. Herrera, J. Kannala, and J. Heikkilä, "Joint depth and color camera calibration with distortion correction," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 10, pp. 2058-2064, Oct. 2012.
11. B. Huhle, T. Schairer, P. Jenke, and W. Straer, "Fusion of range and color images for denoising and resolution enhancement with a non-local filter," Comput. Vis. Image Understand., vol. 114, no. 12, pp. 1336-1345, 2010.
12. S. Izadi et al., "KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera," in Proc. 24th Annu. ACM Symp. User Inter. Softw. Technol. (UIST), 2011, pp. 559-568.
13. A. Janoch et al., "A category-level 3D object dataset: Putting the Kinect to work," in Proc. Int. Conf. Comput. Vis. (ICCV), 2011, pp. 1168-1174.
14. J. Jung, Y. Jeong, J. Park, H. Ha, J. D. Kim, and I.-S. Kweon, "A novel 2.5D pattern for extrinsic calibration of ToF and camera fusion system," in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), Sep. 2011, pp. 3290-3296.
15. J. Kennedy and R. Eberhart, "Particle swarm optimization," in Proc. Int. Conf. Neural Netw., Nov./Dec. 1995, pp. 1942-1948.
16. K. Khoshelham and S. O. Elberink, "Accuracy and resolution of Kinect depth data for indoor mapping applications," Sensors, vol. 12, no. 2, pp. 1437-1454, 2012.
17. J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, "Joint bilateral upsampling," ACM Trans. Graph., vol. 26, no. 3, p. 96, 2007.
18. A. Levin, D. Lischinski, and Y.Weiss, "Colorization using optimization," ACM Trans. Graph., vol. 23, no. 3, pp. 689-694, Aug. 2004.
19. C. Liu, "Beyond pixels: Exploring new representations and applications for motion analysis," Ph.D. Dissertation, Dept. Elect. Eng. Comput. Sci., Massachusetts Inst. Technol., Cambridge, MA, USA, May 2009.
20. S. Meister, S. Izadi, P. Kohli, M. Hämmerle, C. Rother, and D. Kondermann, "When can we use kinectfusion for ground truth acquisition?" in Proc. Workshop Color-Depth Camera Fusion Robot. IROS, 2012.
21. G. Pandey, J. R. McBride, S. Savarese, and R. M. Eustice, "Automatic extrinsic calibration of vision and lidar by maximizing mutual information," J. Field Robot., Special Issue Calibration Field Robot., Sep. 2014. [Online]. Available: http://onlinelibrary.wiley.com/doi/10.1002/rob.21542/pdf
22. J. Park, H. Kim, Y.-W. Tai, M. S. Brown, and I. Kweon, "High quality depth map upsampling for 3D-ToF cameras," in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Nov. 2011, pp. 1623-1630.
23. D. Scharstein and R. Szeliski, "A taxonomy and evaluation of dense twoframe stereo correspondence algorithms," Int. J. Comput. Vis., vol. 47, no. 1, pp. 7-42, 2002.
24. N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, "Indoor segmentation and support inference from RGBD images," in Proc. Eur. Conf. Comput. Vis. (ECCV), 2012, pp. 746-760.
25. Z. Taylor and J. Nieto, "A mutual information approach to automatic calibration of camera and lidar in natural environments," in Proc. Austral. Conf. Robot. Autom., 2012.
26. A. Torralba and W. T. Freeman, "Properties and applications of shape recipes," in Proc. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2003, pp. 383-390.
27. A. Vedaldi and B. Fulkerson. (2008). VLFeat: An Open and Portable Library of Computer Vision Algorithms. [Online]. Available: http://www.vlfeat.org/
28. J. Wang, M. Agrawala, and M. F. Cohen, "Soft scissors: An interactive tool for realtime high quality matting," ACM Trans. Graph., vol. 26, no. 3, pp. 1-9, 2007.
29. Q. Yang, R. Yang, J. Davis, and D. Nistér, "Spatial-Depth super resolution for range images," in Proc. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2007, pp. 1-8.
30. J. Zhu, L. Wang, R. Yang, and J. Davis, "Fusion of time-of-flight depth and stereo for high accuracy depth maps," in Proc. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2008, pp. 1-8.
31. A. Zomet and S. Peleg, "Multi-sensor super-resolution," in Proc. 6th IEEE Workshop Appl. Comput. Vis. (WACV), Dec. 2002, pp. 27-31.