Object-based 2D-to-3D video conversion for effective stereoscopic content generation in 3D-TV applications

IEEE Transactions on Broadcasting

Volumn 57, Issue 2 PART 2, 2011, Pages 500-509

  Feng, Yue ^a   Ren, Jinchang ^b   Jiang, Jianmin ^a  

^a UNIVERSITY OF BRADFORD   (United Kingdom)

^b UNIVERSITY OF STRATHCLYDE   (United Kingdom)

Author keywords

2D to 3D conversion; 3D TV; Broadcasting; computer vision; multimedia systems

Indexed keywords

2D-TO-3D CONVERSION; 3D-TV; CONTENT-BASED; CONVERSION METHODS; DEPTH ESTIMATION; DEPTH LAYERS; DEPTH MEASUREMENTS; MONOSCOPIC; NOVEL METHODS; OBJECT BASED; OBJECT SEGMENTATION; OCCLUSION REASONING; PRACTICAL SOLUTIONS; REAL SCENES; REGION GROWING; STEREO IMAGE PAIRS; STEREOSCOPIC VIDEO; STEREOSCOPIC VIDEO GENERATION; THREE-DIMENSIONAL TELEVISION (3D-TV);

COMPUTER VISION; IMAGE SEGMENTATION; MULTIMEDIA SYSTEMS; TELEVISION APPLICATIONS; THREE DIMENSIONAL;

THREE DIMENSIONAL COMPUTER GRAPHICS;

EID: 79957582725     PISSN: 00189316     EISSN: None     Source Type: Journal    
DOI: 10.1109/TBC.2011.2131030     Document Type: Article

References (58)

1. C. Fehn and P. Kauff, "An evolutionary and optimized approach on 3D-TV," in Proc. Int. Broadcast. Conf., 2002, pp. 357-365, etc.
2. "IMAX," [Online]. Available: http://en.wikipedia.org/wiki/IMAX
3. L. K. J. Meesters, W. A. Ijsselsteijn, and P. J. H. Seuntiens, "A survey of perceptual evaluations and requirements of three-dimensional TV," IEEE Trans. Circuits Syst. Video Technol., vol. 14, no. 3, pp. 381-391, Mar. 2004.
4. S. Barnard and T. B. William, "Disparity analysis of images," IEEE Trans. Pattern Anal. Mach. Intell., vol. 2, no. 4, pp. 333-340, Jul. 1980.
5. A. Kubota, A. Smolic, and M. Magnor, "Multiview imaging and 3DTV," IEEE Signal Process. Mag., vol. 24, no. 6, pp. 10-21, Nov. 2007, etc. (Pubitemid 350201595)
6. R. P. Wildes, "Direct recovery of three-dimensional scene geometry from binocular stereo disparity," IEEE Trans. Pattern Anal. Mach. Intell., vol. 13, no. 8, pp. 761-774, Aug. 1991.
7. T. Kanade and M. Okutomi, "A stereo matching algorithm with an adaptive window: Theory and experiment," IEEE Trans. Pattern Anal. Mach. Intell., vol. 16, no. 9, pp. 920-932, Sep. 1994.
8. X. Jiang and M. Lambers, "DIBR-based 3D videos using non video rate range image stream," in Proc. IEEE Int. Conf. Multimedia Expo (ICME), Jul. 9-12, 2006, pp. 1873-1876.
9. Y. Taguchi, T. Koike, and K. Takahashi, "TransCAIP: A live 3DTV system using a camera array and an integral photography display with interactive control of viewing parameters," IEEE Trans. Vis. Comput. Graphics, vol. 15, no. 5, pp. 841-852, 2009, etc.
10. M. B. Kim, J. Nam, and W. Baek, "The adaptation of 3D stereoscopic video in MPEG-21 DIA," Signal Proc.: Image Commun., vol. 18, no. 8, pp. 685-697, Sep. 2003, Special Issue on Multimedia Adaptation, etc.
11. L. Zhang and W. J. Tam, "Stereoscopic image generation based on depth images for 3D TV," IEEE Trans. Broadcast., vol. 51, no. 2, pp. 191-199, 2005. (Pubitemid 40819422)
12. Y. L. Murphey and J. Chen, "A real-time depth detection system using monocular vision," in In Proc. SSGRR Conf., 2000, etc.
13. G.Wei and G. Hirzinger, "Learning shape from shading by amultilayer network," IEEE Trans. Neural Netw., vol. 7, no. 4, pp. 985-995, 1996. (Pubitemid 126776369)
14. C.-H. Choi, B.-H. Kwon, and M.-R. Choi, "A real-time field-sequential stereoscopic image converter," IEEE Trans. Consum. Electron., vol. 50, no. 3, pp. 903-910, 2004.
15. A.Yokota, T.Yoshida, and H. Kashiyama, "High-speed sensing system for depth estimation based on depth-from-focus by using smart imager," in Proc. IEEE Int. Symp. Circuits Syst., 2005, vol. 1, pp. 564-567, etc.
16. A. S. Ogale, C. Fermuller, and Y. Aloimonos, "Motion segmentation using occlusions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 6, pp. 988-992, 2005. (Pubitemid 40889848)
17. T. Gautama and M. A. V. Hulle, "A phase-based approach to the estimation of the optical flow field using spatial filtering," IEEE Trans. Neural Netw., vol. 13, no. 5, pp. 1127-1136, Sep 2002.
18. M. T. Pourazad, P. Nasiopoulos, and R. K. Ward, "An H.264-based scheme for 2D to 3D video conversion," IEEE Trans. Consum. Electron., vol. 55, no. 2, pp. 742-748, May 2009.
19. T. Okino, H. Murata, and K. Taima, "New television with 2D/3D image conversion technologies," in Proc. SPIE, Stereoscopic Displays Virtual Reality Syst. III, 1996, vol. 2653, pp. 96-103, etc.
20. K. Moustakas, D. Tzovaras, and M. G. Strintzis, "Stereoscopic video generation based on efficient layered structure and motion estimation from a monoscopic image sequence," IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 8, pp. 1065-1073, Aug. 2005. (Pubitemid 41343378)
21. S. H. Lai, C. W. Fu, and S. Chang, "A generalized depth estimation algorithm with a single image," PAMI, vol. 14, no. 4, pp. 405-411, 1992.
22. M. Kim, S. Park, and Y. Cho, "Object-based stereoscopic conversion of MPEG-4 encoded data," Lecture Notes Comput. Sci., vol. 3333, pp. 491-498, 2004. (Pubitemid 39749855)
23. J. Ens and P. Lawrence, "An investigation of methods for determining depth from focus," IEEE Trans. Pattern Anal. Mach. Intell., vol. 15, no. 2, pp. 97-108, 1993. (Pubitemid 23609373)
24. G. C. Feng and J. Jiang, "Image extraction in DCT domain," IEE Proc. Vision, Image Signal Proc., vol. 150, no. 1, pp. 20-27, 2003.
25. W. J. Tam, F. Speranza, and L. Zhang, "Depth image based rendering for multiview stereoscopic displays: Role of information at object boundaries," in Proc. SPIE, 2005, vol. 6016, pp. 75-85, etc.
26. W. J. Tam, A. S. Yee, and J. Ferreira, "Stereoscopic image rendering based on depth maps created from blur and edge information," Stereoscopic Displays & Appl. XII, vol. 5664, pp. 104-115, 2005, etc. (Pubitemid 40949104)
27. S. Battiato, A. Capra, and S. Curti, "3D stereoscopic image pairs by depth-map generation," in Proc. 2nd Int. Symp. 3D Data Proc., Vis. Transmiss., 2004, pp. 124-131, etc. (Pubitemid 40456287)
28. F. Pitie and A. Kokaram, "Matting with a depth map," in Proc. IEEE ICIP, 2010, pp. 21-24.
29. M. Kunter, S. Knorr, and A. Krutz, "Unsupervised object segmentation for 2D to 3D conversion," in Proc. SPIE (Stereoscopic Displays Appl. XX), 2009, vol. 7237, p. 7231B, etc.
30. D. Sun, S. Roth, and M. J. Black, "Secrets of optical flow estimation and their principles," in Proc. CVPR, 2010, pp. 2432-2439.
31. S. Battiato, S. Curti, and M. La Cascia, "Depth map generation by image classification," in Proc. SPIE, 2004, vol. 5302, pp. 95-104, etc.
32. P. Harman, J. Flack, and S. Fox, "Rapid 2D to 3D conversion," in Proc. SPIE, 2002, vol. 4660, pp. 78-86, etc. (Pubitemid 34918012)
33. D. Kim, D. Min, and K. Sohn, "Stereoscopic video generation method using motion analysis," in Proc. 3DTV Conf., 2007, pp. 1-4.
34. I. Ideses, L. P. Yaroslavsky, and B. Fishbain, "Real-time 2D to 3D video conversion," J. Real-Time Image Proc., vol. 2, no. 1, pp. 3-9, 2007. (Pubitemid 47551867)
35. D. fC. Burr and J. Ross, "How does binocular delay give information about depth?," Vis. Res., vol. 19, pp. 523-532, 1979. (Pubitemid 9160784)
36. S. Baker, D. Scharstein, and J. P. Lewis, "A database and evaluation methodology for optical flow," Microsoft Res. Rep. TR-2009-179, 2009 [Online]. Available: http://research.microsoft. com/pubs/117766/ofevaltr2.pdf, etc.
37. D. Kim, D. Min, and K. Sohn, "A stereoscopic video generation method using stereoscopic display characterization and motion analysis," IEEE Trans. Broadcast., vol. 54, no. 2, pp. 188-197, June 2008. (Pubitemid 351780176)
38. Y. Feng, J. Jiang, and S. S. Ipson, "A shape-match based algorithm for pseudo-3D conversion of 2D videos," in Proc. ICIP, 2005, pp. 808-811. (Pubitemid 44479603)
39. D. C. Hutshison and H. W. Neal, "The design and implementation of a stereoscopic microdisplay television," IEEE Trans. Consum. Electron., vol. 54, no. 2, pp. 254-261, May 2008.
40. Y. Feng, D. Li, K. Luo, and M. Zhang, "Asymmetric bidirectional view synthesis for free viewpoint and three-dimensional video," IEEE Trans. Consum. Electron., vol. 55, no. 4, pp. 2349-2355, Nov. 2009.
41. Y. K. Park, K. Jung, and Y. Oh, "Depth-image-based rendering for 3DTV service over T-DMB," Signal Proc.: Image Commun., vol. 24, no. 1/2, pp. 122-136, Jan. 2009, etc.
42. S.-Y. Kim, E.-K. Lee, and Y.-S. Ho, "Generation of ROI enhanced depth maps using stereoscopic cameras and a depth camera," IEEE Trans. Broadcast., vol. 54, no. 4, pp. 732-740, Dec. 2008.
43. S. Y. Kim, S. B. Lee, and Y. S. Ho, "Three-dimensional natural video system based on layered representation of depth maps," IEEE Trans. Consum. Electron., vol. 52, no. 3, pp. 1035-1042, Aug. 2006.
44. G. Wang and Q. M. J. Wu, "Perspective 3-D Euclidean reconstruction with varying camera parameters," IEEE Trans. Circuits Syst. Video Technol., vol. 19, no. 12, pp. 1793-1803, Dec. 2009.
45. Y. Feng, J. Jayaseelan, and J. Jiang, "Cue based disparity estimation for possible 2D to 3D video conversion," in Proc. Int. Conf. Vis. Inf. Eng.(VIE), 2006.
46. Y. Feng and J. Jiang, "Pseudo-stereo conversion from 2D video," in Proc. Int. Conf. Adv. Concepts Intell. Vis. Syst., Lecture Notes Comput. Sci., Oct 2005, vol. 3708, pp. 268-275.
47. L. Zhang, B. Lawrence, and D. Wang, "Comparison study on feature matching and block matching for automatic 2D to 3D video conversion," in Proc. CVMP, 2005, pp. 122-129, etc.
48. E. Stoykova, A. Alatan, and P. Benzie, "3-D time-varying scene capture technologies-A survey," IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 11, pp. 1568-1586, Nov. 2007, etc.
49. W. J. Tam and L. Zhang, "3D-TV content generation: 2D-to-3D conversion," in Proc. ICME, 2006, pp. 1869-1872. (Pubitemid 46680104)
50. X. L. Deng, X. H Jiang, and Q. G. Liu, "Automatic depth map estimation of monocular indoor environments," in Proc. Int. Conf. MMIT, 2008, pp. 646-649, etc.
51. M. Fieseler and X. Jiang, "Registration of depth and video data in depth image based rendering," in Proc. 3rd 3DTV-Conf., 2009.
52. Y. J. Jung, A. Baik, and J. Kim, "A novel 2D-to-3D conversion technique based on relative height depth cue," in Proc. SPIE, 2009, vol. 7237, etc.
53. J. Kim, A. Baik, and Y. J. Jung, "2D-to-3D conversion by using visual attention analysis," in Proc. SPIE, 2010, vol. 7524, etc.
54. W. J. Tam, F. Speranza, C. Vázquez, and L. Zhang, "Temporal subsampling of depth maps in depth image based rendering of stereoscopic image sequences," in Proc. SPIE, 2009, vol. 7237.
55. W. J. Tam, C. Vázquez, and F. Speranza, "Three-dimensional TV: A novel method for generating surrogate depth maps using colour information," in Proc. SPIE, 2009, vol. 7237.
56. A. Smolic, K. Mueller, and P. Merkle, "3D video and free viewpoint video-technologies, applications and MPEG standards," in Proc. ICME, 2006, pp. 2161-2164, etc. (Pubitemid 46680177)
57. Y. Feng, H. Fang, and J. Jiang, "Region growing with automatic seeding for semantic video object segmentation," in Lecture Notes in Comput. Sci., 2005, vol. 3687, pp. 542-549. (Pubitemid 41520508)
58. M. T. Coimbra and M. Davies, "Approximating optical flow within the MPEG-2 compressed domain," IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 1, pp. 103-107, 2005. (Pubitemid 40135881)