3D-TV content creation: Automatic 2D-to-3D video conversion

IEEE Transactions on Broadcasting

Volumn 57, Issue 2 PART 2, 2011, Pages 372-383

  Zhang, Liang ^a   Vázquez, Carlos ^a   Knorr, Sebastian ^b  

^a Industry Canada   (Canada)

^b TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

Author keywords

3D TV; Broadcasting; data conversion; image generation; stereo displays; stereo synthesis; stereo vision; three dimensional displays

Indexed keywords

3D-TV; DATA CONVERSION; IMAGE GENERATION; STEREO DISPLAYS; STEREO SYNTHESIS; THREE-DIMENSIONAL DISPLAY;

DATA HANDLING; DISPLAY DEVICES; STEREO VISION;

THREE DIMENSIONAL;

EID: 79957608462     PISSN: 00189316     EISSN: None     Source Type: Journal    
DOI: 10.1109/TBC.2011.2122930     Document Type: Article

References (95)

1. S. Yano and I. Yuyama, "Stereoscopic HDTV: Experimental system and psychological effects," J. SMPTE, vol. 100, pp. 14-18, 1991.
2. N. S. Holliman, N. A. Dodgson, and G. Favalora, "Three-dimensional display technologies: An analysis of technical performance characteristics, " IEEE Trans. Broadcast., 2011, to be published.
3. H. Tilton, "Broadcast standards for 3D HDTV," in Int. Symp. 3-D Image Technol. Arts, Tokyo, Feb. 1992, pp. 187-191.
4. P. V. Harman, "An architecture for digital 3D broadcasting," in SPIE Conf. Stereoscopic Displays Virtual Reality Syst. VI, San Jose, California, Jan. 1999, vol. 3639, pp. 254-259.
5. P. V. Harman, "Home-based 3D entertainment-An overview," in IEEE Int. Conf. Image Process., Vancouver, 2000, pp. 1-4.
6. W. J. Tam and L. Zhang, "3D-TV Content Generation: 2D-To-3D Conversion," in IEEE Int. Conf. Multimedia Expo, Toronto, ON, Jul. 9-12, 2006, pp. 1869-1872. (Pubitemid 46680104)
7. P. Harman, J. Flack, S. Fox, and M. Dowley, "Rapid 2D to3D conversion," in SPIE Conf. Stereoscopic Displays and Virtual Reality Systems IX, 2002, vol. 4660, pp. 78-86.
8. G. Iddan and G. Yahav, "3D imaging in the studio," in SPIE Conf. 3-D Image Capture Appl. IV, 2001, vol. 4298, pp. 48-55.
9. L. Zhang, J. Tam, and D.Wang, "Stereoscopic image generation based on depth images," in IEEE Conference on Image Processing, Singapore, Oct. 2004, pp. 2993-2996. (Pubitemid 40820703)
10. O. Grau, M. Price, and G. A. Thomas, "Use of 3-D techniques for virtual production," in Proc. SPIE, 2001, vol. 4309, pp. 40-50, Invited Paper. (Pubitemid 32327120)
11. S. C. Chan, H.-Y. Shum, and K.-T. Ng, "Image-based rendering and synthesis," IEEE Signal Process. Mag., pp. 22-33, Nov. 2007. (Pubitemid 350201596)
12. C. Liang and K. K. Wong, "3D reconstruction using silhouettes from unordered viewpoints," Image Vis. Comput., vol. 28, no. 4, pp. 579-589, 2010.
13. J. F. Blinn and M. E. Newell, "Texture and reflection in computer generated images," Commun. ACM, vol. 19, no. 10, pp. 542-547, Oct. 1976.
14. M. Levoy and P. Hanrahan, "Light field rendering," in Proc. SIGGRAPH, Aug. 1996, pp. 31-42.
15. S.J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, "The lumigraph," in Proc. SIGGRAPH, Aug. 1996, pp. 43-54.
16. C. Fehn, "Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV," in SPIE Conf. Stereoscopic Displays Virtual Reality Syst. XI, CA, Jan. 2004, vol. 5291, pp. 93-104.
17. J. Shade, S. Gortler, L. He, and R. Szeliski, "Layered depth image," in Proc. SIGGRAPH'98, Jul. 1998, pp. 231-242.
18. C. L. Zitnick, S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, "High-quality video view interpolation using a layered representation, " ACM Trans. Graphics, vol. 23, no. 3, pp. 600-608, 2004.
19. L. Zhang, D.Wang, and A. Vincent, "Adaptive reconstruction of intermediate views from stereoscopic images," IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 1, pp. 102-113, Jan. 2006. (Pubitemid 43172619)
20. A. Redert, M. Op de Beeck, C. Fehn, W. Ijsselsteijn, M. Pollefeys, L. Van Gool, E. Ofek, I. Sexton, and P. Surman, "ATTEST -advanced three-dimensional television system techniques," in Proc. 3DPVT, Padova, Italy, Jun. 2002, pp. 313-319.
21. Q. Wei, "Converting 2D to 3D: A survey," Delft University of Technology, The Netherlands, Project Report, Dec. 2005.
22. "Tutorial: Radar and stereoscopy visual ability in 3-D depth perception," Canada Centre for Remote Sensing [Online]. Available: http:// cct.rncan.gc.ca/resource/tutor/stereo/chap2/chapter2-5-e.php
23. R. Ostnes, V. Abbott, and S. Lavender, "Visualisation techniques: An overview-Part 1," Hydrographic J., no. 113, pp. 4-7, Jul. 2004.
24. J. Ens and P. Lawrence, "An investigation of methods of determining depth from focus," IEEE Trans. Pattern Anal. Mach. Intell., vol. 15, no. 2, pp. 523-531, 1993.
25. J. Ko, M. Kim, and C. Kim, "2D-To-3D stereoscopic conversion: Depth-map estimation in a 2D single-view image," in Proc. SPIE, 2007, vol. 6696.
26. G. Guo, N. Zhang, L. Huo, and W. Gao, "2D to 3D conversion based on edge defocus and segmentation," in IEEE Int. Conf. Acoust., Speech Signal Process., Mar. 31-April, 4 2008, pp. 2181-2184.
27. Favaro, "Shape from focus/defocus," Washington University, Department of Electrical Engineering Electronic Signals and Systems Research Lab, Jun. 25, 2002.
28. A. P. Pentland, "A new sense for depth of field," IEEE Trans. Pattern Anal. Mach. Intell., vol. 9, pp. 523-531, 1987.
29. J. H. Elder and S. W. Zucker, "Local scale control for edge detection and blur estimation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 7, pp. 699-716, 1998. (Pubitemid 128744855)
30. S. A. Valencia and R. M. Rodriguez-Dagnino, "Synthesizing stereo 3D views from focus cues in monoscopic 2D images," in Proc. SPIE, 2003, vol. 5006, pp. 377-388.
31. S. Battiato, S. Curti, M. La Cascia, M. Tortora, and E. Scordato, "Depth map generation by image classification," in Proc. SPIE, Apr. 2004, vol. 5302, pp. 95-104.
32. X. Huang, L.Wang, J. Huang, D. Li, and M. Zhang, "A depth extraction method based on motion and geometry for 2D to 3D conversion," in 3rd Int. Symp. Intell. Inf. Technol. Appl., 2009, pp. 294-298.
33. Y.-J. Jung, A. Baik, and D. Park, "A novel 2D-to-3D conversion technique based on relative height-depth-cue," in SPIE Conf. Stereoscopic Displays Appl. XX, San José, CA, 2009, vol. 7237, 72371U.
34. K. Yamada, K. Suehiro, and H. Nakamura, "Pseudo 3D image generation with simple depth models," in Int. Conf. Consum. Electron., Dig. Techn. Papers, Jan. 8-12, 2005, pp. 277-278. (Pubitemid 41688113)
35. D. A. Forsyth, "Shape from texture without boundaries," in ECCV02, 2002, pp. 225-239.
36. A. Torralba and A. Oliva, "Depth estimation from image structure," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 9, Sept. 2002.
37. A. M. Loh and R. Hartley, "Shape from non-homogeneous, non-stationary, anisotropic, perspective texture," in Proc. British Mach. Vis. Conf., 2005.
38. F. Cozman and E. Krotkov, "Depth from scattering," in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 1997, pp. 801-806.
39. W. J. Tam, C. Vázquez, and F. Speranza, "3D-TV: A novel method for generating surrogate depth maps using colour information," in SPIE Conf. Stereoscopic Displays Appl. XX Conf., San José, CA, 2009, vol. 7237.
40. C. Vázquez and W. J. Tam, "CRC-CSDM: 2D to 3D conversion using colour-based surrogate depth maps," in Int. Conf. 3D Syst. Appl. (3DSA), Tokyo, Japan, May 2010.
41. R. Zhang, P.-S. Tsai, J. E. Cryer, and M. Shah, "Shape-from-shading: A survey," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 8, pp. 690-706, 1999.
42. E. Rubin, "Figure and ground," in Visual Perception Philadelphia, S. Yantis, Ed. London , U.K.: Psychology, 2001, pp. 225-229.
43. W. J. Tam, A. S. Yee, J. Ferreira, S. Tariq, and F. Speranza, "Stereoscopic image rendering based on depth maps created from blur and edge information," in Proc. Stereoscopic Disp. Appl., Jan. 2005, vol. 5664, pp. 104-115. (Pubitemid 40949104)
44. J. Kim, A. Baik, Y. J. Jung, and D. Park, "2D-to-3D conversion by using visual attention analysis," in Proc. SPIE 7524, Feb. 2010, 752412.
45. B. J. Rogers and M. E. Graham, "Motion parallax as an independent cue for depth perception," Perception, vol. 8, pp. 125-134, 1979.
46. E. Arce and J. Marroquin, "High-precision stereo disparity estimation usingHMMFmodels," Image Vis. Comput., vol. 25, no. 5, pp. 623-636, May 2007.
47. N. Atzpadin, P. Kauff, and O. Schreer, "Stereo analysis by hybrid recursive matching for real-time immersive video conferencing," IEEE Trans. Circuits Syst. Video Technol., vol. 14, pp. 321-334, 2004.
48. D. Scharstein and R. Szeliski, "A taxonomy and evaluation of dense two-frame stereo correspondence algorithms," Int. J. Comput. Vis., no. 47, pp. 7-42, 2002. (Pubitemid 34428419)
49. T. Jebara, A. Azarbayejani, and A. Pentland, "3D structure from 2D motion," IEEE Signal Process. Magaz., vol. 16, no. 3, pp. 66-84, May 1999.
50. R.-I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, Second ed. Cambridge, U.K.: Cambridge Univ. Press, 2004.
51. E. Imre, S. Knorr, A. A. Alatan, and T. Sikora, "Prioritized sequential 3D reconstruction in video sequences of dynamic scenes," in IEEE Int. Conf. Image Process. (ICIP), Atlanta, GA, 2006.
52. I. Ideses, L. P. Yaroslavsky, and B. Fishbain, "Real-time 2D to 3D video conversion," J. Real-Time Image Process., vol. 2, no. 1, pp. 3-7, 2007.
53. M. T. Pourazad, P. Nasiopoulos, and R. K. Ward, "Generating the depth map from the motion information of H.264-encoded 2D video sequence," EURASIP J. Image Video Process., Article ID 108584, 2010.
54. G. Zhang, J. Jia, T. T. Wong, and H. Bao, "Consistent depth maps recovery from a video sequence," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 6, pp. 974-988, 2009.
55. K. Moustakas, D. Tzovaras, and M. G. Strintzis, "Stereoscopic video generation based on efficient layered structure and motion estimation from a monoscopic image sequence," IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 8, pp. 1065-1073, Aug. 2005. (Pubitemid 41343378)
56. P. Gargallo and P. F. Sturm, "Bayesian 3D modeling from images using multiple depth maps," in IEEE Conf. Comput. Vis. Pattern Recog., 2005, vol. 2, pp. 885-891.
57. P. H. S. Torr, "Bayesian model estimation and selection for epipolar geometry and generic manifold fitting," TechReport MSR-TR-2002-29, Apr. 2002, Microsoft Research.
58. L. Alvarez, R. Deriche, T. Papadopoulo, and J. Sánchez, "Symmetrical dense optical flow estimation with occlusions detection," Int. J. Comput. Vis., vol. 75, no. 3, pp. 371-385, 2007. (Pubitemid 47383856)
59. J. Shi and C. Tomasi, "Good features to track," in IEEE Int. Conf. Comput. Vis. Pattern Recog., Seattle, WA, Jun. 1994.
60. D. G. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Comput. Vis., vol. 60, no. 2, pp. 91-110, 2004.
61. C. Tomasi and T. Kanade, "Detection and tracking of point features," Tech. Rep. CMU-CS-91-132, 1991, Carnegie Mellon University Technical.
62. M. Fischler and R. Bolles, "Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography," Commun. ACM, vol. 1981, pp. 381-385.
63. S. Knorr, E. Ýmre, A. A. Alatan, and T. Sikora, "A geometric segmentation approach for the 3D reconstruction of dynamic scenes in 2D video sequences," in EUSIPCO, Florence, Italy, Sep. 2006.
64. C.-C. Cheng, C.-T. Li, Y.-M. Tsai, and L.-G. Chen, "Hybrid depth cueing for 2D-to-3D conversion system," in SPIE Conf. Stereoscopic Displays Appl. XX, San José, CA, 2009, vol. 7237.
65. M. Kunter, S. Knorr, A. Krutz, and T. Sikora, "Unsupervised object segmentation for 2D to 3D conversion," in SPIE Conf. Stereoscopic Displays Appl. XX, San José, CA, 2009, vol. 7237.
66. B. Triggs, P. McLauchlan, R. Hartley, and A. Fitzgibbon, "Bundle adjustment-A modern synthesis," in Vision Algorithms: Theory & Practice. Berlin: Springer-Verlag, 2000, vol. 1883, LNCS, pp. 298-372.
67. M. B. Kim and M. S. Song, "Stereoscopic conversion of monoscopic video by the transformation of vertical to horizontal disparity," Proc. SPIE, vol. 3295, pp. 65-75, 1998.
68. L. Zhang, B. Lawrence, D. Wang, and A. Vincent, "Comparison study on feature matching and block matching for automatic 2D to 3D video conversion," in 2nd IEE Eur. Conf. Visual Media Prod., London, U.K., 2005, pp. 122-129.
69. D. Kim, D. Min, and K. Sohn, "A stereoscopic video generation method using stereoscopic display characterization and motion analysis," IEEE Trans. Broadcast., vol. 54, no. 2, pp. 188-197, Jun. 2008. (Pubitemid 351780176)
70. Y. Matsumoto, H. Terasaki, K. Sugimoto, and T. Arakawa, "Conversion system of monocular image sequence to stereo using motion parallax," in SPIE Conf. Stereoscopic Displays Virtual Reality Syst. IV, May 1997, vol. 3012, pp. 108-112.
71. Q.-T. Luong and O. Faugeras, "Self calibration of a moving camera from point correspondences and fundamental matrices," Int. J. Comput. Vis., vol. 22, no. 3, 1997.
72. M. Pollefeys, "Self-calibration and metric 3D reconstruction from uncalibrated image sequences," PhD thesis, K.U. Leuven, Belgium, 1999.
73. P. R. S. Mendonca and R. Cipolla, "A simple technique for self-calibration," in IEEE Comput. Soc. Conf. Comput.Vis. Pattern Recog. (CVPR), 1999.
74. R. Hartley and P. Sturm, "Triangulation," Comput. Vis. Image Understanding, vol. 68, no. 2, pp. 146-157, 1997.
75. P. H. S. Torr, A. W. Fitzgibbon, and A. Zisserman, "The problem of degeneracy in structure and motion recovery from uncalibrated image sequences," Int. J. Comput. Vis., vol. 32, no. 1, pp. 27-44, Aug. 1999.
76. C. Fehn, "A 3D-TV approach using depth-image-based rendering (DIBR)," in Proc. Vis., Imaging, Image Process., Benalmadena, Spain, Sep. 2003, pp. 482-487.
77. A. Woods, T. Docherty, and R. Koch, "Image distortions in stereoscopic video systems," in SPIE Conf. Stereoscopic Displays Appl., San Jose, CA, Feb. 1993, pp. 36-48.
78. W. R. Mark, "Post-Rendering 3D imagewarping: visibility, reconstruction, and performance for depth-image warping," PhD thesis, University of North Carolina at Chapel Hill, Chapel Hill, NC, Apr. 1999.
79. L. MacMillan, "An image based approach to three-dimensional computer graphics," Ph.D dissertation, University of North Carolina, , 1997.
80. C. Vázquez, W. J. Tam, and F. Speranza, "Stereoscopic imaging: Filling disoccluded areas in depth image-based rendering," in SPIE Conf. 3-D TV, Video, Display V,2006, vol. 6392, 63920D.
81. A. Colombari, A. Fusiello, and V. Murino, "Continuous parallax adjustment for 3D-TV," in IEE Eur. Conf. Vis. Media Prod., London, U.K., Nov. 2005, pp. 194-200.
82. K. Luo, D.-X. Li, Y.-M. Feng, and M. Zhang, "Depth-aided inpainting for disocclusion restoration of multi-view images using depth-image-based rendering," J. Zhejiang Univ. -Sci. A, vol. 10, no. 12, pp. 1738-1749, Dec. 2009.
83. M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, "Image inpainting," in ACM SIGGRAPH, Int. Conf. Comput. Graphics Interactive Tech., New Orleans, LA, Jul. 2000, vol. 174, pp. 417-424. (Pubitemid 32394006)
84. R. Klein Gunnewiek, R.-P. M. Berretty, B. Barenbrug, and J. P. Magalhães, "Coherent spatial and temporal occlusion generation," in Proc. SPIE 7237, 2009, p. 723713.
85. L. Jovanov, A. Pizurica, and W. Philips, "Wavelet-based stereo images reconstruction using depth images," Proc. SPIE, vol. 6701, p. 67012A, 2007.
86. W. J. Tam, G. Alain, L. Zhang, T. Martin, and R. Renaud, "Smoothing depth maps for improved stereoscopic image quality," in SPIE Conf. 3-D TV, Video, Display III, Philadelphia, PA, Oct. 2004, vol. 5599, pp. 162-172. (Pubitemid 40512677)
87. L. Zhang and W. J. Tam, "Stereoscopic image generation based on depth images for 3D TV," IEEE Trans. Broadcast, vol. 51, no. 2, pp. 191-199, Jun. 2005. (Pubitemid 40819422)
88. W.-Y. Chen, Y.-L. Chang, S.-F. Lin, L.-F. Ding, and L.-G. Chen, "Efficient depth image based rendering with edge dependent depth filter and interpolation," in IEEE Int. Conf. Multimedia Expo (ICME), Amsterdam, The Netherlands, Jul. 2005, pp. 1314-1317. (Pubitemid 44669115)
89. I. Daribo, C. Tillier, and B. Pesquet-Popescu, "Distance dependent depth filtering in 3D warping for 3DTV," in IEEE 9th Workshop Multimedia Signal Process., Crete, Greece, Oct. 2007, pp. 312-315.
90. S.-B. Lee and Y.-S. Ho, "Discontinuity-adaptive depth map filtering for 3D view generation," in 2nd Int. Conf. Immersive Telecommun., Berkeley, CA, 2009.
91. W. Wang et al., "Depth image segmentation for improved virtual view image quality in 3-D TV," in 2007 Int. Symp. Intell. Signal Process. Commun. Syst., Xiamen, China, Nov. 2007, pp. 784-787.
92. W. J. Tam and L. Zhang, "Non-uniform smoothing of depth maps before image-based rendering," in SPIE Conf. 3-D TV, Video Display III (ITCOM), 2004, vol. 5599, pp. 173-183. (Pubitemid 40512678)
93. S. Knorr and T. Sikora, "An image-based rendering (IBR) approach for realistic stereo view synthesis of TV broadcast based on structure from motion," in IEEE Int. Conf. Image Process. (ICIP), San Antonio, TX, 2007.
94. S. Knorr, M. Kunter, and T. Sikora, "Stereoscopic 3D from 2D video with super-resolution capability," Signal Process.: Image Commun., vol. 23, no. 9, pp. 665-676, Oct. 2008.
95. Y. L. Chang, J. Y. Chang, Y. M. Tsai, C. L. Lee, and L. G. Chen, "Priority depth fusion for 2D-to-3D conversion systems," in SPIE Proc. 3-D Image Capture Appl., 2008, vol. 6805.