3D video and free viewpoint video - From capture to display

Pattern Recognition

Volumn 44, Issue 9, 2011, Pages 1958-1968

  Smolic, Aljoscha ^a  

^a DISNEY RESEARCH   (United States)

Author keywords

3D video; 3DTV; Free viewpoint video; Stereo video

Indexed keywords

2D VIDEO; 3D VIDEO; 3DTV; FREE-VIEWPOINT VIDEO; MEDIA FORMATS; MEDIA PROCESSING; STEREO VIDEO; TAKE-ALL; VIRTUAL WORLDS; VISUAL MEDIA; VISUAL SCENE;

COMMUNICATION; COMPUTER VISION; DISPLAY DEVICES; IMAGE COMMUNICATION SYSTEMS; THREE DIMENSIONAL; VIRTUAL REALITY; VISUAL COMMUNICATION;

THREE DIMENSIONAL COMPUTER GRAPHICS;

EID: 79957485086     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2010.09.005     Document Type: Conference Paper

References (83)

1. A. Smolic, K. Mueller, P. Merkle, C. Fehn, P. Kauff, P. Eisert, T. Wiegand: 3D video and free view-point video - technologies, applications and MPEG standards, in: ICME 2006, International Conference on Multimedia and Expo, Toronto, Ontario, Canada, July 2006.
2. C. Wheatstone Contributions to the physiology of vision - Part the First. On some remarkable, and hitherto unobserved, phenomena of binocular vision Philosophical Transactions of the Royal Society of London 128 1838 371 394 Received and Read June 21
3. B. Mendiburu 3D Movie Making - Stereoscopic Digital Cinema from Script to Screen 2008 Elsevier
4. A. Smolic, K. Mueller, P. Merkle, T. Rein, P. Eisert, T. Wiegand, Free viewpoint video extraction, representation, coding, and rendering, in: Proceedings of the ICIP 2004, IEEE International Conference on Image Processing, Singapore, October 2427 2004.
5. 〈 http://www.ri.cmu.edu/events/sb35/tksuperbowl.html 〉.
6. T. Kanade, P.J. Narayanan, P.W. Rander, Virtualized reality: concepts and early results, in: IEEE Workshop on the Representation of Visual Scenes (in conjunction with ICCV'95), Boston, MA, June 1995.
7. 〈 http://www.liberovision.com/ 〉.
8. A. Smolic, P. Kauff, Interactive 3D video representation and coding technologies, in: Proceedings of the IEEE, Special Issue on Advances in Video Coding and Delivery, vol. 93(1), January 2005.
9. S.B. Kang, R. Szeliski, P. Anandan: The geometry-image representation tradeoff for rendering, in: ICIP 2000, IEEE International Conference on Image Processing, Vancouver, Canada, September 2000.
10. S. Würmlin, E. Lamboray, and M. Gross 3D video fragments: dynamic point samples for real-time free-viewpoint video Computers and Graphics, Special Issue on Coding, Compression and Streaming Techniques for 3D and Multimedia Data 28 1 2004 3 14 Elsevier Ltd
11. S.M. Seitz, and C.R. Dyer Photorealistic scene reconstruction by voxel colouring International Journal of Computer Vision 35 2 1999 151 173 (Pubitemid 32210849)
12. M. Levoy, P. Hanrahan, Light field rendering, in: Proceedings of the ACM SIGGRAPH, August 1996, pp. 3142.
13. Masayuki Tanimoto, Free viewpoint television - FTV, in: Proceedings of the PCS 2004, Picture Coding Symposium, San Francisco, CA, USA, December 1517, 2004.
14. T. Fujii, and M. Tanimoto Free-viewpoint TV system based on Ray-Space representation SPIE ITCom 4864 22 2002 175 189
15. C. Fehn, P. Kauff, M. Op de Beeck, F. Ernst, W. Ijsselsteijn, M. Pollefeys, L. Vangool, E. Ofek, and I. Sexton An Evolutionary and Optimised Approach on 3D-TV Proceedings of the of IBC 2002 2002 Int. Broadcast Convention Amsterdam, Netherlands
16. C. Fehn, 3D-TV using depth-image-based rendering (DIBR), in: Proceedings of the of Picture Coding Symposium (PCS), San Francisco, CA, USA, December 2004.
17. C. L. Zitnick, S.B. Kang, M. Uyttendaele, S. Winder, R. Szeliski, High-quality video view interpolation using a layered representation, in: Proceedings of the ACM SIGGRAPH and ACM Transactions on Graphics, Los Angeles, CA, USA, August 2004.
18. P. Kauff, N. Atzpadin, C. Fehn, M. Müller, O. Schreer, A. Smolic, and R. Tanger Depth map creation and image based rendering for advanced 3DTV services providing interoperability and scalability Signal Processing: Image Communication. Special Issue on 3DTV 2007
19. A. Smolic, K. Müller, K. Dix, P. Merkle, P. Kauff, T. Wiegand, Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems, in: Proceedings of the ICIP 2008, IEEE International Conference on Image Processing, San Diego, CA, USA, October 2008.
20. K. Mueller, A. Smolic, K. Dix, P. Merkle, P. Kauff, T. Wiegand, View synthesis for advanced 3D video systems, EURASIP Journal on Image and Video Processing, 2008 (2008) doi:10.1155/2008/438148.
21. K. Müller, A. Smolic, K. Dix, P. Merkle, P. Kauff, T. Wiegand, Reliability-based generation and view synthesis in layered depth video, in: Proceedings of the MMSP 2008, IEEE International Workshop on Multimedia Signal Processing, Cairns, Australia, October 2008.
22. J. Shade, S. Gortler, L.W. He, R. Szeliski, Layered depth images, in: Proceedings of the SIGGRAPH'98, Orlando, FL, USA, July 1998.
23. A. Smolic, K. Mueller, P. Merkle, T. Rein, P. Eisert, T. Wiegand: Free viewpoint video extraction, representation, coding, and rendering, in: ICIP 2004, IEEE International Conference on Image Processing, Singapore, October 2427, 2004.
24. T. Matsuyama, X. Wu, T. Takai, and T. Wada Real-time dynamic 3-D object shape reconstruction and high-fidelity texture mapping for 3-D video IEEE Transactions on Circuits and Systems for Video Technology 14 3 2004 357 369
25. M. Kunter, S. Knorr, A. Krutz, T. Sikora, Unsupervised object segmentation for 2D to 3D Conversion, in: Proceedings of the of the SPIE: Stereoscopic Displays and Applications XX Conference, San José, USA, 2009.
26. A. Smolic, P. Kauff, S. Knorr, A. Hornung, M. Kunter, M. Mueller, M. Lang, 3D video post-production and processing, Invited Paper, Proceedings of the IEEE, Special Issue on 3D Media and Displays, in press.
27. A. Kubota, A. Smolic, M. Magnor, T. Chen, and M. Tanimoto Multi-view imaging and 3DTV - special issue overview and introduction IEEE Signal Processing Magazine, Special Issue on Multi-view Imaging and 3DTV 24 6 2007
28. F. Zilly, P. Eisert, P. Kauff, Real-time analysis and correction of stereoscopic HDTV sequences, in: Proceedings of CVMP, London, November 2009.
29. 〈 http://www.3alitydigital.com/ 〉.
30. 〈 http://www.binocle.com/ 〉.
31. 〈 http://www.3d4you.eu/index.php 〉.
32. B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A. Adams, M. Levoy, M. Horowitz, High performance imaging using large camera arrays, in: Proceedings of the SIGGRAPH 2005, July 31August 4, Los Angeles, CA, USA.
33. Fabio Remondino, Clive Fraser, Digital camera calibration methods: considerations and comparisons, IAPRS vol. XXXVI, Part 5, Dresden, 2527 September 2006.
34. C. Loop, and Z. Zhang Computing rectifying homographies for stereo vision Computer Vision and Pattern Recognition 1 1999 131
35. W. Hu, T. Tan, L. Wang, and S. Maybank A survey on visual surveillance of object motion and behaviours IEEE Transactions on Systems, Man and Cybernetics 2004
36. L. Lucchese, and S.K. Mitra Color Image Segmentation: A State-of-the-Art Survey 2001 Department of Electrical and Computer Engineering, USCB
37. A. Yilmaz, O. Javed, and M. Shah Object Tracking: A Survey 2006 ACM Computing Surveys
38. X. Bai, J. Wang, D. Simons, G. Sapiro, Video snapcut: robust video object cutout using localized classifiers, in: Proceedings of the ACM SIGGRAPH, 2009.
39. D.G. Lowe, Object recognition from local scale-invariant features, in: Proc.eedings of the International Conference on Computer Vision, Corfu, Greece, 2004, pp. 11501157.
40. A. Laurentini The visual hull concept for silhouette-based image understanding IEEE Transactions on Pattern Analysis and Machine Intelligence 16 2 1994 150 162
41. W. Matusik, C. Buehler, R. Raskar, S.J. Gortler, L. McMillan, Image based visual hulls, in: SIGGRAPH'00 Proceedings, July 2000, pp. 369374.
42. T. Lewiner Efficient implementation of marching cubes cases with topological guarantee Journal of Graphics Tools 8 2003 1 15
43. G. Taubin, Curve and surface smoothing without shrinkage, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV 95), 1995.
44. H. Hoppe Efficient implementation of progressive meshes Computers & Graphics 22 2 1998 27 36 (Pubitemid 128412903)
45. C. Liang, and K.K. Wong 3D reconstruction using silhouettes from unordered viewpoints Image and Vision Computing 28 4 2010 579 589
46. E. de Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H.P. Seidel, S. Thrun, Performance capture from sparse multi-view video, in: SIGGRAPH'08: ACM SIGGRAPH 2008 papers, 2008, pp. 110.
47. Y.-F. Liu, A unified approach to image focus and defocus analysis, Ph.D. Thesis, State University of New York at Stony Brook, New York, 1998.
48. T.-C. Wei, Three dimensional machine vision using image defocus, Ph.D. Thesis, State University of New York at Stony Brook, New York, 1994.
49. R. Zhang, P.-S. Tsai, J.E. Cryer, and M. Shah Shape from shading: a survey IEEE Transactions on Pattern Analysis and Machine Intelligence 21 8 1999 690 706
50. J.-D. Durou, M. Falcone, and M. Sagona Numerical methods for shape-from- shading: a new survey with benchmarks Computer Vision and Image Understanding 109 1 2008 22 43 (Pubitemid 350251232)
51. P. Beardsley, P.H.S. Torr, A. Zisserman, 3D model acquisition from extended image sequences, in: Proceedings of the European Conference on Computer Vision (ECCV), 1996, pp. 683695. (Pubitemid 126064255)
52. T. Jebara, A. Azarbayejani, and A. Pentland 3D structure from 2D motion IEEE Signal Processing Magazine 16 3 1999 66 84
53. R.-I. Hartley, and A. Zisserman Multiple View Geometry in Computer Vision 2 Edition 2004 Cambridge University Press
54. M. Pollefeys, L. Van Gool, M. Vergauwen, F. Verbiest, K. Cornelis, J. Tops, and R. Koch Visual modeling with a hand-held camera International Journal of Computer Vision 59 3 2004 207 232
55. D. Hoiem, A.A. Efros, and M. Hebert Automatic photo pop-up ACM Transactions on Graphics 24 3 2005 577 584
56. S. Lee, D. Feng, Gooch, perception-based construction of 3D models from line drawings, in: Proceedings of the Symposium on Interactive 3D Graphics and Games, 2008.
57. P. Parodi, and G. Piccioli 3D shape reconstruction by using vanishing points IEEE Transactions on Pattern Analysis and Machine Intelligence 18 2 1996 211 217 (Pubitemid 126772808)
58. E. Arce, and J. Marroquin High-precision stereo disparity estimation using HMMF models Image and Vision Computing 25 5 2007 623 636
59. N. Atzpadin, P. Kauff, and O. Schreer Stereo analysis by hybrid recursive matching for real-time immersive video conferencing IEEE Transactions on Circuits and Systems for Video Technology 14 2004 321 334
60. S.B. Kang, R. Szeliski, J. Chai, Handling occlusions in dense multi-view stereo, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2001, pp. 103110.
61. D. Scharstein, and R. Szeliski A taxonomy and evaluation of dense two-frame stereo correspondence algorithms International Journal of Computer Vision 47 2002 7 42 (Pubitemid 34428419)
62. M.Z. Brown, D. Burschka, and G.D. Hager Advances in computational stereo IEEE Transactions on Pattern Analysis and Machine Intelligence 25 8 2003 993 1008
63. S.M. Seitz, B. Curless, J. Diebel, D. Scharstein, R. Szeliski, A comparison and evaluation of multi-view stereo reconstruction algorithms, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2006, pp. 519528.
64. V. Kolmogorov, R. Zabih, Computing visual correspondence with occlusions using graph cuts, in: International Conference on Computer Vision (ICCV), July 2001.
65. M. Bleyer, and M. Gelautz A layered stereo matching algorithm using image segmentation and global visibility constraints ISPRS Journal of Photogrammetry and Remote Sensing 59 3 2005 128 150 (Pubitemid 40717591)
66. O.P. Gangwal, R.P. Beretty, Depth map post-processing for 3D-TV, digest of technical papers, in: International Conference on Consumer Electronics, 2010.
67. 〈 http://vision.middlebury.edu/stereo/ 〉
68. C. Zitnick, and T. Kanade A cooperative algorithm for stereo matching and occlusion detection IEEE Transactions on Pattern Analysis and Machine Intelligence 22 7 2000 675 684
69. A. Smolic, K. Mueller, N. Stefanoski, J. Ostermann, A. Gotchev, G.B. Akar, G. Triantafyllidis, and A. Koz Coding algorithms for 3DTV - a survey IEEE Transactions on Circuits and Systems for Video Technology, Special Issue on Multiview Video Coding and 3DTV 17 11 2007
70. B. Bartczak, R. Koch, Dense depth maps from low resolution time-of-flight depth and high resolution color views, in: Proceedings of the ISVC 09, Las Vegas, Nevada, USA, December 13, 2009.
71. A. Smolic, K. Mueller, P. Merkle, A. Vetro, Development of a new MPEG Standard for Advanced 3D Video Applications, in: ISPA 2009, 6th International Symposium on Image and Signal Processing and Analysis, Salzburg, Austria, September 2009.
72. P. Merkle, A. Smolic, K. Mueller, and T. Wiegand Efficient prediction structures for multiview video coding IEEE Transactions on Circuits and Systems for Video Technology, Special Issue on Multiview Video Coding and 3DTV 17 11 2007
73. A. Smolic, K. Mueller, P. Merkle, M. Kautzner, T. Wiegand: 3D video objects for interactive applications, in: EUSIPCO 2005, Antalya, Turkey, September 48, 2005.
74. M. Bourges-Sévenier, and E.S. Jang An introduction to the MPEG-4 animation framework extension IEEE Transactions on Circuits and Systems on Video Technology 14 7 2004 928 936
75. K. Mamou, N. Stefanoski, H. Kirchhoffer, K. Mueller, T. Zaharia, F. Preteux, D. Marpe, J. Ostermann, The new MPEG-4/FAMC Standard for Animated 3D Mesh Compression, in: 3DTV-CON'08, May 2830, 2008, Istanbul, Turkey.
76. M. Bertalmío, G. Sapiro, V. Caselles, C. Ballester, Image inpainting, in: Proceedings of the of ACM SIGGRAPH, New Orleans, USA, July 2000.
77. A. Hornung, and L. Kobbelt Interactive pixel-accurate free viewpoint rendering from images with silhouette aware sampling Computer Graphics Forum 28 2009 2090 2103 8 2009 2090 2103
78. S.-W. Shih, and J. Liu A novel approach to 3-D Gaze tracking using stereo cameras IEEE Transactions on Systems, Man, and Cybernatics - Part B: Cybernatics 34 1 2004
79. J. Konrad, and M. Halle 3-D displays and signal processing - an answer to 3-D Ills? IEEE Signal Processing Magazine 24 6 2007
80. A. Woods et al., State of the art in stereo and auto-stereo 3D displays, Invited Paper, in: Proceedings of the IEEE, Special Issue on 3D Media and Displays, in press.
81. T. Balogh, T. Forgacs, O. Balet, E. Bouvier, F. Bettio, E. Gobbetti, G. Zanetti, A large scale interactive holographic display, in: Proceedings of the IEEE VR 2006 Workshop on Emerging Display Technologies, Alexandria, VA, USA, March 2529, 2006.
82. B. Javidi et al., Integral Imaging 3D Display Technologies, Invited Paper, in: Proceedings of the IEEE, Special Issue on 3D Media and Displays, in press
83. L. Onural et al., Holographic 3D Display Technologies, Invited Paper, in: Proceedings of the IEEE, Special Issue on 3D Media and Displays, in press.