Survey of image-based representations and compression techniques

IEEE Transactions on Circuits and Systems for Video Technology

Volumn 13, Issue 11, 2003, Pages 1020-1037

  Shum, Heung Yeung ^a   Kang, Sing Bing ^b   Chan, Shing Chow ^c  

^a MICROSOFT RESEARCH ASIA   (China)

^b MICROSOFT RESEARCH   (United States)

^c UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Image based modeling; Image based rendering (IBR); Image based representations; Survey

Indexed keywords

ALGORITHMS; COMPUTER VISION; IMAGE COMPRESSION; INTERPOLATION; MATHEMATICAL MODELS; MATRIX ALGEBRA; THREE DIMENSIONAL COMPUTER GRAPHICS;

IMAGE BASED MODELING; IMAGE BASED RENDERING; IMAGE BASED REPRESENTATIONS;

IMAGE ANALYSIS;

EID: 0242636443     PISSN: 10518215     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCSVT.2003.817360     Document Type: Article

References (98)

1. E. H. Adelson and J. Bergen, "The plenoptic function and the elements of early vision," in Computational Models of Visual Processing. Cambridge, MA: MIT Press, 1991, pp. 3-20.
2. D. G. Aliaga and I. Carlbom, "Plenoptic stitching: A scalable method for reconstructing 3D interactive walkthroughs," in Proc. ACM Annu. Computer Graphics Conf., Aug. 2001, pp. 443-450.
3. S. Avidan and A. Shashua, "Novel view synthesis in tensor space," in Computer Vision and Pattern Recognition Conf., San Juan, PR, June 1997, pp. 1034-1040.
4. H. Aydinoglu and M. H. Hayes, "Stereo image coding: A projection approach," in IEEE Int. Image Processing Conf., vol. 8, 1998, pp. 506-516.
5. C. Buehler, M. Bosse, L. McMillan, S. Gortler, and M. Cohen, "Unstructured lumigraph rendering," in Proc. ACM Annu. Computer Graphics Conf., Aug. 2001, pp. 425-432.
6. E. Camahort, "4D light-field modeling and rendering," Univ. Texas at Austin, Austin, TX, Tech. Rep. TR01-52, 2001.
7. E. Camahort, A. Lerios, and D. Fussell, "Uniformly sampled light fields," in 9th Eurographics Rendering Workshop, Vienna, Austria, June-July 1998, pp. 117-130.
8. D. Capel and A. Zisserman, "Super-resolution from multiple views using learnt image models," in Computer Vision and Pattern Recognition Conf., vol. 2, Kauai, HI, Dec. 2001, pp. 627-634.
9. J.-X. Chai, X. Tong, S.-C. Chan, and H.-Y. Shum, "Plenoptic sampling," in Proc. ACM Annu. Computer Graphics Conf., July 2000, pp. 307-318.
10. S. C. Chan, K. T. Ng, Z. F. Gan, K. L. Chan, and H.-Y. Shum, "The data compression of simplified dynamic light fields," presented at the IEEE Int. Acoustics, Speech, and Signal Processing Conf., Hong Kong, Apr. 2003.
11. C. Chang, G. Bishop, and A. Lastra, "LDI tree: A hierarchical representation for image-based rendering," in Proc. ACM Annu. Computer Graphics Conf., Aug. 1999, pp. 291-298.
12. S. Chen and L. Williams, "View interpolation for image synthesis," in Proc. ACM Annu. Computer Graphics Conf., Aug. 1993, pp. 279-288.
13. S. E. Chen, "QuickTime VR-an image-based approach to virtual environment navigation," in Proc. ACM Annu. Computer Graphics Conf., Aug. 1995, pp. 29-38.
14. W.-C. Chen, J. V. Bouguet, M. H. Chu, and R. Grzeszczuk, "Light field mapping: Efficient representation and hardware rendering of surface light fields," ACM Trans. Graph., pp. 447-456, July 2002.
15. P. Debevec, Y. Yu, and G. Borshukov, "Efficient view-dependent image-based rendering with projective texture-mapping," in Proc. 9th Eurographics Rendering Workshop, 1998, pp. 105-116.
16. P. E. Debevec, C. J. Taylor, and J. Malik, "Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach," in Proc. ACM Annu. Computer Graphics Conf., Aug. 1996, pp. 11-20.
17. M. Deering, "Geometry compression," in Proc. SIGGRAPH'99, Aug. 1995, pp. 13-20.
18. J. Duan and J. Li, "Compression of the layered depth image," in IEEE Data Compression Conf., Mar. 2001, pp. 331-340.
19. J. Eyles, S. Molnar, J. Poulton, T. Greer, A. Lastra, N. England and L. Westover, "Pixelfiow: The realization," presented at the SIGGRAPH Eurographics Graphics Hardware Workshop, Los Angeles, CA, Aug. 1997.
20. O. Faugeras, Three-Dimensional Computer Vision: A Geometric View-Point. Cambridge, MA: MIT Press, 1993.
21. B. Girod, P. Eisert, M. Magnor, U. Steinback, and T. Wiegand, "3-D image models and compression synthetic hybrid or natural fit?," in IEEE Int. Image Processing Conf., Kobe, Japan, Oct. 1999, pp. 525-529.
22. S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, "The lumigraph," in Proc. ACM Annu. Computer Graphics Conf., New Orleans, LA, Aug. 1996, pp. 43-54.
23. B. Heigl, R. Koch, M. Pollefeys, S. Denzler, and L. Van Goal, "Plenoptic modeling and rendering from image sequences taken by hand-held camera," in Die Deutsche Arbeitsgemeinschaft für Mustererkennung, 1999, pp. 94-101.
24. I. Ihm, S. Park, and R. Lee, "Rendering of spherical light fields," in Pacific Graphics, Seoul, Korea, Oct. 1997, pp. 59-68.
25. M. Irani and S. Peleg, "Improving resolution by image registration," Graph. Models Image Process., vol. 53, no. 3, pp. 231-239, May 1991.
26. A. Isaksen, L. McMillan, and S. Gortler, "Dynamically reparameterized light fields," in Proc. ACM Annu. Computer Graphics Conf., July 2000, pp. 297-306.
27. Digital Compression and Coding of Continuous-Tone Still Images, ISO/IEC Int. Standard DIS 10 918.
28. ISO/IEC JTC I/SC 29/WG II, 1999.
29. JPEG 2000 Image Coding System: Core Coding System [WG 1 N 1646], ISO/IEC JTC 1/SC 29/WG1, ISO/IEC FCD 15 444-1, 2000.
30. Generic Coding of Moving Pictures and Associated Auto Information: Video, ITU-T Rec. H.262/ISO/IEC 1381 8-2, 1994.
31. Video Coding for Low Bit Rate Communication, ITU-T Rec. H.263, 1998.
32. T. Kanade, P. Rander, and P. J. Narayanan, "Virtualized reality: Constructing virtual worlds from real scenes," IEEE Multimedia, vol. 4, pp. 34-47, Jan.-Mar. 1997.
33. S. B. Kang, "A survey of image-based rendering techniques," VideoMetrics, vol. SPIE 3641, pp. 2-16, 1999.
34. S. B. Kang and P. K. Desikan, "Virtual navigation of complex scenes using clusters of cylindrical panoramic images," in Graphics Interface, Vancouver, BC, Canada, June 1998, pp. 223-232.
35. S. B. Kang and R. Szeliski, "3-D scene data recovery using omni- directional multibaseline stereo," in IEEE Computer Vision and Pattern Recognition Soc. Conf., San Francisco, CA, June 1996, pp. 364-370.
36. S. B. Kang, R. Szeliski, and P. Anandan, "The geometry-image representation tradeoff for rendering," in Int. Image Processing Conf., vol. 2, Vancouver, BC, Canada, Sept. 2000, pp. 13-16.
37. A. Katayama, K. Tanaka, T. Oshino, H. Tamura, and S. Fisher, "A viewpoint dependent stereoscopic display using interpolation of multi-viewpoint images," in Proc. SPIE, Stereoscopic Displays and Virtual Reality Systems II, S. Merritt and B Balsa, Eds., 1995, vol. 2409, pp. 11-20.
38. P. Lalonde and A. Fournier, "Interactive rendering of wavelet projected light fields," in Graphics Interface, Kingston, ON, Canada, June 1999, pp. 107-114.
39. S. Laveau and O. Faugeras, "3-D scene representation as a collection of images and fundamental matrices," INRIA-Sophia Antipolis, Sophia Antipolis, Italy, Tech. Rep. 2205, 1994.
40. S. Laveau and O. D. Faugeras, "3-D scene representation as a collection of images," in 12th Pattern Recognition Int. Conf., vol. A, Jerusalem, Israel, Oct. 1994, pp. 689-691.
41. J. Lengyel, "The convergence of graphics and vision," IEEE Computer Soc., Los Alamitos, CA, Tech. Rep., July 1998.
42. H. W. Leung and T. Chen, "Compression with mosaic prediction for image-based rendering applications," presented at the IEEE Int. Multimedia Expo. Conf., New York, NY, July 2000.
43. M. Levoy and P. Hanrahan, "Light field rendering," in Proc. ACM Annu. Computer Graphics Conf., New Orleans, LA, Aug. 1996, pp. 31-42.
44. J. Li, H.-Y. Shum, and Y. Q. Zhang, "On the compression of image based rendering scene," IEEE Int. Image Processing Conf., vol. 2, pp. 21-24, Sept. 2000.
45. M. E. Lukacs, "Predictive coding of multi-viewpoint image sets," in IEEE Int. Acoustics, Speech, and Signal Processing Conf., 1986, pp. 521-524.
46. L. Luo, Y. Wu, J. Li, and Y. Q. Zhang, "Compression of concentric mosaic scenery with alignment and 3D wavelet transform," presented at the SPIE Image and Video Communications and Processing Conf., San Jose, CA, Jan. 2000.
47. M. Manor and B. Girod, "Adaptive block-based light field coding," in 3rd Int. Synthetic and Natural Hybrid Coding and Three-Dimensional Imaging Workshop, Santorini, Greece, Sept. 1999, pp. 140-143.
48. ____, "Hierarchical coding of light fields with disparity maps," presented at the IEEE Int. Image Processing Conf., Kobe, Japan, Oct. 1999.
49. ____, "Data compression for light-field rendering," IEEE Trans. Circuits Syst. Video Technol., vol. 10, pp. 338-343, Apr. 2000.
50. ____, "Model-aided coding of multi-viewpoint image data," in IEEE Int. Image Processing Conf., vol. 2 Vancouver, BC, Canada, Sept. 2000, pp. 919-922.
51. ____, "Model-based coding of multi-viewpoint imagery," in Proc. SPIE Visual Communication and Image Processing Conf., vol. 4067, Perth, Australia, June 2000, pp. 14-22.
52. S. Mann and R. W. Picard, "Virtual bellows: Constructing high-quality images from video," in 1st IEEE Int. Image Processing Conf., vol. I, Austin, TX, Nov. 1994, pp. 363-367.
53. W. Mark, L. McMillan, and G. Bishop, "Post-rendering 3D warping," in Proc. 13D Graphics Symp., 1997, pp. 7-16.
54. W. Matusik, C. Buehler, R. Raskar, S. Gortler, and L. McMillan, "Image-based visual hulls," in Proc. ACM Annu. Computer Graphics Conf., July 2000, pp. 369-374.
55. D. K. McAllister, L. Nyland, V. Popescu, A. Lastra, and C. McCue, "Real-time rendering of real world environments," presented at the Eurographics Rendering Workshop, Granada, Spain, June 1999.
56. L. McMillan, "An image-based approach to three-dimensional computer graphics," Ph.D. dissertation, Dept. Comput Sci., Univ. North Carolina, Chapel Hill, NC, 1999.
57. L. McMillan and U. Bishop, "Plenoptic modeling: An image-based rendering system," in Proc. ACM Annu. Computer Graphics Conf., Aug. 1995, pp. 39-46.
58. G. Miller, S. Rabin, and D. Ponceleon, "Lazy decompression of surface light fields for precomputed global illumination," in Eurographics Rendering Workshop, Oct. 1998, pp. 281-292.
59. T. Naemua, M. Kaneko, and H. Harashima, "Compression and representation of 3-D images," IEICE Trans. Inform. Syst., vol. E82-D, no. 3, pp. 558-567, 1999.
60. V.S. Nalwa, "A true omnidirectional viewer," Bell Labs., Holmdel, NJ, Tech. Rep., 1996.
61. S. Nayar, "Catadioptric omnidirectional camera," in IEEE Computer Soc. Computer Vision and Pattern Recognition Conf., San Juan, PR, June 1997, pp. 482-488.
62. K. T. Ng, S. C. Chan, and H.-Y. Shum, "Scalable coding and progressive transmission of concentric mosaic using nonlinear filter banks," in IEEE Int. Image Processing Conf., vol. 2, Thessaloniki, Greece, Oct. 2001, pp. 113-116.
63. K. T. Ng, S. C. Chan, H.-Y. Shum, and S. B. Kang, "On the data compression and transmission aspects of panoramic video," in IEEE Int. Image Processing Conf., vol. 2, Thessaloniki, Greece, Oct. 2001, pp. 105-108.
64. J. R. Ohm, "Encoding and reconstruction of multiview video objects: Looking at data compression in the context of the MPEG-4 multimedia standard," IEEE Signal Processing Mag., vol. 16, pp. 47-54, May 1999.
65. ____, "Stereo/multiview encoding using the MPEG family of standards," presented at the Electron. Imaging, San Diego, CA, Jan. 1999.
66. M. Oliveira and G. Bishop, "Relief textures," Univ. North Carolina, Chapel Hill, NC, Comput. Sci. Tech. Rep. TR99-015, Mar. 1999.
67. S. Peleg and M. Ben-Ezra, "Stereo panorama with a single camera," presented at the Computer Vision and Pattern Recognition Conf., 1999.
68. S. Peleg and J. Herman, "Panoramic mosaics by manifold projection," in IEEE Computer Soc. Computer Vision and Pattern Recognition Conf., San Juan, PR, June 1997, pp. 338-343.
69. I. Peter and W. Strasser, "The wavelet stream: Interactive multi resolution light field rendering," in Eurographics Rendering Workshop, June 2001, pp. 262-273.
70. A. Puri, R. V. Kollarits, and B. G. Haskell, "Basics of stereoscopic video, new compression results with MPEG-2 and a proposal for MPEG-4," Signal Processing, vol. 10, pp. 201-234, 1997.
71. P. Rademacher, "View-dependent geometry," in SIGGRAPH'99, Aug. 1999, pp. 439-446.
72. P. Rademacher and G. Bishop, "Multiple-center-of-projection images," in Proc. ACM Annu. Computer Graphics Conf., Orlando, FL, July 1998, pp. 199-206.
73. A. Said and W. Pearlman, "A new, fast and efficient image codec based on set partitioning in hierarchical trees," IEEE Trans. Circuits Syst. Video Technol., vol. 6, pp. 243-250, June 1996.
74. D. Scharstein, "Stereo vision for view synthesis," in IEEE Computer Soc. Computer Vision and Pattern Recognition Conf., San Francisco, CA, June 1996, pp. 852-857.
75. S. M. Seitz and C. M. Dyer, "View morphing," in Proc. ACM Annu. Computer Graphics Conf., New Orleans, LA, Aug. 1996, pp. 21-30.
76. J. Shade, S. Gortler, L.-W. He, and R. Szeliski, "Layered depth images," in Proc. ACM Annu. Computer Graphics Conf., Orlando, FL, July 1998, pp. 231-242.
77. H.-Y. Shum and L.-W. He, "Rendering with concentric mosaics," in ACM SIGGRAPH'99, Los Angeles, CA, Aug. 1999, pp. 299-306.
78. H.-Y. Shum, K. T. Ng, and S. C. Chan, "Virtual reality using the concentric mosaic: Construction, rendering and data compression," in IEEE Int. Image Processing Conf., vol. 3, Sept. 2000, pp. 644-647.
79. ____, "A virtual reality system using the concentric mosaic: Construction, rendering, and data compression," IEEE Trans. Multimedia, 2003, to be published.
80. H.-Y. Shum and R. Szeliski, "Construction and refinement of panoramic mosaics with global and local alignment," in 6th Int. Computer Vision Conf., Bombay, India, Jan. 1998, pp. 953-958.
81. P. P. Sloan, M. F. Cohen, and S. J. Gortler, "Time critical lumigraph rendering," in Interactive 3D Graphics Symp., Providence, RI, 1997, pp. 17-23.
82. M. G. Strintzis and S. Malasiotis, "Object-based coding of stereoscopic and 3D image sequences: A review," IEEE Signal Processing Mag., vol. 16, pp. 14-28, May 1999.
83. R. Szeliski, "Video mosaics for virtual environments," IEEE Comput. Graph. Appl., vol. 16, pp. 22-30, Mar. 1996.
84. R. Szeliski and H.-Y. Shum, "Creating full view panoramic image mosaics and texture-mapped models," in Proc. ACM Computer Graphics Conf., Aug. 1997, pp. 251-258.
85. X. Tong and R. M. Gray, "Coding of multi-view images for immersive viewing," in IEEE Int. Acoustics, Speech, Signal Processing Conf., vol. 4, June 2000, pp. 1879-1882.
86. B. Wilburn, M. Smulski, H. H. K. Lee, and M. Horowitz, "The light field video camera," SPIE Electronic Imaging: Media Processors, vol. 4674, pp. 1-8, 2002.
87. T. Wong, P. Heng, S. Or, and W. Ng, "Image-based rendering with controllable illumination," in Proc. 8th Eurographics Rendering Workshop, St. Etienne, France, June 1997, pp. 13-22.
88. D. N. Wood, D. I. Azuma, K. Aldinger, B. Curless, T. Duchamp, D. H. Salesin, and W. Stuetzle, "Surface light fields for 3D photography," in Proc. SIGGRAPH'00, July 2000, pp. 287-296.
89. Y. Wu, L. Luo, J. Li, and Y. Q. Zhang, "Rendering of 3D wavelet compressed concentric mosaic scenery with progressive inverse wavelet synthesis (PIWS)," in Proc. SPIE Visual Commun. Image Processing, vol. 4067, Perth, Australia, June 2000.
90. Y. Wu, C. Zhang, J. Li, and J. Xu, "Smart rebinning for compression of the concentric mosaics," in ACM Int. Multimedia Conf., Los Angeles, CA, Oct. 2000, pp. 201-209.
91. Y. Xiong and K. Turkowski, "Creating image-based VR using a self-calibrating flsheye lens," in IEEE Computer Soc. Computer Vision and Pattern Recognition Conf., San Juan, PR, June 1997, pp. 237-243.
92. H. Yamaguchi, Y. Tatehira, K. Akiyama, and Y. Kobayashi, "Stereoscopic images disparity for predictive coding," in IEEE Int. Acoustics, Speech, Signal Processing Conf., 1989, pp. 1976-1979.
93. J. Yu, L. McMillan, and S. Gortler, "Scan light field rendering," in Proc. Pacific Graphics'02, 2002, pp. 137-144.
94. C. Zhang and T. Chen, "Generalized plenoptic sampling," Carnegie-Mellon Univ., Pittsburgh, PA, Tech. Rep. AMPOI-06, Sept. 2001.
95. C. Zhang and J. Li, "Compression and rendering of concentric mosaics with reference block codec (RBC)," in Proc. SPIE Visual Communication and Image Processing Conf., vol. 4067, Perth, Australia, June 2000.
96. ____, "Compression of lumigraph with multiple reference frame (MRF) prediction and just-in-time rendering," in IEEE Data Compression Conf., Snowbird, UT, Mar. 2000, pp. 254-263.
97. ____, "Interactive browsing of 3D environment over the internet," in Proc. SPIE Visual Communication and Image Processing Conf., vol. 4310, San Jose, CA, Jan. 2001.
98. J. Y. Zheng and S. Tsuji, "Panoramic representation of scenes for route understanding," in Proc. 10th Int. Pattern Recognition Conf., June 1990, pp. 161-167.