An encoder-decoder texture replacement method with application to content-based movie coding

IEEE Transactions on Circuits and Systems for Video Technology

Volumn 14, Issue 6, 2004, Pages 825-840

  Dumitraş, Adriana ^a,b   Haskell, Barry G ^a,b  

^a AT AND T LABS RESEARCH   (United States)

^b APPLE INC   (United States)

Author keywords

Content based; Movie coding; Texture replacement; Texture synthesis; Video compression

Indexed keywords

BIT ERROR RATE; COMPUTER GRAPHICS; COSTS; DECODING; EXPERT SYSTEMS; IMAGE COMPRESSION; MATHEMATICAL MODELS; MOTION PICTURES; VIDEO SIGNAL PROCESSING;

CONTENT-BASED; MOVIE CODING; TEXTURE REPLACEMENT; TEXTURE SYNTHESIS;

ENCODING (SYMBOLS);

EID: 2942672620     PISSN: 10518215     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCSVT.2004.828336     Document Type: Article

References (96)

1. W. Liao and V. O. K. Li, 'The split and merge protocol for interactive video-on-demand," IEEE Multimedia, vol. 4, pp. 51-62, Oct.-Dec. 1997.
2. J. Y. B. Lee, "UVoD: An unified architecture for video-on-demand services," IEEE Commun. Lett., vol. 3, pp. 277-279, Sept. 1999.
3. L. Cloetens, "Broadband access: The last mile," in Proc. IEEE Int. Solid-State Circuits Conf., 2001, pp. 18-21.
4. B. G. Haskell, A. Puri, and A. N. Netravali, Digital Video: An Introduction to MPEG-2. London, U.K.: Chapman & Hall, 1997.
5. R.Koenen, Ed.,"Overview of the MPEG-4 Standard Ver. 18 "ISO/IEC, JTC1/SC29/WG11 (MPEG), Doc. N4030, Mar. 2001.
6. ITU-T Video Coding for Low Bit Rate Communication, H.263, Feb. 1998.
7. "Joint Final Committee Draft of Joint Video Specification ISO/IEC/JTC1/SC29/WG11 (MPEG) 14496-10 Advanced Video Coding and ITU-T Rec. H.264," ISO/TEC, JTC1/SC29/WG11 (MPEG) and TTU-T Video Coding Experts Group (VCEG), July 2002.
8. H. G. Musmann, M. Hötter, and J. Ostermann, "Object-oriented analysis-synthesis coding of moving images," Signal Process. Image Commun., vol. 1, no. 2, pp. 117-138, Nov. 1989.
9. D. E. Pearson, "Developments in model-based video coding," Proc. IEEE, vol. 83, pp. 892-906, June 1995.
10. P. Gerken, "Object-based analysis-synthesis coding of image sequences at very low bit rates," IEEE Trans. Circuits Syst. Video Technol., vol. 4, pp. 228-235, June 1994.
11. M. Hötter, "Optimization and efficiency of an object-oriented analysis-synthesis coder," IEEE Trans. Circuits Syst. Video Technol, vol. 4, pp. 181-194, Apr. 1994.
12. A. Kaup, "Object-based texture coding of moving video in MPEG-4," IEEE Trans. Circuits Syst. Video Technol., vol. 9, pp. 5-15, Feb. 1999.
13. J. Ostermann and M. Kampmann, "Source models for content-based video coding," in Proc. IEEE Int. Conf. Image Processing, vol. 3, 2000, pp. 62-65.
14. J. Ostermann, "Differences between an object-based analysis-synthesis coder and a block-based hybrid coder," in Proc. IEEE Int. Conf. Image Processing, vol. 2, 1995, pp. 398-401.
15. S.-S. Hao, M.-F. Lee, and J.-F. Yang, "Smoothing algorithms for clip-and-paste model-based video coding," IEEE Trans. Consumer Electron., vol. 45, pp. 427-435, May 1999.
16. K.-C. Fan and K.-S. Kan, "An active scene analysis-based approach for pseudoconstant bit-rate video coding," IEEE Trans. Circuits Syst. Video Technol., vol. 8, pp. 159-170, Apr. 1998.
17. T. Chen, C. Swain, and B. G. Haskell, "Coding of subregions for content-based scalable video," IEEE Trans. Circuits Syst. Video Technol., vol. 7, pp. 256-260, Feb. 1997.
18. D. Bordwell and K.Thompson, Film Art. An Introduction. New York: McGraw-Hill, 1997.
19. K. Reisz and G. Millar, The Technique of Film Editing. London, U.K.: Focal Press, 2000.
20. A. M. Dawood and M. Ghanbari, "Scene content classification for MPEG coded bit streams," in Proc. IEEE Workshop Multimedia Signal Processing, 1999, pp. 253-258.
21. P. Campisi, D. Hatzinakos, and A. Neri, "A perceptually lossless, modelbased, texture compression technique," IEEE Trans. Image Processing, vol. 9, pp. 1325-1336, Aug. 2000.
22. B. DeKnuydt, S. Desmet, and L. Van Eycken, "Coding of dynamic texture for mapping on 3-D scenes," IEEE Trans. Circuits Systems Video Technol., vol. 9, pp. 210-217, Mar. 1999.
23. C. Toklu, A. T. Erdem, and A. M. Tekalp, "Two-dimensional mesh-based mosaic representation for manipulation of video objects with occlusion," IEEE Trans. Image Processing, vol. 9, pp. 1617-1630, Sept. 2000.
24. N. Vasconcelos and A. Lippman, "Statistical models of video structure for content analysis and characterization," IEEE Trans. Image Processing, vol. 9, pp. 3-19, Jan. 2000.
25. G. Qiu, "Pattern color separable image coding for content-based indexing," in Proc. IEEE Workshop Multimedia Signal Processing, 1999, pp. 407-412.
26. G. Schaefer and G. Qiu, "Color image coding by visually meaningful image patterns," in Proc. IEEE Int. Conf. Image Processing and Its Applications, vol. 2, 1999, pp. 586-590.
27. Y. Q. Chen, "Novel Techniques for Image Texture Classification, " Ph.D. thesis, Dept. Electron. Comput. Sci., Faculty Eng. Appl. Sci., Univ. Southampton, Southampton, U.K., 1995.
28. P. P. Raghu and B. Yegnanarayana, "Segmentation of Gabor-filtered textures using deterministic relaxation," IEEE Trans. Image Processing, vol. 5, pp. 1625-1636, Dec. 1996.
29. R. M. Haralick, K. Shanmugam, and I. Dinstein, "Textural features for image classification," IEEE Trans. Syst., Man, Cybern., vol. SMC-3, pp. 610-621, Nov. 1973.
30. A. K. Jain and K. Karu, "Learning texture discrimination masks," IEEE Trans. Pattern Anal. Machine Intell., vol. 18, pp. 195-205, Feb. 1996.
31. W. Y. Ma and B. S. Manjunath, "Texture features and learning similarity," presented at the IEEE Int. Conf. Computer Vision and Pattern Recognition, San Francisco, CA, 1996.
32. J. P. Eakins, J. M. Boardman, and M. E. Graham, "Similarity retrieval of trademark images," IEEE Multimedia, vol. 5, pp. 53-63, Apr.-June 1998.
33. H. Huang, Image Repositories, 1997. Preprint, available on the Internet.
34. F. Dell'Acqua and P. Gamba, "Simplified modal analysis and search for reliable shape retrieval," IEEE Trans. Circuits Syst. Video Technol., vol. 8, pp. 654-666, Sept. 1998.
35. X. Wan and C.-C. J. Kuo, "A new approach to image retrieval with hierarchical color clustering," IEEE Trans. Circuits Syst. Video Technol., vol. 8, pp. 628-643, Sept. 1998.
36. C. Y. Yee, K.-L. Tan, T. S. Chua, and B. C. Ooi, "An empirical study of color-spatial retrieval techniques for large image databases," in Proc. IEEE Int. Conf. Multimedia Computing and Systems, Austin, TX, 1998, pp. 218-221.
37. J. Mao and K.-K. Ma, "Semantic spatio-temporal segmentation for extracting video objects," in Proc. IEEE Int. Conf. Multimedia Computing and Systems, vol. 1, 1999, pp. 738-743.
38. W. Tanavapong and S. Krishnamohan, "A characteristics-based bandwidth reduction technique for pre-recorded videos," in Proc. IEEE Int. Conf. Multimedia and Expo, vol. 3, 2000, pp. 1751-1754.
39. A. M. Dawood and M. Ghanbari, "MPEG video model based on scene content," Electron. Lett., vol. 34, no. 9, pp. 868-870, Apr. 1998.
40. _, "MPEG video modeling based on scene description," in Proc. IEEE Int. Conf. Image Processing, vol. 2, 1998, pp. 351-355.
41. T. Liu, X. Zhang, D. Wang, J. Feng, and K.-T. Lo, "Inertia-based cut detection techniques: A step to the integration of video coding and content-based retrieval," in Proc. IEEE Int. Conf. Signal Processing, vol. 2, pp. 1018-1025.
42. T. Keaton and R. Goodman, "A compression framework for content analysis," in Proc. IEEE Workshop Content-Based Access of Image and Video Libraries, 1999, pp. 69-73.
43. N. Vasconcelos and A. Lippman, "Libray-based coding: A representation for efficient video compression and retrieval," in Proc. Data Compression Conf., 1997, pp. 121-130.
44. _, "Content-based pre-indexed video," in Proc. IEEE Int. Conf. Image Processing, vol. 2, 1997, pp. 542-545.
45. H. J. Zhang, J. Y. A. Wang, and Y. Altunbasak, "Content-based video retrieval and compression: A unified solution," in Proc. IEEE Int. Conf. Image Processing, vol. 1, 1997, pp. 13-16.
46. R. S. Jasinschi and J. M. F. Moura, "Content-based video sequence representation," in Proc. IEEE Int. Conf. Image Processing, vol. 2, 1995, pp. 229-232.
47. J. D. Foley, A. van Dam, S. K. Feiner, and J. F. Hughes, Computer Graphics Principles and Practice. Reading, MA: Addison-Wesley, 1997,
48. A. Watt and M. Watt, Advanced Animation and Rendering Techniques. Theory and Practice. Reading, MA: Addison-Wesley, 1992.
49. A. Watt, 3D Computer Graphics. Reading, MA: Addison-Wesley, 2000.
50. Z. Bar-Joseph, "Statistical learning of multi-dimensional textures," Master's thesis, Inst. Comp. Sci., The Hebrew Univ. of Jerusalem, Jerusalem, Israel, Apr. 1999.
51. D. J. Heeger and J. R. Bergen, "Pyramid-based texture analysis/synthesis," in Proc. ICIP, vol. 3,1995, pp. 648-651.
52. J. S. DeBonet, "Multiresolution sampling procedure for analysis and synthesis of texture images," in Proc. SIGGRAPH, Los Angeles, CA, 1997, pp. 361-368.
53. A. A. Efros and T. K. Leung, "Texture synthesis by nonparametric sampling," in Proc. IEEE Int. Conf. Computer Vision, Corfu, Greece, 1999, pp. 1033-1038.
54. J. Portilla and E. P. Simoncelli, "A parametric texture model based on joint statistics of complex wavelet coefficients," Int. J. Comput. Vis., vol. 40, no. 1, pp. 49-71, 2000.
55. Z. Bar-Joseph, R. El-Yaniv, D. Lischinski, and M. Werman, "Texture mixing and texture movie synthesis using statistical learning," IEEE Trans. Visual. Comput. Graphics, vol. 7, pp. 120-135, Apr.-June 2001.
56. H. Greenspan, S. Belongie, R. Goodman, and P. Perona, "Rotation invariant texture recognition using a steerable pyramid," in Proc. IEEE Int. Conf. Pattern Recognition: Computer Vision and Image Processing, vol. 2, 1994, pp. 162-167.
57. H. Greenspan, S. Belongie, R. Goodman, P. Perona, S. Rakshit, and C. H. Anderson, "Overcomplete steerable pyramid filters and rotation invariance," in Proc. IEEE Int. Conf. Pattern Recognition: Computer Vision and Image Processing, 1994, pp. 222-228.
58. J. A. Cadzow, D. M. Wilkes, R. A. Peters II, and Z. Li, "Image texture synthesis-by-analysis using moving-average models," IEEE Trans. Aerosp. Electron. Syst., vol. 29, pp. 1110-1122, Oct. 1993.
59. G. Jacovitti, A. Neri, and G. Scarano, "Texture synthesis-by- analysis with hard-limited gaussian processes," IEEE Trans. Image Processing, vol. 7, pp. 1615-1621, Nov. 1998.
60. S. Horbelt, P. Thevenaz, and M. Unser, "Texture mapping by successive refinement," in Proc. IEEE Int. Conf. Image Processing, vol. 2, 2000, pp. 307-310.
61. D. F. DeBry, "Computer graphics system for creating and enhancing texture maps," U.S. Patent 5 838 331, Nov. 17, 1998.
62. T. Sakaibara, T. Kawasaki, and T. Kuwana, "Method and apparatus for mapping texture on an object displayed at a varying view angle from an observer," U.S. Patent 5 786 822, July 28, 1998.
63. B. A. Alcorn and D. N. Emmot, "System for managing texture mapping data in a computer graphics system," U.S. Patent 5 886 706, Mar. 23, 1999.
64. S.-G. Choi, "Texture mapping system and method using a texture cache for minimum capacity and minimum access time," U.S. Patent 6 111 585, Aug. 29, 2000.
65. C. P. Gossett, "Computer graphics system and method for texture mapping using triangular interpolation," U.S. Patent 5 844 567, Dec. 1, 1998.
66. M. Suzuoki and M. Furuhashi, "Texture mapping method in which 3-D image data is transformed into 2-D data and mapped onto a surface of an object for display," U.S. Patent 5 774 125, June 30, 1998.
67. T. Ueda, "Texture mapping apparatus," U.S. Patent 5 029 225, July 2, 1991.
68. S. Yang, "Method and apparatus for efficient digital modeling and texture mapping," U.S. Patent 5719599, Feb. 17, 1998.
69. W. L. Luken and C. Narayanswami, "Computer graphics system having efficient texture mapping with perspective correction," U.S. Patent 6 057 851, May 2, 2000.
70. R. Szeliski and H.-Y. Shum, "Texture map construction method and apparatus for displaying panoramic image mosaics," U.S. Patent 6 009 190, Dec. 28, 1999.
71. G. Wolberg and T. Boult, "Separable image warping methods and systems using spatial lookup tables," U.S. Patent 5 204 944, Apr. 20, 1993.
72. K. Kaneko, "Texture mapping apparatus computing texture address by fill address," U.S. Patent 6 754 237, June 1998.
73. M. Vecchione, "Method and apparatus for mapping surface texture," U.S. Patent 5 333 245, July 26, 1994.
74. J. P. Ewins, M. D. Waller, M. White, and P. F. Lister, "Mip-map level selection for texture mapping," IEEE Trans. Visual. Comput. Graphics, vol. 4, pp. 317-329, Oct.-Dec. 1998.
75. G. Sannier and N. M. Thalmann, "A user-friendly texture-fitting methodology for virtual humans," in Proc. IEEE Int. Conf. Computer Graphics, 1997, pp. 167-176.
76. M. Oka and M. Suzuoki, "Method and apparatus for mapping texture," U.S. Patent 5933 148, Aug. 3, 1999.
77. G. P. Vaswani, D. P. Wilde, and T. A. Dye, "Dynamically selectable texture filter for computer graphics," U.S. Patent 6 130 674, Oct. 10, 2000.
78. T. Yutaka, M. Suzuoki, M. Furuhashi, and M. Tanaka, "Method and apparatus for generating images," U.S. Patent 5 659 672, Aug. 19, 1997.
79. J. J. Kichury, "Blended texture illumination mapping," U.S. Patent 6057850, May 2, 2000.
80. K. Matsui, "Texture mapping method for placing texture pieces successively on object," U.S. Patent 5 657 432, Aug. 12, 1997.
81. H. P. A. Lensch, W. Heidrich, and H.-P. Seidel, "Automated texture registration and stitching for real world models," in Proc. IEEE Conf. Computer Graphics and Applications, 2000, pp. 317-326.
82. M. Adachi and K. T. Miura, "Modification of n-sided patches based on variation of blending functions," in Proc. IEEE 5th Pacific Conf. Computer Graphics and Applications, 1997, pp. 91-220.
83. D. DiFrancesco, "Method for borderless mapping of texture images," U.S. Patent 5 194 969, Mar. 16, 1993.
84. A. Dumitraş and B. G. Haskell, "A texture replacement method at the encoder for bit-rate reduction of compressed video," IEEE Trans. Circuits Syst. Video Technol., vol. 13, pp. 163-175, Feb. 2003.
85. B. Julesz and R. Bergen, "Testons, the fundamental elemenets in preattentive vision and perception of textures," Bell Syst. Tech. J., vol. 62, pp. 1619-1645, July-Aug. 1983.
86. A. Mojsilovic, J. Kovacevic, D. Kall, R. J. Safranek, and S. K. Ganapathy, "The vocabulary and grammar of color patterns," IEEE Trans. Image Processing, vol. 9, pp. 417-431, Mar. 2000.
87. P. Brigger, J. Hoeg, and M. Unser, "B-spline snakes: A flexible tool for parametric contour detection," IEEE Trans. Image Processing, vol. 9, pp. 1484-1496, Sept. 2000.
88. J. Mannos and D. Sakrison, "The effects of a visual fidelity criterion on the encoding of images," IEEE Trans. Inform. Theory, vol. 20, pp. 525-535, July 1974.
89. T. Mitsa and K. Varkur, "Evaluation of contrast sensitivity functions for the formulation of quality measures incorporated in halftoning algorithms," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, vol. 5, 1993, pp. 301-304.
90. N. Damera-Venkata, T. D. Kite, W. S. Geisler, B. L. Evans, and A. C. Bovik, "Image quality assessment based on a degradation model," IEEE Trans. Image Processing, vol. 9, pp. 636-650, Apr. 2000.
91. T. Weyers and J. J. D. Van Shalkwyk, "Picture quality assessment model," Electron. Lett., vol. 29, pp. 287-288, Feb. 1993.
92. D. Lauzon, A. Vincent, and L. Wang, "Performance evaluation of MPEG-2 video coding for HDTV," IEEE Trans. Broadcast, vol. 42, pp. 88-94, June 1996.
93. L. Contin and S. Battista, "Performance evaluation of video coding schemes working at very low bit rates," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, vol. 5, 1994, pp. 409-412.
94. M. Ardito, M. Gunetti, and M. Visca, "Influence of display parameters on perceived HDTV quality," IEEE Trans. Consumer Electron., vol. 42, pp. 145-155, Feb. 1996.
95. S. H. Tan, K. K. Pang, and K. N. Ngan, "Classified perceptual coding with adaptive quantization," IEEE Trans. Circuits Syst. Video Technol., vol. 6, pp. 375-388, Aug. 1996.
96. P. Daras, T. Raptis, I. Kompatsiaris, and M. Strintzis, "MPEG-4 Toolbox v1.0, Authoring Toolbox Software," Informatics and Telematics Inst., Thessaloniki, Greece, 2001.