Perceptual video compression: A survey

IEEE Journal on Selected Topics in Signal Processing

Volumn 6, Issue 6, 2012, Pages 684-697

  Lee, Jong Seok ^a   Ebrahimi, Touradj ^b  

^a Yonsei University   (South Korea)

^b EPFL   (Switzerland)

Author keywords

Coding efficiency; focus of attention; human visual system; perceptual video compression; quality of experience (QoE); region of interest; visual perception

Indexed keywords

CODING EFFICIENCY; FOCUS OF ATTENTION; HUMAN VISUAL SYSTEM; QUALITY OF EXPERIENCE (QOE); REGION OF INTEREST; VISUAL PERCEPTION;

QUALITY OF SERVICE;

IMAGE COMPRESSION;

EID: 84866521527     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2012.2215006     Document Type: Article

References (105)

1. D. Agrafiotis, N. Canagarajah, D. R. Bull, and M. Dye, "Perceptually optimised sign language video coding based on eye tracking analysis," Electron. Lett., vol. 39, no. 24, pp. 1703-1705, Nov. 2003.
2. D. Agrafiotis, S. J. C. Davies, N. Canagarajah, and D. R. Bull, "Towards efficient context-specific video coding based on gaze-tracking analysis," ACMTrans. Multimedia Comput. Commun. Appl., vol. 3, no. 4, pp. 22:1-22:15, Dec. 2007.
3. J. Ballé, A. Stojanovic, and J.-R. Ohm, "Models for static and dynamic texture synthesis in image and video compression," IEEE J. Sel. Topics Signal Process., vol. 5, no. 7, pp. 1353-1365, Nov. 2011.
4. G. Boccignone, A. Marcelli, P. Napoletano, G. D. Fiore, G. Iacovoni, and S. Morsa, "Bayesian integration of face and low-level cues for foveated video coding," IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 12, pp. 1727-1740, Dec. 2008.
5. M. Bosch, F. Zhu, and E. J. Delp, "Segmentation-based video compression using texture and motion models," IEEE J. Sel. Topics Signal Process., vol. 5, no. 7, pp. 1366-1377, Nov. 2011.
6. B. Breitmeyer and H. Ö?gmen, Visual Masking: Time Slices through Conscious and Unconscious Vision, 2nd ed. Oxford, U.K.: Oxford Univ. Press, 2006.
7. X. Cai, F. H. Ali, and E. Stipidis, "Object-based video coding with dynamic quality control," Image Vis. Comput., vol. 28, no. 3, pp. 285-297, Mar. 2010.
8. R. Carmi and L. Itti, "The role of memory in guiding attention during natural vision," J. Vision, vol. 6, no. 9, pp. 898-914, Aug. 2006.
9. A. Cavallaro, O. Steiger, and T. Ebrahimi, "Semantic video analysis for adaptive content delivery and automatic description," IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 10, pp. 1200-1209, Oct. 2005. (Pubitemid 41640558)
10. M. Cerf, J. Harel, W. Einhauser, and C. Koch, "Predicting human gaze using low-level saliency combined with face detection," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2007, vol. 20.
11. D. Chai and K. N. Ngan, "Face segmentation using skin-color map in videophone applications," IEEE Trans. Circuits Syst. Video Technol., vol. 9, no. 4, pp. 551-934 Jun. 1999.
12. Q. Chen, X. Yang, L. Song, and W. Zhang, "Robust video region-ofinterest coding based on leaky prediction," IEEE Trans. Circuits Syst. Video Technol., vol. 19, no. 9, pp. 1389-1394, Sep. 2009.
13. Q. Chen, G. Zhai, X. Yang, and W. Zhang, "Application of scalable visual sensitivity profile in image and video coding," in Proc. IEEE Int. Symp. Circuits Syst., Seattle, WA, May 2008, pp. 268-271.
14. Z. Chen and C. Guillemot, "Perceptually-friendly H.264/AVC video coding based on foveated just-noticeable distortion model," IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 6, pp. 806-819 Jun. 2010.
15. Z. ChPren, J. Han, and K. N. Ngan, "Dynamic bit allocation for multiple video object coding," IEEE Trans. Multimedia, vol. 8, no. 6, pp. 1117-1124, Dec. 2006. (Pubitemid 44955677)
16. Z. Chen, M. Li, and Y.-P. Tan, "Perception-aware multiple scalable video streaming over WLANs," IEEE Signal Process. Lett., vol. 17, no. 7, pp. 675-678, Jul. 2010.
17. Z. Chen, W. Lin, and K. N. Ngan, "Perceptual video coding: Challenges and approaches," in Proc. ICME, Singapore, Jul. 2010, pp. 784-789.
18. M.-C. Chi, M.-J. Chen, C.-H. Yeh, and J.-A. Jhu, "Region-of-interest video coding based on rate and distortion variations for H.263+," Signal Process.: Image Commun., vol. 23, pp. 127-142, 2008.
19. M.-C. Chi, C.-H. Yeh, and M.-J. Chen, "Robust region-of-interest determination based on user attention model through visual rhythm analysis," IEEE Trans. Circuits Syst. Video Technol., vol. 19, no. 7, pp. 1025-1038, Jul. 2009.
20. C.-H. Chou and C.-W. Chen, "A perceptually optimized 3-D sub-band codec for video communication over wireless channels," , vol. 6, no. 2, pp. 143-156, Apr. 1996. (Pubitemid 126786260)
21. C.-H. Chou and Y.-C. Li, "A perceptually tuned sub-band image coder based on the measure of just-noticeable-distortion profile," IEEE Trans. Circuits Syst. Video Technol., vol. 5, no. 6, pp. 467-476, Dec. 1995.
22. M. Corbetta and G. L. Shulman, "Control of goal-oriented and stimulus-driven attention in the brain," Nature Reviews Neurosci., vol. 3, no. 3, pp. 201-215, Mar. 2002. (Pubitemid 135706590)
23. S. Daly and J. Ribas-Corbera, "As plain as the noise on your face: Adaptive video compression using face detection and visual eccentricity models," J. Electron. Imaging, vol. 10, no. 1, pp. 30-46, Jan. 2001. (Pubitemid 32279359)
24. C. Dikici and H. I. Bozma, "Attention-based video streaming," Signal Process.: Image Commun., vol. 25, no. 10, pp. 745-760, Nov. 2010.
25. N. Doulamis, A. Doulamis, D. Kalogeras, and S. Kollias, "Low bit-rate coding of image sequences using adaptive regions of interest," IEEE Trans. Circuits Syst. Video Technol., vol. 8, no. 8, pp. 928-934, Dec. 1998. (Pubitemid 128743744)
26. J. Driver, "A selective review of selective attention research from the past century," British J. Psychol., vol. 92, pp. 53-78, Feb. 2001. (Pubitemid 33602718)
27. A. Dumitras and B. G. Haskell, "An encoder-decoder texture replacement method with application to content-based movie coding," IEEE Trans. Circuits Syst. Video Technol., vol. 14, no. 6, pp. 825-840 Jun. 2004.
28. J. T. Enns and V. D. Lollo, "What's new in visual masking," Trends in Cognitive Sci., vol. 4, no. 9, pp. 345-352, Sep. 2000.
29. S. Frintrop, E. Rome, and H. I. Christensen, "Computational visual attention systems and their cognitive foundations:A survey," ACMTrans. Applied Percept., vol. 7, no. 1, pp. 6:1-6:39, Jan. 2010.
30. W. S. Geisler and J. S. Perry, "A real-time foveated multiresolution system for low-bandwidth video communication," in Proc. SPIE, San Jose, CA, Jan. 1998, vol. 3299, pp. 294-305.
31. D. Gibson, M. Spann, and S. I. Woolley, "A wavelet-based region of interest encoder for the compression of angiogram video sequences," IEEE Trans. Inf. Technol. Biomed., vol. 8, no. 2, pp. 103-113 Jun. 2004.
32. R. B. Goldstein, R. L. Woods, and E. Peli, "Where people look when watching movie: Do all viewers look at the same place?," Comput. Biol. Med., vol. 37, no. 7, pp. 957-964, Jul. 2007. (Pubitemid 46617479)
33. C. Guo and L. Zhang, "A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression," IEEE Trans. Image Process., vol. 19, no. 1, pp. 185-198, Jan. 2010.
34. U. Hasson, O. Landesman, B. Knappmeyer, I. Vallines, N. Rubin, and D. J. Heeger, "Neurocinematics: The neuroscience of film," Projections, vol. 2, no. 1, pp. 1-26, 2009.
35. S. S. Hemami and A. R. Reibman, "No-reference image and video quality estimation: Applications and human-motivated design," Signal Process.: Image Commun., vol. 25, no. 7, pp. 469-481, Aug. 2010.
36. O. Hershler and S. Hochstein, "At first sight: A high-level pop out effects for faces," Vis. Res., vol. 45, no. 13, pp. 1707-1724 Jun. 2005. (Pubitemid 40403830)
37. C.-C. Ho, J.-L. Wu, and W.-H. Cheng, "A practical foveation-based rate-shaping mechanism for MPEG videos," IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 11, pp. 1365-1372, Nov. 2005. (Pubitemid 41640545)
38. Information Technology-Coding of Audio/Visual Objects, Part 2: Visual, ISO/IEC 14496-2:2004, 2004.
39. L. Itti, "Automatic foveation for video compression using a neurobiological model of visual attention," IEEE Trans. Image Process., vol. 13, no. 10, pp. 1304-1318, Oct. 2004.
40. A. K. Kannur and B. Li, "An enhanced rate control scheme with motion assisted slice grouping for low bit rate coding in H.264," in Proc. ICIP, San Diego, CA, Oct. 2008, pp. 2100-2103.
41. A. K. Kannur and B. Li, "Power-aware content-adaptive H.264 video encoding," in Proc. ICASSP, Taipei, Taiwan, Apr. 2009, pp. 925-928.
42. L. S. Karlsson and M. Sjöström, "Improved ROI video coding using variable Gaussian pre-filters and variance in intensity," in Proc. ICIP, Genoa, Italy, Sep. 2006, vol. 2, pp. 313-316. (Pubitemid 44538939)
43. D. H.Kelly, "Motion and vision. II. Stabilzed spatio-temporal threshold surface," J. Opt. Soc. Amer., vol. 69, no. 10, pp. 1340-1349, Oct. 1979.
44. A. Khan, L. Sun, and E. Ifeachor, "QoE prediction model and its application in video quality adaptation over UMTS networks," IEEE Trans. Multimedia, vol. 14, no. 2, pp. 431-442, Apr. 2012.
45. E. I. Knudsen, "Fundamental components of attention," Annu. Rev. Neurosci., vol. 30, pp. 57-78, 2007. (Pubitemid 47218751)
46. O. V.Komogortsev and J. I. Khan, "Predictive real-time perceptualcompression based on eye-gaze-position analysis," ACMTrans. Multimedia Comput. Commun. Appl., vol. 4, no. 3, pp. 23:1-23:16, Sep. 2008.
47. J.-S. Lee and T. Ebrahimi, "Efficient video coding in H.264/AVC by using audio-visual information," in Proc. IEEE Int. Conf. Multimedia Signal Process., Rio de Janeiro, Brazil, Oct. 2009, pp. 1-6.
48. J.-S. Lee, F. D. Simone, and T. Ebrahimi, "Influence of audio-visual attention on perceived quality of standard definition multimedia content," in Proc. Int. Workshop Quality of Multimedia Experience, San Diego, CA, Jul. 2009, pp. 13-19.
49. J.-S. Lee, F. D. Simone, and T. Ebrahimi, "Video coding based on audio-visual attention," in Proc. Int. Conf. Multimedia and Expo, New York Jun. 2009, pp. 57-60.
50. J.-S. Lee, F. D. Simone, and T. Ebrahimi, "Efficient video coding based on audio-visual focus of attention," J. Vis. Commun. Image R., vol. 22, no. 8, pp. 704-711, Nov. 2011.
51. J.-S. Lee, F. D. Simone, and T. Ebrahimi, "Subjective quality evaluation of foveated video coding using audio-visual focus of attention," IEEE J. Sel. Topics Signal Process., vol. 5, no. 7, pp. 1322-1331, Nov. 2011.
52. J.-S. Lee, F. D. Simone, N. Ramzan, E. Izquierdo, and T. Ebrahimi, "Quality assessment of multidimensional video scalability," IEEE Commun. Mag., vol. 50, no. 4, pp. 38-46, Apr. 2012.
53. S. Lee and A. C. Bovik, "Fast algorithms for foveated video processing," IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 2, pp. 149-162, Feb. 2003.
54. S. Lee, A. C. Bovik, and Y. Y. Kim, "High quality, low delay foveated visual communications over mobile channels," J. Vis. Commun. Image R., vol. 16, no. 2, pp. 180-211, Apr. 2005.
55. S. Lee, M. S. Pattichis, and A. C. Bovik, "Foveated video compression with optimal rate control," IEEE Trans. Image Process., vol. 10, no. 7, pp. 977-992, Jul. 2001. (Pubitemid 32650874)
56. S. Lee, M. S. Pattichis, and A. C. Bovik, "Foveated video quality assessment," IEEE Trans. Multimedia, vol. 4, no. 1, pp. 129-132, Mar. 2002. (Pubitemid 34291535)
57. Z. Li, S. Qin, and L. Itti, "Visual attention guided bit allocation in video compression," Image Vis. Comput., vol. 29, no. 1, pp. 1-14, Jan. 2011.
58. Y. Liu, Z. G. Li, and Y. C. Soh, "Region-of-interest based resource allocation for conversational video communication of H.264/AVC," IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 1, pp. 134-139, Jan. 2008.
59. E. Macaluso and J. Driver, "Multisensory spatial interactions: A window onto functional integration in the human brain," Trends in Neurosci., vol. 28, no. 5, pp. 264-271, May 2005. (Pubitemid 41556911)
60. M. G. Martini and C. T. E. R. Hewage, "Flexible macroblock ordering for context-aware ultrasound video transmission over mobile WiMAX," Int. J. Telemed. Appl., vol. 2010, pp. 1-14, 2010.
61. V. Mazza, M. Turatto, M. Rossi, and C. Umilta, "How automatic are audiovisual links in exogenous spatial attention?," Neuropsychologia, vol. 45, no. 3, pp. 514-522, 2007. (Pubitemid 44894459)
62. J. D. McCarthy, M. A. Sasse, and D. Miras, "Sharp or smooth? Comparing the effects of quantization vs. frame rate for streamed video," in Proc. SIGCHI Conf. Human Factors in Comput. Syst., Vienna, Austria, Apr. 2004, pp. 535-542.
63. S. Nasir, C. T. E. R. Hewage, Z. Ahmad, M. Mrak, S. Worrall, and A. Kondoz, "Quality-driven coding and prioritization of 3D video over wireless networks," in High-Quality Visual Experience, ser. Signals and Communication Technology, M. Mrak, M. Grgic, and M. Kunt, Eds. New York: Springer, 2010, ch. 21, pp. 477-495.
64. P. Ndjiki-Nya, D. Bull, and T. Wiegand, "Perception-oriented video coding based on texture analysis and synthesis," in Proc. ICIP, Cairo, Egypt, Nov. 2009, pp. 2273-2276.
65. P. Ndjiki-Nya, B. Mokai, G. Blättermann, A. Smolic, H. Schwarz, and T. Wiegand, "Improved H.264/AVC coding using texture analysis and synthesis," in Proc. ICIP, Barcelona, Spain, Sep. 2003, pp. 849-852.
66. E. Nguyen, C. Labit, and J.-M. Odobez, "A ROI approach for hybrid image sequence coding," in Proc. ICIP, Austin, TX, Nov. 1994, vol. 3, pp. 245-249.
67. M. Nyström and K. Holmqvist, "Effect of compressed offline foveated video on viewing behavior and subjective quality," ACM Trans. Multimedia Comput. Commun. Appl., vol. 6, no. 1, pp. 1-14, 2010.
68. B. T. Oh, Y. Su, A. Segall, and C.-C. J. Kuo, "Synthesis-based texture coding for video compression with side information," in Proc. ICIP, San Diego, CA, Oct. 2008, pp. 1628-1683.
69. W. Osberger, A. J. Maeder, and N. Bergmann, "A perceptually based quantization technique for MPEG encoding," in Proc. SPIE, San Jose, CA, Jan. 1998, vol. 3299, pp. 148-159.
70. "Methodology for the subjective assessment of the quality of television pictures," Rec. ITU-R BT.500-11, 2002.
71. "Subjective video quality assessment methods for multimedia applications," Rec. ITU-R P.910, 1999.
72. S. P. Roi, N. S. Jayant, M. E. Stachura, E. Astapova, and A. Pearson-Shaver, "Delivering diagnostic quality video over mobile wireless networks for telemedicine," Int. J. Telemed. Appl., vol. 2009, pp. 1-9, 2009.
73. D. M. Saxe and R. A. Foulds, "Robust region of interest coding for improved sign language telecommunication," IEEE Trans. Inf. Technol. Biomed., vol. 6, no. 4, pp. 310-316, Dec. 2002.
74. H. Schwarz, D. Marpe, and T. Wiegand, "Overview of the scalable video coding extension of the H.264/AVC standard," IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 9, pp. 1103-1120, Sep. 2007. (Pubitemid 47483052)
75. R. Sharma, V. I. Pavlović, and T. S. Huang, "Toward multimodal human-computer interface," Proc. IEEE, vol. 86, no. 5, pp. 853-869, May 1998. (Pubitemid 128720228)
76. H. R. Sheikh, B. L. Evans, and A. C. Bovik, "Real-time foveation techniques for low bit rate video coding," Real-Time Imaging, vol. 9, pp. 27-40, 2003.
77. L. Shen, Z. Liu, Z. Zhang, and H. Yan, "Motion attention based frame-level bit allocation scheme for H.264," in Proc. Int. Conf. Internet Multimedia Comput. Service, Kunming, Yunnan, China, Nov. 2009.
78. D. J. Simons and C. F. Chabris, "Gorillas in our midst: Sustained inattentional blindness for dynamic events," Perception, vol. 28, no. 9, pp. 1058-1074, 1999.
79. C. Spence, "Crossmodal spatial attention," Ann. New York Acad. Sci., vol. 1191, pp. 182-200, Mar. 2010.
80. C. Spence and J. Driver, "Audiovisual links in exogenous covert spatial orienting," Percept. Psychophys., vol. 59, no. 1, pp. 1-22, 1997. (Pubitemid 127456363)
81. L. Stelmach, W. Tam, and P. Hearty, "Static and dynamic spatial resolution in image coding: An investigation of eye movements," in Proc. SPIE, San Jose, CA, Feb. 1992, vol. 1453, pp. 147-152. (Pubitemid 21688317)
82. Y. Sun, I. Ahmad, D. Li, and Y.-Q. Zhang, "Region-based rate control and bit allocation for wireless video transmission," IEEE Trans. Multimedia, vol. 8, no. 1, pp. 1-10, Feb. 2006.
83. C.-W. Tang, "Spatiotemporal visual considerations for video coding," IEEE Trans. Multimedia, vol. 9, no. 2, pp. 231-238, 2007. (Pubitemid 46188196)
84. C.-W. Tang, C.-H. Chen, Y.-H. Yu, and C.-J. Tsai, "Visual sensitivity guided bit allocation for video coding," IEEE Trans. Multimedia, vol. 8, no. 1, pp. 11-18, Feb. 2006. (Pubitemid 43146126)
85. D. J. Tellinghuisen and E. J. Nowak, "The inability to ignore auditory distractors as a function of visual task perceptual load," Percept. Psychophys., vol. 65, no. 5, pp. 817-828, 2003. (Pubitemid 37544856)
86. L. Tong and K. R. Rao, "Region-of-interest based rate control for lowbit-rate video conferencing," J. Electron. Imaging, vol. 15, no. 3, pp. 1-12, 2006.
87. N. Tsapatsoulis, C. Loizou, and C. Pattichis, "Region of interest video coding for low bit-rate transmission of carotid ultrasound videos over 3G wireless networks," in Proc. Int. Conf. IEEE Eng. Med. Biol. Soc., Lyon, France, Aug. 2007, pp. 3717-3720.
88. N. Tsapatsoulis, K. Rapantzikos, and C. Pattichis, "An embedded saliency map estimator scheme: Application to video coding," Int. J. Neural Syst., vol. 17, no. 4, pp. 289-304, 2007. (Pubitemid 47258350)
89. H. Wang, Y. Liang, and K. El-Maleh, "Real-time region-of-interest video coding using content-adaptive background skipping with dynamic bit reallocation," in Proc. ICASSP, Toulouse, France, May 2006, pp. 45-48.
90. M.Wang, T. Zhang, C. Liu, and S. Goto, "Region-of-interest based dynamical parameter allocation for H.264/AVC encoder," in Proc. Picture Coding Symp., Chicago, IL, May 2009, pp. 1-4.
91. Z. Wang and A. C. Bovik, "Embedded foveation image coding," IEEE Trans. Image Process., vol. 10, no. 10, pp. 1397-1410, Oct. 2001. (Pubitemid 32981815)
92. Z. Wang, A. C. Bovik, and B. L. Evans, "Blind measurement of blocking artifacts in images," in Proc. ICIP, Vancouver, BC, Canada, Oct. 2000, vol. 3, pp. 981-984.
93. Z. Wang, L. Lu, and A. C. Bovic, "Foveation scalable video coding with automatic fixation selection," IEEE Trans. Image Process., vol. 12, no. 2, pp. 243-254, Feb. 2003.
94. Z. Wang, L. Lu, and A. C. Bovik, "Video quality assessment based on structural distortion measurement," Signal Process.: Image Commun., vol. 19, no. 2, pp. 121-132, Feb. 2004.
95. S. Winkler and C. Faller, "Perceived audiovisual quality of low bitrate multimedia content," IEEE Trans. Multimedia, vol. 8, no. 5, pp. 973-980, Oct. 2006. (Pubitemid 44523111)
96. S.Winkler and M. Mohandas, "The evolution of video quality measurement: From PSNR to hybrid metrics," IEEE Trans. Broadcast., vol. 54, no. 3, pp. 660-668, Sep. 2008.
97. L. Yang, L. Zhang, S. Ma, and D. Zhao, "A ROI quality adjustable rate control scheme for low bitrate video coding," in Proc. Picture Coding Symp., Chicago, IL, May 2009, pp. 1-4.
98. X. Yang, W. Lin, Z. Lu, X. Lin, S. Rahardja, E. Ong, and S. Yao, "Rate control for videophone using local perceptual cues," IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 4, pp. 496-507, Apr. 2005.
99. A. Yarbus, Eye Movements and Vision. New York: Plenum, 1967.
100. J. You, U. Reiter,M.M. Hannuksela, M. Gabbouj, and A. Perkis, "Perceptual-based quality assessment for audio-visual services: A survey," Signal Process.: Image Commun., vol. 25, no. 7, pp. 482-501, Aug. 2010.
101. G. Zhai, J. Cai, W. Lin, X. Yang, and W. Zhang, "Three dimensional scalable video adaptation via user-end perceptual quality assessment," IEEE Trans. Broadcast., vol. 54, no. 3, pp. 719-727, Sep. 2008.
102. G. Zhai, Q. Chen, X.Yang, and W. Zhang, "Scalable visual significance profile estimation," in Proc. ICASSP, Las Vegas, NV, Apr. 2008, pp. 268-271.
103. F. Zhang and D. R. Bull, "A parametric framework for video compression using region-based texture models," IEEE J. Sel. Topics Signal Process., vol. 5, no. 7, pp. 1378-1392, Nov. 2011.
104. Y. Zhang, G. Jiang, M. Yu, Y. Yang, Z. Peng, and K. Chen, "Depth perceptual region-of-interest based multiview video coding," J. Vis. Commun. Image R., vol. 21, no. 5-6, pp. 498-512, 2010.
105. C. Zhu, X. Sun, F.Wu, and H. Li, "Video coding with spatio-temporal texture synthesis and edge-based inpainting," in Proc. ICME, Hannover, Germany Jun. 2008, pp. 813-816.