Objective video quality assessment methods: A classification, review, and performance comparison

IEEE Transactions on Broadcasting

Volumn 57, Issue 2 PART 1, 2011, Pages 165-182

  Chikkerur, Shyamprasad ^a   Sundaram, Vijay ^a   Reisslein, Martin ^a   Karam, Lina J ^a  

^a Arizona State University   (United States)

Author keywords

Full reference metric; objective video quality; perceptual video quality; reduced reference metric

Indexed keywords

CHARACTERISTICS METHOD; CLASSIFICATION SCHEME; FREQUENCY DOMAINS; FULL-REFERENCE METRIC; HIGH DEFINITION VIDEO; HUMAN VISUAL SYSTEM; INTEGRITY EVALUATIONS; LIVE VIDEO; OBJECTIVE VIDEO QUALITY; PERCEPTUAL VIDEO QUALITY; PERFORMANCE COMPARISON; PIXEL DOMAIN; REDUCED REFERENCE; SIMILARITY INDICES; SPATIO-TEMPORAL; VIDEO QUALITY; VIDEO QUALITY ASSESSMENT; VIDEO QUALITY METRIC; VISUAL FEATURE;

COMPUTATIONAL COMPLEXITY; RATING; VIDEO SIGNAL PROCESSING;

IMAGE QUALITY;

EID: 79957780748     PISSN: 00189316     EISSN: None     Source Type: Journal    
DOI: 10.1109/TBC.2011.2104671     Document Type: Article

References (96)

1. G. Van der Auwera, P. David, and M. Reisslein, "Traffic characteristics of H.264/AVC variable bit rate video," IEEE Commun. Mag., vol. 46, no. 11, pp. 164-174, Nov. 2008.
2. M. Isnardi, "Historical overview of video compression in consumer electronic devices," in Proc. Int. Conf. Consum. Electron.(ICCE), 2007, pp. 1-2.
3. M. Wien, H. Schwarz, and T. Oelbaum, "Performance analysis of SVC," IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 9, pp. 1194-1203, Sep. 2007. (Pubitemid 47483058)
4. B. Ciubotaru and G.-M. Muntean, "SASHA-A quality-oriented handover algorithm for multimedia content delivery to mobile users," IEEE Trans. Broadcast., vol. 55, no. 2, pp. 437-450, Jun. 2009.
5. J. Monteiro, C. Calafate, and M. Nunes, "Evaluation of the H.264 scalable video coding in error prone IP networks," IEEE Trans. Broadcast., vol. 54, no. 3, pp. 652-659, Sep. 2008.
6. M. Pinson, S. Wolf, and G. Cermak, "HDTV subjective quality of H.264 vs. MPEG-2, with and without packet loss," IEEE Trans. Broadcast., vol. 56, no. 1, pp. 86-91, Mar. 2010.
7. F. Speranza, A. Vincent, and R. Renaud, "Bit-rate efficiency of H.264 encoders measured with subjective assessment techniques," IEEE Trans. Broadcast., vol. 55, no. 4, pp. 776-780, Dec. 2009.
8. T.Wiegand, L. Noblet, and F. Rovati, "Scalable video coding for IPTV services," IEEE Trans. Broadcast., vol. 55, no. 2, pp. 527-538, Jun. 2009.
9. Cisco, Inc., "Visual networking index: Global mobile data traffic forecast update, 2009-2014," Feb. 2010.
10. L. Karam, T. Ebrahimi, S. Hemami, T. Pappas, R. Safranek, Z. Wang, and A. Watson, "Introduction to the special issue on visual media quality assessment," IEEE J. Sel. Topics Signal Process., vol. 3, no. 2, pp. 189-192, Mar./Apr. 2009.
11. G. Van der Auwera and M. Reisslein, "Implications of smoothing on statistical multiplexing of H.264/AVC and SVC video streams," IEEE Trans. Broadcast., vol. 55, no. 3, pp. 541-558, Sep. 2009.
12. N. Staelens, S. Moens, W. Van den Broeck, I. Marien, B. Vermeulen, P. Lambert, R. Van de Walle, and P. Demeester, "Assessing quality of experience of IPTV and video on demand services in real-life environments," IEEE Trans. Broadcast., vol. 56, no. 4, pp. 458-466, Sep. 2010.
13. L. Guo and Y. Meng, "What is wrong and right with MSE?," in Proc. 8th Int. Conf. Signal Image Process., 2006, pp. 212-215.
14. Methodology for the Subjective Assessment of the Quality of Television Pictures, ITU-R Recommendation BT.500-11.
15. Subjective video quality assessment [Online]. Available: http://www.acceptv.com
16. Subjective Video Quality Assessment Methods for Multimedia Applications, ITU-T Recommendation-P.910, Sep. 1999.
17. K. Brunnstrom, D. Hands, F. Speranza, and A. Webster, "VQEG validation and ITU standardization of objective perceptual video quality metrics," IEEE Signal Process. Mag., vol. 26, no. 3, pp. 96-101, May 2009.
18. A. Takahashi, D. Hands, and V. Barriac, "Standardization activities in the ITU for a QoE assessment of IPTV," IEEE Commun. Mag., vol. 46, no. 2, pp. 78-84, Feb. 2008.
19. User Requirements for Objective Perceptual Video Quality Measurements in Digital Cable Television, ITU-T Recommendation J.1443m, May 2000.
20. B. Ciubotaru, G.-M. Muntean, and G. Ghinea, "Objective assessment of region of interest-Aware adaptive multimedia streaming quality," IEEE Trans. Broadcast., vol. 55, no. 2, pp. 202-212, Jun. 2009.
21. S.Winkler, A. Sharma, and D. McNally, "Perceptual video quality and blockiness metrics for multimedia streaming applications," in Proc. of Int. Symp.Wireless Personal Multimedia Commun., 2001, pp. 553-556.
22. M. Siller and J.Woods, "QoE in multimedia services transmission," in Proc. 7th World Multiconf. Systemics, Cybernetics Inf., 2003, vol. 7, pp. 74-76.
23. M. Venkataraman, S. Sengupta, M. Chatterjee, and R. Neogi, "Towards a video QoE definition in converged networks," in Proc. Int. Conf. Digital Telecommun., 2007, pp. 92-97.
24. K. Yamagishi and T. Hayashi, "Parametric packet-layer model for monitoring video quality of IPTV services," in Proc. Int. Conf. Commun., 2008, pp. 1026-1030.
25. J. Kim, H. Lee, M. Lee, H. Lee, W. Lyu, and G. Choi, "The QoE evaluation method through the QoS-QoE correlation model," in Proc. 4th Int. Conf. Networked Comput. Adv. Inf. Manage. (NCM), 2008, vol. 2, pp. 719-725.
26. P. Simoens, S. Latre, B. De Vleeschauwer, W. Van de Meerssche, F. De Turck, B. Dhoedt, P. Demeester, S. Van Den Berghe, and E. Gilon, "Design of an autonomic QoE reasoner for improving access network performance," in Proc. Int. Conf. Autonomic Autonomous Syst., 2008, pp. 233-240.
27. M. Garcia and A. Raake, "Impairment-factor-based audio-visual quality model for IPTV," in Proc. Int. Workshop Quality Multimedia Exper. (QoMEX), Jul. 2009.
28. G. W. Cermak, "Subjective video quality as a function of bit rate, frame rate, packet loss, and codec," in Proc. Int. Workshop Quality Multimedia Exper. (QoMEX), Jul. 2009.
29. P. Calyam, P. Chandrasekaran, G. Trueb, N. Howes, D. Yu, Y. Liu, L. Xiong, R. Ramnath, and D. Yang, "Impact of router queuing disciplines on multimedia QoE in IPTV deployments," in Proc. Int. Workshop Quality Multimedia Exper. (QoMEX), Jul. 2009.
30. U. Engelke and H.-J. Zepernick, "Perceptual-based quality metrics for image and video services: A survey," in Next Gen. Internet Netw. (NGI)-3rd EuroNGI Conf. Next Gen. Internet Networks: Design Eng. Heterogeneity , 2007, pp. 190-197. (Pubitemid 47439987)
31. H. Cheng and J. Lubin, "Reference-free objective quality metrics for MPEG-coded video," in Proc. SPIE-Int. Soc. Opt. Eng., Mar. 2005, vol. 5666, no. 1, pp. 160-167. (Pubitemid 40949011)
32. Y. Kawayoke and Y. Horita, "NR objective continuous video quality assessment model based on frame quality measure," in Proc. Int. Conf. Image Process., 2008, pp. 385-388.
33. M. A. Saad and A. C. Bovik, "Natural motion statistics for no-reference video quality assessment," in Proc. Int. Workshop Quality Multimedia Experience (QoMEX), Jul. 2009.
34. Y. Tian and M. Zhu, "Analysis and modelling of no-reference video quality assessment," in Proc. Int. Conf. Computer and Automation Eng., 2009, pp. 108-112.
35. T. Oelbaum, C. Keimel, and K. Diepold, "Rule-based no-reference video quality evaluation using additionally coded videos," IEEE J. Sel. Topics Signal Process., vol. 3, no. 2, pp. 294-303, Apr. 2009.
36. C. Keimel, T. Oelbaum, and K. Diepold, "No-reference video quality evaluation for high-definition video," in Proc. IEEE Int. Conf. Acoust., Speech and Signal Process., 2009, pp. 1145-1148.
37. S. Hemami and A. Reibman, "No-reference image and video quality estimation: Applications and human-motivated design," Signal Process.: Image Commun., vol. 25, no. 7, pp. 469-481, Aug. 2010.
38. S. Olsson, M. Stroppiana, and J. Baina, "Objective methods for assessment of video quality : State of the art," IEEE Trans. Broadcast., vol. 43, no. 4, pp. 487-495, Dec 1997. (Pubitemid 127764530)
39. S. Winkler, Digital Video Quality: Vision Models and Metrics. Hoboken: Wiley, 2005.
40. H. Wu and K. R. Rao, Digital Video Image Quality and Perceptual Coding. Boca Raton: CRC Press, 2005.
41. S.Winkler, "Issues in vision modeling for perceptual video quality assessment," Signal Process., vol. 78, no. 2, pp. 231-252, 1999. (Pubitemid 32078185)
42. S. Rihs, "The influence of audio on perceived picture quality and subjective audio-video delay tolerance," in MOSAIC Handbook 1996, pp. 183-187.
43. T. Virtanen, J. Radun, P. Lindroos, S. Suomi, T. Saamanen, T. Vuori, M. Vaahteranoksa, and G. Nyman, "Forming valid scales for subjective video quality measurement based on a hybrid qualitative/quantitative methodology," in Proc. SPIE, Vol. 6808, Image Quality Syst. Perform. V, Jan. 2008.
44. P. Corriveau, C. Gojmerac, B. Hughes, and L. Stelmach, "All subjective scales are not created equal: The effects of context on different scales," Signal Process., vol. 77, no. 1, pp. 1-9, Aug. 1999.
45. H. R.Wu, Z. Yu, and B. Qiu, "Multiple reference impairment scale subjective assessment method for digital video," in Proc. Int. Conf. Digital Signal Process. (DSP), 2002, pp. 185-189.
46. Final report from the video quality experts group on the validation of objective models of video quality assessment, Phase II 2003Video Quality, Experts Group (VQEG).
47. A. Stuart, K. Ord, and S. Arnold, Kendall's Advanced Theory of Statistics, Volume 2A: Classical Inference and the Linear Model, 6th ed. Hoboken: Wiley, 2009.
48. "Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment," Study Group 9, 2000ITU-T, Jun. [Online]. Available: http://www.its.bldrdoc.gov/vqeg/projects/ frtv-phaseI
49. Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE Trans. Image Process., vol. 13, no. 4, pp. 600-612, Apr. 2004.
50. Z. Wang, L. Lu, and A. Bovik, "Video quality assessment based on structural distortion measurement," Signal Process. Image Commun., vol. 19, no. 2, pp. 121-132, Feb. 2004.
51. Z.Wang, E. Simoncelli, and A. Bovik, "Multiscale structural similarity for image quality assessment," in Conf. Rec. 37th Asilomar Conf. Signals, Syst. Comput., 2003, vol. 2, pp. 1398-1402.
52. Z.Wang and Q. Li, "Video quality assessment using a statistical model of human visual speed perception," J. Opti. Soc. America A (Optics, Image Sci., Vision), vol. 24, no. 12, pp. B61-B69, Dec. 2007. (Pubitemid 351325125)
53. H. Sheikh and A. Bovik, "Image information and visual quality," IEEE Trans. Image Process., vol. 15, no. 2, pp. 430-444, Feb. 2006. (Pubitemid 43126168)
54. L. Lu, Z. Wang, A. Bovik, and J. Kouloheris, "Full-reference video quality assessment considering structural distortion and no-reference quality evaluation of MPEG video," in Proc. IEEE Int. Conf. Multimedia Expo, 2002, vol. 1, pp. 61-64.
55. A. Shnayderman, A. Gusev, and A. Eskicioglu, "Multidimensional image quality measure using singular value decomposition," in Proc. SPIE-Int. Soc. Opt. Eng., 2003, vol. 5294, no. 1, pp. 82-92.
56. P. Tao and A. M. Eskicioglu, "Video quality assessment using M-SVD," in Proc. Int. Soc. Opt. Eng. (SPIE), 2007, vol. 6494.
57. A. Pessoa, A. Falcao, R. Nishihara, A. Silva, and R. Lotufo, "Video quality assessment using objective parameters based on image segmentation," Soc. Motion Pictures Television Eng. (SMPTE) J., vol. 108, no. 12, pp. 865-872, Dec. 1999. (Pubitemid 30528021)
58. M. Pinson and S. Wolf, "A new standardized method for objectively measuring video quality," IEEE Trans. Broadcast., vol. 50, no. 3, pp. 312-322, Sep. 2004.
59. J. Okamoto, T. Hayashi, A. Takahashi, and T. Kurita, "Proposal for an objective video quality assessment method that takes temporal and spatial information into consideration," Electron. Commun. Japan, Part 1 (Commun.), vol. 89, no. 12, pp. 97-108, 2006. (Pubitemid 44085730)
60. S.-O. Lee and D.-G. Sim, "New full-reference visual quality assessment based on human visual perception," in Proc. Int. Conf. Consum. Electron. (ICCE), 2008, pp. 75-76.
61. A. Bhat, I. Richardson, and S. Kannangara, "A new perceptual quality metric for compressed video," in IEEE Int. Conf. Acoust., Speech, Signal Process., 2009, pp. 933-936.
62. F. Lukas and Z. Budrikis, "Picture quality prediction based on a visual model," IEEE Trans. Commun., vol. 30, no. 7, pp. 1679-1692, Jul. 1982. (Pubitemid 13474449)
63. C. van den Branden Lambrecht and O. Verscheure, "Perceptual quality measure using a spatio-temporal model of the human visual system," in Proc. Int. Soc. Opt. Eng. (SPIE), 1996, vol. 2668, pp. 450-461.
64. A.Watson, J. Hu, and J. McGowan, "Digital video quality metric based on human vision," J. Electron. Imaging, vol. 10, no. 1, pp. 20-29, Jan. 2001. (Pubitemid 32279358)
65. F. Xiao, "DCT-based video quality evaluation," Winter 2000.
66. C. Lee and O. Kwon, "Objective measurements of video quality using the wavelet transform," Optical Eng., vol. 42, no. 1, pp. 265-272, Jan. 2003.
67. K. Seshadrinathan and A. C. Bovik, "Motion tuned spatio-temporal quality assessment of natural videos," IEEE Trans. Image Process., vol. 19, no. 2, pp. 335-350, Feb. 2010.
68. A. Hekstra, J. Beerends, D. Ledermann, F. de Caluwe, S. Kohler, R. Koenen, S. Rihs, M. Ehrsam, and D. Schlauss, "PVQM-A perceptual video quality measure," Signal Process. Image Commun., vol. 17, no. 10, pp. 781-798, Nov. 2002. (Pubitemid 35282146)
69. Z. Lu, W. Lin, E. Ong, X. Yang, and S. Yao, "PQSM-based RR and NR video quality metrics," in Proc. Int. Soc. Opt. Eng. (SPIE), 2003, vol. 5150, pp. 633-640.
70. E. Ong, X. Yang, W. Lin, Z. Lu, and S. Yao, "Video quality metric for low bitrate compressed videos," in Proc. Int. Conf. Image Process., 2004, vol. 5, pp. 3531-3534. (Pubitemid 40820837)
71. E. Ong, W. Lin, Z. Lu, and S. Yao, "Colour perceptual video quality metric," in Proc. Int. Conf. Image Process., 2006, pp. 1172-1175.
72. P. Ndjiki-Nya, M. Barrado, and T. Wiegand, "Efficient full-reference assessment of image and video quality," in Proc. IEEE Int. Conf. Image Process. (ICIP), 2007, pp. 125-128.
73. D. Chandler and S. Hemami, "VSNR: A wavelet-based visual signal-to-noise ratio for natural images," IEEE Trans. Image Process., vol. 16, no. 9, pp. 2284-2298, Sep. 2007. (Pubitemid 47316675)
74. Objective perceptual multimedia video quality measurement in the presence of a full reference, ITU-T Recommendation J.247, Aug 2008.
75. M. Pinson and S. Wolf, "Application of the NTIA General Video Quality Metric VQM to HDTV quality monitoring," in 3rd Int. Workshop Video Process. Quality Metrics Consum. Electron. (VPQM-07), Jan. 2007 [Online]. Available: http://www.its.bldrdoc. gov/n3/video/documents.htm
76. O. Sugimoto, S. Naito, S. Sakazawa, and A. Koike, "Objective perceptual picture quality measurement method for high-definition video based on full reference framework," in Proc. Int. Soc. Opt. Eng. (SPIE), 2009, vol. 7242, p. 72421A, (9 pp.).
77. J. Okamoto, K.Watanabei, A. Hondaii, M. Uchidaiii, and S. Hangaiiv, "HDTV objective video quality assessment method applying fuzzy measure," in Proc. Int.Workshop Quality Multimedia Exper. (QoMEX), Jul. 2009.
78. Objective Perceptual Multimedia Video Quality Measurement of HDTV for Digital Cable Television in the Presence of a Full Reference, ITU-T Recommendation J.341, Jan. 2011.
79. "Final report from VQEG on the validation of objective models of video quality assessment," ITU-T Study Group 12 Temporary Document 8 (WP2/12), May 2000.
80. H. Sheikh, M. Sabir, and A. Bovik, "A statistical evaluation of recent full reference image quality assessment algorithms," IEEE Trans. Image Process., vol. 15, no. 11, pp. 3440-3451, Nov. 2006. (Pubitemid 44615139)
81. A. A. Stocker and E. P. Simoncelli, "Noise characteristics and prior expectations in human visual speed perception," Nature Neuroscience, vol. 9, pp. 578-585, 2006.
82. "Video quality measurement techniques," NTIA report 02-392, 2002 [Online]. Available: http://www.its.bldrdoc.gov/pub/ntia-rpt/02-392/vqm- techniques-v2.pdf
83. American National Standard for Telecommunications-Digital Transport of One-Way Video Signals-Parameters for Objective Performance Analysis, ANSI T1.801.03-1996, 1996.
84. "Final report from the video quality experts group on the validation of objective models of multimedia quality assessment, Phase I," Video Quality Experts Group (VQEG), 2008.
85. T. Oelbaum, K. Diepold, and W. Zia, "A generic method to increase the prediction accuracy of visual quality metrics," in Picture Coding Symp. (PCS), 2007.
86. O. Faugeras, "Digital color image processing within the framework of a human vision model," IEEE Trans. Acoust., Speech, Signal Process., vol. 27, no. 4, pp. 380-393, 1979. (Pubitemid 10431923)
87. R. T. Born and D. C. Bradley, "Structure and function of visual area MT," Annu. Rev. Neurosci., vol. 28, no. 1, pp. 157-189, 2005. (Pubitemid 41098903)
88. "Report on the validation of video quality models for high definition video content," Video Quality Experts Group (VQEG), 2010.
89. S. Borer, "A model of jerkiness for temporal impairments in video transmission," in Proc. Int. Workshop Quality Multimedia Exper. (QoMEX), Jun. 2010, pp. 218-223.
90. K. Ferguson, "An adaptable human vision model for subjective video quality rating prediction among CIF, SD, HD, and e-cinema," in Proc. 3rd Int. Workshop Video Process. Quality Metrics for Consum. Electron. (VPQM), 2007 [Online]. Available: http://enpub.fulton.asu.edu/resp/vpqm2007/
91. "VQEG FRTV phase 1 database," 2000 [Online]. Available: ftp://ftp. crc.ca/crc/vqeg/TestSequences/
92. "LIVE video quality database," 2009 [Online]. Available: http://live. ece.utexas.edu/research/quality/live-video.html
93. K. Seshadrinathan, R. Soundararajan, A. Bovik, and L. Cormack, "Study of subjective and objective quality assessment of video," IEEE Trans. Image Process., vol. 19, no. 6, pp. 1427-1441, Jun. 2010.
94. A. Murthy and L. Karam, "IVQUEST-Image and video quality evaluation software," [Online]. Available: http://ivulab.asu.edu/Quality/IVQUEST
95. A. Murthy and L. Karam, "A MATLAB based framework for image and video quality evaluation," in Proc. Int. Workshop Quality Multimedia Exper. (QoMEX), Jun. 2010, pp. 242-247.
96. C. Keimel, T. Oelbaum, and K. Diepold, "Improving the verification process of video quality metrics," in Proc. Int. Workshop Quality Multimedia Exper. (QoMEX), Jul. 2009.