Study of rating scales for subjective quality assessment of high-definition video

IEEE Transactions on Broadcasting

Volumn 57, Issue 1, 2011, Pages 1-14

  Huynh Thu, Quan ^a,b   Garcia, Marie Neige ^c   Speranza, Filippo ^c   Corriveau, Philip ^e   Raake, Alexander ^d  

^a Psytechnics Ltd   (United Kingdom)

^b Technicolor   (France)

^c DEUTSCHE TELEKOM LABORATORIES   (Germany)

^d Industry Canada   (Canada)

^e INTEL CORPORATION   (United States)

Author keywords

ACR; MOS; rating scale; Subjective assessment; Video quality

Indexed keywords

ACR; MOS; RATING SCALE; SUBJECTIVE ASSESSMENTS; VIDEO QUALITY;

IMAGE QUALITY; RATING;

SUBJECTIVE TESTING;

EID: 79952009304     PISSN: 00189316     EISSN: None     Source Type: Journal    
DOI: 10.1109/TBC.2010.2086750     Document Type: Article

References (40)

1. ITU-R, "Methodology for the subjective assessment of the quality of television pictures," Rec. BT.500, Geneva, 2002.
2. ITU-R, "Subjective assessment methods for image quality in high-definition television," Rec. BT.710, Geneva, 1998.
3. ITU-T, "Subjective video quality assessment methods for multimedia applications," Rec. P.910, Geneva, 2008.
4. S. Winkler and P. Mohandas, "The evolution of video quality measurement: From PSNR to hybrid metrics," IEEE Trans. Broadcast., vol. 54, no. 3(2), pp. 660-668, Sep. 2008.
5. H. Sohn, H.Yoo, W. De Neve, C. S. Kim, and Y. M. Ro, "Full-reference video quality metric for fully scalable and mobile SVC content," IEEE Trans. Broadcast., vol. 56, no. 3, pp. 269-280, Sep. 2010.
6. I. P. Gunawan and M. Ghanbari, "Efficient reduced-reference video quality meter," IEEE Trans. Broadcast., vol. 54, no. 3(2), pp. 669-679, Jun. 2008.
7. J. Asghar, I. Hood, and F. Le Faucheur, "Preserving video quality in IPTV networks," IEEE Trans. Broadcast., vol. 55, no. 2(2), pp. 386-395, Jun. 2009.
8. R. Shmueli, O. Hadar, R. Huber, M. Maltz, and M. Huber, "Effects of an encoding scheme on perceived video quality transmitted over lossy internet protocol networks," IEEE Trans. Broadcast., vol. 54, no. 3(2), pp. 628-640, Sep. 2008.
9. Q. Huynh-Thu and M. Ghanbari, "Temporal aspect of perceived quality in mobile video broadcasting," IEEE Trans. Broadcast., vol. 54, no. 3(2), pp. 641-651, Sep. 2008.
10. "Final report from the Video Quality Experts Group on the validation of objective models of video quality assessment, Phase II (FR-TV2)," Video Quality Experts Group, Aug. 2003 [Online]. Available: http://www.vqeg.org
11. "Final report from the Video Quality Experts Group on the validation of objective models of multimedia quality assessment, Phase I," Video Quality Experts Group, Sep. 2008 [Online]. Available: http:// www.vqeg.org
12. Q. Huynh-Thu, "Perceptual quality assessment of communicationsgrade video with temporal artefacts," Ph.D. thesis, University of Essex, Colchester, Jun. 2009.
13. J. W. Mayne, "Accuracy, precision, and reliableness in operational research," Operations Res., vol. 11, no. 6, pp. 990-996, Nov./Dec. 1963.
14. P. Corriveau, C. Gojmerac, B. Hughes, and L. Stelmach, "All subjective scales are not created equal: The effects of context on different scales," Signal Process., vol. 77, no. 1, pp. 1-9, Aug. 1999.
15. M. Pinson and S. Wolf, "Comparing subjective video quality testing methodologies," in Proc. SPIE Visual Commun. Image Process., Lugano, Jul. 2003, vol. 5150, pp. 573-582.
16. F. Speranza, T. Martin, and R. Renaud, "Subjective quality assessment and the effect of context in expert and non-expert viewers," in Proc. SPIE Image Quality Syst. Perform., San Jose, Jan. 2004, vol. 5294, pp. 201-210.
17. Q. Huynh-Thu and M. Ghanbari, "A comparison of subjective video quality assessment methods for low-bit rate and low-resolution video," in Proc. IASTED Int. Conf. Signal Image Process., Honolulu, Aug. 2005, vol. 479, pp. 70-76.
18. C. Lee, H. Choi, E. Lee, S. Lee, and J. Choe, "Comparison of various subjective video quality assessment methods," in Proc. SPIE Image Quality Syst. Perform. III, San Jose, Jan. 2006, vol. 6059.
19. M. D. Brotherton, Q. Huynh-Thu, D. S. Hands, and K. Brunnstrm, "Subjective multimedia quality assessment," IEICE Trans. Fundam. Electron., Commun.Comput. Sci. (Spec. Section Image Media Quality), vol. EA-89, no. 11, pp. 2920-2932, Nov. 2006. (Pubitemid 44934394)
20. Q. Huynh-Thu, M. Ghanbari, D. S. Hands, and M. Brotherton, "Subjective video quality evaluation for multimedia applications," in Proc. SPIE Human Vision Electron. Imaging XI, San Jose, Jan. 2006, vol. 6057, pp. 464-474.
21. M. S. Matell and J. Jacoby, "Is there an optimal number of alternatives for Likert scale items? Study 1: reliability and validity," Educational and Psychological Measurement, vol. 31, pp. 657-674, 1971.
22. B. L. Jones and P. R. McManus, "Graphic scaling of qualitative terms," SMPTE Journal, vol. 95, no. 11, pp. 1166-1171, 1986.
23. M. T. Virtanen, N. Gleiss, and M. Goldstein, "On the use of evaluative category scales in telecommunications," in Proc. Human Factors Telecommun., Melbourne, Australia, 1995, pp. 253-260.
24. K. Teunissen, "The validity of CCIR quality indicators along a graphical scale," SMPTE J., vol. 105, no. 3, pp. 144-149, Mar. 1996. (Pubitemid 126610948)
25. N. Narita, "Graphic scaling and validity of Japanese descriptive terms used in subjective evaluation tests," SMPTE J., vol. 102, no. 7, pp. 616-622, Jul. 1993. (Pubitemid 23693360)
26. A. Watson, "Assessing audio and video quality in desktop multimedia conferencing systems," Ph.D. thesis, University College London, London, U.K., 2001.
27. A. Watson and A. Sasse, "Measuring perceived quality of speech and video in multimedia conferencing applications," in Proc. ACM Multimedia Conf., Sep. 1998, pp. 55-60.
28. S. Zieliñski, P. Brooks, and F. Rumsey, "On the use of graphic scales in modern listening tests," in Proc. 123rd AES Convention, New York, Oct. 2007.
29. ITU-R, "Method for the subjective assessment of intermediate quality level of coding systems," Rec. BS.1534, Geneva, 2003.
30. S. J. McKelvie, "Graphic rating scales-How many categories?," British J. Psych., vol. 69, no. 2, pp. 185-202, May 1978.
31. E. P. Cox, "The optimal number of response alternatives for a scale: A review," J. Marketing Res., vol. 17, no. 4, pp. 407-422, Nov. 1980.
32. G. Albaum, R. Best, and D. Hawkins, "Continuous vs. discrete semantic differential rating scales," Psych. Reports, vol. 49, pp. 83-86, 1981.
33. C. C. Preston and A. M. Colman, "Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences," Acta Psychologica, vol. 104, no. 1, pp. 1-15, 2000.
34. E. Svensson, "Comparison of the quality of assessments using continuous and discrete ordinal rating scales," Biometrical J., vol. 42, no. 4, pp. 417-434, 2000.
35. S. Winkler, "On the properties of subjective ratings in video quality experiments," in Proc. Int. Workshop Quality Multimedia Exper. (QoMEX), San Diego, CA, Jul. 29-31, 2009.
36. Video Quality Experts Group, "VQEG HDTV test plan," Jun. 2009 [Online]. Available:
37. IEC, "Sound system equipment-Part 13: Listening tests on loudspeakers," Publication 60268-13, Geneva, 1985.
38. ITU-T, "Requirement specification for P.Objective Listening Quality Assessment (P.OLQA)," ITU-T Study Group 12 Temporary Document 90 Rev.4, May 2008.
39. ITU-T, "Advanced video coding for generic audiovisual services," Rec. H.264, Geneva, 2009.
40. M. R. Spiegel, Outline of theory and problems of statistics. New York: McGraw-Hill, 1989.