The influence of subjects and environment on audiovisual subjective tests: An international study

IEEE Journal on Selected Topics in Signal Processing

Volumn 6, Issue 6, 2012, Pages 640-651

  Pinson, Margaret H ^a   Janowski, Lucjan ^b   Pepion, Romuald ^c   Huynh Thu, Quan ^d   Schmidmer, Christian ^e   Corriveau, Phillip ^f   Younkin, Audrey ^f   Le Callet, Patrick ^c   Barkowsky, Marcus ^c   Ingram, William ^a  

^a Natl Telecom and Info Admin   (United States)

^b AGH UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Poland)

^c UNIVERSITÉ DE NANTES   (France)

^d Technicolor Research and Innovation   (France)

^e OPTICOM GmbH   (Germany)

^f INTEL CORPORATION   (United States)

Author keywords

Audiovisual quality; environment effect; language effect; Mean Opinion Scores (MOS); subjective testing

Indexed keywords

ABSOLUTE CATEGORY RATINGS; AUDIO QUALITY; AUDIOVISUAL DEVICES; AUDIOVISUAL QUALITY; BACKGROUND NOISE; CONTROLLED ENVIRONMENT; ENVIRONMENT EFFECTS; ENVIRONMENTAL FACTORS; INDIVIDUAL DIFFERENCES; INTERNET VIDEO; LABORATORY ENVIRONMENT; LANGUAGE EFFECT; MEAN OPINION SCORES; MONITOR CALIBRATION; PUBLIC ENVIRONMENT; QUALITY ASSESSMENT; SUBJECTIVE TESTS; SYSTEMATIC STUDY; VIDEO QUALITY;

CELLULAR TELEPHONE SYSTEMS; EXPERIMENTS; LABORATORIES; STATISTICAL TESTS; SUBJECTIVE TESTING;

QUALITY CONTROL;

EID: 84866514046     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2012.2215306     Document Type: Article

References (46)

1. F. N. Rahayu et al., "Subjective video quality assessment in the presence of audio for digital cinema," in Proc. Int. Workshop Quality of Multimedia Experience (QoMEX), Sep. 2011.
2. N. Staelens et al., "Assessing quality of experience of IPTV and video on demand services in real-life environments," IEEE Trans. Broadcasting, vol. 56, no. 4, pp. 458-466, Dec. 2010.
3. Q. Huynh-Thu et al., "Study of rating scales for subjective quality assessment of high-definition video," IEEE Trans. Broadcasting, vol. 57, no. 1, pp. 1-14, Mar. 2011.
4. S. Péchard et al., "Suitable methodology in subjective video quality assessment: A resolution dependent paradigm," in Proc. Int. Workshop Image Quality and Its Application, Kyoto, Sep. 2008.
5. M. Brotherton et al., "Subjective multimedia quality assessment," IEICE Trans. Fundamentals, Electron. Commun. Comput. Sci., vol. E89-A, no. 11, pp. 2920-2932, 2006. (Pubitemid 44934394)
6. T. Tominaga et al., "Performance comparisons of subjective quality assessment methods for mobile video," in Proc. Int. Workshop Quality of Multimedia Experience (QoMEX) Jun. 2010.
7. P. Corriveau et al., "All subjective scales are not created equal: The effects of context on different scales," Signal Process., vol. 77, no. 1, pp. 1-9, 1999.
8. Q. Huynh-Thu and M. Ghanbari, "A comparison of subjective video quality assessment methods for low-bit rate and low-resolution video," in Proc. Signal Image Process., Honolulu, HI, 2005, vol. 479.
9. E. P. Cox, "The optimal number of response alternatives for a scale: A review," J. Market. Res., vol. XVII, pp. 407-422, Nov. 1980.
10. S. Zielinski et al., "On some biases encountered in modern audio quality listening tests-A review," J. Audio Eng. Soc., vol. 56, no. 6, pp. 427-451 Jun. 2008.
11. M. Pinson et al., "Audiovisual quality components," IEEE Signal Process. MPrag., vol. 28, no. 6, pp. 60-67, Nov. 2011.
12. J. G. Beerends et al., "The influence of video quality on perceived audio quality and vice versa," J. Audio Eng. Soc., vol. 47, no. 5, pp. 355-362, May 1999.
13. G. Cermak, "Four suggestions for research on multimedia QoE using subjective evaluations," IEEE Commun. Soc. Multimedia Commun. Tech. Committee (COMSOC MMTC) E-Lett. vol. 4, no. 10, Nov. 2009. Available: http://committees.comsoc.org/mmc/eletters.asp
14. D. E. Pearson, "Viewer response to time-varying video quality," in Proc. SPIE Human Vis. Electron. Imag. III, Jan. 26-29, 1998, vol. 3299.
15. R. Aldridge R et al., "Measurement of scene-dependent quality variations in digitally coded television pictures," IEE Proc.: Vision, Image Signal Process., vol. 142, no. 3, pp. 149-154 Jun. 1995.
16. R. Hamburg and H. De Ridder, "Time-varying image quality: Modeling the relation between instantaneous and overall quality," SMPTE J., vol. 108, no. 11, pp. 2888-2899, Nov. 1999.
17. D. S. Hands and S. E. Avons, "Recency and duration neglect in subjective assessment of television picture quality," Appl. Cognitive Psychol., vol. 15, no. 6, pp. 639-657, Nov. 2001. (Pubitemid 33673463)
18. L. Grosand and N. Chateau, "Instantaneous and overall judgements for time-varying speech quality: Assessments and relationships," Acta Acustica united With Acustica, vol. 87, no. 3, pp. 367-377, May-Jun. 2001. (Pubitemid 32699292)
19. V. Seferidis et al., "Forgiveness effect in subjective assessment of packet video," Electron. Lett., vol. 28, no. 21, pp. 2013-2014, Oct. 1992. (Pubitemid 23576583)
20. D. S. Hands, "Temporal characterisation of forgiveness effect," Electron. Lett., vol. 37, no. 12, pp. 752-754 Jun. 2001. (Pubitemid 32576748)
21. Y. Liang et al., "Analysis of packet loss for compressed video: Does burst-length matter?," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Apr. 2003, pp. 684-687.
22. R. R. Pastrana-Vidal, "Sporadic frame dropping impact on quality perception," in Proc. SPIE Human Vis. Electron. Imag. IX, Jan. 2004, vol. 5292, pp. 182-193.
23. R. R. Pastrana et al., "Temporal masking effect on dropped frames at video scene cuts," in Proc. SPIE Human Vis. Electron. Imag. IX, Jan. 2004, vol. 5292, pp. 194-201.
24. U. Reiter et al., "Comparing apples and oranges: Assessment of the relative video quality in the presence of different types of distortions," EURASIP J. Image Video Process., 2011.
25. O. Le Meur and P. Le Callet, "What we see most is likely to be what matters: Visual attention and applications," in Proc. IEEE Int. Conf. Image Process., Cairo, Egypt, 2009, pp. 3085-3088.
26. Rec. ITU-T H.264, 2011, Advanced Video Coding For Generic Audiovisual Services. Geneva, 2011 [Online].Available: http://www.itu.int
27. ISO/IEC 14496-10:2010, Information Technology-Coding of Audio-Visual Objects-Part 10: Advanced Video Coding.
28. M. Pinson et al., Report on the Validation of Video Quality Models for High Definition Video Contentm VQEG, 2010 [Online]. Available: http://www.vqeg.org
29. Rec. ITU-R BT.500-12, Methodology for the Subjective Assessment of the Quality of Television Pictures. Geneva, 2009 [Online]. Available: http://www.itu.int
30. Rec. ITU-T P.78, 1996, Subjective Testing Method for Determination of Loudness Ratings in Accordance With Recommendation P.76. Geneva, 1996 [Online]. Available: http://www.itu.int
31. S. Winkler, "On the properties of subjective ratings in video quality experiments," in Proc. Int.Workshop Quality of Multimedia Experience (QoMEX) 2009, Jul. 29-31, 2009.
32. D. Huff and I. Geis, How to Lie With Statistics. New York: W. W. Norton, 1993.
33. A. G?owacz et al., "Automated qualitative assessment of multi-modal distortions in digital images based on GLZ," in Special Issue of Annals of Telecommunications on Quality of Experience and Socio-Economic Issues of Network-Based Services. New York: Springer, Feb. 2010, vol. 65, pp. 3-17, no. 1-2.
34. L. Janowski and Z. Papir, "Modeling subjective tests of quality of experience with a generalized linear model," in Proc. Quality of Multimedia Experience (QoMEX), Jul. 2009.
35. M. Pinson and S. Wolf, "An objective method for combining multiple subjective data sets," in Proc. SPIE Video Commun. Image Process. Conf., Lugano, Jul. 2003.
36. Y. Pitrey et al., "Evaluation of MPEG4-SVC for QoE protection in the context of transmission errors," in Proc. SPIE Opt. Eng. , San Diego, CA, Aug. 2010.
37. K.Wang et al., "Subjective evaluation of HDTV stereoscopic videos in IPTV scenarios using absolute category rating," in Proc. SPIE Stereoscopic Displays Applicat. XXII, San Francisco, CA, Jan. 2011.
38. Z. Cai et al., "Comparison of MOS evaluation characteristics for Chinese, Japanese, and English in IP Telephony," in Proc. 4th Int. Universal Commun. Symp. (IUCS), Oct. 2010.
39. D. J. Goodman and R. D. Nash, "Subjective quality of the same speech transmission conditions in seven different countries," IEEE Trans. Commun., vol. COM-30, no. 4, pp. 642-654, 1982.
40. S. Zieliński et al., "On the use of graphic scales in modern listening tests," in Proc. 123rd Conv. Audio Eng. Soc., Oct. 2007.
41. Rec. ITU-R BS.1116-1, Methods for the Subjective Assessment of Small Impairments in Audio Systems Including Multichannel Sound Systems. Geneva, 1997 [Online]. Available: http://www.itu.int
42. Rec. ITU-T P.910, Subjective Video Quality Assessment Methods for Multimedia Applications. Geneva, 2008 [Online]. Available: http://www.itu.int
43. Rec. ITU-T P.800, Methods for Subjective Determination of Transmission Quality. Geneva, 1996 [Online]. Available: http://www.itu.int
44. J. Neter et al., Applied Linear Statistical Models. New York: Mc-Graw-Hill, 1974, p. 81.
45. Rec. ITU-R BS.1534-1, Method for the Subjective Assessment of Intermediate Quality Levels of Coding Systems. Geneva, 2003 [Online]. Available: http://www.itu.int
46. A. Catellier et al., "Impact of mobile devices and usage location on perceived multimedia quality," in Proc. Int. Workshop on Quality of Multimedia Experience (QoMEX), Jul. 2012.