VQAMap: A Novel Mechanism for Mapping Objective Video Quality Metrics to Subjective MOS Scale

IEEE Transactions on Broadcasting

Volumn 62, Issue 3, 2016, Pages 610-627

  Moldovan, Arghir Nicolae ^a   Ghergulescu, Ioana ^b   Muntean, Cristina Hava ^a  

^a NATIONAL COLLEGE OF IRELAND   (Ireland)

^b Adaptemy   (Ireland)

Author keywords

mean opinion score (MOS); objective metrics; performance evaluation; public video quality databases; subjective methods; Video quality assessment (VQA); video quality mapping

Indexed keywords

DATABASE SYSTEMS; IMAGE QUALITY; MAPPING;

MEAN OPINION SCORES; OBJECTIVE METRICS; PERFORMANCE EVALUATIONS; SUBJECTIVE METHODS; VIDEO QUALITY; VIDEO QUALITY ASSESSMENTS (VQA); VIDEO QUALITY DATABASE;

QUALITY CONTROL;

EID: 84973883015     PISSN: 00189316     EISSN: None     Source Type: Journal    
DOI: 10.1109/TBC.2016.2570002     Document Type: Article

References (73)

1. Cisco Systems, "Cisco visual networking index: The zettabyte era- Trends and analysis, " Cisco Systems, Inc., San Jose, CA, USA, Tech. Rep. 1459358220166184, May 2015. [Online]. Available: http://www.cisco.com/c/en/us/solutions/collateral/service-provider/ visual-networking-index-vni/VNI-Hyperconnectivity-WP.html
2. A.-N. Moldovan, S. Weibelzahl, and C. H. Muntean, "Energy-aware mobile learning: Opportunities and challenges, " IEEE Commun. Surveys Tuts., vol. 16, no. 1, pp. 234-265, Mar. 2014.
3. C. Hall. (Jul. 2015). The 4K UHD TV Trend Line is Spiking Up. [Online]. Available: http://www.onlinereporter.com/ 2015/07/03/the-4k-uhd-tv-trend-line-is-spiking-up/
4. Phone Arena. (Jun. 2015). Quad HD vs 1080p Display Resolution: Can People Actually See the Difference?. [Online]. Available: http:// www.phonearena.com/news/Quad-HD-vs-1080p-display-resolution-canpeople- actually-see-the-difference-id69856
5. A.-N. Moldovan and C. H. Muntean, "Towards personalised and adaptive multimedia in M-learning systems, " in Proc. 16th World Conf. E-Learn. Corp. Gov. Healthcare Higher Educ. (E-Learn). Honolulu, HI, USA, Oct. 2011, pp. 782-791. [Online]. Available: http://editlib.org/p/38807
6. I. Sodagar, "The MPEG-DASH standard for multimedia streaming over the Internet, " IEEE MultiMedia, vol. 18, no. 4, pp. 62-67, Apr. 2011.
7. G. Sullivan, J. Ohm, W.-J. Han, and T. Wiegand, "Overview of the high efficiency video coding (HEVC) standard, " IEEE Trans. Circuits Syst. Video Technol., vol. 22, no. 12, pp. 1649-1668, Dec. 2012.
8. A.-N. Moldovan, I. Ghergulescu, S. Weibelzahl, and C. H. Muntean, "User-centered EEG-based multimedia quality assessment, " in Proc. 8th IEEE Int. Symp. Broadband Multimedia Syst. Broadcast. (BMSB), London, U.K., Jun. 2013, pp. 1-8.
9. S. Chikkerur, V. Sundaram, M. Reisslein, and L. J. Karam, "Objective video quality assessment methods: A classification, review, and performance comparison, " IEEE Trans. Broadcast., vol. 57, no. 2, pp. 165-182, Jun. 2011.
10. A.-N. Moldovan and C. H. Muntean, "Subjective assessment of BitDetect-A mechanism for energy-aware multimedia content adaptation, " IEEE Trans. Broadcast., vol. 58, no. 3, pp. 480-492, Sep. 2012.
11. S. Winkler, "Analysis of public image and video databases for quality assessment, " IEEE J. Sel. Topics Signal Process., vol. 6, no. 6, pp. 616-625, Oct. 2012.
12. C.-H. Ke, C.-K. Shieh, W.-S. Hwang, and A. Ziviani, "An evaluation framework for more realistic simulations of MPEG video transmission, " J. Inf. Sci. Eng., vol. 24, no. 2, pp. 425-440, 2008.
13. T. Zinner, O. Abboud, O. Hohlfeld, T. Hossfeld, and P. Tran-Gia, "Towards QoE management for scalable video streaming, " in Proc. 21st ITC Spec. Seminar Multimedia Appl. Traffic Perform. QoE, Miyazaki, Japan, Mar. 2010, pp. 64-69. [Online]. Available: http://www.net.t-labs.tu-berlin.de/papers/ZAHHT-TQMSVS-10.pdf
14. A.-N. Moldovan, I. Ghergulescu, and C. H. Muntean, "A novel methodology for mapping objective video quality metrics to the subjective MOS scale, " in Proc. 9th IEEE Int. Symp. Broadband Multimedia Syst. Broadcast. (BMSB). Beijing, China, Jun. 2014, pp. 1-7.
15. G. Ghinea and S. Y. Chen, "Measuring quality of perception in distributed multimedia: Verbalizers vs. imagers, " Comput. Human Behav., vol. 24, no. 4, pp. 1317-1329, Jul. 2008.
16. P. Le Callet, S. Möller, and A. Perkis, "Qualinet white paper on definitions of quality of experience, " Eur. Netw. Qual. Experience Multimedia Syst. Services (COST Action IC 1003), Lausanne, Switzerland, Tech. Rep. 1.2, Mar. 2013. [Online]. Available: https://hal.archives-ouvertes.fr/hal-00977812
17. ITU-T, "P.10/G.100 Amendment 2: New definitions for inclusion in Recommendation ITU-T P.10/G.100-Vocabulary for performance and quality of service, " Int. Telecommun. Union, Geneva, Switzerland, Tech. Rep. E 33770, Jul. 2008. [Online]. Available: http://www.itu.int/rec/T-REC-P.10-200807-I!Amd2/en
18. ITU-R, "BT.500-12: Methodology for the subjective assessment of the quality of television pictures, " Int. Telecommun. Union, Geneva, Switzerland, Tech. Rep. E 34387, Sep. 2009. [Online]. Available: http://www.itu.int/rec/R-REC-BT.500-12-200909-I/en
19. ITU-T, "P.910: Subjective video quality assessment methods for multimedia applications, " Int. Telecommun. Union, Geneva, Switzerland, Tech. Rep. E 33690, Apr. 2008. [Online]. Available: http://www.itu.int/rec/T-REC-P.910-200804-I/en
20. J. L. Blin, "New quality evaluation method suited to multimedia context: Samviq, " in Proc. 2nd Int. Workshop Video Process. Qual. Metrics Consum. Electron. (VPQM), vol. 6. Scottsdale, AZ, USA, Jan. 2006, pp. 1-5. [Online]. Available: http://enpub.fulton.asu.edu/resp/vpqm2006/papers06/313.pdf
21. Q. Huynh-Thu, M. N. Garcia, F. Speranza, P. Corriveau, and A. Raake, "Study of rating scales for subjective quality assessment of highdefinition video, " IEEE Trans. Broadcast., vol. 57, no. 1, pp. 1-14, Mar. 2011.
22. Nidhi and N. Aggarwal, "A review on video quality assessment, " in Proc. Recent Adv. Eng. Comput. Sci. (RAECS), Chandigarh, India, Mar. 2014, pp. 1-6.
23. M. Pinson et al., "The influence of subjects and environment on audiovisual subjective tests: An international study, " IEEE J. Sel. Topics Signal Process., vol. 6, no. 6, pp. 640-651, Oct. 2012.
24. T. Hoßfeld et al., "Survey of Web-based crowdsourcing frameworks for subjective quality assessment, " in Proc. IEEE 16th Int. Workshop Multimedia Signal Process. (MMSP), Jakarta, Indonesia, Sep. 2014, pp. 1-6.
25. G.-M. Muntean, G. Ghinea, and T. N. Sheehan, "Region of interestbased adaptive multimedia streaming scheme, " IEEE Trans. Broadcast., vol. 54, no. 2, pp. 296-303, Jun. 2008.
26. M. Kennedy, H. Venkataraman, and G.-M. Muntean, "Battery and stream-aware adaptive multimedia delivery for wireless devices, " in Proc. IEEE 35th Conf. Local Comput. Netw. (LCN), Denver, CO, USA, Oct. 2010, pp. 843-846.
27. S. Winkler and P. Mohandas, "The evolution of video quality measurement: From PSNR to hybrid metrics, " IEEE Trans. Broadcast., vol. 54, no. 3, pp. 660-668, Sep. 2008.
28. Z. Wang and A. C. Bovik, "Mean squared error: Love it or leave it? A new look at signal fidelity measures, " IEEE Signal Process. Mag., vol. 26, no. 1, pp. 98-117, Jan. 2009.
29. K. Egiazarian et al., "A new full-reference quality metrics based on HVS, " in Proc. 2nd Int. Workshop Video Process. Qual. Metrics, vol. 4. Scottsdale, AZ, USA, 2006, pp. 1-4. [Online]. Available: http://enpub.fulton.asu.edu/resp/vpqm/vpqm2006/papers06/270.pdf
30. N. Ponomarenko et al., "On between-coefficient contrast masking of DCT basis functions, " in Proc. 3rd Int. Workshop Video Process. Qual. Metrics, vol. 4. 2007, pp. 1-4. [Online]. Available: http://ponomarenko.info/vpqm07-p.pdf
31. Z. Wang, L. Lu, and A. C. Bovik, "Video quality assessment based on structural distortion measurement, " Signal Process. Image Commun., vol. 19, no. 2, pp. 121-132, Feb. 2004.
32. Z. Wang, E. P. Simoncelli, and A. C. Bovik, "Multiscale structural similarity for image quality assessment, " in Proc. Conf. Record 37th Asilomar Conf. Signals Syst. Comput., vol. 2. Pacific Grove, CA, USA, 2003, pp. 1398-1402.
33. H. Sheikh and A. Bovik, "Image information and visual quality, " IEEE Trans. Image Process., vol. 15, no. 2, pp. 430-444, Feb. 2006.
34. K. Gu et al., "Content-weighted mean-squared error for quality assessment of compressed images, " Signal Image Video Process., pp. 1-8, Sep. 2015.
35. K. Gu, M. Liu, G. Zhai, X. Yang, and W. Zhang, "Quality assessment considering viewing distance and image resolution, " IEEE Trans. Broadcast., vol. 61, no. 3, pp. 520-531, Sep. 2015.
36. K. Gu et al., "Analysis of distortion distribution for pooling in image quality prediction, " IEEE Trans. Broadcast., vol. PP, no. 99, pp. 1-11, 2016.
37. L. Li, H. Zhu, and G. Yang, "Image quality assessment based on low order moment features, " IEICE Trans. Fundament. Electron. Commun. Comput. Sci., vol. E97-A, no. 2, pp. 538-542, Feb. 2014.
38. J. Qian, D. Wu, L. Li, D. Cheng, and X. Wang, "Image quality assessment based on multi-scale representation of structure, " Digit. Signal Process., vol. 33, pp. 125-133, Oct. 2014.
39. I. P. Gunawan and M. Ghanbari, "Efficient reduced-reference video quality meter, " IEEE Trans. Broadcast., vol. 54, no. 3, pp. 669-679, Sep. 2008.
40. R. Soundararajan and A. C. Bovik, "Survey of information theory in visual quality assessment, " Signa Image Video Process., vol. 7, no. 3, pp. 391-401, Mar. 2013.
41. S. S. Hemami and A. R. Reibman, "No-reference image and video quality estimation: Applications and human-motivated design, " Signal Process. Image Commun., vol. 25, no. 7, pp. 469-481, 2010.
42. M. Shahid, A. Rossholm, B. Lövström, and H.-J. Zepernick, "No-reference image and video quality assessment: A classification and review of recent approaches, " EURASIP J. Image Video Process., vol. 2014, no. 1, p. 40, Aug. 2014.
43. M. Shahid, J. Panasiuk, G. Van Wallendael, M. Barkowsky, and B. Lovstrom, "Predicting full-reference video quality measures using HEVC bitstream-based no-reference features, " in Proc. 7th Int. Workshop Qual. Multimedia Experience (QoMEX), Pylos-Nestor, Greece, May 2015, pp. 1-2.
44. U. Engelke, M. Kusuma, H.-J. Zepernick, and M. Caldera, "Reducedreference metric design for objective perceptual quality assessment in wireless imaging, " Signal Process. Image Commun., vol. 24, no. 7, pp. 525-547, Aug. 2009.
45. D. M. Chandler, "Seven challenges in image quality assessment: Past, present, and future research, " Int. Scholarly Res. Notices, vol. 2013, pp. 1-53, Feb. 2013.
46. M. H. Pinson, N. Staelens, and A. Webster, "The history of video quality model validation, " in Proc. IEEE 15th Int. Workshop Multimedia Signal Process. (MMSP), Pula, Croatia, 2013, pp. 458-463.
47. VQEG, "Final report on the validation of video quality models for high definition video content, " Video Qual. Experts Group, Boulder, CO, USA, Tech. Rep. 2.0, Jun. 2010. [Online]. Available: http://www.its.bldrdoc.gov/vqeg/projects/hdtv/hdtv.aspx
48. K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack, "Study of subjective and objective quality assessment of video, " IEEE Trans. Image Process., vol. 19, no. 6, pp. 1427-1441, Jun. 2010.
49. A. K. Moorthy, L. K. Choi, A. C. Bovik, and G. de Veciana, "Mobile video quality assessment database, " in Proc. IEEE ICC Workshop Realizing Adv. Video Optim. Wireless Netw., Ottawa, ON, Canada, Jun. 2012, 1-5. [Online]. Available: http://live.ece.utexas.edu/publications/2012/moorthy-icc2012.pdf
50. A. K. Moorthy, L. K. Choi, A. C. Bovik, and G. de Veciana, "Video quality assessment on mobile devices: Subjective, behavioral and objective studies, " IEEE J. Sel. Topics Signal Process., vol. 6, no. 6, pp. 652-671, Oct. 2012.
51. M. Vranješ, S. Rimac-Drlje, and D. Vranješ, "ECVQ and EVVQ video quality databases, " in Proc. 54th Int. Symp. (ELMAR), Zadar, Croatia, Sep. 2012, pp. 1-5. [Online]. Available: http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6338461
52. M. Vranješ, S. Rimac-Drlje, and K. Grgič, "Review of objective video quality metrics and performance comparison using different databases, " Signal Process. Image Commun., vol. 28, no. 1, pp. 1-19, 2013.
53. J.-S. Lee, F. De Simone, and T. Ebrahimi, "Subjective quality evaluation via paired comparison: Application to scalable video coding, " IEEE Trans. Multimedia, vol. 13, no. 5, pp. 882-893, Oct. 2011.
54. Y. Gitman, M. Erofeev, D. Vatolin, B. Andrey, and F. Alexey, "Semiautomatic visual-attention modeling and its application to video compression, " in Proc. IEEE Int. Conf. Image Process. (ICIP), Paris, France, Oct. 2014, pp. 1105-1109.
55. K. Fliegel. (Oct. 2014). QUALINET Multimedia Databases V5.5. [Online]. Available: http://dbq-wiki.multimediatech.cz/
56. S. Winkler. (Jun. 2015). Image and Video Quality Resources. [Online]. Available: http://stefan.winklerbros.net/resources.html
57. A.-N. Moldovan, I. Ghergulescu, and C. H. Muntean, "Learning assessment for different categories of educational multimedia clips in a mobile learning environment, " in Proc. 25th Soc. Inf. Technol. Teacher Educ. Int. Conf. (SITE). Jacksonville, FL, USA, Mar. 2014, pp. 1687-1692. [Online]. Available: http://www.editlib.org/p/131015
58. A.-N. Moldovan and C. H. Muntean, "User QoE assessment on mobile devices for natural and non-natural multimedia clips, " in Proc. 23rd Int. Conf. Telecommun. (ICT), Thessaloniki, Greece, May 2016, pp. 116-120.
59. S. Winkler, F. M. Savoy, and R. Subramanian, "X-Eye: A reference format for eye tracking data to facilitate analyses across databases, " in Proc. SPIE Int. Soc. Opt. Eng., vol. 9014. San Francisco, CA, USA, Feb. 2014, Art. no. 90140L.
60. R. Trestian, A.-N. Moldovan, C. H. Muntean, O. Ormond, and G.-M. Muntean, "Quality utility modelling for multimedia applications for android mobile devices, " in Proc. 7th IEEE Int. Symp. Broadband Multimedia Syst. Broadcast. (BMSB). Seoul, South Korea, Jun. 2012, pp. 1-6.
61. N. Brace, R. Kemp, and R. Snelgar, SPSS for Psychologists, 5th ed. New York, NY, USA: Routledge, Jul. 2012. [Online]. Available: http://www.amazon.com/SPSSPsychologists- Edition-Nicola-Brace/dp/1848726007/
62. D. W. Stockburger, Introductory Statistics: Concepts, Models, and Applications, 3rd ed. Cincinnati, OH, USA: Atomic Dog, 2013. [Online]. Available: http://www.psychstat.missouristate.edu/IntroBook3/ sbk.htm
63. D. Vatolin. (Mar. 2015). MSU Video Quality Measurement Tool. [Online]. Available: http://compression.ru/video/quality-measure/ video-measurement-tool-en.html
64. P. Hanhart. (Mar. 2013). VQMT: Video Quality Measurement Tool Multimedia Signal Processing Group (MMSPG). [Online]. Available: http://mmspg.epfl.ch/vqmt
65. R. F. Nau. (2015). What's a Good Value for R-Squared? [Online]. Available: http://people.duke.edu/~rnau/rsquared.htm
66. D. Hasler and S. E. Suesstrunk, "Measuring colorfulness in natural images, " in Proc. SPIE, vol. 5007, Jun. 2003, pp. 87-95.
67. OpenCV. (Jun. 2015). Open Source Computer Vision. [Online]. Available: http://opencv.org/
68. A. Balakhnin. (Jun. 2015). MVTools. [Online]. Available: http://avisynth.org.ru/mvtools/mvtools2.html
69. B. Rudiak-Gould. (Jun. 2015). AviSynth. [Online]. Available: http://avisynth.nl
70. Groucho. (2004). AVSMeter. [Online]. Available: http://www.videohelp.com/software/AVSMeter
71. A.-N. Moldovan, "Device and quality-aware adaptive multimedia for mobile learning, " Ph.D. dissertation, School of Comput., Nat. College Ireland, Dublin, Ireland, Jun. 2014. [Online]. Available: http://trap.ncirl.ie/1567/
72. D. M. Bates and S. DebRoy. (Feb. 2015). R: Nonlinear Least Squares. [Online]. Available: http://stat.ethz.ch/R-manual/ R-patched/library/stats/html/nls.html
73. A.-N. Moldovan, I. Ghergulescu, and C. H. Muntean, "Performance evaluation of EMOS model for mapping-based Video Quality estimation, " in Proc. 9th Int. Symp. Image Signal Process. Anal. (ISPA). Zagreb, Croatia, Sep. 2015, pp. 120-125.