Measurement of quality of experience of video-on-demand services: A survey

IEEE Communications Surveys and Tutorials

Volumn 18, Issue 1, 2016, Pages 401-418

  Juluri, Parikshit ^a   Tamarapalli, Venkatesh ^b   Medhi, Deep ^a  

^a UNIVERSITY OF MISSOURI KANSAS CITY   (United States)

^b INDIAN INSTITUTE OF TECHNOLOGY GUWAHATI   (India)

Author keywords

[No Author keywords available]

Indexed keywords

QUALITY OF SERVICE; SURVEYS; VIDEO STREAMING;

MEASUREMENT METHODOLOGY; ON-DEMAND STREAMING; ONLINE VIDEO STREAMING; QUALITY OF EXPERIENCE (QOE); SERVICE PROVIDER; USERS' PERCEPTION; VIDEO ON DEMAND SERVICES; VIDEO STREAMING SERVICES;

VIDEO ON DEMAND;

EID: 84962426303     PISSN: None     EISSN: 1553877X     Source Type: Journal    
DOI: 10.1109/COMST.2015.2401424     Document Type: Review

References (105)

1. Cisco, "Cisco Visual Networking Index: Forecast and Methodology, 2012-2017," San Jose, CA, USA, Retrieved 4. 3. 2014. Online. Available: http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/ns537/ns705/ns827/white-paper-c11-481360.pdf
2. Sandvine, "Global Internet Phenomena Report: 1H 2013," Waterloo, ON, Canada, Retrieved 4. 3. 2014. Online. Available: https://www.sandvine.com/downloads/general/global-internet-phenomena/2013/sandvine-global-internet-phenomena-report-1h-2013.pdf
3. ITU-T Recommendation P. 10/G. 100-Amendment 3, "Vocabulary for performance and quality of service," Recommendations of the ITU, Telecommunications Sector, 12, 2012.
4. ITU-T Recommendation P. 10/G. 100-Amendment 2, "Vocabulary for performance and quality of service," Recommendations of the ITU, Telecommunications Sector, 12, 2012.
5. R. Schatz, T. Hoßfeld, L. Janowski, and S. Egger, "From packets to people: Quality of experience as a new measurement challenge," in Data Traffic Monitoring and Analysis, vol. 7754, E. Biersack, C. Callegari, and M. Matijasevic, Eds. Berlin, Germany: Springer-Verlag, 2013, ser. Lecture Notes in Computer Science, pp. 219-263. Online. Available: http://dx.doi.org/10.1007/978-3-642-36784-7-10
6. P. L. Callet, S. Moller, and A. Perkis, "Qualinet white paper on definitions of quality of experience," in European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003). Wadern, Germany: Dagstuhl, 2013.
7. T. Hoßfeld, R. Schatz, E. Biersack, and L. Plissonneau, "Internet video delivery in YouTube: From traffic measurements to quality of experience," in Data Traffic Monitoring and Analysis. Berlin, Germany: Springer-Verlag, 2013.
8. N. Staelens et al., "Assessing quality of experience of IPTV and video on demand services in real-life environments," IEEE Trans. Broadcast., vol. 56, no. 4, pp. 458-466, Dec. 2010.
9. Alliance for Telecommunications Industry Solutions (ATIS), "ATIS IPTV Exploratory Group Report and Recommendation to the TOPS Council," Washington, DC, USA, Jul. 2006.
10. HTTP Archive. Online. Available: http://httparchive.org/trends.php
11. P. Gill, M. Arlitt, Z. Li, and A. Mahanti, "YouTube traffic characterization: A view from the edge," in Proc. IMC, 2007, pp. 15-28.
12. D. K. Krishnappa, D. Bhat, and M. Zink, "DASHing YouTube: An analysis of using DASH in YouTube video service," in Proc. 38th Annu. IEEE Conf. Local Comput. Netw., 2013, pp. 407-415.
13. H. Schulzrinne, A. Rao, and R. Lanphier, RFC2326-Real Time Streaming Protocol (RTSP), Apr. 1998.
14. H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, "RTP: A transport protocol for real-time applications," rFC 3550, 2003.
15. Real Networks Inc., "Real player 8 plus user manual," Seattle, WA, USA, 2000. Online. Available: http://docs.real.com/docs/playerplus/plus-manual.8-pdfs/rp8manualfull.pdf
16. Real Networks Inc., "Configuring RealPlayer Version 7 to Work From Behind a Firewall," Seattle, WA, USA, Retrieved 4. 3. 2014, 2014. Online. Available: http://service.real.com/firewall/configRP7.html
17. Apple, "Quicktime 7. 3," Retrieved 4. 3. 2014. Online. Available: http://www.apple.com/quicktime/
18. Microsoft, "Which protocols does windows media player use for streaming?" Retrieved 10. 13. 2014. Online. Available: http://windows. microsoft.com/en-us/windows/which-streaming-protocols-media-player
19. Adobe, "Adobes Real Time Messaging Protocol," Retrieved 4. 3. 2014. Online. Available: http://www.adobe.com/devnet/rtmp.html
20. V. Adhikari, Y. Guo, F. Hao, V. Hilt, and Z.-L. Zhang, "A tale of three CDNs: An active measurement study of Hulu and its CDNs," in Proc. IEEE INFOCOM WKSHPS, Mar. 2012, pp. 7-12.
21. Cisco, HTT versus RTMP: Which way to go and why?" San Jose, CA, USA. Online. Available: http://www.cisco.com/c/en/us/products/collateral/video/as5100-series-media-processor/white-paper-c11-675935.html
22. B. Staehle, M. Hirth, F. Wamser, R. Pries, and D. Staehle, "YoM: A YouTube application comfort monitoring tool," Inst. Informatik, Berlin, Germany, Tech. Rep. 467, Mar. 2010.
23. M. Ghobadi, Y. Cheng, A. Jain, and M. Mathis, "Trickle: Rate limiting YouTube video streaming," in Proc. USENIX Conf. ATC, Berkeley, CA, USA, 2012, p. 17.
24. S. Alcock and R. Nelson, "Application flow control in YouTube video streams," SIGCOMM Comput. Commun. Rev., vol. 41, no. 2, pp. 24-30, Apr. 2011. Online. Available: http://doi.acm.org/10.1145/1971162.1971166
25. A. Finamore, M. Mellia, M. M. Munafo, R. Torres, and S. G. Rao, "YouTube everywhere: Impact of device and infrastructure synergies on user experience," in Proc. ACM SIGCOMM Conf. IMC, 2011, pp. 345-360.
26. R. Leider, "MPEG DASH Supersession-A session sponsored by DASH," IBC, London, U. K., Tech. Rep., 2014. Online. Available: http://www.ibc.org/
27. I. Sodagar, "The MPEG-DASH standard for multimedia streaming over the Internet," IEEE Trans. Multimedia, vol. 18, no. 4, pp. 62-67, Apr. 2011.
28. L. Yitong et al., "A study on quality of experience for adaptive streaming service," in Proc. ICC Workshops, Jun. 2013, pp. 682-686.
29. T. Arsan, "Review of bandwidth estimation tools and application to bandwidth adaptive video streaming," in Proc. 9th Int. Conf. HONET, Dec. 2012, pp. 152-156.
30. K. Miller, E. Quacchio, G. Gennari, and A. Wolisz, "Adaptation algorithm for adaptive streaming over HTTP," in Proc. 19th Int. PV Workshop, May 2012, pp. 173-178.
31. C. Liu, I. Bouazizi, and M. Gabbouj, "Rate adaptation for adaptive HTTP streaming," in Proc. 2nd Annu. ACM Conf. MMSys, New York, NY, USA, 2011, pp. 169-174. Online. Available: http://doi.acm.org/10.1145/1943552.1943575
32. Adobe, "AdobeHTTPDynamicStreaming,"Retrieved 4. 3. 2014. Online. Available: http://www.adobe.com/products/httpdynamicstreaming/
33. Microsoft, "IIS smooth streaming transport protocol," Retrieved 4.3.2014. Online. Available: http://go.microsoft.com/?linkid=9682896
34. R. Pantos and E. W. May, "HTTP Live Streaming, Informational Internet Draft," Mar. 2011. Online. Available: http://tools.ietf.org/html/draft-pantos-http-live-streaming-06
35. "Netflix quarterly earnings," Accessed 17-April-2013. Online. Available: http://ir.netflix.com/results.cfm
36. "Microsoft Silverlight," Retrieved 4. 3. 2014. Online. Available: http://www.microsoft.com/silverlight/
37. V. Adhikari et al., "Unreeling Netflix: Understanding and improving multi-CDN movie delivery," in Proc. IEEE INFOCOM, 2012, pp. 1620-1628.
38. M. Fiedler, T. Hoßfeld, and P. Tran-Gia, "A generic quantitative relationship between quality of experience and quality of service," IEEE Netw., vol. 24, no. 2, pp. 36-41, Mar./Apr. 2010.
39. R. Serral-Graci et al., "An overview of quality of experience measurement challenges for video applications in IP networks," in Proc. Int. Conf. WWIC, 2010, pp. 252-263.
40. Z. Wang, L. Lu, and A. Bovik, "Video quality assessment using structural distortion measurement," in Proc. Int. Conf. Image Process., 2002, vol. 3, pp. III-65-III-68.
41. Y. Chen, K. Wu, and Q. Zhang, "From QoS to QoE: A survey and tutorial on state of art, evolution and future directions of video quality analysis," IEEE Commun. Surveys Tuts., 2014.
42. M. Saxena, U. Sharan, and S. Fahmy, "Analyzing video services in Web 2. 0: A global perspective," in Proc. Int. Workshop NOSSDAV, 2008, pp. 39-44.
43. S. S. Krishnan and R. K. Sitaraman, "Video stream quality impacts viewer behavior: Inferring causality using quasi-experimental designs," in Proc. ACM Conf. IMC, New York, NY, USA, 2012, pp. 211-224. Online. Available: http://doi.acm.org/10.1145/2398776.2398799
44. T. Hoßfeld et al., "Initial delay vs. interruptions: Between the devil and the deep blue sea," in Proc. 4th Int. Workshop QoMEX, Jul. 2012, pp. 1-6.
45. T. Hoßfeld et al., "Quantification of YouTube QoE via crowdsourcing," in Proc. IEEE ISM, 2011, pp. 494-499.
46. Y. Qi and M. Dai, "The effect of frame freezing and frame skipping on video quality," in Proc. Int. Conf. IIH-MSP, 2006, pp. 423-426.
47. B. Lewcio, B. Belmudez, T. Enghardt, and S. Moller, "On the way to high-quality video calls in future mobile networks," in Proc. QoMEX, 2011, pp. 43-48.
48. M. Montenovo, A. Perot, M. Carli, P. Cicchetti, and A. Neri, "Objective quality evaluation of video services," in Proc. 3rd Int. Workshop Video Process. Qual. Metrics Consum. Electron., 2006, pp. 235-242.
49. S. Winkler, A. Sharma, and D. McNally, "Perceptual video quality and blockiness metrics for multimedia streaming applications," in Proc. Int. Symp. Wireless Pers. Multimedia Commun., 2001, pp. 547-552.
50. W. Lin and C.-C. J. Kuo, "Perceptual visual quality metrics: A survey," J. Vis. Commun. Image Represent., vol. 22, no. 4, pp. 297-312, May 2011.
51. U. Engelke and H.-J. Zepernick, "Perceptual-based quality metrics for image and video services: A survey," in Proc. 3rd EuroNGI Conf. Netw., May 2007, pp. 190-197.
52. J. You, U. Reiter, M. M. Hannuksela, M. Gabbouj, and A. Perkis, "Perceptual-based quality assessment for audio-visual services: A survey," Image Commun., vol. 25, no. 7, pp. 482-501, Aug. 2010.
53. F. Dobrian et al., "Understanding the impact of video quality on user engagement," in Proc. ACM SIGCOMM, 2011, vol. 41, pp. 362-373.
54. M. Sidibe et al., "A novel monitoring architecture for media services adaptation based on network QoS to perceived QoS mapping," Signal Image Video Process., vol. 2, no. 4, pp. 307-320, Dec. 2008.
55. O. Oyman and S. Singh, "Quality of experience for HTTP adaptive streaming services," IEEE Commun. Mag., vol. 50, no. 4, pp. 20-27, Apr. 2012.
56. M. Seufert et al., "A survey on quality of experience of HTTP adaptive streaming," IEEE Commun. Surveys Tuts., 2014.
57. A. Takahashi, D. Hands, and V. Barriac, "Standardization activities in the ITU for a QoE assessment of IPTV," IEEE Commun. Mag., vol. 46, no. 2, pp. 78-84, Feb. 2008.
58. F. Yang and S. Wan, "Bitstream-based quality assessment for networked video: A review," IEEE Commun. Mag., vol. 50, no. 11, pp. 203-209, Nov. 2012.
59. S. Chikkerur, V. Sundaram, M. Reisslein, and L. Karam, "Objective video quality assessment methods: A classification, review, performance comparison," IEEE Trans. Broadcast., vol. 57, no. 2, pp. 165-182, Jun. 2011.
60. A. Csizmar Dalal and E. Perry, "A new architecture for measuring and assessing streaming media quality," in Proc. PAM, San Diego, CA, USA, 2003, pp. 223-231.
61. A. Csizmar Dalal et al., "Predicting user-perceived quality ratings from streaming media data," in Proc. IEEE ICC, 2007, pp. 65-72.
62. A. Csizmar Dalal and K. Purrington, "Discerning user-perceived media stream quality through application-layer measurements," in Proc. Int. Conf. MSAN, 2005, pp. 44-48.
63. R. K. Mok, E. W. Chan, X. Luo, and R. K. Chang, "Inferring the QoE of HTTP video streaming from user-viewing activities," in Proc. ACM SIGCOMM W-MUST, 2011, pp. 31-36.
64. R. Mok, E. Chan, and R. Chang, "Measuring the quality of experience of HTTP video streaming," in Proc. IFIP/IEEE Int. Symp. IM Netw., 2011, pp. 485-492.
65. X. Liu et al., "A case for a coordinated internet video control plane," in Proc. ACM SIGCOMM, 2012, pp. 359-370.
66. P. Juluri, L. Plissonneau, and D. Medhi, "Pytomo: A tool for analyzing playback quality of YouTube videos," in Proc. ITC, 2011, pp. 304-305.
67. L. Plissonneau, E. Biersack, and P. Juluri, "Analyzing the impact of YouTube delivery policies on user experience," in Proc. ITC, 2012, pp. 1-8.
68. P. Juluri, L. Plissonneau, Y. Zeng, and D. Medhi, "ViewingYouTube from a metropolitan area: What do users accessing from residential ISPs experience?" in Proc. IFIP/IEEE Int. Symp. IM Netw., 2013, pp. 589-595.
69. S. Jumisko-Pyykko, V. Kumar, M. V. M. Liinasuo, and M. Hannuksela, "Acceptance of audiovisual quality in erroneous television sequences over a DVB-H channel," in Proc. Int. Workshop Video Process. Qual. Metrics Consum. Electron., 2006, pp. 1-5.
70. M. Claypool and J. Tanner, "The effects of jitter on the perceptual quality of video," in Proc. ACM Multimedia Conf., 1999, pp. 115-118.
71. O. Teyeb, T. B. Sorensen, P. Mogensen, and J. Wigard, "Subjective evaluation of packet service performance in UMTS and heterogeneous networks," in Proc. ACM Int. Workshop Q2SWinet, 2006, pp. 95-102.
72. S. Rugel, T. Bauschert, T. M. Knoll, and M. Eckert, "A network-based method for measurement of Internet video streaming quality," presented at the ITG-FG 5. 2. 3-Next Generation Networks, Darmstadt, Germany, 2011.
73. Tstat, "Tstat: TCP statistic and analysis tool," Retrieved: 4. 3. 2014. Online. Available: http://tstat.tlc.polito.it/index.shtml
74. S. Latre et al., "On-line estimation of the QoE of progressive download based services in multimedia access networks," in Proc. Int. Conf. Internet Comput., 2008, pp. 181-187.
75. B. D. Vleeschauwer et al., "Enabling autonomic access network QoE management through TCP connection monitoring," in Proc. IEEE Workshop ACNM, 2007, pp. 56-63.
76. R. Huysegems et al., "Session reconstruction for HTTP adaptive streaming: Laying the foundation for network-based QoE monitoring," in Proc. IEEE IWQoS, 2012, pp. 1-9.
77. R. Schatz, T. Hoßfeld, and P. Casas, "Passive YouTube QoE monitoring for ISPs," in Proc. Int. Conf. Ubiquitous Comput., 2012, pp. 358-364.
78. ITU-T Recommendation P. 910, "Subjective video quality assessment methods for multimedia applications," Recommendations of the ITU, Telecommunications Sector.
79. Y. Wang, M. Claypool, and Z. Zuo, "An empirical study of RealVideo performance across the Internet," in Proc. 1st ACM SIGCOMM Workshop IMW, 2001, pp. 295-309.
80. Y. Wang and M. Claypool, "RealTracer-Tools for measuring the performance of RealVideo on the Internet," Multimedia Tools Appl., vol. 27, no. 3, pp. 411-430, Dec. 2005.
81. Amazon, "Amazon Mechanical Turk," Retrieved: 4. 3. 2014. Online. Available: https://www.mturk.com/mturk/welcome
82. Microworkers, Retrieved 4. 3. 2014. Online. Available: http://microworkers.com/
83. T. Hoßfeld, R. Schatz, and U. Krieger, "QoE of YouTube video streaming for current Internet transport protocols," in Measurement, Modelling, Evaluation of Computing Systems and Dependability and Fault Tolerance, vol. 8376, K. Fischbach and U. Krieger, Eds. Zurich, Switzerland: Springer-Verlag, 2014, ser. Lecture Notes in Computer Science, pp. 136-150. Online. Available: http://dx.doi.org/10.1007/978-3-319-05359-2-10
84. D. C. Robinson, Y. Jutras, and V. Craciun, "Subjective video quality assessment of HTTP adaptive streaming technologies," Bell Labs Tech. J., vol. 16, no. 4, pp. 5-23, Mar. 2012. Online. Available: http://dx.doi.org/10.1002/bltj.20531
85. F. Yang, S. Wan, Q. Xie, and H. R. Wu, "No-reference quality assessment for networked video via primary analysis of bit stream," IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 11, pp. 1544-1554, Nov. 2010.
86. S. Khirman and P. Henriksen, "Relationship between quality-of-service and quality-of-experience for public Internet service," in Proc. 3rd Passive Active Measurement Workshop, Fort Collins, CO, USA, 2002.
87. J. Shaikh, M. Fiedler, and D. Collange, "Quality of experience from user and network perspectives," Ann. Telecommun., vol. 65, no. 1/2, pp. 47-57, Feb. 2010.
88. S. Aroussi, T. Bouabana-Tebibel, and A. Mellouk, "Empirical QoE/QoS correlation model based on multiple parameters for VoD flows," in Proc. IEEE GLOBECOM, 2012, pp. 1963-1968.
89. C. Alberti et al., "Automated QoE evaluation of dynamic adaptive streaming over HTTP," in Proc. QoMEX, 2013, pp. 58-63.
90. B. Shao et al., "An adaptive system for real-time scalable video streaming with end-to-end QoS control," in Proc. 11th Int. WIAMIS, 2010, pp. 1-4.
91. ITU-TP. 1201 (2012) Amd. 2., "Amendment 2:Newappendix III-Use of p. 1201 for non-adaptive, progressive download type media streaming," Recommendations of the ITU, Telecommunications Sector, 12, 2013.
92. D. Vera, P. Rodrguez-Bocca, and G. Rubino, "QoE monitoring platform for video delivery networks," in IP Operations and Management, vol. 4786, D. Medhi, J. Nogueira, T. Pfeifer, and S. Wu, Eds. Berlin, Germany: Springer-Verlag, 2007, ser. Lecture Notes in Computer Science, pp. 131-142.
93. A. Csizmar Dalal, E. Kawaler, and S. Tucker, "Towards Real-Time stream quality prediction: Predicting video stream quality from partial stream information," in Quality of Service in Heterogeneous Networks, vol. 22. Berlin, Germany: Springer-Verlag, 2009, pp. 20-33.
94. H. French, J. Lin, T. Phan, and A. Dalal, "Real time video QoE analysis of RTMP streams," in Proc. 30th IPCCC, 2011, pp. 1-2.
95. W. Cherif, A. Ksentini, D. Negru, and M. Sidibe, "A-PSQA: Efficient real-time video streaming QoE tool in a future media Internet context," in Proc. IEEE ICME, 2011, pp. 1-6.
96. K. Singh, Y. Hadjadj-Aoul, and G. Rubino, "Quality of experience estimation for adaptive HTTP/TCP video streaming using H. 264/AVC," in Proc. CCNC, 2012, pp. 127-131.
97. A. Balachandran et al., "Developing a predictive model of quality of experience for Internet video," in Proc. ACM SIGCOMM Conf. SIGCOMM, New York, NY, USA, 2013, pp. 339-350.
98. A. L. Vlado Menkovski and G. Exarchakos, "Online learning for quality of experience management," in Proc. 19th Mach. Learn. Conf. Belgium and The Netherlands, 2010, pp. 1-6.
99. F. Agboma and A. Liotta, "QoE-aware QoS management," in Proc. 6th Int. Conf. Adv. MoMM, 2008, pp. 111-116.
100. F. Yang, J. Song, S. Wan, and H. R. Wu, "Content-adaptive packet-layer model for quality assessment of networked video services," IEEE J. Sel. Topics Signal Process., vol. 6, no. 6, pp. 672-683, Oct. 2012.
101. Netflix, "The ISP Speed Index From Netflix," Retrieved: 4. 3. 2014. Online. Available: http://ispspeedindex.netflix.com/
102. L. Skorin-Kapov and M. Varela, "A multi-dimensional view of QoE : The ARCU model," in Proc. 35th Int. Conv. MIPRO, May 2012, pp. 662-666.
103. A. Moorthy, L. K. Choi, A. Bovik, and G. de Veciana, "Video quality assessment on mobile devices: Subjective, behavioral and objective studies," IEEE J. Sel. Topics Signal Process., vol. 6, no. 6, pp. 652-671, Oct. 2012.
104. S. Xiang, L. Cai, and J. Pan, "Adaptive scalable video streaming in wireless networks," in Proc. 3rd MMSys Conf., New York, NY, USA, 2012, pp. 167-172.
105. A. Bokani, M. Hassan, and S. Kanhere, "HTTP-based adaptive streaming for mobile clients using Markov decision process," in Proc. 20th Int. PV Workshop, Dec. 2013, pp. 1-8.