Best practices for qoe crowdtesting: Qoe assessment with crowdsourcing

IEEE Transactions on Multimedia

Volumn 16, Issue 2, 2014, Pages 541-558

  Hossfeld, Tobias ^a   Keimel, Christian ^b   Hirth, Matthias ^a   Gardlo, Bruno ^c   Habigt, Julian ^b   Diepold, Klaus ^b   Tran Gia, Phuoc ^a  

^a UNIVERSITY OF WÜRZBURG   (Germany)

^b TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^c TELECOMMUNICATIONS RESEARCH CENTER VIENNA FTW   (Austria)

Author keywords

Best practices; cognition; human computer inter action; instrumentation and measurement multimedia systems; testing

Indexed keywords

BEST PRACTICES; COGNITION; ENVIRONMENTAL CONTEXTS; INFLUENCE OF INCENTIVES; INSTRUMENTATION AND MEASUREMENTS; MULTIMEDIA APPLICATIONS; QUALITY OF EXPERIENCE (QOE); VIDEO QUALITY ASSESSMENT;

MULTIMEDIA SYSTEMS; TESTING;

QUALITY OF SERVICE;

EID: 84893421670     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2013.2291663     Document Type: Article

References (65)

1. E. Estellés-Arolas and F. G.-L. Guevara, "Towards an integrated crowdsourcing definition," J. Inf. Sci., vol. 38, no. 2, pp. 189-200, 2012.
2. ReCaptcha, Jul. 2013. [Online]. Available: http://www.google.com/ recaptcha.
3. InnoCentive. Jul. 2013. [Online]. Available: http://www.innocentive. com.
4. T. Hoßfeld, M. Hirth, and P. Tran-Gia, "Modeling of crowdsourcing platforms and granularity of work organization in future internet," in Proc. Int. Teletraffic Congr. (ITC), San Francisco,CA,USA, Sep. 2011.
5. Crowdflower, Jul. 2013. [Online]. Available: http://crowdflower.com.
6. CrowdSource, Jul. 2013. [Online]. Available: http://www.crowdsource. com.
7. Microtask, Jul. 2013. [Online]. Available: http://www.microtask.com.
8. TaskRabbit, Jul. 2013. [Online].Available: http://www.taskrabbit.com.
9. Amazon Mechanical Turk, Feb. 2013. [Online]. Available: http://mturk.com.
10. Microworkers, Feb. 2013. [Online]. Available: http://microworkers. com.
11. Facebook, Jul. 2013. [Online]. Available: http://www.facebook.com.
12. P. L. Callet, S. Möller, and A. Perkis, Eds., Qualinet White Paper on Definitions of Quality of Experience (2012), Lausanne, Switzerland, 2012.
13. T. Hoßfeld, R. Schatz, and E. P. BiersackL, "Internet video delivery in youtube: From traffic measurements to quality of experience," in Data Traffic Monitoring and Analysis: From Measurement, Classification and Anomaly Detection to Quality of Experience,M.M. Ernst Biersack and C. Callegari, Eds. New York, NY, USA: Springer's Computer Communications and Networks series, 2013.
14. T. Hoßfeld, R. Schatz, M. Varela, and C. Timmerer, "Challenges of qoe management for cloud applications," IEEE Commun. Mag., vol. 50, no. 4, pp. 28-36, 2012.
15. S. Ickin, K. Wac, M. Fiedler, L. Janowski, J.-H. Hong, and A. Dey, "Factors influencing quality of experience of commonly used mobile applications," IEEE Commun. Mag., vol. 50, no. 4, pp. 48-56, 2012.
16. J.-S. Lee, F. D. Simone, T. Ebrahimi, N. Ramzan, and E. Izquierdo, "Quality assessment of multidimensional video scalability," IEEE Commun. Mag., vol. 50, no. 4, pp. 38-46, 2012.
17. R. Schatz, T. Hoßfeld, L. Janowski, and S. Egger, "From packets to people: Quality of experience as new measurement challenge," in Data Traffic Monitoring and Analysis: From Measurement, Classification and Anomaly Detection to Quality of Experience,M.M. Ernst Biersack and C. Callegari, Eds. New York, NY, USA: Springer's Computer Communications and Networks series, 2013.
18. ITU-R BT.500 Methodology for the Subjective Assessment of the Quality for Television Pictures, ITU-R Std., Rev. 12, Sep. 2009.
19. ITU-T P.910 Subjective Video Quality Assessment Methods for Multimedia Applications, ITU-T Std., Apr. 2008.
20. T. Hoßfeld, M. Seufert, M. Hirth, T. Zinner, P. Tran-Gia, and R. Schatz, "Quantification of youtube qoe via crowdsourcing," in Proc. 2011 IEEE Int. Symp. Multimedia (ISM), Dec. 2011, pp. 494-499.
21. C. Keimel, J. Habigt, C. Horch, and K. Diepold, "Qualitycrowd-a framework for crowd-based quality evaluation," in Proc. Picture Coding Symp. (PCS) 2012, Krakow, Poland, May 2012, pp. 245-248.
22. I. R. Assembly, ITU-R BT.500-12 Methodology for the Subjective Assessment of the Quality of Television Pictures 2009.
23. C. Keimel, J. Habigt, C. Horch, and K. Diepold, "Quality evaluation in the cloud," in Proc. Int. Packet Video Workshop (PV) 2012, Munich, Germany, May 2012, pp. 155-160.
24. K.-T. Chen, C.-C. Wu, Y.-C. Chang, and C.-L. Lei, "A crowdsourceable QoE evaluation framework for multimedia content," in Proc. 17th ACM Int. Conf. Multimedia, ser. MM '09, 2009, pp. 491-500.
25. K.-T. Chen, C.-J. Chang, C.-C. Wu, Y.-C. Chang, and C.-L. Lei, "Quadrant of Euphoria: a crowdsourcing platform for QoE assessment," IEEE Netw., vol. 24, no. 2, pp. 28-35, Mar. 2010.
26. C. Wu, K. Chen, Y. Chang, and C. Lei, "Crowdsourcing multimedia qoe evaluation: A trusted framework," IEEE Trans. Multimedia, to be published.
27. T. Hoßfeld, S. Egger, R. Schatz, M. Fiedler, K. Masuch, and C. Lorentzen, "Initial delay vs. Interruptions: Between the devil and the deep blue sea," in Proc. QoMEX 2012, Yarra Valley, Australia, Jul. 2012.
28. Q. Xu, Q. Huang, T. Jiang, B. Yan, W. Lin, and Y. Yao, "Hodgerank on random graphs for subjective video quality assessment," IEEE Trans. Multimedia, vol. 14, no. 3, pp. 844-857, Jun. 2012.
29. C. Keimel, J. Habigt, and K. Diepold, "Challenges in crowd-based video quality assessment," in Proc. 4th Int. Workshop Quality of Multimedia Experience (QoMEX 2012), Yarra Valley, Australia, Jul. 2012.
30. R. Schatz, S. Egger, and K. Masuch, "The impact of test duration on user fatigue and reliability of subjective quality ratings," J. Audio Eng. Soc. (JAES), 2012.
31. T. Hoßfeld, R. Schatz, S. Biedermann, A. Platzer, S. Egger, and M. Fiedler, "The memory effect and its implications on web QoE modeling," in Proc. 23rd Int. Teletraffic Congr. (ITC 2011), San Francisco, CA, USA, Sep. 2011.
32. M. Hirth, T. Hoßfeld, and P. Tran-Gia, "Anatomy of a crowdsourcing platform-using the example of microworkers.com," in Proc. Workshop Future Inter. and Next Generation Networks (FINGNet), Seoul, Korea, Jun. 2011.
33. S. Suri, D. Goldstein, and W. Mason, "Honesty in an online labor market," in Human Computation: Papers from the 2011 AAAI Workshop (WS-11-11), 2011.
34. J. Le, A. Edmonds, V. Hester, and L. Biewald, "Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution," in Proc. SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, 2010, pp. 21-26.
35. D. Oleson, A. Sorokin, G. Laughlin, V. Hester, J. Le, and L. Biewald, "Programmatic gold: Targeted and scalable quality assurance in crowdsourcing," Proc. HComp, 2011.
36. P. Ipeirotis, F. Provost, and J.Wang, "Quality management on Amazon Mechanical Turk," in Proc. ACM SIGKDD Workshop Human Computation, 2010, pp. 64-67.
37. A. Kittur, E. Chi, and B. Suh, "Crowdsourcing user studies with mechanical turk," in Proc. 26th Annu. ACM SIGCHI Conf. Human Factors in Computing Syst., 2008, pp. 453-456.
38. L. V. Ahn and L. Dabbish, "Labeling images with a computer game," in Proc. ACM SIGCHI Conf. Human Factors in Comput. Syst, 2004, pp. 319-326.
39. C. Eickhoff and A. P. d. Vries, "How crowdsourcable is your task," in Proc. Workshop Crowdsourcing for Search and Data Mining (CSDM) at the 4th ACM Int. Conf.Web Search and Data Mining (WSDM), 2011, pp. 11-14.
40. S. Dow, A. Kulkarni, B. Bunge, T. Nguyen, S. Klemmer, and B. Hartmann, "Shepherding the crowd: managing and providing feedback to crowd workers," in Proc. 2011 Annu. ACM Conf. Extended Abstracts on Human Factors in Comput. Syst., 2011, pp. 1669-1674.
41. G. Little, L. Chilton, M. Goldman, and R. Miller, "Turkit: tools for iterative tasks on mechanical turk," in Proc. ACM SIGKDD Workshop Human Computation, 2009, pp. 29-30.
42. P. Dai et al., "Decision-theoretic control of crowd-sourced workflows," in Proc. 24th AAAI Conf. Artificial Intell., 2010.
43. A. Kittur and R. Kraut, "Harnessing the wisdom of crowds in Wikipedia: quality through coordination," in Proc. 2008 ACM Conf. Comput. Supported Cooperative Work, 2008, pp. 37-46.
44. M. Hirth, T. Hoßfeld, and P. Tran-Gia, "Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms," Math. Comput. Model., 2012.
45. C. Eickhoff and A. P. de Vries, "Increasing cheat robustness of crowdsourcing tasks," Inf. Retriev., pp. 1-17, 2012.
46. M. Sabou, K. Bontcheva, and A. Scharl, "Crowdsourcing research opportunities: lessons from natural language processing," in Proc. 12th ACM Int. Conf. Knowledge Management and Knowledge Technologies, 2012, p. 17.
47. C. Harris, "You're hired! an examination of crowdsourcing incentive models in human resource tasks," in Proc. WSDM Workshop Crowdsourcing for Search and Data Mining (CSDM), 2011, pp. 15-18.
48. A. D. Shaw, J. J. Horton, and D. L. Chen, "Designing incentives for inexpert human raters," in Proc. ACM 2011 Conf. Computer Supported Cooperative Work, ser. CSCW '11, New York, NY, USA, 2011, pp. 275-284. [Online]. Available: http://doi.acm.org/10.1145/1958824. 1958865.
49. L. Von Ahn, "Games with a purpose," Comput., vol. 39, no. 6, pp. 92-94, 2006.
50. C. Eickhoff, C. G. Harris, A. P. de Vries, and P. Srinivasan, "Quality Through Flow And Immersion: Gamifying Crowdsourced Relevance Assessments," in Proc. ACM SIGIR Conf. Res. and Develop. in Inf. Retrieval 2012, Aug. 2012.
51. S. Owl, Java market share, Oct. 2012. [Online]. Available: http://www. statowl.com/java.php.
52. B. Gardlo, M. Ries, T. Hoßfeld, and R. Schatz, "Microworkers vs. Facebook: The impact of crowdsourcing platform choice on experimental results," in Proc. QoMEX 2012, Yarra Valley, Australia, Jul. 2012.
53. I. R. Assembly, ITU-R BS.1116-1, Methods for the Subjective Assessment of Small Impairments in Audio Systems Including Multichannel Sound Systems, 1997.
54. M. Soleymani and M. Larson, "Crowdsourcing for affective annotation of video: Development of a viewer-reported boredom corpus," in Proc. ACM SIGIR 2010 Workshop Crowdsourcing for Search Evaluation (CSE 2010), Jul. 2010, pp. 4-8.
55. F.D. Simone,M. Tagliasacchi,M.Naccari, S. Tubaro, and S. Ebrahimi, "A H.264/AVC video database for the evaluation of quality metrics," in Proc. IEEE ICASSP, 2010, pp. 2430-2433.
56. F. D. Simone, M. Naccari, M. Tagliasacchi, F. Dufaux, S. Tubaro, and T. Ebrahimi, "Subjective assessment of H.264/AVC video sequences transmitted over a noisy channel," in Proc. QoMEX 2009, San Diego, CA, USA, Jul. 2009.
57. O. Alonso, D. E. Rose, and B. Stewart, "Crowdsourcing for relevance evaluation," SIGIR Forum, vol. 42, no. 2, pp. 9-15, Nov. 2008. [Online]. Available: http://doi.acm.org/10.1145/1480506.1480508.
58. J. S. Downs, M. B. Holbrook, S. Sheng, and L. F. Cranor, "Are your participants gaming the system?: screening mechanical turk workers," in Proc. ACM SIGCHI Conf.Human Factors in Comput. Syst., ser. CHI '10, New York, NY, USA, 2010, pp. 2399-2402. [Online]. Available: http://doi.acm.org/10.1145/ 1753326.1753688.
59. T. Hoßfeld, R. Schatz, and S. Egger, "SOS: The MOS is not enough!," in Proc. QoMEX 2011, Mechelen, Belgium, Sep. 2011.
60. F. Ribeiro, D. Florencio, C. Zhang, and M. Seltzer, "Crowdmos: An approach for crowdsourcing mean opinion score studies," in Proc. 2011 IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), May 2011, pp. 2416-2419.
61. S.-H. Kim, H. Yun, and J. S.Yi, "Howto filter out random clickers in a crowdsourcing-based study? (Research Paper)," in Proc. BELIV 2012 Workshop: Beyond Time and Errors-Novel Evaluation Methods for Visualization, Seattle, WA, USA, Oct. 2012.
62. K. L. Gwet, Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Multiple Raters. Gaithersburg, MD, USA: Advanced Analytics Press, 2012.
63. A. F. Hayes and K. Krippendorff, "Answering the call for a standard reliability measure for coding data," Commun. Meth. Meas., vol. 1, no. 1, pp. 77-89, 2007.
64. B.Gardlo, M. Ries, and T. Hoßfeld, "Impact of screening technique on crowdsourcing qoe assessments," in Proc. 22nd Int. Conf. Radioelektronika 2012, Special Session on Quality in Multimedia Syst., Brno, Czech Republic, Apr. 2012.
65. P.-Y. Hsueh, P.Melville, and V. Sindhwani, "Data quality fromcrowdsourcing: a study of annotation selection criteria," in Proc. NAACL HLT 2009Workshop Active Learning for Natural Language Processing, ser. HLT '09: Association for Computational Linguistics, Stroudsburg, PA, USA, 2009, pp. 27-35. [Online]. Available: http://dl.acm.org/citation. cfm?id=1564131.1564137.