Survey of web-based crowdsourcing frameworks for subjective quality assessment

2014 IEEE International Workshop on Multimedia Signal Processing, MMSP 2014

Volumn , Issue , 2014, Pages

  Hoßfeld, Tobias ^a   Hirth, Matthias ^a   Korshunov, Pavel ^b   Hanhart, Philippe ^b   Gardlo, Bruno ^c   Keimel, Christian ^d   Timmerer, Christian ^e  

^a UNIVERSITY OF WÜRZBURG   (Germany)

^b EPFL   (Switzerland)

^c TELECOMMUNICATIONS RESEARCH CENTER VIENNA FTW   (Austria)

^d TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^e UNIVERSITY OF KLAGENFURT   (Austria)

Author keywords

[No Author keywords available]

Indexed keywords

CROWDSOURCING; DIGITAL STORAGE; QUALITY CONTROL; SIGNAL PROCESSING; WEBSITES;

AMAZON MECHANICAL TURKS; CROWDSOURCING PLATFORMS; MULTIMEDIA CONTENTS; ON-LINE ASSESSMENT; QUALITY OF EXPERIENCE (QOE); SUBJECTIVE QUALITY ASSESSMENTS; TESTING METHODOLOGY; WEB-BASED FRAMEWORK;

QUALITY OF SERVICE;

EID: 84914141916     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/MMSP.2014.6958831     Document Type: Conference Paper

References (26)

1. T. Hoßfeld, M. Seufert, M. Hirth, T. Zinner, P. Tran-Gia, and R. Schatz, "Quantification of YouTube QoE via crowdsourcing," in Symposium on Multimedia, Dana Point, USA, Dec. 2011.
2. J. Redi, T. Hoßfeld, P. Korshunov, F. Mazza, I. Povoa, and C. Keimel, "Crowdsourcing-based multimedia subjective evaluations: A case study on image recognizability and aesthetic appeal," in Workshop on Crowdsourcing for Multimedia, Barcelona, ES, Oct. 2013.
3. I. Hupont, P. Lebreton, T. Mäki, E. Skodras, and M. Hirth, "Is it possible to crowdsource emotions?" in International Conference on Communications and Electronics, Da Nang, VN, Jul. 2014.
4. P. Hanhart, P. Korshunov, and T. Ebrahimi, "Crowd-based quality assessment of multiview video plus depth coding," in IEEE International Conference on Image Processing, ICIP'2014, Paris France, Apr. 2014.
5. P. Korshunov, H. Nemoto, A. Skodras, and T. Ebrahimi, "Crowdsourcing-based evaluation of privacy in HDR images," in SPIE Photonics Europe 2014, Optics, Photonics and Digital Technologies for Multimedia Applications, Brussels, Belgium, Apr. 2014.
6. F. Ribeiro, D. Florencio, C. Zhang, and M. Seltzer, "CrowdMOS: An approach for crowdsourcing mean opinion score studies," in International Conference on Acoustics, Speech and Signal Processing, Prague, CZ, May 2011.
7. ITU-R BT. 500 Methodology for the Subjective Assessment of the Quality for Television Pictures, ITU-R Std., Rev. 12, Sep. 2009.
8. T. Hossfeld, C. Keimel, M. Hirth, B. Gardlo, J. Habigt, K. Diepold, and P. Tran-Gia, "Best practices for qoe crowdtesting: QoE assessment with crowdsourcing," Transactions on Multimedia, vol. 16, no. 2, Feb 2014.
9. F. Wei, J. Xu, T. Wen, X. Liu, and H. Yan, "Smart phone based online QoE assessment for end-to-end multimedia services on 3G mobile Internet," in Consumer Electronics, Communications and Networks, Yichang, CN, Apr. 2012.
10. A. K. Jain, C. Bal, and T. Q. Nguyen, "Tally: A web-based subjective testing tool," in Workshop on Quality of Multimedia Experience, Klagenfurth, AU, Jul. 2013.
11. O. Figuerola Salas, V. Adzic, A. Shah, and H. Kalva, "Assessing internet video quality using crowdsourcing," in Workshop on Crowdsourcing for multimedia, Barcelona, ES, Oct. 2013.
12. K.-T. Chen, C.-C. Wu, Y.-C. Chang, and C.-L. Lei, "A crowdsourceable QoE evaluation framework for multimedia content," in Multimedia, Beijing, CN, Oct. 2009.
13. K.-T. Chen, C.-J. Chang, C.-C. Wu, Y.-C. Chang, and C.-L. Lei, "Quadrant of euphoria: A crowdsourcing platform for QoE assessment," Network, vol. 24, no. 2, Mar. 2010.
14. C. Wu, K. Chen, Y. Chang, and C. Lei, "Crowdsourcing multimedia QoE evaluation: A trusted framework," Transactions on Multimedia, vol. 15, no. 99, Jul. 2013.
15. ITU-T P. 800 Methods for subjective determination of transmission quality, ITU-T Std., Aug. 1996.
16. ITU-R BS. 1534-1 Method for the subjective assessment of intermediate quality levels of coding systems, ITU-R Std., Oct. 2003.
17. F. Ribeiro, D. Florencio, and V. Nascimento, "Crowdsourcing subjective image quality evaluation," in Image Processing, Brussels, BE, Sep. 2011.
18. ITU-T P. 910 Subjective video quality assessment methods for multimedia applications, ITU-T Std., Apr. 2008.
19. C. Keimel, J. Habigt, C. Horch, and K. Diepold, "QualityCrowd-a framework for crowd-based quality evaluation," in Picture Coding Symposium, Krakow, PL, May 2012.
20. B. Rainer, M. Waltl, and C. Timmerer, "A web based subjective evaluation platform," in Workshop on Quality of Multimedia Experience, Klagenfurth, AT, Jul. 2013.
21. ITU Radiocommunication Assembly, "ITU-R BT. 500-12 Methodology for the subjective assessment of the quality of television pictures," 2009.
22. S. Kraft and U. Zölzer, "BeaqleJS: HTML5 and JavaScript based framework for the subjective evaluation of audio quality," in Linux Audio Conference, Karlsruhe, DE, May 2014.
23. B. Gardlo, S. Egger, M. Seufert, and R. Schatz, "Crowdsourcing 2. 0: Enhancing execution speed and reliability of web-based QoE testing," in International Conference on Communications, Sydney, AU, Jun. 2014.
24. S.-H. Kim, H. Yun, and J. S. Yi, "How to filter out random clickers in a crowdsourcing-based study?" in BELIV Workshop: Beyond Time and Errors-Novel Evaluation Methods for Visualization, Seattle, USA, Oct. 2012.
25. Q. Xu, Q. Huang, T. Jiang, B. Yan, W. Lin, and Y. Yao, "HodgeRank on random graphs for subjective video quality assessment," Transactions on Multimedia, vol. 14, no. 3, Jun. 2012.
26. Q. Xu, Q. Huang, and Y. Yao, "Online crowdsourcing subjective image quality assessment," in Multimedia, Nara, JP, Oct. 2012.