Robust evaluation for quality of experience in crowdsourcing

MM 2013 - Proceedings of the 2013 ACM Multimedia Conference

Volumn , Issue , 2013, Pages 43-52

  Xu, Qianqian ^a   Xiong, Jiechao ^a   Huang, Qingming ^b   Yao, Yuan ^a  

^a PEKING UNIVERSITY   (China)

^b INSTITUTE OF COMPUTING TECHNOLOGY   (China)

Author keywords

Crowdsourcing; Hodge decomposition; LASSO; Outlier detection; Paired comparison; Quality of experience (qoe); Random graph; Robust evaluation

Indexed keywords

CROWDSOURCING; HODGE DECOMPOSITION; LASSO; OUTLIER DETECTION; PAIRED COMPARISON; QUALITY OF EXPERIENCE (QOE); RANDOM GRAPHS; ROBUST EVALUATION;

EXPERIMENTS; STATISTICS;

QUALITY OF SERVICE;

EID: 84887440006     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2502081.2502083     Document Type: Conference Paper

References (33)

1. ITU-R Recommendation P.800. Methods for subjective determination of transmission quality, 1996.
2. Subjective quality assessment irccyn/ivc database. http://www2.irccyn.ec- nantes.fr/ivcdb/2005.
3. LIVE image & video quality assessment database. http://live.ece. utexas.edu/research/quality/2008.
4. K. Arrow. A difficulty in the concept of social welfare. Journal of Political Economy, 58(4):328-346, 1950.
5. A.-L. Barabasi and R. Albert. Emergence of scaling in random networks. Science, 286(5439):509-512, 1999.
6. K.-T. Chen, C.-C. Wu, Y.-C. Chang, and C.-L. Lei. A crowdsourceable QoE evaluation framework for multimedia content. pages 491-500. ACM Multimedia, 2009.
7. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression. Annals of Statistics, 32(2):407-499, 2004.
8. P. Erdös and A. Rényi On random graphs I. Publicationes Mathematicae-Debrecen, 6:290-297, 1959.
9. J. Fan and R. Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of American Statistical Association, pages 1348-1360, 2001.
10. I. Gannaz. Robust estimation and wavelet thresholding in partial linear models. Statistics and Computing, 17:293-310, 2007. arXiv:math/0612066v1.
11. B. Gardlo, M. Ries, and T. Hoßfeld. Impact of screening technique on crowdsourcing QoE assessments. International Conference Radioelektronika 2012, Special Session on Quality in multimedia systems, 2012.
12. Y. Grandvalet, C. Ambroise, and J. Chiquet. Sparsity by worst-case quadratic penalties. 2012. arXiv:1210.2077.
13. R. Herbrich, T. Graepel, and K. Obermayer. Large margin rank boundaries for ordinal regression. MIT Press, 2000.
14. V. Hodge and J. Austin. A survey of outlier detection methodologies. Artificial Intelligence Review, 22(2):85-126, 2004.
15. J. Howe. The rise of crowdsourcing. Wired Magazine, 14(6):176-183, 2006.
16. P. J. Huber. Robust Statistics. New York: Wiley, 1981.
17. X. Jiang, L.-H. Lim, Y. Yao, and Y. Ye. Statistical ranking and combinatorial Hodge theory. Mathematical Programming, 2010.
18. C. Keimel, J. Habigt, and K. Diepold. Challenges in crowd-based video quality assessment. In Forth International Workshop on Quality of Multimedia Experience (QoMEX 2012), 2012.
19. M. Kendall and B. Smith. On the method of paired comparisons. Biometrika, 31(3-4):324-345, 1940.
20. A. Kittur, E. Chi, and B. Suh. Crowdsourcing user studies with Mechanical Turk. pages 453-456. SIGCHI conference on Human factors in computing systems, 2008.
21. W. Lin and C.-C. J. Kuo. Perceptual visual quality metrics: A survey. Journal of Visual Communication and Image Representation, 22(4):297-312, 2011.
22. B. Osting, J. Darbon, and S. Osher. Statistical ranking using the L1-norm on graphs. AIMS', 2012.
23. M. Penrose. Random Geometric Graphs (Oxford Studies in Probability). Oxford University Press, 2003.
24. T. Saaty. A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology, 15(3):234-281, 1977.
25. Y. She and A. B. Owen. Outlier detection using nonconvex penalized regression. Journal of the American Statistical Association, 106(494):626-639, 2011.
26. R. Tibshirani. Regression shrinkage and selection via the LASSO. J. of the Royal Statistical Society, Series B, 58(1):267-288, 1996.
27. M. J. Wainwright. Sharp thresholds for high-dimensional and noisy sparsity recovery using L1-constrained quadratic programming (LASSO). IEEE Transactions on Information Theory, 55(5):2183-2202, 2009.
28. D. Watts and S. Strogatz. Collective dynamics of 'small-world' networks. Nature, (393):440-442, 1998.
29. N. Wormald. Models of random regular graphs. pages 239-298. In Surveys in Combinatorics, 1999.
30. J. Xiong, X. Cheng, and Y. Yao. Robust ranking on graphs. Journal of Machine Learning Research, submitted, 2013.
31. Q. Xu, Q. Huang, T. Jiang, B. Yan, W. Lin, and Y. Yao. HodgeRank on random graphs for subjective video quality assessment. IEEE Transactions on Multimedia, 14(3):844-857, 2012.
32. Q. Xu, Q. Huang, and Y. Yao. Online crowdsourcing subjective image quality assessment. pages 359-368. ACM Multimedia, 2012.
33. Q. Xu, T. Jiang, Y. Yao, Q. Huang, B. Yan, and W. Lin. Random partial paired comparison for subjective video quality assessment via HodgeRank. pages 393-402. ACM Multimedia, 2011.