Robust Subjective Visual Property Prediction from Crowdsourced Pairwise Labels

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 38, Issue 3, 2016, Pages 563-577

  Fu, Yanwei ^a   Hospedales, Timothy M ^b   Xiang, Tao ^b   Xiong, Jiechao ^c   Gong, Shaogang ^b   Wang, Yizhou ^d   Yao, Yuan ^c  

^a FUDAN UNIVERSITY   (China)

^b QUEEN MARY UNIVERSITY OF LONDON   (United Kingdom)

^c PEKING UNIVERSITY   (China)

^d Peking University   (China)

Author keywords

outlier detection; robust ranking; Subjective visual properties

Indexed keywords

DATA HANDLING; FORECASTING;

INTERMEDIATE REPRESENTATIONS; LEARNING TO RANK; OUTLIER DETECTION; PAIR-WISE COMPARISON; PREDICTION MODEL; ROBUST RANKING; VISUAL PROPERTIES; VISUAL RECOGNITION;

STATISTICS;

EID: 84962128645     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2015.2456887     Document Type: Article

References (63)

1. J. Donahue and K. Grauman, "Annotator rationales for visual recognition," in Proc. Int. Conf. Comput. Vis., 2011, pp. 1395-1402.
2. A. Kovashka, D. Parikh, and K. Grauman, "Whittlesearch: Image search with relative attribute feedback" in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2973-2980.
3. S. Dhar, V. Ordonez, and T. L. Berg, "High level describable attributes for predicting aesthetics and interestingness" in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 1657-1664.
4. M. Gygli, H. Grabner, H. Riemenschneider, F. Nater, and L. V. Gool, "The interestingness of images" in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1633-1640.
5. Y.-G. Jiang, Y. Wang, R. Feng, X. Xue, Y. Zheng, and H. Yang, "Understanding and predicting interestingness of videos" in Proc. AAAI, 2013, pp. 1113-1119.
6. D. Parikh and K. Grauman, "Relative attributes," in Proc. Int. Conf. Comput. Vis., 2011, pp. 503-510.
7. Z. Zhang, C. Wang, B. Xiao, W. Zhou, and S. Liu, "Robust relative attributes for human action recognition," in Proc. Pattern Anal. Appl., 2015, vol. 18, pp. 157-171.
8. K. Chen, C. Wu, Y. Chang, and C. Lei, "Crowdsourceable QoE evalutation framework for multimedia content," in Proc. 17th ACM Int. Conf. Multimedia, 2009, pp. 491-500.
9. Y. Ma, T. Xiong, Y. Zou, and K. Wang, "Person-specific age estimation under ranking framework," in Proc. 1st ACM Int. Conf. Multimedia Retrieval, 2011, pp. 38:1-38:7.
10. O. Chapelle and S. S. Keerthi, "Efficient algorithms for ranking with svms," Inf. Retrieval, vol. 13, pp. 201-215, 2010.
11. X. Chen and P. N. Bennett, "Pairwise ranking aggregation in a crowdsourced setting," in Proc. 6th ACM Int. Conf. Web Search Data Mining, 2013, pp. 193-202.
12. O. Wu, W. Hu, and J. Gao, "Learning to rank under multiple annotators," in Proc. 22nd Int. Joint Conf. Artif. Intell., 2011, pp. 1571-1576.
13. C. Long, G. Hua, and A. Kapoor, "Active visual recognition with expertise estimation in crowdsourcing," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 3000-3007.
14. A. Kittur, E. H. Chi, and B. Suh, "Crowdsourcing user studies with mechanical turk," in Proc. SIGCHI Conf. Human Factors Comput. Syst., 2008, pp. 453-456.
15. A. Kovashka and K. Grauman, "Attribute adaptation for personalized image search," in Proc. IEEE Int. Conf. Comput. Vis., Dec. 2013, pp. 3432-3439.
16. P. Welinder, S. Branson, S. Belongie, and P. Perona, "The multidimensional wisdom of crowds," in Proc. Adv. Neural Inf. Process. Syst. 23, 2010, pp. 2424-2432.
17. V. C. Raykar, S. Yu, L. H. Zhao, A. Jerebko, C. Florin, G. H. Valadez, L. Bogoni, and L. Moy, "Supervised learning from multiple experts: Whom to trust when everyone lies a bit," in Proc. 26th Annu. Int. Conf. Mach. Learn., 2009, pp. 889-896.
18. J. Whitehill, T. fan Wu, J. Bergsma, J. R. Movellan, and P. L. Ruvolo, "Whose vote should count more: Optimal integration of labels from labelers of unknown expertise," in Proc. Adv. Neural Inf. Process. Syst. 22, 2009, pp. 2035-2043.
19. X. Jiang, L.-H. Lim, Y. Yao, and Y. Ye, "Statistical ranking and combinatorial hodge theory," in Proc. Math. Program., 2011, vol. 127, no. 1, pp. 203-244.
20. R. Tibshirani, "Regression shrinkage and selection via the lasso," J. Roy. Statist. Soc., Series B, vol. 58, pp. 267-288, 1994.
21. I. Gannaz, "Robust estimation and wavelet thresholding in partial linear models," Statist. Comput., vol. 17, pp. 293-310, 2007.
22. F. L. Wauthier, N. Jojic, and M. I. Jordan, "A comparative framework for preconditioned lasso algorithms," in Proc. Neural Inf. Process. Syst., 2013, pp. 1061-1069.
23. Y. Ke, X. Tang, and F. Jing, "The design of high-level features for photo quality assessment," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog., 2006, pp. 419-426.
24. P. Isola, J. Xiao, A. Torralba, and A. Oliva, "What makes an image memorable" in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog., 2011, pp. 145-152.
25. F. Liu, Y. Niu, and M. Gleicher, "Using web photos for measuring video frame interestingness," in Proc. 21st Int. Jont Conf. Artif. Intell., 2009, pp. 2058-2063.
26. C. H. Lampert, H. Nickisch, and S. Harmeling, "Learning to detect unseen object classes by between-class attribute transfer," in Proc. IEEE Comput. Soc. Conf. Comput.Vis. Pattern Recog., 2009, pp. 951-958.
27. A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth, "Describing objects by their attributes," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog., 2009, pp. 1778-1785.
28. A. Shrivastava, S. Singh, and A. Gupta, "Constrained semi-supervised learning via attributes and comparative attributes," in Proc. 12th Eur. Conf. Comput. Vis., 2012, pp. 369-383.
29. A. Parkash and D. Parikh, "Attributes for classifier feedback," in Proc. 12th Eur. Conf. Comput. Vis., 2012, pp. 354-368.
30. A. Biswas and D. Parikh, "Simultaneous active learning of classifiers and attributes via relative feedback," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 644-651.
31. A. Sorokin and D. Forsyth, "Utility data annotation with Amazon mechanical turk," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog. Workshops, 2008, pp. 1-8.
32. G. Patterson and J. Hays, "Sun attribute database: Discovering, annotating, and recognizing scene attributes," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2751-2758.
33. W. V. Gehrlein, "Condorcet's paradox," in Proc. Theory Dec., vol. 15, no. 2, pp. 161-197, 1983.
34. L. Liang and K. Grauman, "Beyond comparing image pairs: Setwise active learning for relative attributes," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 208-215.
35. Q. Xu, Q. Huang, T. Jiang, B. Yan, W. Lin, and Y. Yao, "Hodgerank on random graphs for subjective video quality assessment," IEEE Trans. Multimedia, vol. 14, no. 3, pp. 844-857, Jun. 2012.
36. Q. Xu, Q. Huang, and Y. Yao, "Online crowdsourcing subjective image quality assessment," in Proc. 20th ACM Int. Conf. Multimedia, 2012, pp. 359-368.
37. M. Maire, S. X. Yu, and P. Perona, "Object detection and segmentation from joint embedding of parts and pixels," in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 2142-2149.
38. Z. Cao, T. Qin, T.-Y. Liu, M.-F. Tsai, and H. Li, "Learning to rank: From pairwise approach to listwise approach," in Proc. 24th Int. Conf. Mach. Learn., 2007, pp. 129-136.
39. Y. Liu, B. Gao, T.-Y. Liu, Y. Zhang, Z. Ma, S. He, and H. Li, "Browserank: Letting web users vote for page importance," in Proc. ACM SIGIR, 2008.
40. Z. Sun, T. Qin, Q. Tao, and J. Wang, "Robust sparse rank learning for non-smooth ranking measures," in Proc. 32nd Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2009, pp. 259-266.
41. Y. Fu, T. M. Hospedales, T. Xiang, S. Gong, and Y. Yao, "Interestingness prediction by robust learning to rank," in Proc. Eur. Conf. Comput. Vis., 2014, pp. 488-503.
42. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning:Data Mining, Inference, and Prediction, 2nd ed. New York, NY, USA: Springer, 2009.
43. A. N. Hirani, K. Kalyanaraman, and S. Watts, "Least squares ranking on graphs," arXiv:1011.1716, 2010.
44. P. J. Huber, Robust Statistics. New York, NY, USA: Wiley, 1981.
45. T. Sun and C.-H. Zhang, "Scaled sparse linear regression," Biometrika, vol. 99, no. 4, pp. 879-898, 2012.
46. A. Belloni, V. Chernozhukov, and L. Wang, "Pivotal recovery of sparse signals via conic programming," Biometrika, vol. 98, pp. 791-806, 2011.
47. Y. She and A. B. Owen, "Outlier detection using nonconvex penalized regression," J. Am. Statist. Assoc., vol. 106, pp. 626-639, 2011.
48. J. Friedman, T. Hastie, and R. Tibshirani, "Regularization paths for generalized linear models via coordinate descent," J. Statist. Softw., vol. 33, no. 1, pp. 1-22, 2010.
49. J. Fan and R. Li, "Variable selection via nonconcave penalized likelihood and its oracle properties," J. Am. Statist. Assoc., vol. 96, pp. 1348-1360, 2001.
50. J. Fan, R. Tang, and X. Shi, "Partial consistency with sparse incidental parameters," arXiv:1210.6950, 2012.
51. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, "Least angle regression," Ann. Statist., vol. 32, pp. 407-499, 2004.
52. N. Meinshausen and P. Bühlmann, "High-dimensional graphs and variable selection with the lasso," Ann. Statist., vol. 34, pp. 1436-1462, 2006.
53. Q. Xu, J. Xiong, Q. Huang, and Y. Yao, "Robust evaluation for quality of experience in crowdsourcing," in Proc. 21st ACM Int. Conf. Multimedia, 2013, pp. 43-52.
54. N. Kumar, A. C. Berg, P. N. Belhumeur, and S. K. Nayar, "Attribute and simile classifiers for face verification," in Proc. IEEE 12th Int. Conf. Comput. Vis., 2009, pp. 365-372.
55. A. Oliva and A. Torralba, "Modeling the shape of the scene: A holistic representation of the spatial envelope," Int. J. Comput. Vis., vol. 42, pp. 145-175, 2001.
56. Y. Fu, G. Guo, and T. Huang, "Age synthesis and estimation via faces: A survey," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 11, pp. 1955-1976, Nov. 2010.
57. S. Mika, B. Scholkopf, A. Smola, K.-R. Muller, M. Scholz, and G. Ratsch, "Kernel PCA and de-noising in feature spaces," in Proc. Adv. Neural Inf. Process. Systems II, 1999, pp. 536-542.
58. A. Vedaldi and A. Zisserman, "Efficient additive kernels via explicit feature maps," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 3, pp. 480-492, Mar. 2011.
59. B. Carterette, "On rank correlation and the distance between rankings," in Proc. 32nd Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2009, pp. 436-443.
60. R. Hunter, "Mm algorithms for generalized Bradley-Terry models," The Ann. Statist., vol. 32, pp. 384-406, 2004.
61. H. Azari Soufiani, W. Chen, D. C. Parkes, and L. Xia, "Generalized method-of-moments for rank aggregation," in Proc. Adv. Neural Inf. Process. Syst. 26, 2013, pp. 2706-2714.
62. F. Caron and A. Doucet, "Efficient Bayesian inference for generalized Bradley-Terry models," J. Comput. Graph. Statist.,vol. 21, pp. 174-196, 2012.
63. S. X. Yu, "Angular embedding: A robust quadratic criterion," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 1, pp. 158-173, Jan. 2012.