Spectral methods meet EM: A provably optimal algorithm for crowdsourcing

Journal of Machine Learning Research

Volumn 17, Issue , 2016, Pages 1-44

  Zhang, Yuchen ^a   Chen, Xi ^b   Zhou, Dengyong ^c   Jordan, Michael I ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b NEW YORK UNIVERSITY   (United States)

^c MICROSOFT RESEARCH   (United States)

Author keywords

Crowdsourcing; Dawid Skene model; EM; Minimax rate; Non convex optimization; Spectral methods

Indexed keywords

CONVEX OPTIMIZATION; CROWDSOURCING; SPECTROSCOPY;

EMPIRICAL APPROACH; LOG-LIKELIHOOD FUNCTIONS; MINIMAX RATES; NONCONVEX OPTIMIZATION; OBJECTIVE FUNCTIONS; OPTIMAL ALGORITHM; OPTIMAL CONVERGENCE; SPECTRAL METHODS;

OPTIMIZATION;

EID: 84988975385     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (30)

1. Anima Anandkumar, Daniel Hsu, and Sham M. Kakade. A method of moments for mixture models and hidden Markov models. In Proceedings of the Annual Conference on Learning Theory, 2012.
2. Anima Anandkumar, Rong Ge, Daniel Hsu, and Sham M. Kakade. A tensor spectral approach to learning mixed membership community models. In Proceedings of the Annual Conference on Learning Theory, 2013.
3. Anima Anandkumar, Rong Ge, Daniel Hsu, Sham M. Kakade, and Matus Telgarsky. Tensor decompositions for learning latent variable models. Journal of Machine Learning Research, 15:2773-2832, 2014.
4. Animashree Anandkumar, Dean P. Foster, Daniel Hsu, Sham M. Kakade, and Yi-Kai Liu. A spectral algorithm for latent Dirichlet allocation. Algorithmica, 72(1):193-214, 2015.
5. Sivaraman Balakrishnan, Martin J. Wainwright, and Bin Yu. Statistical guarantees for the EM algorithm: From population to sample-based analysis. Annals of Statisics (to appear), 2016.
6. Arun Tejasvi Chaganty and Percy Liang. Spectral experts for estimating mixtures of linear regressions. In Proceedings of the International Conference on Machine Learning, 2013.
7. Xi Chen, Qihang Lin, and Dengyong Zhou. Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. In Proceedings of the International Conference on Machine Learning, 2013.
8. Nilesh Dalvi, Anirban Dasgupta, Ravi Kumar, and Vibhor Rastogi. Aggregating crowd-sourced binary ratings. In Proceedings of the World Wide Web Conference, 2013.
9. Sanjoy Dasgupta and Leonard Schulman. A probabilistic analysis of EM for mixtures of separated, spherical Gaussians. Journal of Machine Learning Research, 8:203-226, 2007.
10. Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society, Series C, 28 (1):20-28, 1979.
11. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009.
12. Chao Gao and Dengyong Zhou. Minimax optimal convergence rates for estimating ground truth from crowdsourced labels. arXiv preprint arXiv:1310.5764, 2014.
13. Arpita Ghosh, Satyen Kale, and Preston McAfee. Who moderates the moderators? Crowdsourcing abuse detection in user-generated content. In Proceedings of the ACM Conference on Electronic Commerce, 2011.
14. Daniel Hsu, Sham M. Kakade, and Tong Zhang. A spectral algorithm for learning hidden Markov models. Journal of Computer and System Sciences, 78(5):1460-1480, 2012.
15. Prateek Jain and Sewoong Oh. Learning mixtures of discrete product distributions using spectral decompositions. In Proceedings of the Conference on Learning Theory, 2014.
16. David R. Karger, Sewoong Oh, and Devavrat Shah. Efficient crowdsourcing for multi-class labeling. In Proceedings of the ACM SIGMETRICS, 2013.
17. David R. Karger, Sewoong Oh, and Devavrat Shah. Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 62(1):1-24, 2014.
18. Matthew Lease and Gabriella Kazai. Overview of the TREC 2011 crowdsourcing track. In Proceedings of TREC 2011, 2011.
19. Erich L. Lehmann and George Casella. Theory of Point Estimation. Springer, 2nd edition, 2003.
20. Percy Liang. Partial information from spectral methods. NIPS Spectral Learning Workshop, 2013.
21. Qiang Liu, Jian Peng, and Alexander T Ihler. Variational inference for crowdsourcing. In Advances in Neural Information Processing Systems, 2012.
22. Fabio Parisi, Francesco Strino, Boaz Nadler, and Yuval Klugera. Ranking and combining multiple predictors without labeled data. Proceedings of the National Academy of Sciences, 111(4):1253-1258, 2014.
23. Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 11:1297-1322, 2010.
24. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y Ng. Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2008.
25. Peter Welinder, Steve Branson, Serge Belongie, and Pietro Perona. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems, 2010.
26. Jacob Whitehill, Ting F. Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems, 2009.
27. Bin Yu. Assouad, Fano, and Le Cam. In Festschrift for Lucien Le Cam, pages 423-435. Springer, 1997.
28. Dengyong Zhou, John C. Platt, Sumit Basu, and Yi Mao. Learning from the wisdom of crowds by minimax entropy. In Advances in Neural Information Processing Systems, 2012.
29. Dengyong Zhou, Qiang Liu, John C. Platt, and Christopher Meek. Aggregating ordinal labels from crowds by minimax conditional entropy. In Proceedings of the International Conference on Machine Learning, 2014.
30. James Zou, Daniel Hsu, David Parkes, and Ryan Adams. Contrastive learning using spectral methods. In Advances in Neural Information Processing Systems, 2013.