Sequential crowdsourced labeling as an epsilon-greedy exploration in a Markov Decision Process

Journal of Machine Learning Research

Volumn 33, Issue , 2014, Pages 832-840

  Raykar, Vikas C ^a   Agrawal, Priyanka ^a  

^a IBM RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; MARKOV PROCESSES;

ANNOTATED DATASETS; BAYESIAN DECISION; CROWDSOURCING; GREEDY EXPLORATION; LABELING PROCEDURES; MARKOV DECISION PROCESSES; UTILITY FUNCTIONS;

COSTS;

EID: 84955471643     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (29)

1. URL https://www.mturk.com.
2. URL https://sites.google.com/site/nlpannotations/.
3. M. J. Beal. Variational algorithms for approximate Bayesian inference. PhD thesis, University of London, 2003.
4. R. Bellman. Dynamic Programming. Princeton University Press, Princeton, NJ, 1957.
5. T. Berg, A. Sorokin, G. Wang, D. Forsyth, D. Hoiem, I. Endres, and A. Farhadi. It's All About the Data. Proceedings of the IEEE, 98(8):1434-1452, 2010.
6. L. D. Brown, T. T. Cai, and A. DasGupta. Interval estimation for a binomial proportion. Statistical Science, 16:101-133, 2001.
7. X. Chen, Q. L., and D. Z. Optimistic knowledge gradient policy for optimal budget allocation in crowd-sourcing. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pages 64-72, 2013.
8. A. P. Dawid and A. M. Skene. Maximum likeihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28(1):20-28, 1979.
9. M. O. Duff and A. G. Barto. Local bandit approximation for optimal learning problems. In Advances in Neural Information Processing Systems. 1997.
10. P. I. Frazier, W. B. Powell, and S. Dayanik. A knowledge-gradient policy for sequential information collection. SIAM Journal on Control and Optimization, 47(5):2410-2439, 2008.
11. P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP '10, pages 64-67, 2010.
12. D. Karger, S. Oh, and D. Shah. Budget-optimal crowdsourcing using low-rank matrix approximations. In 49th Annual Allerton Conference on Communication, Control, and Computing, pages 284-291. 2011.
13. D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In Advances in Neural Information Processing Systems 24, pages 1953-1961. 2011.
14. D. V. Lindley. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics, 27(4):986-1005, 1956.
15. Q. Liu, J. Peng, and A. Ihler. Variational inference for crowdsourcing. In Advances in Neural Information Processing Systems 25, pages 701-709. 2012.
16. W. P., S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems 23, pages 2424-2432. 2010.
17. M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley and Sons, New York, NY, 1994.
18. V. C. Raykar and S. Yu. Ranking annotators for crowdsourced labeling tasks. In Advances in Neural Information Processing Systems 24, pages 1809-1817. 2011.
19. V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. Journal of Machine Learning Research, 11: 1297-1322, April 2010.
20. S. J. Russell, P. Norvig, J. F. Canny, J. M. Malik, and D. D. Edwards. Artificial intelligence: a modern approach. Englewood Cliffs: Prentice hall, 1995.
21. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and Fast - but is it good? Evaluating NonExpert Annotations for Natural Language Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '08, pages 254-263, 2008.
22. A. L. Strehl and M. L. Littman. An empirical evaluation of interval estimation for Markov Decision Processes. In 16th IEEE International Conference on Tools with Artificial Intelligence, pages 128-135, 2004.
23. J. Vermorel and M. Mohri. Multi-armed bandit algorithms and empirical evaluation. In Proceedings of the European Conference on Machine Learning (ECML 2005), pages 437-448. 2005.
24. S. Vijayanarasimhan and K. Grauman. What's it going to cost you?: Predicting effort vs. informativeness for multi-label image annotations. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2262-2269, 2009.
25. C. Watkins. Learning from Delayed Rewards. PhD thesis, Cambridge University, 1989.
26. P. Welinder and P. Perona. Online crowdsourcing: rating annotators and obtaining cost-effective labels. In Workshop on Advancing Computer Vision with Humans in the Loop at CVPR 2010, 2010.
27. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22, pages 2035-2043. 2009.
28. Y. Yan, R. Rosales, G. Fung, M. Schmidt, G. Hermosillo, L. Bogoni, L. Moy, and J. Dy. Modeling an notator expertise: Learning when everybody knows a bit of something. In Proceedings of The Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS) 2010, pages 932-939, 2010.
29. O. F. Zaidan and C. Callison-Burch. Crowdsourcing Translation: Professional Quality from NonProfessionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1220-1229, 2011.