Efficient crowdsourcing of unknown experts using bounded multi-armed bandits

Artificial Intelligence

Volumn 214, Issue , 2014, Pages 89-111

  Tran Thanh, Long ^a   Stein, Sebastian ^a   Rogers, Alex ^a   Jennings, Nicholas R ^a  

^a UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

Author keywords

Budget limitation; Crowdsourcing; Machine learning; Multi armed bandits

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COSTS; LEARNING SYSTEMS;

BUDGET LIMITATION; CRISIS SITUATIONS; CROWDSOURCING; DEVELOPMENT AND TESTING; EXPLORATION AND EXPLOITATION; MULTI ARMED BANDIT; PRODUCT MARKETING; QUALITY CHARACTERISTIC;

BUDGET CONTROL;

EID: 84901941951     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.artint.2014.04.005     Document Type: Article

References (40)

1. R. Agrawal Sample mean based index policies with O (log n) regret for the multi-armed bandit problem Adv. Appl. Probab. 27 1995 1054 1078
2. V. Anantharam, P. Varaiya, and J. Walrand Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays - part I: I.i.d. rewards IEEE Trans. Autom. Control 32 11 1987 977 982 (Pubitemid 18521626)
3. J.-Y. Audibert, S. Bubeck, and R. Munos Best arm identification in multi-armed bandits Proceedings of the Twenty-Third Annual Conference on Learning Theory 2010 41 53
4. P. Auer, N. Cesa-Bianchi, and P. Fischer Finite-time analysis of the multiarmed bandit problem Mach. Learn. 47 2002 235 256 (Pubitemid 34126111)
5. M.S. Bernstein, G. Little, R.C. Miller, B. Hartmann, M.S. Ackerman, D.R. Karger, D. Crowell, and K. Panovich Soylent: a word processor with a crowd inside Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology 2010 313 322
6. A. Beygelzimer, J. Langford, L. Li, L. Reyzin, and R. Schapire Contextual bandit algorithms with supervised learning guarantees Proceeding of the Fourteenth International Conference on Artificial Intelligence and Statistics 2011 19 26
7. F.P. Brooks The Mythical Man-Month: Essays on Software Engineering 1995 Addison-Wesley Pub. Co.
8. S. Bubeck, R. Munos, and G. Stoltz Pure exploration for multi-armed bandit problems Proceedings of the Twentieth international conference on Algorithmic Learning Theory 2009 23 37
9. D. Clery Galaxy zoo volunteers share pain and glory of research Science 333 6039 2011 173 175
10. P. Dai, Mausam, and D.S. Weld Artificial intelligence for artificial artificial intelligence Proceedings of the 25th Conference on Artificial Intelligence AAAI 2011 2011 1153 1159
11. A. Doan, R. Ramakrishnan, and A.Y. Halevy Crowdsourcing systems on the world-wide web Commun. ACM 54 4 2011 86 96
12. H. Gao, G. Barbier, and R. Goolsby Harnessing the crowdsourcing power of social media for disaster relief IEEE Intell. Syst. 26 3 2011 10 14
13. S. Guha, and K. Munagala Multi-armed bandits with metric switching costs S. Albers, A. Marchetti-Spaccamela, Y. Matias, S. Nikoletseas, W. Thomas, Automata, Languages and Programming Lect. Notes Comput. Sci. vol. 5556 2009 Springer Berlin, Heidelberg 496 507
14. C.-J. Ho, and J.W. Vaughan Online task assignment in crowdsourcing markets Proceedings of the 26th Conference on Artificial Intelligence AAAI 2012 2012 45 51
15. W. Hoeffding Probability inequalities for sums of bounded random variables J. Am. Stat. Assoc. 58 1963 13 30
16. J.J. Horton, and L.B. Chilton The labor economics of paid crowdsourcing Proceedings of the 11th ACM Conference on Electronic Commerce EC'10 2010 209 218
17. E. Huang, H. Zhang, D.C. Parkes, K.Z. Gajos, and Y. Chen Toward automatic task design: a progress report Proceedings of the ACM SIGKDD Workshop on Human Computation HCOMP'10 2010 77 85
18. C. Juhn, K.M. Murphy, and B. Pierce Wage inequality and the rise in returns to skill J. Polit. Econ. 101 3 1993 410 442 (Pubitemid 24385467)
19. H. Kellerer, U. Pferschy, and D. Pisinger Knapsack Problems 2004 Springer
20. A. Kittur, E.H. Chi, and B. Suh Crowdsourcing user studies with mechanical turk Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI'08 2008 453 456
21. V. Kuleshov, D. Precup, 2010, Algorithms for the multi-armed bandit problem. Unpublished.
22. G. Little, L.B. Chilton, M. Goldman, and R.C. Miller Exploring iterative and parallel human computation processes Proceedings of the ACM SIGKDD Workshop on Human Computation HCOMP'10 2010 68 76
23. S. Marcello, and M. Toth Knapsack Problems: Algorithms and Computer Implementations 1990 Wiley
24. M. Marge, S. Banerjee, and A. Rudnicky Using the amazon mechanical turk for transcription of spoken language Proceedings of the 35th International IEEE Conference on Acoustics, Speech, and Signal Processing ICASSP'10 2010 5270 5273
25. O. Maron, and A.W. Moore Hoeffding races: accelerating model selection search for classification and function approximation Proceedings of the Seventh Annual Conference on Neural Information Processing Systems 1993 59 66
26. W. Mason, and D.J. Watts Financial incentives and the performance of crowds Proceedings of the ACM SIGKDD Workshop on Human Computation HCOMP'09 2009 77 85
27. V. Mnih, C. Szepesvari, and J. Audibert Empirical Bernstein stopping Proceedings of the 25th International Conference on Machine Learning 2008 672 679
28. R.R. Morris, M. Dontcheva, and E.M. Gerber Priming for better performance in microtask crowdsourcing environments IEEE Internet Comput. 16 2012 13 19
29. H. Robbins Some aspects of the sequential design of experiments Bull. Am. Meteorol. Soc. 55 1952 527 535
30. R.S. Sutton, and A.G. Barto Reinforcement Learning: An Introduction 1998 MIT Press
31. O. Tokarchuk, R. Cuel, and M. Zamarian Analyzing crowd labor and designing incentives for humans in the loop IEEE Internet Comput. 16 5 2012 45 51
32. L. Tran-Thanh Budget-limited multi-armed bandits Ph.D. thesis 2012 University of Southampton, School of Electronics and Computer Science Southampton UK
33. L. Tran-Thanh, A. Chapman, J.E.M. de Cote, A. Rogers, and N.R. Jennings Epsilon-first policies for budget-limited multi-armed bandits Proceedings of the 24th Conference on Artificial Intelligence AAAI 2010 2010 1211 1216
34. L. Tran-Thanh, S. Stein, A. Rogers, and N.R. Jennings Efficient crowdsourcing of unknown experts using multi-armed bandits 20th European Conference on Artificial Intelligence ECAI 2012 2012 768 773
35. L. Tran-Thanh, A. Chapman, A. Rogers, and N.R. Jennings Knapsack based optimal policies for budget-limited multi-armed bandits Proceedings of the 26th Conference on Artificial Intelligence AAAI 2012 2012 1134 1140
36. J. Vermorel, and M. Mohri Multi-armed bandit algorithms and empirical evaluation Proceedings of the 16th European Conference on Machine Learning ECML'05 2005 437 448
37. J. Vuurens, and A. de Vries Obtaining high-quality relevance judgments using crowdsourcing IEEE Internet Comput. 16 5 2012 20 27
38. P. Welinder, S. Branson, S. Belongie, and P. Perona The multidimensional wisdom of crowds Advances in Neural Information Processing Systems NIPS 2010 vol. 24 2010 2424 2432
39. O.F. Zaidan, and C. Callison-Burch Crowdsourcing translation: professional quality from non-professionals Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, vol. 1 HLT'11 2011 1220 1229
40. M. Zook, M. Graham, T. Shelton, and S. Gorman Volunteered geographic information and crowdsourcing disaster relief: a case study of the Haitian earthquake World Med. Heal. Policy 2 2 2010 7 33