POMDP-based control of workflows for crowdsourcing

Artificial Intelligence

Volumn 202, Issue , 2013, Pages 52-85

  Dai, Peng ^a   Lin, Christopher H ^a   Mausam, ^a   Weld, Daniel S ^a  

^a University of Washington   (United States)

Author keywords

Crowdsourcing; Partially Observable Markov Decision; Planning under uncertainty; POMDP; Process

Indexed keywords

AMAZON MECHANICAL TURKS; BAYESIAN NETWORK LEARNING; CROWDSOURCING; MARKOV DECISION; MARKOV DECISION PROCESSES; NON-ADAPTIVE CONTROLLERS; PLANNING UNDER UNCERTAINTY; POMDP;

BAYESIAN NETWORKS; ITERATIVE METHODS; LEARNING ALGORITHMS; MARKOV PROCESSES; PROCESSING;

DESIGN;

EID: 84880360544     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.artint.2013.06.002     Document Type: Article

References (69)

1. Radha-Krishna Balla, and Alan Fern UCT for tactical assault planning in real-time strategy games IJCAI 2009 40 45
2. R. Bellman Dynamic Programming 1957 Princeton University Press Princeton, NJ
3. Michael S. Bernstein, Joel Brandt, Robert C. Miller, and David R. Karger Crowds in two seconds: Enabling realtime crowd-powered interfaces UIST 2011
4. Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich Soylent: A word processor with a crowd inside UIST 2010 313 322
5. Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich Soylent: A word processor with a crowd inside UIST 2010
6. Dimitri P. Bertsekas Dynamic Programming and Optimal Control, vol. 1 2nd ed. 2000 Athena Scientific
7. Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samuel White, and Tom Yeh Vizwiz: Nearly real-time answers to visual questions UIST 2010 333 342
8. Emma Brunskill, Leslie Kaelbling, Tomas Lozano-Perez, and Nicholas Roy Continuous-state POMDPs with hybrid dynamics Symposium on Artificial Intelligence and Mathematics 2008
9. Urszula Chajewska, Daphne Koller, and Ronald Parr Making rational decisions using adaptive utility elicitation AAAI 2000
10. Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beene, Andrew Leaver-Fay, David Baker, Zoran Popovic, and Foldit players Predicting protein structures with a multiplayer online game Nature 446 2010 756 760
11. Dan Cosley, Shyong K. Lam, Istvan Albert, Joseph A. Konstan, and John Riedl Is seeing believing?: How recommender system interfaces affect usersÊ opinions Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 2003 585 592
12. Peng Dai Mausam, and Daniel S. Weld Decision-theoretic control of crowd-sourced workflows AAAI 2010
13. Peng Dai Mausam, and Daniel S. Weld Artificial intelligence for artificial intelligence AAAI 2011
14. A.P. Dawid, and A.M. Skene Maximum likelihood estimation of observer error-rates using the em algorithm Appl. Stat. 28 1 1979 20 28
15. Pinar Donmez, Jaime G. Carbonell, and Jeff Schneider Efficiently learning the accuracy of labeling sources for selective sampling KDD 2009
16. Pinar Donmez, Jaime G. Carbonell, and Jeff Schneider A probabilistic framework to learn from multiple annotators with time-varying accuracy SIAM International Conference on Data Mining (SDM) 2010 826 837
17. Arnaud Doucet, Nando de Freitas, and Neil Gordon Sequential Monte Carlo Methods in Practice 2001 Springer
18. Sylvain Gelly, and Yizao Wang Exploration exploitation in go: UCT for Monte Carlo go NIPS On-line Trading of Exploration and Exploitation Workshop 2006
19. David Alan Grier Error identification and correction in human computation: Lessons from the WPA HCOMP 2011
20. Chien-Ju Ho, and Jennifer Wortman Vaughan Online task assignment in crowdsourcing markets AAAI 2012
21. Leah Hoffmann Crowd control Comm. ACM 52 3 2009 16 17
22. Eric Huang, Haoqi Zhang, David C. Parkes, Krzysztof Z. Gajos, and Yiling Chen Toward automatic task design: A progress report Proceedings of the ACM SIGKDD Workshop on Human Computation 2010 77 85
23. Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang Quality management on Amazon Mechanical Turk Proceedings of the ACM SIGKDD Workshop on Human Computation 2010 64 67
24. Hyun Joon Jung, and Matthew Lease Spam worker filtering and featured-voting based consensus accuracy improvement Proc. of CrowdConf 2011
25. Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra Planning and acting in partially observable stochastic domains Artificial Intelligence 101 1-2 1998 99 134 (Pubitemid 128387390)
26. Hiroshi Kajino, Yuta Tsuboi, and Hisashi Kashima A convex formulation for learning from crowds AAAI 2012
27. Ece Kamar, Severin Hacker, and Eric Horvitz Combining human and machine intelligence in large-scale crowdsourcing AAMAS 2012
28. David R. Karger, Sewoong Oh, and Devavrat Shah Budget-optimal crowdsourcing using low-rank matrix approximations Allerton 2011
29. Levente Kocsis, and Csaba Szepesvári Bandit based Monte Carlo planning ECML 2006 282 293 (Pubitemid 44618839)
30. Anand Kulkarni, Matthew Can, and Bjorn Hartmann Collaboratively crowdsourcing workflows with Turkomatic Proceedings of CSCW 2012
31. Anand Kulkarni, Philipp Gutheim, Prayag Narula, David Rolnitzky, Tapan Parikh, and Bjoern Hartmann Mobileworks Designing for quality in a managed crowdsourcing architecture HCOMP 2012
32. Walter, S. Lasecki, Kyle I. Murray, Samuel White, Robert C. Miller, and Jeffrey P. Bigham Real-time crowd control of existing interfaces Proceedings of UIST 2011
33. Edith Law, and Luis von Ahn Human Computation. Synthesis Lectures on Artificial Intelligence and Machine Learning 2011 Morgan & Claypool Publishers
34. Beatrice Liem, Haoqi Zhang, and Yiling Chen An iterative dual pathway structure for speech-to-text transcription HCOMP 2011
35. Christopher H. Lin Mausam, Mausam, and Daniel S. Weld Crowdsourcing control: Moving beyond multiple choice UAI 2012
36. Christopher H. Lin, Mausam, and Daniel S. Weld Dynamically switching between synergistic workflows for crowdsourcing AAAI 2012
37. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller TurKit: Tools for iterative tasks on Mechanical Turk KDD Workshop on Human Computation 2009 29 30
38. Omid Madani, Steve Hanks, and Anne Condon On the undecidability of probabilistic planning and related stochastic optimization problems Artificial Intelligence 147 1-2 2003 5 34
39. Mausam, Emmanuel Benazera, Ronen I. Brafman, Nicolas Meuleau, and Eric A. Hansen Planning with continuous resources in stochastic domains IJCAI 2005 1244 1251
40. Mausam, and Andrey Kolobov Planning with Markov Decision Processes: An AI Perspective 2012 Morgan and Claypool
41. David Milne, and Ian H. Witten Learning to link with Wikipedia Proceedings of the ACM Conference on Information and Knowledge Management 2008
42. John Noronha, Eric Hysen, Haoqi Zhang, and Krzysztof Z. Gajos Platemate: Crowdsourcing nutrition analysis from food photographs UIST 2011
43. Aditya Parameswaran, Hector Garcia-Molina, Hyunjung Park, Neoklis Polyzotis, Aditya Ramesh, and Jennifer Widom Crowdscreen: Algorithms for filtering data with humans VLDB 2010
44. Joelle Pineau, Geoffrey Gordon, and Sebastian Thrun Anytime point-based approximations for large POMDPs J. Artificial Intelligence Res. 27 1 2006 335 380
45. Josep M. Porta, Nikos Vlassis, Matthijs T.J. Spaan, and Pascal Poupart Point-based value iteration for continuous POMDPs J. Mach. Learn. Res. 7 2006 2329 2367 (Pubitemid 44708007)
46. Lev Ratinov, Dan Roth, Doug Downey, and Mike Anderson Local and global algorithms for disambiguation to Wikipedia Proceedings of the Annual Meeting of the Association of Computational Linguistics 2011
47. Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, and Gerardo Valadez Learning from crowds J. Mach. Learn. Res. 11 2010 1297 1322
48. A. Kimball Romney, Susan C. Weller, and William H. Batchelder Culture as consensus: A theory of culture and informant accuracy Am. Anthropol. 88 2 1986 313 338
49. Joel Ross, Lilly Irani, M. Six Silberman, Andrew Zaldivar, and Bill Tomlinson Who are the crowdworkers? Shifting demographics in Mechanical Turk CHI 2010
50. Nicholas Roy, and Sebastian Thrun Coastal navigation with mobile robots Advances in Neural Processing Systems 12 1999 1043 1049
51. Jeffery Rzeszotarski, Ed Chi, Praveen Paratosh, Peng Dai, And now for something completely different: Introducing micro-breaks into crowdsourcing workflows, in: Submission, 2013.
52. E. Tjong, Kim Sang, and F. De Meulder Introduction to the CoNNL-2003 shared task: Language-independent named entity recognition Proceedings of CoNNL 2003
53. Dafna Shahaf, and Eric Horvitz Generalized markets for human and machine computation AAAI 2010
54. Guy Shani, Ronen I. Brafman, and Solomon Eyal Shimony Prioritizing point-based POMDP solvers IEEE Trans. Syst. Man Cybern., Part B, Cybern. 38 6 2008 1592 1605
55. Victor, S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis Get another label? Improving data quality and data mining using multiple, noisy labelers Proceedings of the Fourteenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2008
56. David Silver, and Joel Veness Monte Carlo planning in large POMDPs NIPS 2010 2164 2172
57. Trey Smith, and Reid G. Simmons Focused real-time dynamic programming for MDPs: Squeezing more out of a heuristic AAAI 2006
58. Rion Snow, Brendan OÊConnor, Daniel Jurafsky, and Andrew Y. Ng Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks EMNLP 2008 254 263
59. Edward J. Sondik The optimal control of partially observable Markov processes PhD thesis 1971 Stanford
60. Alexander Sorokin, and David Forsyth Utility data annotation with Amazon Mechanical Turk Computer Vision and Pattern Recognition Workshop 2008 1 8
61. Matthijs T.J. Spaan, and Nikos Vlassis Perseus: Randomized point-based value iteration for POMDPs J. Artificial Intelligence Res. 24 2005 195 220 (Pubitemid 43130936)
62. Richard S. Sutton, and Andrew G. Barto Reinforcement Learning 1998 The MIT Press
63. Luis von Ahn Games with a purpose IEEE Comput. Mag. June 2006 96 98
64. Fabian L. Wauthier, and Michael I. Jordan Bayesian bias mitigation for crowdsourcing NIPS 2011
65. Daniel S. Mausam, Mausam, and Peng Dai Human intelligence needs artificial intelligence HCOMP 2011
66. Peter Welinder, Steve Branson, Serge Belongie, and Pietro Perona The multidimensional wisdom of crowds NIPS 2010
67. Jacob Whitehill, Paul Ruvolo, Jacob Bergsma, Tingfan Wu, and Javier Movellan Whose vote should count more: Optimal integration of labels from labelers of unknown expertise NIPS 2009
68. Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier Movellan Whose vote should count more: Optimal integration of labels from labelers of unknown expertise Proc. of NIPS 2009 2035 2043
69. Yan Yan, Romer Rosales, Glenn Fung, and Jennifer G. Dy Active learning from crowds ICML 2011