Crowdsourcing complex workflows under budget constraints

Proceedings of the National Conference on Artificial Intelligence

Volumn 2, Issue , 2015, Pages 1298-1304

  Tran Thanh, Long ^a   Dong Huynh, Trung ^a   Rosenfeld, Avi ^b   Ramchurn, Sarvapali D ^a   Jennings, Nicholas R ^a  

^a UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

^b JERUSALEM COLLEGE OF TECHNOLOGY   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CROWDSOURCING;

AMAZON MECHANICAL TURKS; BUDGET CONSTRAINT; COMPLEX WORKFLOWS; MICROTASKS; TASK ALLOCATION;

BUDGET CONTROL;

EID: 84959925225     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (22)

1. AMT. 2010. Amazon mechanical turk. http://www.mtu rk.com/.
2. Audibert, J.-Y.; Bubeck, S.; and Munos, R. 2010. Best arm identification in multi-armed bandits. COLT 41-53.
3. Azaria, A.; Aumann, Y; and Kraus, S. 2014. Automated agents for reward determination for human work in crowdsourcing applications. Journal of Autonomous Agents and Multi-Agent Systems 28:934-955.
4. Bernstein, M. S.; Little, G.; Miller, R. C.; Hartmann, B.; Ackerman, M. S.; Karger, D. R.; Crowell, D.; and Panovich, K. 2010. Soylent: a word processor with a crowd inside. In UIST, 313-322.
5. Boyd, S., and Vandenberghe, L. 2004. Convex Optimization. New York, NY, USA: Cambridge University Press.
6. Dai. P.; Lin, C. H.; Mausam; and Weld, D. S. 2013. POMDP-based control of workflows for crowdsourcing. Artificial Intelligence (AU) 202:52-85.
7. Flesch, R. 1948. A new readability yardstick. Journal of Applied Psychology 32(3):221-233.
8. Ho, C.-J., and Wortman-Vaughan, J. 2012. Online task assignment in crowdsourcing markets. In AAAI'12, 45-51.
9. Kalyanakrishnan, S.; Tewari, A.; Auer, P.; and Stone, P. 2012. PAC subset selection in stochastic multi-armed bandits. In Langford, J., and Pineau, J., eds., ICML, 655-662. New York, NY, USA: Omnipress.
10. Kamar, E.; Hacker, S.; and Horvitz, E. 2012. Combining human and machine intelligence in large-scale crowdsourcing. AAMAS Abl-Ml
11. Karger, D. R.; Oh, S.; and Shah, D. 2011. Budget-optimal task allocation for reliable crowdsourcing systems. CoRR abs/1110.3564.
12. Kaufmann, E., and Kalyanakrishnan, S. 2013. Information complexity in bandit subset selection. In COLT, volume 30 of JMLR Workshop and Conference Proceedings, 228-251. JMLR.
13. Kincaid, J. P.; Fishburne, R. P.; Rogers, R. L.; and Chissom, B. S. 1975. Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Technical report.
14. Lasecki, W. S.; Bigham, J. P.; Allen, J. F.; and Ferguson, G. 2012. Real-time collaborative planning with the crowd. In AAAI.
15. Lin, C. H.; Mausam; and Weld, D. S. 2012. Dynamically switching between synergistic workflows for crowdsourcing. AAAI'12 87-93.
16. Little, G.; Chilton, L. B.; Goldman, M.; and Miller, R. C. 2009. Turkit: tools for iterative tasks on mechanical turk. KDD-HCOMP 29-30.
17. Ramchurn, S. D.; Huynh, T. D.; Venanzi, M.; and Shi, B. 2013. Collabmap: Crowdsourcing maps for emergency planning. In 5th ACM Web Science Conference (WebSci '13).
18. Simpson, E.; Roberts, S.; Psorakis, I.; Smith, A.; and Lintott, C. 2011. Bayesian combination of multiple imperfect classifiers. NIPS.
19. Tran-Thanh, L.; Stein, S.; Rogers, A.; and Jennings, N. R. 2012. Efficient crowdsourcing of unknown experts using multi-armed bandits. In ECAI'12, 768-773.
20. Tran-Thanh, L.; Venanzi, M.; Rogers, A.; and Jennings, N. R. 2013. Efficient budget allocation with accuracy guarantees for crowdsourcing classification tasks. AAMAS'13 901-908.
21. Tran-Thanh, L.; Huynh, T. D.; Rosenfield, A.; Ramchurn, S.; and Jennings, N. R. 2014. Budgetfix: budget limited crowdsourcing for interdependent task allocation with quality guarantees. In AAMAS'14, 477-484.
22. Zhang, H.; Law, E.; Gajos, K.; Miller, R.; Parkes, D.; and Horvitz, E. 2012. Human computation tasks with global constraints. ACM CHI.