Dynamically switching between synergistic workflows for crowdsourcing

Proceedings of the National Conference on Artificial Intelligence

Volumn 1, Issue , 2012, Pages 87-93

  Lin, Christopher H ^a   Mausam, ^a   Weld, Daniel S ^a  

^a University of Washington   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPLEX WORKFLOWS; CROWDSOURCING; ERROR REDUCTION; LEARNING MODELS; MECHANICAL TURKS; NATURAL DESIGN; OFFLINE; ONLINE METHODS; PROBABILISTIC GRAPHICAL MODELS; RETURNS ON INVESTMENT; TRAINING DATA; WORK-FLOWS;

DESIGN; EXPERIMENTS;

ARTIFICIAL INTELLIGENCE;

EID: 84868278309     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (29)

1. Barowy, D. W.; Berger, E. D.; and McGregor, A. 2012. Automan: A platform for integrating human-based and digital computation. Technical report, University of Massachusetts, Amherst.
2. Bernstein, M. S.; Little, G.; Miller, R. C.; Hartmann, B.; Ackerman, M. S.; Karger, D. R.; Crowell, D.; and Panovich, K. 2010. Soylent: A word processor with a crowd inside. In UIST.
3. Dai, P.; Mausam; and Weld, D. S. 2010. Decision-theoretic control of crowd-sourced workflows. In AAAI.
4. Dai, P.; Mausam; and Weld, D. S. 2011. Artificial intelligence for artificial intelligence. In AAAI.
5. Grier, D. A. 2011. Error identification and correction in human computation: Lessons from the WPA. In HCOMP.
6. Kamar, E.; Hacker, S.; and Horvitz, E. 2012. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS.
7. Karger, D. R.; Oh, S.; ; and Shah, D. 2011. Budget-optimal crowd-sourcing using low-rank matrix approximations. In Allerton.
8. Kulkarni, A.; Can, M.; and Hartmann, B. 2012. Collaboratively crowdsourcing workflows with turkomatic. In Proceedings of CSCW.
9. Lasecki, W. S.; Murray, K. I.; White, S.; Miller, R. C.; and Bigham, J. P. 2011. Real-time crowd control of existing interfaces. In Proceedings of UIST.
10. Liem, B.; Zhang, H.; and Chen, Y. 2011. An iterative dual pathway structure for speech-to-text transcription. In HCOMP.
11. Little, G.; Chilton, L. B.; Goldman, M.; and Miller, R. C. 2009. Turkit: tools for iterative tasks on mechanical turk. In KDD-HCOMP, 29-30.
12. Milne, D., and Witten, I. H. 2008. Learning to link with wikipedia. In Proceedings of the ACM Conference on Information and Knowledge Management.
13. Noronha, J.; Hysen, E.; Zhang, H.; and Gajos, K. Z. 2011. Platemate: Crowdsourcing nutrition analysis from food photographs. In UIST.
14. Parameswaran, A.; Garcia-Molina, H.; Park, H.; Polyzotis, N.; Ramesh, A.; and Widom, J. 2010. Crowdscreen: Algorithms for filtering data with humans. In VLDB.
15. Ratinov, L.; Roth, D.; Downey, D.; and Anderson, M. 2011. Local and global algorithms for disambiguation to wikipedia. In Proceedings of the Annual Meeting of the Association of Computational Linguistics.
16. Raykar, V. C.; Yu, S.; Zhao, L. H.; and Valadez, G. 2010. Learning from crowds. Journal of Machine Learning Research 11:1297-1322.
17. Romney, A. K.; Weller, S. C.; and Batchelder, W. H. 1986. Culture as consensus: A theory of culture and informant accuracy. American Anthropologist 88(2):313-338.
18. Ross, S.; Chalb-draa, B.; and Pineau, J. 2008. Bayes-adaptive POMDPs. In NIPS.
19. Russell, S., and Norvig, P. 2002. Artificial Intelligence: A Modern Approach. Prentice Hall.
20. Sang, E. T. K., and Meulder, F. D. 2003. Introduction to the CoNNL-2003 shared task: Language-independent named entity recognition. In Proceedings of CoNLL.
21. Shahaf, D., and Horvitz, E. 2010. Generlized markets for human and machine computation. In AAAI.
22. Sheng, V. S.; Provost, F.; and Ipeirotis, P. G. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the Fourteenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
23. Smith, T., and Simmons, R. G. 2006. Focused real-time dynamic programming for MDPs: Squeezing more out of a heuristic. In AAAI.
24. Snow, R.; O'Connor, B.; Jurafsky, D.; and Ng, A. Y. 2008. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In EMNLP, 254-263.
25. Sondik, E. J. 1971. The Optimal Control of Partially Observable markov Processes. Ph.D. Dissertation, Stanford.
26. Sutton, R. S., and Barto, A. G. 1998. Reinforcement Learning. The MIT Press.
27. Weld, D. S.; Mausam; and Dai, P 2011. Human intelligence needs artificial intelligence. In HCOMP.
28. Welinder, P.; Branson, S.; Belongie, S.; and Perona, P. 2010. The multidimensional wisdom of crowds. In NIPS.
29. Whitehill, J.; Ruvolo, P.; Bergsma, J.; Wu, T.; and Movellan, J. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS.