Combining human and machine intelligence in large-scale crowdsourcing

11th International Conference on Autonomous Agents and Multiagent Systems 2012, AAMAS 2012: Innovative Applications Track

Volumn 1, Issue , 2012, Pages 520-527

  Kamar, Ece ^a   Hacker, Severin ^b   Horvitz, Eric ^a  

^a MICROSOFT RESEARCH   (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

Complementary computing; Consensus tasks; Crowdsourcing; Decision theoretic reasoning

Indexed keywords

AUTONOMOUS AGENTS; MULTI AGENT SYSTEMS;

COMPLEMENTARY COMPUTING; COMPUTATIONAL AGENTS; CONSENSUS TASKS; CROWDSOURCING; DECISION-THEORETIC; MACHINE INTELLIGENCE; PREDICTIVE MODELS; PROBABILISTIC MODELS;

COMPUTER VISION;

EID: 84899442104     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (14)

1. Galaxy Zoo, 2007, http://zool.galaxyzoo.org/.
2. D. Chickering, D. Heckerman, and C. Meek. A Bayesian approach to learning Bayesian networks with local structure. In UAI'97, 1997.
3. P. Dai et al. Artificial intelligence for artificial artificial intelligence. In AAAI, 2011.
4. P. Dai, Mausam, and D. Weld. Decision-theoretic control of crowd-sourced workflows. In AAAI, 2010.
5. D. Heckerman, E. Horvitz, and B. Nathwani. Toward normative expert systems: The Pathfinder project. Stanford University, 1991.
6. P. Ipeirotis. Analyzing the Amazon Mechanical Turk marketplace. XRDS: Crossroads, The ACM Magazine for Students, 17(2):16-21, 2010.
7. L. Kocsis and C. Szepesvári. Bandit based monte-carlo planning. Machine Learning: ECML 2006, pages 282-293, 2006.
8. A. Krause, E. Horvitz, A. Kansal, and F. Zhao. Toward community sensing. In IPSN. IEEE, 2008.
9. C. Lintott et al. Galaxy Zoo: Morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey? Monthly Notices of the Royal Astronomical Society, 389(3):1179-1189, 2008.
10. D. Shahaf and E. Horvitz. Generalized task markets for human and machine computation. In AAAI, 2010.
11. V. Sheng, F. Provost, and P. Ipeirotis. Get another label? Improving data quality and data mining using multiple, noisy labelers. In ACM SIGKDD, 2008.
12. R. Snow, B. O'Connor, D. Jurafsky, and A. Ng. Cheap and fast but is it good?: Evaluating non-expert annotations for natural language tasks. In EMNLP, 2008.
13. L. Von Ahn and L. Dabbish. Designing games with a purpose. Communications of the ACM, 2008.
14. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. NISP, 2009.