Human computation

Synthesis Lectures on Artificial Intelligence and Machine Learning

Volumn 13, Issue , 2011, Pages 1-120

  Law, Edith ^a   Von Ahn, Luis ^a  

^a CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

active learning; crowdsourcing; games with a purpose; human computation; human in the loop algorithms; incentives; labor markets; latent class models; mechanism design; output aggregation; task design; task routing

Indexed keywords

ACTIVE LEARNING; CROWDSOURCING; GAMES WITH A PURPOSE; HUMAN COMPUTATION; HUMAN-IN-THE-LOOP; INCENTIVES; LABOR MARKETS; LATENT CLASS MODEL; MECHANISM DESIGN; OUTPUT AGGREGATION; TASK DESIGN; TASK ROUTING;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; EMPLOYMENT; MACHINE DESIGN; OPTICAL CHARACTER RECOGNITION; USER INTERFACES;

RESEARCH;

EID: 79960222794     PISSN: 19394608     EISSN: 19394616     Source Type: Book Series    
DOI: 10.2200/S00371ED1V01Y201107AIM013     Document Type: Article

References (365)

1. L. A. Adamic, J. Zhang, E. Bakshy, and M. S. Ackerman. Knowledge sharing and yahoo answers: Everyone knows something. In WWW, pages 665-674, 2008. DOI: 10.1145/1367497.1367587 Cited on page(s) 74
2. E.Agichtein, C.Castillo,D. Donato,A.Gionis, and G.Mishne. Finding high-quality content in social media. In WSDM, pages 183-194, 2008. DOI: 10.1145/1341531.1341557 Cited on page(s) 74 (Pubitemid 351606090)
3. N. Ailon, M. Charikar, and A. Newman. Aggregating inconsistent information: ranking and clustering. In STOC, 2005. DOI: 10.1145/1411509.1411513 Cited on page(s) 31
4. G.Akerlof. The market for lemons:Quality uncertainty and the market mechanism. Quarterly Journal of Economics, 84:488-500, 1970. DOI: 10.2307/1879431 Cited on page(s) 74
5. A. Alchian and H. Demsetz. Production, information costs, and economic organization. The American Economic Review, 62(5):777-795, December 1972. DOI: 10.1109/EMR.1975.4306431 Cited on page(s) 62
6. Amazon mechanical turk. https://www.mturk.com/mturk/welcome. Cited on page(s) 46
7. Amazon mechanical turk requestor interface. https://requester.mturk.com. Cited on page(s) 47
8. V. Ambati, S. Vogel, and J. Carbonell. Active learning and crowd-sourcing for machine translation. In LREC, 2010. Cited on page(s) 32, 35
9. D.Ariely,U. Gneezy,G. Loewenstein,and N.Mazar. Large stakes and big mistakes. Review of Economic Studies 75:1-19, 2009. DOI: 10.1111/j.1467-937X. 2009.00534.x Cited on page(s) 61
10. D. Ariely, E. Kamenica, and D. Prelec. Man's search for meaning: the case of legos. Journal of Economic Behavior & Organization, 67:671 - 677, 2008. DOI: 10.1016/j.jebo.2008.01.004 Cited on page(s) 61
11. Dan Ariely, Wing Tung Au, Randall H. Bender, David V. Budescu, Christiane B. Dietz, Hongbin Gu, Thomas S. Wallsten, and Gal Zauberman. The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied, pages 130-147, 2000. DOI: 10.1037/1076-898X.6.2.130 Cited on page(s) 31
12. S. Arora, E. Nyberg, and C.P. Rose. Estimating annotation cost for active learning in a multi-annotator environment. In NAACL HLT Workshop on Active Learning for Natural Language Processing, pages 18-26, 2009. DOI: 10.3115/1564131.1564136 Cited on page(s) 9
13. 3 = alpha (or beta). Technical report, University of Essex, 2005. Cited on page(s) 28
14. D. Autor. Wiring the labor market. Journal of Economics Perspectives, 15:25-40, 2001. DOI: 10.1257/jep.15.1.25 Cited on page(s) 75 (Pubitemid 33603378)
15. C. Avery, P. Resnick, and R. Zeckhauser. The market for evaluations. American Economic Review, 89(3):564-584, 1999. DOI: 10.1257/aer.89.3.564 Cited on page(s) 62
16. M. Babaioff, M. Feldman, and N. Nisan. Combinatorial agency. In ACM EC, pages 18-28, 2006. DOI: 10.1145/1134707.1134710 Cited on page(s) 62 (Pubitemid 44393620)
17. F. Baker. The Basics of Item Response Theory. Heinemann, 1985. Cited on page(s) 29
18. H. E. Bal, J. G. Steiner, and A. S. Tanenbaum. Programming languages for distributed computing systems. ACM Computing Surveys, 21 (3), 1989. DOI: 10.1145/72551.72552 Cited on page(s) 17
19. M. Balabanovic and Y. Shohan. Fab: content-based, collaborative recommendation. CACM, 40(3):66-72, 1997. DOI: 10.1145/245108.245124 Cited on page(s) 40
20. J. Baldridge and M. Osborne. Active learning and the total cost of annotation. In EMNLP, 2004. Cited on page(s) 22
21. S. Bangalore, G. Bordel, and G. Riccardi. Computing consensus translation from multiple machine translation systems. In IEEE Workshop on Automatic Speech Recognition and Understanding, 2001. DOI: 10.1109/ASRU.2001.1034659 Cited on page(s) 32
22. A. Barabasi. The origin of bursts and heavy tails in human dynamics. Nature, 435:207-211, 2005. DOI: 10.1038/nature03459 Cited on page(s) 40 (Pubitemid 40685946)
23. G. Baron. The 106th christmas bird count. In Audubon Society, 2006. Cited on page(s) 51
24. L. Barrington, D. O'Malley, D. Turnbull, and G. Lanckriet. User-centered design of a social game to tag music. In KDD-HCOMP, pages 7-10, 2009. DOI: 10.1145/1600150.1600152 Cited on page(s) 9, 69
25. R. Barzilay. Information Fusion for Multidocument Summarization. PhD thesis, Columbia University, 2003. Cited on page(s) 32
26. R. Barzilay and L. Lee. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In NAACL-HLT, 2003. DOI: 10.3115/1073445. 1073448 Cited on page(s) 30
27. R. Barzilay, K. R. McKeown, and M. Elhadad. Information fusion in the context of multi-document summarization. In ACL, pages 550-557, 1999. DOI: 10.3115/1034678.1034760 Cited on page(s) 32
28. R. Bell,Y. Koren, and C.Volinsky. Matrix factorization techniques for recommender systems. IEEE Computer, 42(8):30-37, 2009. DOI: 10.1109/MC.2009.263 Cited on page(s) 41
29. P. N. Bennett, M. Chickering, and A. Mityagin. Learning consensus opinion: Mining data from a labeling game. In WWW, 2009. DOI: 10.1145/1526709.1526727 Cited on page(s) 64
30. M. Bernstein, J.R. Brandt, R.C. Miller, and D.R. Karger. Crowds in two seconds: Enabling realtime crowd-powered interfaces. In UIST, 2011. Cited on page(s) 22
31. M. Berstein, G. Little, R. Miller, B. Hartmann, M. Ackerman, D. Karger, D. Crowell, and K. Panovich. Soylent: A word processor with a crowd inside. In UIST, 2010. DOI: 10.1145/1866029.1866078 Cited on page(s) 17, 56, 60
32. T. Bertin-Mahieux, D. Eck, F. Maillet, and P. Lamere. Autotagger: a model for predicting social tags from acoustic features on large music databases. TASLP, 37(2):115-135, 2008. DOI: 10.1080/09298210802479250 Cited on page(s) 8
33. J. Bian, Y. Liu, E. Agichtein, and H. Zha. Finding the right facts in the crowd: Factoid question answering over social media. In WWW, pages 467-476, 2008. DOI: 10.1145/1367497.1367561 Cited on page(s) 74
34. J. Bigham and A. Cavender. Evaluating existing audio captchas and an interface optimized for non-visual use. In NIPS, 2009. DOI: 10.1145/1518701. 1518983 Cited on page(s) 49
35. J.P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R.C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, and T. Yeh. Vizwiz: Nearly real-time answers to visual questions. In UIST, 2010. DOI: 10.1145/1866029.1866080 Cited on page(s) 21
36. D. Billsus and M. J. Pazzani. Learning collaborative information filters. In ICML, pages 46-54, 1998. Cited on page(s) 41
37. C. Boehm and G. Jacopini. Flow diagrams, turing machines and languages with only two formation rules. Communications of the ACM, 9(5):366-371, 1966. DOI: 10.1145/355592.365646 Cited on page(s) 17
38. Boinc. http://boinc.berkeley.edu/. Cited on page(s) 4
39. L. Bonetta. New citizens for the life sciences. Cell, 138:1043-1045, 2009. DOI: 10.1016/j.cell.2009.09.007 Cited on page(s) 52
40. R. Bonney, C. Cooper, J. Dickinson, S.Kelling,T.Philips, K. Rosenberg, and J. Shirk. Citizen science: A developing tool for expanding science knowledge and scientific literacy. BioScience, 59:977-984, 2009. DOI: 10.1525/bio.2009.59.11.9 Cited on page(s) 54
41. Boto. http://code.google.com/p/boto. Cited on page(s) 19
42. B. Bouzy and T. Cazenave. Computer go: An ai oriented survey. Artificial Intelligence, 132(1):39-103, October 2001. DOI: 10.1016/S0004-3702(01)00127-8 Cited on page(s) 10 (Pubitemid 32888154)
43. S. Branson, C. Wah, B. Babenko, F. Schroff, P. Welinder, P. Perona, and S. Belongie. Visual recognition with humans in the loop. In ECCV, 2010. Cited on page(s) 52, 74
44. C. Brodley and M. Friedl. Identifying mislabeled training data. Journal of Artificial Intelligence Research, 11:131-167, 1999. DOI: 10.1613/jair.606 Cited on page(s) 30 (Pubitemid 129628762)
45. D. Brossard, B. Lewenstein, and R. Bonney. Scientific knowledge and attitude change: the impact of a citizen science project. International Journal of Science Education, 27:1099-1121, 2005. DOI: 10.1080/09500690500069483 Cited on page(s) 54 (Pubitemid 41030400)
46. J. Brown and J. Morgan. Reputation in online auctions: The market for trust. California Management Review, 49(1):61-81, 2006. Cited on page(s) 74 (Pubitemid 44917127)
47. J. Burke, D. Estrin, M. Hansen, A. Parker, N. Ramanathan, S. Reddy, and M.B. Srivastava. Participatory sensing. In ACM Sensys World Sensor Web Workshop, 2006. Cited on page(s) 4
48. R. Burke. Hybrid recommender systems: survey and experiments. User Modeling and User-adapted Interaction,12(4):331-370, 2002.DOI: 10.1023/A:1021240730564 Cited on page(s) 41
49. R. Burke, B. Mobasher, R. Bhaumik, and C. William. Segment-based injection attacks against collaborative filtering recommender systems. In ICDM, pages 577-580, 2005. DOI: 10.1109/ICDM.2005.127 Cited on page(s) 41 (Pubitemid 47385744)
50. C. Camerer, L. Babcock, G. Loewenstein, and R. Thaler. Labor supply of new york city cabdrivers: One day at a time. The quarterly journal of economics, 112(2):407-441, 1997. DOI: 10.1162/003355397555244 Cited on page(s) 61 (Pubitemid 28004308)
51. C. N. Cardamone, K. Schawinski, M. Sarzi, S. P. Bamford, N. Bennert, C. M. Urry, C. Lintott, W. C. Keel, J. Parejko, R. C. Nichol, D. Thomas, D. Andreescu, P. Murray, M. J. Raddick, A.Slosar, A. Szalay,and J.VandenBerg. Galaxy zoo green peas: Discovery of a class of compact extremely star-forming galaxies. MNRAS, 2009. DOI: 10.1111/j.1365-2966.2009.15383.x Cited on page(s) 51
52. J. Carletta. Assessing agreement on classification tasks: the kappa statistics. Computational Linguistics, 22(2):249-254, 1996. Cited on page(s) 28
53. A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E.R. Hruschka Jr., and T.M. Mitchell. Towards an architecture for never-ending language learning. In AAAI, 2010. Cited on page(s) 74
54. B. Carpenter. Multilevel bayesian models of categorical data annotation. Technical report, Alias-i, 2008. Cited on page(s) 29
55. B. Carterette, P. N. Bennett, D. M. Chickering, and S. Dumais. Here or there: Preference judgments for relevance. In ECIR, pages 16-27, 2008. Cited on page(s) 18 (Pubitemid 351499055)
56. Catbook. http://bjoern.org/projects/catbook/. Cited on page(s) 60
57. J. Chamberlain, M. Poesio, and U. Kruschwitz. Phrase detectives - a web-based collaborative annotation game. In I-Semantics, 2008. Cited on page(s) 69
58. D. Chandler and A. Kepelner. Breaking monotony with meaning: Motivation in crowdsourc-ing markets. Technical report, MIT, 2010. working paper. Cited on page(s) 61
59. K. Che. Design competition through multidimensional auctions. RAND Journal of Economics, 24(4):668-680, 1993. DOI: 10.2307/2555752 Cited on page(s) 75
60. K. Chellapilla, K. Larson, P. Simard, and M. Czerwinski. Designing human friendly human interaction proofs. In CHI, 2005. DOI: 10.1145/1054972.1055070 Cited on page(s) 49
61. Y. Chen, L. Fortnow, N. Lambert, D. Pennock, and J. Wortman. Complexity of combinatorial market makers. In ACM EC, 2008. DOI: 10.1145/1386790.1386822 Cited on page(s) 32
62. Y. Chen and D. Pennock. Designing marekts for prediction. AI Magazine, 31(4):42-52, 2010. Cited on page(s) 32
63. M.Chi.Two approaches to the study of experts'characteristics. In K.A.Ericsson,N.Charness, P. J. Feltovich, and R. R. Hoffman, editors, The Cambridge handbook of expertise and expert performance, pages 21-30. Cambridge University Press, 2006. Cited on page(s) 35
64. L. Chilton, J. Horton, R. Miller, and S. Azenkot. Task search in a human computation market. In KDD-HCOMP, pages 1-9, 2010. DOI: 10.1145/1837885.1837889 Cited on page(s) 22, 40, 45
65. R. Clemen and R. Winkler. Combining probability distributions from experts in risk analysis. Risk Analysis, 19(2):187-203, 1999. DOI: 10.1023/A:1006917509560 Cited on page(s) 31 (Pubitemid 29374646)
66. A. Cobham. Priority assignment in waiting line problems. Journal of Operations Research Society of America, 2:70-76, 1954. Cited on page(s) 40
67. W.Cohen,R.Schapire,andY.Singer. Learning toorder things. Journal of Artificial Intelligence, 10:243-270, 1999. DOI: 10.1613/jair.587 Cited on page(s) 31
68. Community collaborative rain,hail and snownetwork. http://www.cocorahs. org/. Cited on page(s) 52
69. S. Cooper, F. Khatib, A.Treuille, J. Barbero, J. Lee, Michael Beenen, A. Leaver-Fay, D. Baker, Z. Popovic, and FoldIt Players. Predicting protein structures with a multiplayer online game. Nature, 466:756-760, August 2010. DOI: 10.1038/nature09304 Cited on page(s) 10, 59, 62, 69
70. S. Cooper, A. Treuille, J. Barbero, A. Leaver-Fay, K. Tuite, F. Khatib, A. Snyder, M. Beenen, D. Salesin, D. Baker, Z. Popovic, and 000 Foldit players. The challenge of designing scientific discovery games. In FDG, 2010. DOI: 10.1145/1822348.1822354 Cited on page(s) 52
71. T. Cormen, C. Leiserson, and R. Rivest. Introduction to Algorithms. McGraw-Hill, 1999. Cited on page(s) 17
72. Cornell lab of ornithology citizen science project. http:// cornellcitizenscience.org. Cited on page(s) 54
73. J.R. Corney, I. Kowalska, A.P. Jagadeesan, A. Lyn, Hugo Medellin, and W. Regli. Crowd-sourcing human problem solving strategy. In CrowdConf, 2010. Cited on page(s) 10, 62
74. D. Cosley, D. Frankowski, L. Terveen, and J. Riedl. Suggestbot: using intelligent task routing to help people find work in wikiepdia. In IUI, 2007. DOI: 10.1145/1216295.1216309 Cited on page(s) 41 (Pubitemid 47463988)
75. P. Cramton, Y. Shoham, and R. Steinberg, editors. Conbinatorial Auctions. MIT Press, 2006. Cited on page(s) 75
76. Cybertracker. http://www.cybertracker.org/. Cited on page(s) 52
77. P. Dai, Mausam, and D. Weld. Decision-theoretic control of crowd-sourced workflows. In AAAI, 2010. Cited on page(s) 19
78. P. Dai, Mausam, and D. Weld. Artificial intelligence for artificial artificial intelligence. In AAAI, 2011. Cited on page(s) 19
79. A. Dawid and A. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics, 28(1):20-28, 1979. DOI: 10.2307/2346806 Cited on page(s) 28
80. L.T. DeCarlo and D.V. Cross. Sequential effects in magnitude scaling: models and theory. Journal of Experimental Psychology: General, 119:375-396, 1990. DOI: 10.1037/0096-3445.119.4.375 Cited on page(s) 59
81. E. L. Deci, R. Koestner, and R. M. Ryan. A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin, 6(125):627-668, 1999. DOI: 10.1037/0033-2909.125.6.627 Cited on page(s) 61
82. O. Dekel and O. Shamir. Good learners for evil teachers. In ICML, 2009. DOI: 10.1145/1553374.1553404 Cited on page(s) 30
83. O. Dekel and O. Shamir. Vox populi: Collecting high-quality labels from a crowd. In COLT, 2009. Cited on page(s) 29
84. C. Dellarocas. The digitization of word-of-mouth: promise and challenges of online reputation mechanisms. Management Science, 10(49):1407-1424, 2003. DOI: 10.1287/mnsc.49.10.1407.17308 Cited on page(s) 74
85. C. Dellarocas, F. Dini, and G. Spagnolo. Designing reputation mechanisms. In G. Spagnolo N. Dimitri, G. Piga, editor, Handbook of Procurement. Cambridge University Press, 2007. Cited on page(s) 74
86. C. Dellarocas and C. A. Wood. The sound of silence in online feedback: Estimating trading risks in the presence of reporting bias. Management Science, 54(3):460-476, 2008. DOI: 10.1287/mnsc.1070.0747 Cited on page(s) 29 (Pubitemid 351463127)
87. Deneme: a blog of experiments on amazon mechanical turk. http://groups.csail.mit. edu/uid/deneme. Cited on page(s) 21
88. J. Deng, W. Dong, R. Socher, L.J. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. Cited on page(s) 8
89. P. Denning. The great principles of computing. American Scientist, September-October 2010. Cited on page(s) 1
90. D. Dipalantino and M. Vojnovic. Corwdsourcing and all-pay auctions. In ACM EC, 2009. DOI: 10.1145/1566374.1566392 Cited on page(s) 62
91. A. Doan, R. Ramakrishnan, and A. Y. Halevy. Mass collaboration systems on the world-wide web. Communications of the ACM, 54(4), 2011. DOI: 10.1145/1924421.1924442 Cited on page(s) 5
92. P. Donmez and J. Carbonell. Proactive learning: Cost-sensitive active learning with multiple imperfect oracles. In CIKM, 2008. DOI: 10.1145/1458082.1458165 Cited on page(s) 38, 39
93. P. Donmez and J. Carbonell. From active to proactive learning methods. In J. Koronacki, S.T. Wierzchon, Z. Ras, and J. Kacprzyk, editors, Recent Advances in Machine Learning. Springer: Studies in Computational Intelligence, 2009. Cited on page(s) 23
94. P. Donmez, J. Carbonell, and J. Schneider. Efficiently leanring the accuracy of labeling sources for selective sampling. In KDD, 2009. DOI: 10.1145/1557019.1557053 Cited on page(s) 39
95. P. Donmez, J. Carbonell, and J. Schneider. A probabilistic framework to learn from multiple annotators with time-varying accuracy. In SDM, 2010. Cited on page(s) 40
96. C. Dwork, R. Kumar, M. Naor, and D. Sivakumar. Rank aggregation methods for the web. In WWW, 2001. DOI: 10.1145/371920.372165 Cited on page(s) 31
97. E-bird. http://ebird.org/. Cited on page(s) 51
98. N. Eagle. txteagle: Mobile crowdsourcing. Lecture Notes in Computer Science: Internationalization, Design and Global Development, 5623:447-456, 2009. DOI: 10.1007/978-3-642-02767-3-50 Cited on page(s) 39
99. K. Ehrlich, C.Y. Lin, and V. Griffiths-Fisher. Searching for experts in the enterprise: combining text and social network analysis. In GROUP, 2007. DOI: 10.1145/1316624.1316642 Cited on page(s) 41
100. Einstein@home. http://einstein.phys.uwm.edu/. Cited on page(s) 4
101. A.Elo. The Rating of Chessplayers, Past and Present. Arco Publications, 1978. Cited on page(s) 31
102. K. Eriksoon and B. Simpson. Emotional reactions to losing explain gender differences in entering a risky lottery. Judgment and Decision, 5(3):159-163, 2010. Cited on page(s) 46
103. William S. Evans. Information Theory and Noisy Computation. PhD thesis, University of California at Berkeley, 1994. Cited on page(s) 21
104. Evolution megalab. http://www.evolutionmegalab.org. Cited on page(s) 52
105. Eyes of the reef. http://www.reefcheckhawaii.org/eyesofthereef.htm. Cited on page(s) 52
106. R. Fagin, R. Kumar, M. Mahdian, D. Sivakumar, and E. Vee. Comparing partial rankings. SIAMJournal of Discrete Mathematics,3(20):628-648,2006.DOI: 10.1137/05063088X Cited on page(s) 31 (Pubitemid 47250574)
107. H. S. Farber. Reference-dependent preferences and labor supply: The case of new york city taxi drivers. American Economic Review, 98(3):1069-1082, 2008. Cited on page(s) 61
108. S. Faridani, B. Hartmann, and P. Ipeirotis. What's the right price: pricing tasks for finishing on time. In HCOMP, 2011. Cited on page(s) 22
109. E. Fehr and A. Falk. Psychological foundations of incentives. European Economic Review, 46(4-5):687-724, 2002. DOI: 10.1016/S0014-2921(01)00208-2 Cited on page(s) 61 (Pubitemid 34580819)
110. E. Fehr and L. Goette. Do workers work more if wages are high: Evidence from a randomized field experiment. American Economic Review, 97(1):298-317, 2007. DOI: 10.1257/aer.97.1.298 Cited on page(s) 61
111. A. Felstiner. Working the crowd: employment and labor law in the crowdsourcing industry. http://ssrn.com/abstract=1593853, 2011. Cited on page(s) 74
112. P. Floreen, A. Kruger, and M. Spasojevic. Recruitment framework for participatory sensing data collection. In Pervasive, pages 138-155, 2010. DOI: 10.1007/978-3-642-12654-3-9 Cited on page(s) 36, 39
113. J. Fogarty, D. Tan, A. Kapoor, and S. Winder. Cueflik: Interactive concept learning in image search. In CHI, 2008. DOI: 10.1145/1357054.1357061 Cited on page(s) 74
114. Folding@home. http://folding.stanford.edu/. Cited on page(s) 4
115. M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. Answering queries with crowdsourcing. In SIGMOD, 2011. DOI: 10.1145/1989323.1989331 Cited on page(s) 56
116. R. Frederking and S. Nirenburg. Three heads are better than one. In ANLP, pages 95-100, 1994. Cited on page(s) 32
117. B. S. Frey and R. Jegen. Motivation crowding theory. Journal of Economic Surveys, 5(15):589- 611, 2001. DOI: 10.1111/1467-6419.00150 Cited on page(s) 61
118. E. Friedman and P. Resnick. The social cost of cheap pseudonyms. Journal of Economic Management Strategy, 10(1):173-199, 2001. DOI: 10.1162/ 105864001300122476 Cited on page(s) 57
119. E. Friedman, P. Resnick, and R. Sami. Manipulation-resistant reputation systems. In N. Nisan, T. Roughgarden, E. Tardos, and V. Vazirani, editors, Algorithmic game theory. Cambridge University Press, 2007. Cited on page(s) 74
120. J. Froehlich, M.Y. Chen, I.E. Smith, and F. Potter. Voting with your feet: An investigative study of the relationship between place visit behavior and preference. In Ubicomp, 2006. DOI: 10.1007/11853565-20 Cited on page(s) 39
121. D. Gale and L. S. Shapley. College admissions and the stability of marriage. The American Mathematical Monthly, 1(69):9-15, 1962. DOI: 10.2307/2312726 Cited on page(s) 38
122. F. Garcin, B. Faltings, R. Jurca, and N. Joswig. Rating aggregation in collaborative filtering systems. In the third ACM Conference on Recommender Systems, pages 349-352, 2009. DOI: 10.1145/1639714.1639785 Cited on page(s) 31
123. M. R. Garey and D. S. Johnson. Computers and Intractibility: A guide to the theory of NP-completeness. W.H. Freeman, 1979. Cited on page(s) 10
124. A. Ghosh and P. Hummel. A game-theoretic analysis of rank-order mechanisms for user-generated content. In ACM EC, 2011. DOI: 10.1145/1993574.1993603 Cited on page(s) 61
125. A. Ghosh, S. Kale, and P. McAfee. Who moderates the moderators? crowdsourcing abuse detection in user-generated content. In ACM EC, 2011. DOI: 10.1145/1993574.1993599 Cited on page(s) 31
126. A. Ghosh and P. McAfee. Incentivizing high-quality user-generated content. In WWW, 2011. DOI: 10.1145/1963405.1963428 Cited on page(s) 61
127. A. Gionis, H. Mannila, and P. Tsaparas. Clustering aggregation. In ICDE, pages 341-352, 2005. DOI: 10.1145/1217299.1217303 Cited on page(s) 31 (Pubitemid 41731148)
128. U. Gneezy and A. Rustichini. Pay enough or don't pay at all. Quarterly journal of economics, 115:791-810, 2000. DOI: 10.1162/003355300554917 Cited on page(s) 61
129. A. Goder and V. Filkov. Consensus clustering algorithms: Comparison and refinement. In SIAM Ninth Workshop on Algorithm Engineering and Experiments (ALENEX), 2008. Cited on page(s) 31
130. Google image labeler. http://images.google.com/imagelabeler/help.html. Cited on page(s) 8, 50
131. D.A. Grier. When Computers were Human. Princeton University Press, 2005. Cited on page(s) 1, 2, 51
132. G. Gutin and A. Punnen. The Traveling Salesman Problem and Its Variations. Springer, 2006. Cited on page(s) 10
133. S. Hacker and L. von Ahn. Eliciting user preferences with an online game. In CHI, pages 1207-1216, 2009. DOI: 10.1145/1518701.1518882 Cited on page(s) 18, 31, 64
134. J. W. Hatfield and S. D. Kominers. Matching in networks with bilateral contracts. In ACM EC, pages 119-120, 2010. DOI: 10.1145/1807342.1807361 Cited on page(s) 38
135. R. Herbrich, T. Minka, and T. Graepel. Trueskilltm: A bayesian skill rating system. In NIPS, pages 569-576, 2007. Cited on page(s) 31
136. A. Herdagdelen and M. Baroni. The concept game: Better commonsense knowledge extraction by combining text mining and a game with a purpose. In AAAI Fall Symposium on Commonsense Knowledge, 2010. Cited on page(s) 68
137. J. L. Herlocker, J. Konstan, L. Terveen, and J. Riedl. Evaluating collaborative filtering rec-ommender systems. ACM Transactions on Information Systems, 22(1):5-53, 2004. DOI: 10.1145/963770.963772 Cited on page(s) 41
138. B. E. Hermalin. Toward an economic theory of leadership: leading by example. The American Economic Review, 88(5):1188-1206, December 1998. DOI: 10.2139/ssrn.15570 Cited on page(s) 62 (Pubitemid 128453594)
139. P. Heymann and H. Garcia-Molina. Human processing. Technical report, Standard University, July 2010. Cited on page(s) 21
140. A. V. Hill. An experimental comparison of human schedulers and heuristic algorithms for the traveling salesman problem. Journal of Operations Management, 2(4):215-223, August 1982. DOI: 10.1016/0272-6963(82)90010-9 Cited on page(s) 10
141. C.J. Ho, T.H. Chang, J.C. Lee, J. Hsu, and K.T. Chen. Kisskissban: a competitive human computation game for image annotation. In Proceedings of the first Workshop on Human Computation, pages 11-14, 2009. DOI: 10.1145/1600150.1600153 Cited on page(s) 68
142. J. Hodge and R. E. Klima. The Mathematics of Voting and Elections: A Hands-On Approach. American Mathematics Society, 2000. Cited on page(s) 31
143. M. Hoffman, D. Blei, and P. Cook. Easy as cba: A simple probabilistic model for tagging music. In ISMIR, pages 369-374, 2009. Cited on page(s) 8
144. J. Horton. Online labor markets. In the 6th Workshop on Internet and Network Economics (WINE), 2010. DOI: 10.1007/978-3-642-17572-5-45 Cited on page(s) 75
145. J. Horton and L. Chilton. The labor economics of paid crowdsourcing. In EC, 2010. DOI: 10.1145/1807342.1807376 Cited on page(s) 61
146. J. Horton, D. Rand, and R. Zeckhauser. The online laboratory. In WINE, 2010. Cited on page(s) 46, 62
147. E. Horvitz. Reflections on challenges and promises of mixed-initiative interaction. AAAI Magazine, 28, 2007. Cited on page(s) 74
148. E. Horvitz, J. Apacible, and P. Koch. Busybody: Creating and fielding personalized models of the cost of interruption. In CSCW, 2004. DOI: 10.1145/1031607.1031690 Cited on page(s) 39
149. E. Horvitz and T. Paek. Complementary computing: Policies for transferring callers from dialog systems to human receptionists. User Modeling and User Adapted Interaction, 17, 2007. DOI: 10.1007/s11257-006-9026-1 Cited on page(s) 6, 74 (Pubitemid 46327429)
150. E. Huang, H. Zhang, D. Parkes, K. Gajos, and Y. Chen. Toward automatic task design: A progress report. In Proceedings of the second Human Computation Workshop, 2010. DOI: 10.1145/1837885.1837908 Cited on page(s) 23, 55, 60
151. Y. Huang. Mixed-initiative clustering. PhD thesis, Carnegie Mellon University, 2010. Cited on page(s) 31
152. Human computation supplementary resources. http://humancomputation.com/ book. Cited on page(s) 46
153. L. Hurwicz. On informationally decentralized systems. In C.B. McGuire and R. Radner, editors,Decision and Organization: a Volume in Honor of Jacob Marshak,pages 297-336.North-Holland, 1972. Cited on page(s) 63
154. Igda causal games white paper 2008-2009. http://archives.igda.org/casual/ IGDA-Casual-Games-White-Paper-2008.pdf. Cited on page(s) 49, 50
155. Imagenet large scale visual recognition challenge 2010. http://www.image-net.org/ challenges/LSVRC/2010/index. Cited on page(s) 8
156. P. Ipeirotis. Analyzing the amazon mechanical turk marketplace. XRDS, 17(2), 2010. DOI: 10.1145/1869086.1869094 Cited on page(s) 45, 46, 61
157. P. Ipeirotis. Demographics of mechanical turk. Technical report, New York University, 2010. Working Paper. Cited on page(s) 47
158. P. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In HCOMP, 2010. DOI: 10.1145/1837885.1837906 Cited on page(s) 26, 28, 29
159. M. O. Jackson. Mechanism theory. In U. Derigs, editor, Encyclopedia of Life Support Systems. EOLSS Publishers, 2003. Cited on page(s) 62
160. S. Jain, Y. Chen, and D. Parkes. Designing incentives for online question and answers forums. In EC, pages 129-138, 2009. DOI: 10.1145/1566374.1566393 Cited on page(s) 61
161. S. Jain and D. Parkes. A game-theoretic analysis of games with a purpose. In WINE '08: Proceedings of the 4th International Workshop on Internet and Network Economics,pages 342-350, 2008. DOI: 10.1007/978-3-540-92185-1-40 Cited on page(s) 67
162. S. Jain and D. Parkes. Combinatorial agency of threshold functions. In ACM EC Workshop on social computing and user generated content, 2011. Cited on page(s) 62
163. S. Jayaraman and A. Lavie. Multi-engine machine translation guided by explicit word matching. In EAMT, 2005. DOI: 10.3115/1225753.1225779 Cited on page(s) 32
164. T. Joachims. Optimizing search engines using clickthrough data. In KDD, 2002. DOI: 10.1145/775047.775067 Cited on page(s) 31
165. R. Jurca and B. Faltings. An incentive-compatible reputation mechanism. In IEEE Conference on E-Commerce, pages 285-292, 2003. DOI: 10.1145/860575. 860778 Cited on page(s) 74
166. R. Jurca and B. Faltings. Mechanisms for making crowds truthful. Journal of Artificial Intelligence Research, 34(34):209-253, 2009. Cited on page(s) 32
167. L. Kaelbling. Learning in Embedded Systems. The MIT Press, Cambridge, MA, 1993. Cited on page(s) 39
168. L. Kaelbling, M. Littman, and A. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237-285, 1996. Cited on page(s) 39 (Pubitemid 126646155)
169. A. Kapoor and E. Horvitz. On discarding, caching, and recalling samples in active learning. In UAI, 2007. Cited on page(s) 39
170. A. Kapoor and E. Horvitz. Experience sampling for building predictive user models: A comparative study. In CHI, 2008. DOI: 10.1145/1357054.1357159 Cited on page(s) 39
171. G. Karabatsos and W. Batchelder. Markov chain estimation for test theory without an answer key. Psychometrika, 68(3):373-389, 2003. DOI: 10.1007/BF02294733 Cited on page(s) 29 (Pubitemid 37524544)
172. M. Kearn,S. Suri, and N. Montfort. An experimental study of the coloring problem on human subject networks. Science, 313(5788):824-827, August 2006. DOI: 10.1126/science.1127207 Cited on page(s) 4, 10, 18 (Pubitemid 44223348)
173. A. J. Kim. Community building on the web: secret strategies for successful online communities. Peachpit Press, 2000. Cited on page(s) 55
174. Y.E. Kim, E. Schmidt, and L. Emelle. Moodswings: A collaborative game for music mood label collection. In ISMIR, pages 231-236, 2008. Cited on page(s) 69
175. A. Kittur. Crowdsourcing, collaboration and creativity. XRDS, 17(2):22 - 26, 2010. DOI: 10.1145/1869086.1869096 Cited on page(s) 59, 60, 62
176. A. Kittur, E. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In CHI, 2008. DOI: 10.1145/1357054.1357127 Cited on page(s) 60, 62
177. A. Kittur and R. Kraut. Harnessing the wisdom of crowds in wikipedia: Quality through coordination. In CHI, 2008. DOI: 10.1145/1460563.1460572 Cited on page(s) 5
178. A. Kittur, B. Smus, and R. Kraut. Crowdforge: Crowdsourcing complex work. Technical report, Carnegie Mellon University, 2011. DOI: 10.1145/1979742. 1979902 Cited on page(s) 19, 62
179. A. Kittur, B. Suh, B. Pendleton, and E. Chi. He says, she says: Conflict and coordination in wikipedia. In CHI, 2007. Cited on page(s) 5
180. G. W. Klau, N. Lesh, J. Marks, and M. Mitzenmacher. Human-guided tabu search. In AAAI, pages 41-47, 2002. DOI: 10.1007/s10732-009-9107-5 Cited on page(s) 10, 11
181. G. W. Klau, N. Lesh, J. Marks, M. Mitzenmacher, and G. T. Schafer. The hugs platform: a toolkit for interactive optimization. In Advanced Visual Interfaces, pages 324-330, 2002. DOI: 10.1145/1556262.1556314 Cited on page(s) 10, 11
182. D. Knuth. The Art of Computer Programming: Second Edition. Addison-Wesley, 1973. Cited on page(s) 16
183. R. Kowalski. Algorithm logic control. Communications of the ACM, 22(7):424-436, July 1979. DOI: 10.1145/359131.359136 Cited on page(s) 17
184. R. E. Kraut and P. Resnick. Evidence-based social design: Mining the social sciences to build online communities. MIT Press, 2011. Cited on page(s) 55, 61
185. A. Kulkarni and M. Can. Turkomatic: Automatic recursive task design for mechanical turk. In CHI Work-in-Progress, 2011. Cited on page(s) 19, 59
186. Y.L. Kuo, K.Y. Chiang, C.W. Chan, J.C. Lee, R. Wang, E. Shen, and J. Hsu. Community-based game design: Experiments on social games for commonsense data collection. In KDD Human Computation Workshop (HCOMP 2009), 2009. DOI: 10.1145/1600150.1600154 Cited on page(s) 45, 62
187. S.K. Lam and J. Riedl. Shilling recommender systems for fun and profit. In WWW, pages 393-402, 2004. DOI: 10.1145/988672.988726 Cited on page(s) 41 (Pubitemid 40752774)
188. E. Law, P. Bennett, and E. Horvitz. The effects of choice in routing relevance judgments. In SIGIR, 2011. Cited on page(s) 40
189. E. Law, B. Settles, and T. Mitchell. Learning to tag from open vocabulary labels. In ECML, 2010. DOI: 10.1007/978-3-642-15883-4-14 Cited on page(s) 30, 51, 67
190. E.Law and L.von Ahn. Input-agreement:A newmechanism for data collection using human computation games. In CHI,pages 1197-1206,2009.DOI: 10.1145/1518701.1518881 Cited on page(s) 9, 25, 51, 65, 69
191. E. Law, L. von Ahn, R. Dannenberg, and M. Crawford. Tagatune: A game for music and sound annotation. In ISMIR, 2007. Cited on page(s) 64
192. E. Law, K. West, M. Mandel, M. Bay, and S. Downie. Evaluation of algorithms using games: The case of music tagging. In ISMIR, 2009. Cited on page(s) 9
193. E. Law and H. Zhang. Towards large-scale collaborative planning: Answering high-level search queries using human computation. In AAAI, 2011. Cited on page(s) 59, 74
194. J. Lee and B. Hoh. Sell your experiences: A market mechanism based incentive for participatory sensing. In IEEE PerCom, 2010. DOI: 10.1109/PERCOM.2010.5466993 Cited on page(s) 62
195. D. Levy and M. Newborn. How Computers Play Chess. W.H. Freeman, 1990. Cited on page(s) 10
196. D. Lewis and J. Catlett. Heterogeneous uncertainty sampling for supervised learning. In ICML, pages 148-156, 1994. Cited on page(s) 23
197. Lhca@home. http://lhcathome.cern.ch/. Cited on page(s) 4
198. C.Y. Lin, K. Ehrlich, V. Griffiths-Fisher, and C. Desforges. Smallblue:
199. G. Linden, B.Smith, and J.York. Amazon.com recommendations: item-to-item collaborative filtering. IEEE Internet Computing, 7(1):76-80, 2003. DOI: 10.1109/MIC.2003.1167344 Cited on page(s) 41
200. C. Lintott, K. Schawinski, S. Bamford, A. Slosar, K. Land, D. Thomas, E. Edmondson, K. Masters, R. Nichol, J. Raddick, A. Szalay, D. Andreescu, P. Murray, and J. Vandenberg. Galaxy zoo 1 : Data release of morphological classifications for nearly 900,000 galaxies. MNRAS, 2010. DOI: 10.1111/j.1365-2966.2010.17432.x Cited on page(s) 51
201. C.J. Lintott, K. Schawinski, A. Slosar, K. Land, S. Bamford, D. Thomas, M.J. Raddick, R.Nichol,A.S.Szalay,D.Andreescu,P.Murray,and J.Vandenberg. Galaxy zoo:Morphologies derived from visual inspection of galaxies from the sloan digital sky survey. MNRAS,389:1179, 2008. DOI: 10.1111/j.1365-2966.2008.13689.x Cited on page(s) 51
202. G. Little, L. Chilton, M. Goldman, and R. Miller. Exploring iterative and parallel human computationprocesses.InHCOMP,2010.DOI: 10.1145/1837885.1837907 Cited on page(s) 32, 59
203. G. Little, L. Chilton, M. Goldman, and R. Miller. Turkit: Human computation algorithms on mechanical turk. In UIST, 2010. DOI: 10.1145/1866029.1866040 Cited on page(s) 59
204. G. Little, L. Chilton, R. Miller, and M. Goldman. Turkit: Tools for iterative tasks on mechanical turk. In HCOMP, 2009. DOI: 10.1145/1600150.1600159 Cited on page(s) 17, 19, 20
205. D. Lizotte, O. Madani, and R. Greiner. Budgeted learning of naive-bayes classifiers. In UAI, 2003. Cited on page(s) 23
206. R.D. Luce and H. Raiffa. Games and Decisions: Introduction and Critical Survey. Wiley, 1957. Cited on page(s) 38
207. G. Lugosi. Learning with an unreliable teacher. Pattern Recognition, 25(1):79-87, 1992. DOI: 10.1016/0031-3203(92)90008-7 Cited on page(s) 30
208. C. Lundmark. Bioblitz: Getting into backyard biodiversity. Bioscience, 53:329, 2003. DOI: 10.1641/0006-3568(2003)053[0329:BGIBB]2.0.CO;2 Cited on page(s) 52 (Pubitemid 36453245)
209. I. Lynce and J. Ouaknine. Sudoku as a sat problem. In 9th Symposium on Artificial Intelligence and Mathematics, 2006. Cited on page(s) 10
210. C. Macdonald and I. Ounis. Voting for candidates: adapting data fusion techniques for an expert search task. In CIKM, pages 387-396, 2006. DOI: 10.1145/1183614.1183671 Cited on page(s) 41 (Pubitemid 47203584)
211. J.N. MacGregor and T. Ormerod. Human performance on the traveling salesman problem. Perception & Psychophysics, 58(4):527-539, 1996. DOI: 10.3758/BF03213088 Cited on page(s) 10 (Pubitemid 126421137)
212. Magnatagatune. http://tagatune.org/Magnatagatune.html. Cited on page(s) 9
213. T. Malone. Heuristics for designing enjoyable user interfaces. In CHI, pages 63-68, 1982. DOI: 10.1145/800049.801756 Cited on page(s) 63
214. T.W. Malone and R.J. Laubacher. The dawn of the e-lance economy. Harvard Business Review, 76(5):144 - 152, 1998. Cited on page(s) 5, 74
215. M. Mandel and D. Ellis. A web-based game for collecting music metadata. Journal of New Music Research, 37(2):151-165, 2009. DOI: 10.1080/ 09298210802479300 Cited on page(s) 9, 64
216. A.Mao,D.Parkes, A.Procaccia, and H. Zhang. Human computation and multiagent systems: An algorithmic perspective. In working paper, 2011. Cited on page(s) 18
217. A. Marcus, E. Wu, D. Karger, S. Madden, and R. Miller. Crowdsourced databases: Query processing with people. In CIDR, 2011. Cited on page(s) 56
218. M. Marge, S. Banerjee, and A. I. Rudnicky. Using the amazon mechanical turk for transcription of spoken language. In ICASSP, 2010. DOI: 10.1109/ICASSP.2010.5494979 Cited on page(s) 8
219. W. Mason and S. Suri. Conducting behavioral research on amazon's mechanical turk. In Social Science Research Network Working Paper Series, 2010. Cited on page(s) 46, 47
220. W. Mason and D. Watts. Financial incentives and the "performance of the crowds". In Proceedings of the first Human Computation Workshop, 2009. DOI: 10.1145/1600150.1600175 Cited on page(s) 61
221. W. Mason and D. Watts. Collaborative problem solving in networks. In Submission, 2011. Cited on page(s) 60
222. E. Matusov, N. Ueffing,and H. Ney. Computing consensus translation from multiple machine translation systems using enhancedhypotheses alignment. InEACL,pages 33-40,2006. Cited on page(s) 32
223. S.T. Maytal, Prem Melville, and F. Provost. Active feature-value acquisition for model induction. Management Science, 55(4):664-684, 2009. DOI: 10.1287/mnsc.1080.0952 Cited on page(s) 39
224. S.T. Maytal and F. Provost. Decision-centric active learning of binary-outcome models. Information Systems Research, 18(1):1-19, 2007. DOI: 10.1287/isre.1070.0111 Cited on page(s) 39
225. D.W. McDonald and M.S. Ackerman. Expertise recommender: A flexible recommendation architecture. In CSCW, pages 231-240, 2000. DOI: 10.1145/358916.358994 Cited on page(s) 41 (Pubitemid 32400054)
226. C. McMillen and M. Veloso. Thresholded rewards: acting optimally in timed, zero-sum games. In AAAI, 2007. Cited on page(s) 21
227. C. McMillen and M. Veloso. Unknown rewards in finite-horizon domains. In AAAI, 2008. Cited on page(s) 21
228. P. Milgrom. Package auctions and package exchanges. Econometrica, 75(4):935-966, 2007. DOI: 10.1111/j.1468-0262.2007.00778.x Cited on page(s) 38
229. N. Miller, P. Resnick, and R. Zeckhauser. Eliciting honest feedback: the peer prediction method. Management Science,9(51):1359-1373,2005. DOI: 10.1287/mnsc.1050.0379 Cited on page(s) 32, 61 (Pubitemid 41394242)
230. Mirex. http://www.music-ir.org/mirex/wiki/MIREX-HOME. Cited on page(s) 9
231. B. Moldovanu and A. Sela. The optimal allocation of prizes in contests. American economic review, 91:542-558, June 2001. DOI: 10.1257/aer.91.3.542 Cited on page(s) 62 (Pubitemid 33597260)
232. B. Moldovanu and A. Sela. Contest architecture. Journal of economic theory, 126(1):70-96, January 2006. DOI: 10.1016/j.jet.2004.10.004 Cited on page(s) 62 (Pubitemid 43071922)
233. G. Mori and J. Malik. Recognizing objects in adversarial clutter: Breaking a visual catpcha. In CPVR, 2003. DOI: 10.1109/CVPR.2003.1211347 Cited on page(s) 49
234. M. Motoyama, K. Levchenko, C. Kanich, D. McCoy, G. M. Voelker, and S. Savage. Re: Captchas - understanding captcha-solving services in an economic context. In USENIX Security Symposium, 2010. Cited on page(s) 49
235. M. Mozer, H. Pashler, M. Wilder, R. Lindsey, M. Jones, and M. Jones. Decontaminating human judgments by removing sequential dependencies. In NIPS, 2010. Cited on page(s) 59
236. R.Munro. Crowdsourced translation for emergency response in haiti: the global collaboration of local knowledge. In AMTA Workshop on Collaborative Crowdsourcing for Translation, 2010. Cited on page(s) 54
237. K. K. Nam, M. S. Ackerman,and L. A. Adamic. Questions in, knowledge in? a study of naver's question answering community. In CHI, 2009. DOI: 10.1145/1518701.1518821 Cited on page(s) 61
238. G. Nemhauser, L. Wolsey, and M. Fisher. An analysis of approximations for maximizing submodular set functions. Mathematical Programming, 14(1):265-294, 1978. DOI: 10.1007/BF01588971 Cited on page(s) 37
239. J. Nielsen. Usability Engineering. Morgan Kaufmann, 1993. Cited on page(s) 45
240. N.Nisan. Introduction to mechanism design (for computer scientists). In N. Nisan,T.Rough-garden, E. Tardos, and V. Vazirani, editors, Algorithmic Game Theory. Cambridge University Press, 2007. Cited on page(s) 62, 63
241. D.W. North. A tutorial introduction to decision theory. IEEE Transactions on Systems Science and Cybernetics, 4(3):200-210, 1968. DOI: 10.1109/TSSC.1968.300114 Cited on page(s) 38, 39
242. C.Notredame. Recent progresses in multiple sequence alignment: a survey. Pharmacogenomics, 31(1):131-144, 2002. DOI: 10.1517/14622416.3.1.131 Cited on page(s) 10 (Pubitemid 34285245)
243. C. Notredame. Pulsar discovery by global volunteer computing. Science, 329(5997):1305, September 2010. DOI: 10.1126/science.1195253 Cited on page(s) 4
244. P. O'Keefe. Managing online forums: everything you need to know to create and run successful community discussion boards. AMACOM, 2008. Cited on page(s) 55
245. F. Olsson. A literature survey of active machine learning in the context of natural language processing. Technical report, Swedish Institute of Computer Science, 2009. Cited on page(s) 22
246. G. Paolacci, J. Chandler, and P. Ipeirotis. Running experiments on amazon mechanical turk. Judgment and Decision, 5(5):411-419, 2010. Cited on page(s) 46
247. T. Papaioannou and G. D. Stamoulis. An incentives' mechanism promoting truthful feedback in peer-to-peer systems. In ACM CCGRID Workshop on Global P2P Computing, 2005. DOI: 10.1109/CCGRID.2005.1558565 Cited on page(s) 74 (Pubitemid 44910952)
248. A. Parameswaran and N. Polyzotis. Answering queries using humans, algorithms and databases. In CIDR, 2011. Cited on page(s) 56
249. A. Parameswaran, A. Das Sarma, H. Garcia-Molina, N. Polyzotis, and Jennifer Widom. Human-assisted graph search:It's okayto ask questions.Technical report,Stanford University, 2011. Cited on page(s) 10
250. A. Parducci. The relativism of absolute judgements. Scientific American, 219(6):84-90, 1968. DOI: 10.1038/scientificamerican1268-84 Cited on page(s) 59
251. A. Parducci. Category ratings and the relational character of judgment. In H.G. Geissler, editor,Modern issues in perception, volume 1, pages 262-282. Elsevier Science Publishers, 1991. Cited on page(s) 59
252. R. Patz, B. Junker, and M. Johnson. The hierarchical rater model for rated test items and its application to large-scale educational assessment data. Journal of Educational and Behavioral Statistics, 27(4):341-381, 2002. DOI: 10.3102/10769986027004341 Cited on page(s) 29
253. E. Paulos. Designing for doubt: Citizen science and the challenge of change. In Engaging Data: First International Forum on the Application and Management of Personal Electronic Information, 2009. Cited on page(s) 52
254. D. Pennock and R. Sami. Computational aspects of prediction marekts. In N. Nisan, T. Roughgarden, E. Tardos, and V. Vazirani, editors, Algorithmic game theory. Cambridge University Press, 2007. Cited on page(s) 32
255. Phylo - a human computing framework for comparative genomics. http://phylo.cs. mcgill.ca/. Cited on page(s) 10, 69
256. K. Pipineni, S. Roukos, T. Ward, and W. Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318, 2002. DOI: 10.3115/1073083.1073135 Cited on page(s) 32
257. N. Pippenger. Reliable computation by formulas in the presence of noise. IEEE Transactions on Information Theory, 34(2):194-197, 1988. DOI: 10.1109/18.2628 Cited on page(s) 21 (Pubitemid 18639972)
258. D. M. Powazek. Design for community: the art of connecting real people in virtual places. Pearson Technology, 2002. Cited on page(s) 55
259. J. Preece. Online communities: designing usability and supporting sociability. John Wiley & Sons, 2000. Cited on page(s) 55
260. Primegrid. http://www.primegrid.com/. Cited on page(s) 4
261. M.J. Raddick, G. Bracey, P.L. Gay, C.J. Lintott, P. Murray, K. Schawinski, A.S. Szalay, and J. Vandenberg. Galaxy zoo: Exploring the motivations of citizen science volunteers. MNRAS, 2010. DOI: 10.3847/AER2009036 Cited on page(s) 51, 54
262. F. Radlinski and T. Joachims. Query chains: learning to rank from implicit feedback. In KDD, 2005. DOI: 10.1145/1081870.1081899 Cited on page(s) 31 (Pubitemid 43218285)
263. H. Raghavan, O. Madani, and R. Jones. Active learning with feedback on both features and instances. Journal of Machine Learning Research, 7:1655-1686, 2006. Cited on page(s) 23 (Pubitemid 44223089)
264. V. Raykar, S. Yu, L. Zhao, G. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. Journal of Machine Learning Research, 11:1297-1322, 2010. Cited on page(s) 29, 30
265. V.C. Raykar, S. Yu, L.H. Zhao, A. Jerebko, C. Florin, G.H. Valadez, L. Bogoni, and L. Moy. Supervised learning from multiple experts: Whom to trust when everyone lies a bit. In ICML, 2009. DOI: 10.1145/1553374.1553488 Cited on page(s) 29, 30
266. S. Reiss. Multifaceted nature of intrinsic motivation: the theory of 16 basic desires. Review of General Psychology, 8:179-193, 2004. DOI: 10.1037/1089-2680.8.3.179 Cited on page(s) 61 (Pubitemid 39299948)
267. P. Resnick, I. Neophytos, P. Bergstrom, S. Mitesh, and J. Reidl. Grouplens: An open architecture for collaborative filtering of netnews. In CSCW, 1994. DOI: 10.1145/192844.192905 Cited on page(s) 41
268. K. Ristovski, D. Das, V. Ouzienko, Y. Guo, and Z. Obradovic. Regression learning with multiple noisy oracles. In ECAI, 2010. Cited on page(s) 30
269. M. Rodriguez and J. Bollen. An algorithm to determine peer reviewers. In CIKM, 2008. DOI: 10.1145/1458082.1458127 Cited on page(s) 40
270. K. Romney. Culture consensus as a statistical model. Current Anthropology, 40:103-115, 1999. DOI: 10.1086/200063 Cited on page(s) 27
271. K. Romney, S. Weller, and W. Batchelder. Culture as consensus: A theory of culture and informant accuracy. American Anthropology, 88(2):313-338, 1986 DOI: 10.1525/aa.1986.88.2.02a00020 Cited on page(s) 27, 29
272. J. Ross, L. Irani, M.S. Silberman, A. Zaldivar, and B.Tomlinson. Who are the crowdworkers: shifting demographics in mechanical turk. In CHI Work-in-Progress, pages 2863-2872, 2010. DOI: 10.1145/1753846.1753873 Cited on page(s) 46, 47, 74
273. A.-V.I. Rosti, N.F. Ayan, B. Xiang, S. Matsoukas, R. Schwartz, and B.J. Dorr. Combining outputs from multiple machine translation systems. In NAACL-HLT, pages 228-235, 2007. Cited on page(s) 32
274. A. E. Roth. The economics of matching: stability and incentives. Mathetmatics of Operations Research, 4(7):617-628, 1982. DOI: 10.1287/moor.7.4.617 Cited on page(s) 38
275. A. E. Roth. The economist as engineer: game theory, experimentation, and computation as tools for design economics. Econometrica, 4(1):1341-1378, 2002. DOI: 10.1111/1468-0262.00335 Cited on page(s) 38 (Pubitemid 34799523)
276. N. Roy and A. McCallum. Toward optimal active learning through sampling estimation of error reduction. In ICML, pages 441-448, 2001. Cited on page(s) 23
277. T. Sandholm. Expressive commercse and its application to sourcing: how we conducted $35 billion of generalized combinatorial auctions. AI Magazine, 28(3):45-58, 2007. DOI: 10.1145/1282100.1282165 Cited on page(s) 75 (Pubitemid 350229447)
278. B. Sarwar, G. Karypis, J. Konstan, and J. Reidl. Item-based collaborative filtering algorithms. In WWW, pages 285-295, 2001. DOI: 10.1145/371920.372071 Cited on page(s) 41
279. G. Sauer, H. Hochheiser, J. Feng, and J. Lazar. Towards a universally usable captcha. In SOUPS, 2008. Cited on page(s) 49
280. T. Schulze. Worker motivation in crowdsourcing and human computation. In HCOMP, 2011. Cited on page(s) 61
281. Scienceforcitizens.net. http://scienceforcitizens.net/. Cited on page(s) 52
282. N. Seemakurty, J. Chu, L. von Ahn, and A. Tomasic. Word sense disambiguation via human computation. In 2nd KDD Human Computation Workshop, 2010. DOI: 10.1145/1837885.1837905 Cited on page(s) 64
283. Seti@home. http://setiathome.berkeley.edu/. Cited on page(s) 4
284. B. Settles. Active learning literature survey. Technical report, University of Wisconsin-Madison, 2009. Cited on page(s) 6, 22, 23, 38
285. B. Settles, M. Craven, and L. Friedland. Active learning with real annotation costs. In NIPS Workshop on Cost Sensitive Learning, 2008. Cited on page(s) 9
286. H.S. Seung, M. Opper, and H. Sompolinsky. Query by committee. In ACM Workshop on Computational Learning Theory, pages 287-294, 1992. DOI: 10.1145/130385.130417 Cited on page(s) 23 (Pubitemid 23615454)
287. D. Shahaf and E. Amir. Towards a theory of ai completeness. In 8th International Symposium on Logical Formalizations of Commonsense Reasoning, 2007. Cited on page(s) 21
288. D. Shahaf and E. Horvitz. Generalized task markets for human and machine computation. In AAAI, 2010. Cited on page(s) 37, 74
289. A. D. Shaw, J. J. Horton, and D. L. Chen. Designing incentives for inexpert human raters. In CSCW, 2011. DOI: 10.1145/1958824.1958865 Cited on page(s) 61
290. V. Sheng, F. Provost, and P. Ipeirotis. Get another label: improving data quality and data mining using multiple,noisy labelers. InKDD,2008.DOI: 10.1145/1401890.1401965 Cited on page(s) 23
291. M. Shilman, D. Tan, and P. Simard. Cuetip: A mixed-initiative interface for correcting handwriting errors. In CHI, pages 323-332, 2006. DOI: 10.1145/1166253.1166304 Cited on page(s) 74 (Pubitemid 47142458)
292. M. Six Silberman, L. Irani, and J. Ross. Ethics and tactics of professional crowdwork. XRDS, 17(2):39 - 43, 2010. DOI: 10.1145/1869086.1869100 Cited on page(s) 74
293. M.Six Silberman,J.Ross,L. Irani,and B.Tomlinson. Sellers'problems in human computation markets. In HCOMP, 2010. DOI: 10.1145/1837885.1837891 Cited on page(s) 74
294. J. Silvertown. A new dawn for citizen science. Cell, 24(9), 2009. DOI: 10.1016/j.tree.2009.03.017 Cited on page(s) 52
295. Y. Singer and M. Mittal. Pricing mechnaisms for online labor markets. In HCOMP, 2011. Cited on page(s) 22
296. K. Siorpaes and M. Hepp. Games with a purpose for the semantic web. IEEE Intelligent Systems, 23(3):50-60, 2008. DOI: 10.1109/MIS.2008.45 Cited on page(s) 64 (Pubitemid 351751946)
297. D.B.Skillicorn and D.Talia. Models and languages for parallel computation. ACM Computing Surveys, 30(2):123-169, 1998. DOI: 10.1145/280277.280278 Cited on page(s) 17
298. P. Smyth, U. Fayyad, M. Burl, P. Perona, and P. Baldi. Inferring ground truth from subjective labeling of venus images. In NIPS, pages 1-9, 1997. Cited on page(s) 26, 30
299. R. Snow, B. O'Connor, D. Juraksky, and A. Ng. Cheap and fast - but is it good: evaluating non-expert annotations for natural language tasks. In EMNLP, 2008. DOI: 10.3115/1613715.1613751 Cited on page(s) 9, 35
300. R. Speer, C. Havasi, and H. Surana. Using verbosity: Common sense data from games with a purpose. In FLAIRS, 2010. Cited on page(s) 65, 67
301. K. Starbird. Digital volunteerism during disaster: Crowdsourcing information processing. In CHI Workshop on Crowdsourcing and Human Computation, 2011. Cited on page(s) 54
302. N. Stewart, G.D.A. Brown, and N. Chater. Absolute identification by relative judgment. Psychological Review, 112:881-911, 2005. DOI: 10.1037/0033-295X.112.4.881 Cited on page(s) 59 (Pubitemid 41677522)
303. H. S. Stone. Algorithms, turing machines, and programs. In Introduction to Computer Organization and Data Structures. McGraw-Hill, 1972. Cited on page(s) 16
304. A. Strehl and J. Ghosh. Cluster ensembles - a knowledge reuse framework for combining multiple partitions. Journal of Machine Learning Research, 3:583-617, 2002. DOI: 10.1162/153244303321897735 Cited on page(s) 31
305. S. Sueken, J. Tang, and D. Parkes. Accounting mechanisms for distributed work systems. In AAAI, pages 860-866, 2010. Cited on page(s) 62
306. B.L. Sullivan, C.L. Wood, M.J. Iliff, R. E. Bonney, D. Fink, and S. Kelling. ebird: A citizen-based bird observation network in the biological sciences. Biological Conservation, 2009. DOI: 10.1016/j.biocon.2009.05.006 Cited on page(s) 51, 52
307. S. Suri and D. Watts. Cooperation and contagion in networked public goods experiments. PLoS ONE, 6(3), 2011. DOI: 10.1371/journal.pone.0016836 Cited on page(s) 46
308. P. Sweetser and P. Wyeth. Gameflow: a model for evaluating player enjoyment in games. ACM Computers in Entertainment, 3(3), 2005. DOI: 10.1145/1077246.1077253 Cited on page(s) 63
309. G. Takacs, I. Pilaszy, B. Nemeth, and D. Tikk. On the gravity recommendation system. In KDD, pages 22-30, 2007. Cited on page(s) 31
310. J. Tam, J. Simsa, S. Hyde, and L. von Ahn. Breaking audio captchas. In NIPS, 2009. Cited on page(s) 49
311. O. Tamuz, C. Liu, S. Belongie, O. Shamir, and A. Kalai. Adaptively Learning the Crowd Kernel. In Proceedings of the 28th International Conference on Machine Learning, 2011. Cited on page(s) 74
312. L.Terry, V. Roitch, S.Tufail, K. Singh, O.Taraq,W. Luk, and P. Jamieson. Harnessing human computation cycles for the fpga placement problem. In ERSA, 2009. Cited on page(s) 10
313. L.TerveenandD.W.McDonald.Socialmatching:a frameworkandresearchagenda. TOCHI, 12(3):401-434, 2005. DOI: 10.1145/1096737.1096740 Cited on page(s) 41 (Pubitemid 43892810)
314. S.Thaler,E. Simperl,K.Siorpaes, and C.Hofer. A survey on games for knowledge acquisition. Technical report, STI, May 2011. Cited on page(s) 69
315. The great sunflower project. http://www.greatsunflower.org/. Cited on page(s) 52
316. The lost tomb of genghis khan. http://exploration.nationalgeographic.com/ mongolia/. Cited on page(s) 8
317. Trec. http://trec.nist.gov/. Cited on page(s) 9
318. D.J. Trumbull, R. Bonney, D. Bascom, and A. Cabral. Thinking scientifically during participation in a citizen-science project. Science Education, 84:265-275, 2000. DOI: 10.1002/(SICI)1098-237X(200003)84:2%3C265:: AID-SCE7%3E3.0.CO;2-5 Cited on page(s) 54
319. D.J.Trumbull,R.Bonney,and N.Grudens-Schuck. Developing materials to promote inquiry: Lessons learned. Science Education, 89:879-900, 2005. DOI: 10.1002/sce.20081 Cited on page(s) 54 (Pubitemid 41728641)
320. G. Tullock. Efficient rent seeking. In Towards a Theory of the Rent-Seeking Society. Texas A&M University Press, 1980. Cited on page(s) 62
321. A. M. Turing. Computing machinery and intelligence. Mind LIX, 2236:43-60, October 1950. Cited on page(s) 3
322. Turkit. http://groups.csail.mit.edu/uid/turkit/. Cited on page(s) 19, 21
323. D. Turnbull, L. Barrington, D. Torres, and G. Lanckriet. Five approaches to collecting tags for music. In ISMIR, 2008. Cited on page(s) 8
324. D.Turnbull,L. Barrington, D.Torres, and G. Lanckriet. Semantic annotation and retrieval of music and sound effects. TASLP, 16(2):467-476, 2008. DOI: 10.1109/TASL.2007.913750 Cited on page(s) 8
325. D. Turnbull, R. Liu, L. Barrington, and G. Lanckriet. A game-based approach for collecting semantic annotations for music. In ISMIR, pages 535-538, 2007. Cited on page(s) 9, 51, 64
326. A. Tverskey and D. Kahneman. Judgment under uncertainty: Heuristics and biases. Science, 185:1124-1130, 1974. DOI: 10.1126/science.185.4157.1124 Cited on page(s) 59
327. J. S. Uebersax. Statistical modeling of expert ratings on medical treatment appropriateness. Journal of the American Statistical Association,88(422):421-427, 1993. DOI: 10.2307/2290320 Cited on page(s) 29
328. D. Vickrey, A. Bronzan, W. Choi, A Kumar, J.Turner-Maier, A. Wang, and D. Koller. Online word games for semantic data collection. In EMNLP,2008.DOI: 10.3115/1613715.1613781 Cited on page(s) 64
329. D.Vickrey, A. Bronzan,W. Choi, A. Kumar, J.Turner-Maier, A.Wang, and D. Koller. Online word games for semantic data collection. In EMNLP, pages 535-538, 2008. Cited on page(s) 67, 69
330. S. Vijayanarasimhan and K. Grauman. What's it going to cost you: Predicting effort vs. informativeness for multi-label image annotations. CVPR, pages 2262-2269, 2009. DOI: 10.1109/CVPR.2009.5206705 Cited on page(s) 23
331. A. Vlachos. Active annotation. In EACL Workshop on Adaptive Text Extraction, 2006. Cited on page(s) 22
332. L. von Ahn. Human Computation. PhD thesis, Carnegie Mellon University, 2005. Cited on page(s) 3
333. L. von Ahn, M. Blum, and J. Langford. Telling humans and computers apart automatically: How lazy cryptographers do ai. Communications of the ACM, 47(2):57-60, 2004. Cited on page(s) 48, 49
334. L. von Ahn and L.Dabbish. Labeling images with a computer game. In CHI,pages 319-326, 2004. DOI: 10.1145/985692.985733 Cited on page(s) 8, 25, 49, 63, 67, 69
335. L. von Ahn and L. Dabbish. Designing games with a purpose. Communications of the ACM, 51(8), August 2008. DOI: 10.1145/1378704.1378719 Cited on page(s) 65, 67
336. L. von Ahn and L. Dabbish. General techniques for designing games with a purpose. In CACM, pages 58-67, 2008. Cited on page(s) 7, 63
337. L. von Ahn, R. Liu, and M. Blum. Peekaboom: A game for locating objects in images. In CHI Notes, pages 55-64, 2006. DOI: 10.1145/1124772.1124782 Cited on page(s) 65, 69 (Pubitemid 44032086)
338. L.von Ahn,R.Liu,and M.Blum. Verbosity:A game for collecting common-sense knowledge. In CHI Notes, pages 75-78, 2006. DOI: 10.1145/1124772.1124784 Cited on page(s) 25, 65, 69 (Pubitemid 44032088)
339. L. von Ahn, B. Maurer, C. McMillen, D. Abraham, and M. Blum. recaptcha: Human-based character recognition via web security measures. Science, pages 1465-1468, 2008. DOI: 10.1126/science.1160379 Cited on page(s) 49
340. J. von Neumann. Probabilistic logics and the synthesis of reliable organisms from unreliable components. In C. E. Shannon and J. McCarthy, editors, Automata Studies, pages 43 - 98. Princeton University Press, 1956. Cited on page(s) 21
341. B.C. Wallace, K. Small, C.E. Brodley, and T.A. Trikalinos. Who should label what? Instance allocation in multiple expert active learning. In Proc. of SIAM International Conference on Data Mining (SDM), 2011. Cited on page(s) 23
342. J. Wang, S. Faridani, and P. Ipeirotis. Estimating the completion time of crowdsourced tasks using survival analysis models. In CSDM, 2011. Cited on page(s) 40
343. I. Weber, S.Robertson,and M.Vojnovic. Rethinking the esp game. In CHI, pages 3937-3942, 2009. DOI: 10.1145/1520340.1520597 Cited on page(s) 67
344. G.M. Weiss and F. Provost. Learning when training data are costly: the effect of class distribution on tree induction. Journal of Artificial Intelligence Research, 19:315-354, 2003. DOI: 10.1613/jair.1199 Cited on page(s) 23 (Pubitemid 41525924)
345. P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In NIPS, pages 1-9, 2010. Cited on page(s) 28, 29
346. P. Welinder and P. Perona. Online crowdsourcing: rating annotators and obtaining cost-effective labels. In CPVR, 2010. Cited on page(s) 29, 40
347. S.C. Weller. Cross-cultural concept of illness: Variation and validation. American Anthropologist, 86(2):341-351, 1984. DOI: 10.1525/aa.1984.86.2. 02a00090 Cited on page(s) 27
348. S.C. Weller. Cultural consensus theory: Applications and frequently asked questions. Field Methods, 19(4):339-368, 2007. DOI: 10.1177/1525822X07303502 Cited on page(s) 29
349. S.C. Weller and N.C. Mann. Assessing rater performance without "gold standard" using consensus theory. Medical Decision Making, 17(1):71-79, 1997. DOI: 10.1177/0272989X9701700108 Cited on page(s) 27 (Pubitemid 27020141)
350. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, 2009. Cited on page(s) 29
351. Wikiepdia entry for "list of cognitive biases". http://en.wikipedia.org/wiki/ List-of-cognitive-biases. Cited on page(s) 59
352. Wikipedia. http://wikipedia.com. Cited on page(s) 5
353. T.R. Williams. Reconsidering the history of the aavso. Journal of the American Association of Variable Star Observers, 29:132, 2001. Cited on page(s) 51
354. R. Wilson and F. Keil, editors. The MIT Encyclopedia of the Cogntivie Sciences. MIT Press, 1999. Cited on page(s) 29
355. J. Wolfers and E. Zitzewitz. Prediction markets. Journal of Economic Perspectives, 18(2):107- 126, 2004. DOI: 10.1257/0895330041371321 Cited on page(s) 32 (Pubitemid 39057292)
356. J. Yan and A.S. El Ahmad. Usability of captchas. In SOUPS, 2008. DOI: 10.1145/1408664.1408671 Cited on page(s) 49
357. Y. Yan, R. Rosales, G. Fung, and J. Dy. Active learning from crowds. In ICML, 2011. Cited on page(s) 29
358. A. Yates and O. Etzioni. Unsupervised methods for determining object and relation. Journal of Artificial Intelligence Research, 34:255-296, 2009. Cited on page(s) 30
359. Y. Yue and T. Joachims. Interactively optimizing information retrieval systems as a dueling bandits problem. In ECML, 2009. DOI: 10.1145/1553374. 1553527 Cited on page(s) 74
360. O. F. Zaidan and C. Callison-Burch. Feasibility of human-in-the-loop minimum error rate training. In EMNLP, 2009. DOI: 10.3115/1699510.1699518 Cited on page(s) 32
361. H. Zhang, E. Horvitz, Y. Chen, and D. Parkes. Task routing for prediction tasks. In ACM EC Workshop on Social Computing and User Generated Content, 2011. Cited on page(s) 41
362. H. Zhang, E. Horvitz, R. Miller, and D. Parkes. Crowdsourcing general computation. Technical report, Microsoft Research, 2011. Cited on page(s) 19
363. J. Zhang, M. S. Ackerman, and L. Adamic. Expertise networks in online communities: Structure and algorithms. InWWW,2007.DOI: 10.1145/1242572.1242603 Cited on page(s) 41 (Pubitemid 47582252)
364. Y. Zhou, G. Cong, B. Cui, C. Jensen, and J. Yao. Routing questions to the right users in online communities. In ICDE, 2009. Cited on page(s) 41
365. Zooniverse. http://www.zooniverse.org. Cited on page(s) 51