Crowd IQ: Measuring the intelligence of crowdsourcing platforms

Proceedings of the 4th Annual ACM Web Science Conference, WebSci'12

Volumn volume, Issue , 2012, Pages 151-160

  Kosinski, Michal ^a   Bachrach, Yoram ^b   Kasneci, Gjergji ^b   Van Gael, Jurgen ^b   Graepel, Thore ^b  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b MICROSOFT RESEARCH   (United Kingdom)

Author keywords

Crowdsourcing; Incentive schemes; Psychometrics

Indexed keywords

AMAZON'S MECHANICAL TURKS; CROWDSOURCING; INCENTIVE SCHEMES; MAJORITY VOTE; MULTIPLE RESPONSE; PERFORMANCE BASED; PSYCHOMETRICS;

EID: 84869015270     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2380718.2380739     Document Type: Conference Paper

References (44)

1. L.A. Adamic, J. Zhang, E. Bakshy, and M.S. Ackerman. Knowledge sharing and yahoo answers: everyone knows something. In Proceeding of the 17th international conference on World Wide Web, pages 665-674. ACM, 2008.
2. N. Archak and A. Sundararajan. Optimal design of crowdsourcing contests. ICIS 2009 Proceedings, 200, 2009.
3. Y. Bachrach, E. Markakis, E. Resnick, A.D. Procaccia, J.S. Rosenschein, and A. Saberi. Approximating power indices: theoretical and empirical analysis. Autonomous Agents and Multiagent Systems, 2010.
4. Y. Bachrach, A. Parnes, A.D. Procaccia, and J.S. Rosenschein. Gossip-based aggregation of trust in decentralized reputation systems. Autonomous Agents and Multi-Agent Systems, 19(2):153-172, 2009.
5. Y. Bachrach, E. Porat, and J.S. Rosenschein. Sketching techniques for collaborative filtering. Proceedings of the 21st international jont conference on Artifical intelligence, pages 2016-2021, 2009.
6. Y. Bachrach, Graepel T., Kasneci G., Kosinski M., and J. Van-Gael. Crowd IQ - aggregating opinions to boost performance. In AAMAS, 2012.
7. W.A.M. Borst. Understanding Crowdsourcing: Effects of motivation and rewards on participation and performance in voluntary online activities. PhD thesis, Erasmus University Rotterdam, 2010.
8. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28(1):20-28, 1979.
9. D. DiPalantino and M. Vojnovic. Crowdsourcing and all-pay auctions. In Proceedings of the 10th ACM conference on Electronic commerce, pages 119-128. ACM, 2009.
10. A. Doan, R. Ramakrishnan, and A.Y. Halevy. Crowdsourcing systems on the world-wide web. Communications of the ACM, 54(4):86-96, 2011.
11. A. Galland, S. Abiteboul, A. Marian, and A. Senellart. Corroborating information from disagreeing views. In WSDM, pages 131-140, 2010.
12. A. Gao, Y. Bachrach, Key P., and T. Graepel. Quality expectation-variance tradeoffs in crowdsourcing contests. AAAI, 2012.
13. D. Goldberg, D. Nichols, B.M. Oki, and D. Terry. Using collaborative filtering to weave an information tapestry. Communications of the ACM, 35(12):61-70, 1992.
14. L.S. Gottfredson. Why g matters: The complexity of everyday life. Intelligence, 24(1):79-132, 1997.
15. T. Gowers and M. Nielsen. Massively collaborative mathematics: The Polymath project. Nature, 461(7266):879-881, October 2009.
16. P.Y. Hsueh, P. Melville, and V. Sindhwani. Data quality from crowdsourcing: a study of annotation selection criteria. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, pages 27-35. Association for Computational Linguistics, 2009.
17. P.G. Ipeirotis, F. Provost, and J. Wang. Quality management on Amazon Mechanical Turk. In Proceedings of the ACM SIGKDD workshop on human computation, pages 64-67. ACM, 2010.
18. A.R. Jensen. The g factor: The science of mental ability. London: Westport, 1998.
19. A. Jøsang, R. Ismail, and C. Boyd. A survey of trust and reputation systems for online service provision. Decision support systems, 43(2):618-644, 2007.
20. G. Kasneci, J. Van Gael, D. H. Stern, and T. Graepel. Cobayes: bayesian knowledge corroboration with assessors of unknown areas of expertise. In WSDM, pages 465-474, 2011.
21. G. Kasneci, J. Van Gael, D. H. Stern, R. Herbrich, and T. Graepel. Bayesian knowledge corroboration with logical rules and user feedback. In ECML/PKDD (2), pages 1-18, 2010.
22. G. Linden, B. Smith, and J. York. Amazon. com recommendations: Item-to-item collaborative filtering. Internet Computing, IEEE, 7(1):76-80, 2003.
23. C. List and C. Puppe. Judgment aggregation: A survey. Handbook of Rational and Social Choice, 2009.
24. D. Lubinski. Introduction to the special section on cognitive abilities: 100 years after spearman's (1904) "general intelligence, objectively determined and measured". Journal of Personality and Social Psychology, 86(1):96, 2004.
25. J.A. Lyle. Collective problem solving: Are the many smarter than the few? 2008.
26. W. Mason and D.J. Watts. Financial incentives and the performance of crowds. ACM SIGKDD Explorations Newsletter, 11(2):100-108, 2010.
27. A. McLennan. Consequences of the condorcet jury theorem for beneficial information aggregation by rational agents. American Political Science Review, pages 413-418, 1998.
28. D.M. Pennock and R. Sami. Computational aspects of prediction markets, 2007.
29. J.C. Raven. Standard progressive matrices plus, sets a-e.
30. J.C. Raven. Progressive matrices. Éditions Scientifiques et Psychotechniques, 1938.
31. J.C. Raven. The raven's progressive matrices: Change and stability over culture and time. Cognitive Psychology, 41(1):1-48, 2000.
32. V.C. Raykar, S. Yu, L.H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. Journal of Machine Learning Research, 11:1297-1322, 2010.
33. B. Sarwar, G. Karypis, J. Konstan, and J. Reidl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web, pages 285-295. ACM, 2001.
34. G. Sautter and K. Böhm. High-throughput crowdsourcing mechanisms for complex tasks. Social Informatics, pages 240-254, 2011.
35. F.L. Schmidt and J. Hunter. General mental ability in the world of work: occupational attainment and job performance. Journal of Personality and Social Psychology, 86(1):162, 2004.
36. A. Sen. Social choice theory. Handbook of mathematical economics, 3:1073-1181, 1986.
37. L. S. Shapley. A value for n-person games. Contrib. to the Theory of Games, pages 31-40, 1953.
38. L. S. Shapley and M. Shubik. A method for evaluating the distribution of power in a committee system. American Political Science Review, 48:787-792, 1954.
39. C. Spearman. The abilities of man. 1927.
40. L. von Ahn. Games with a Purpose. Computer, 39(6):92-94, June 2006.
41. M. Vukovic. Crowdsourcing for enterprises. In Services-I, 2009 World Conference on, pages 686-692. Ieee, 2009.
42. A. Whitby, A. Jøsang, and J. Indulska. Filtering out unfair ratings in bayesian reputation systems. In Proc. 7th Int. Workshop on Trust in Agent Societies, 2004.
43. J. Wolfers and E. Zitzewitz. Prediction markets. Technical report, National Bureau of Economic Research, 2004.
44. A.W. Woolley, C.F. Chabris, A. Pentland, N. Hashmi, and T.W. Malone. Evidence for a collective intelligence factor in the performance of human groups. Science, 330(6004):686, 2010.