Quizz: Targeted crowdsourcing with a billion (potential) users

WWW 2014 - Proceedings of the 23rd International Conference on World Wide Web

Volumn , Issue , 2014, Pages 143-153

  Ipeirotis, Panagiotis G ^a   Gabrilovich, Evgeniy ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

FEEDBACK; INFORMATION THEORY; MARKETING; PUBLISHING;

ADVERTISING CAMPAIGN; ADVERTISING NETWORKS; ADVERTISING SYSTEMS; CONTROLLED EXPERIMENT; CROWDSOURCING PLATFORMS; INCENTIVE MECHANISM; KNOWLEDGEABLE USERS; TARGETING CAPABILITY;

CROWDSOURCING;

EID: 84909594574     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2566486.2567988     Document Type: Conference Paper

References (40)

1. Abhishek, V., and Hosanagar, K. Keyword generation for search engine advertising using semantic similarity between terms. In Proceedings of the ninth international conference on Electronic commerce (2007), ACM, pp. 89-94.
2. Anderson, A., Huttenlocher, D., Kleinberg, J., and Leskovec, J. Steering user behavior with badges. In Proceedings of the 22nd international conference on World Wide Web (2013), International World Wide Web Conferences Steering Committee, pp. 95-106.
3. Archer, E., Park, I. M., and Pillow, J. W. Bayesian entropy estimation for countable discrete distributions. CoRR abs/1302.0328 (2013).
4. Ariely, D. Predictably irrational: The hidden forces that shape our decisions. HarperCollins, 2009.
5. Attfield, S., Kazai, G., Lalmas, M., and Piwowarski, B. Towards a science of user engagement (position paper). In WSDM Workshop on User Modelling for Web Applications (2011).
6. Baeza-Yates, R., and Lalmas, M. User engagement: the network effect matters! In Proceedings of the 21st ACM international conference on Information and knowledge management (2012), ACM, pp. 1-2.
7. Berger, R. L. Multiparameter hypothesis testing and acceptance sampling. Technometrics 24, 4 (1982), 295-300.
8. Dawid, A. P., and Skene, A. M. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics 28, 1 (Sept. 1979), 20-28.
9. Dodge, H. F. Notes on the Evolution of Acceptance Sampling. American Society for Quality Control, 1973.
10. Dupret, G., and Lalmas, M. Absence time and user engagement: evaluating ranking functions. In Proceedings of the sixth ACM international conference on Web search and data mining (2013), ACM, pp. 173-182.
11. Easley, D., and Ghosh, A. Incentives, gamification, and game theory: an economic approach to badge design. In Proceedings of the fourteenth ACM conference on Electronic commerce (2013), EC '13, ACM, pp. 359-376.
12. Embretson, S. E., and Reise, S. P. Item response theory. Psychology Press, 2000.
13. Fuxman, A., Tsaparas, P., Achan, K., and Agrawal, R. Using the wisdom of the crowds for keyword generation. In Proceedings of the 17th international conference on World Wide Web (2008), ACM, pp. 61-70.
14. Gelman, A., Carlin, J. B., Stern, H. S., and Rubin, D. B. Bayesian data analysis. CRC press, 2003.
15. Genest, C., and Zidek, J. V. Combining probability distributions: A critique and an annotated bibliography. Statistical Science 1, 1 (1986), 114-135.
16. Gneezy, U., and Rustichini, A. Pay enough or don't pay at all. The Quarterly Journal of Economics 115, 3 (2000), 791-810.
17. Hoffmann, R., Amershi, S., Patel, K., Wu, F., Fogarty, J., and Weld, D. S. Amplifying community content creation with mixed initiative information extraction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2009), ACM, pp. 1849-1858.
18. Horton, J. Online labor markets. Internet and Network Economics (2010), 515-522.
19. Ipeirotis, P. Demographics of mechanical turk. Tech. rep., New York University, 2010. Available at http://ssrn.com/abstract=1585030.
20. Joshi, A., and Motwani, R. Keyword generation for search engine advertising. In Data Mining Workshops, 2006. ICDM Workshops 2006. Sixth IEEE International Conference on (2006), IEEE, pp. 490-496.
21. Kamar, E., Hacker, S., and Horvitz, E. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS (2012), International Foundation for Autonomous Agents and Multiagent Systems, pp. 467-474.
22. Kuznetsov, S. Motivations of contributors to wikipedia. ACM SIGCAS computers and society 36, 2 (2006), 1.
23. Lehmann, J., Lalmas, M., Yom-Tov, E., and Dupret, G. Models of user engagement. In User Modeling, Adaptation, and Personalization. Springer, 2012, pp. 164-175.
24. Li, S.-M., Mahdian, M., and McAfee, R. P. Value of learning in sponsored search auctions. In Internet and Network Economics. Springer, 2010, pp. 294-305.
25. Malone, T. W. Toward a theory of intrinsically motivating instruction. Cognitive science 5, 4 (1981), 333-369.
26. McCay-Peet, L., Lalmas, M., and Navalpakkam, V. On saliency, affect and focused attention. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2012), ACM, pp. 541-550.
27. Murphy, K. P. Machine learning: a probabilistic perspective. The MIT Press, 2012.
28. Nov, O. What motivates wikipedians? Communications of the ACM 50, 11 (2007), 60-64.
29. Puterman, M. L. Markov decision processes: Discrete stochastic dynamic programming, vol. 414. Wiley. com, 2009.
30. Raykar, V. C., Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., and Moy, L. Learning from crowds. The Journal of Machine Learning Research 99 (2010), 1297-1322.
31. Robertson, S., Vojnovic, M., and Weber, I. Rethinking the esp game. In CHI'09 Extended Abstracts on Human Factors in Computing Systems (2009), ACM, pp. 3937-3942.
32. Schilling, E. G. Acceptance Sampling in Quality Control, vol. 42. CRC PressI Llc, 1982.
33. Von Ahn, L., and Dabbish, L. Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human factors in computing systems (2004), ACM, pp. 319-326.
34. Von Ahn, L., Maurer, B., McMillen, C., Abraham, D., and Blum, M. Recaptcha: Human-based character recognition via web security measures. Science 321, 5895 (2008), 1465-1468.
35. Wang, J., Ipeirotis, P., and Provost, F. Quality-based pricing for crowdsourced workers. Tech. rep., New York University, 2013. Available at papers.ssrn.com/abstract=2283000.
36. Waterhouse, T. P. Pay by the bit: an information-theoretic metric for collective human judgment. In Proceedings of the 2013 conference on Computer supported cooperative work (2013), ACM, pp. 623-638.
37. Welinder, P., Branson, S., Belongie, S., and Perona, P. The multidimensional wisdom of crowds. Advances in Neural Information Processing Systems 23 (2010), 2424-2432.
38. Wetherill, G., and Chiu, W. A review of acceptance sampling schemes with emphasis on the economic aspect. International Statistical Review/Revue Internationale de Statistique (1975), 191-210.
39. Whitehill, J., Wu, T.-f., Bergsma, J., Movellan, J. R., and Ruvolo, P. L. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems (2009), pp. 2035-2043.
40. Yang, H.-L., and Lai, C.-Y. Motivations of wikipedia content contributors. Computers in Human Behavior 26, 6 (2010), 1377-1383.