No oops, you won't do it again: Mechanisms for self-correction in crowdsourcing

33rd International Conference on Machine Learning, ICML 2016

Volumn 1, Issue , 2016, Pages 1-21

  Shah, Nihar B ^a   Zhou, Dengyong ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CROWDSOURCING; ERRORS; LEARNING SYSTEMS;

LABELED DATA; LARGE AMOUNTS; MODERN MACHINES; NUMERICAL EXPERIMENTS; SELF-CORRECTION;

LEARNING ALGORITHMS;

EID: 84997755074     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (57)

1. Aggarwal, V., Srikant, S., and Shashidhar, V. Principles for using machine learning in the assessment of open response items: Programming assessment as a case study. In NIPS Workshop on Data Driven Education, 2013.
2. Alonso, O. and Mizzaro, S. Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. In SIGIR Workshop on the Future of IR Evaluation, pp. 15-16, 2009.
3. Ayewah, N. and Pugh, W. Using checklists to review static analysis warnings. In Workshop on Defects in Large Software Systems (at ACM ISSTA), 2009.
4. Bearden, W. and Rose, R. Attention to social comparison information: An individual difference factor affecting consumer conformity. Journal of Consumer Research, 1990.
5. Bernstein, M. S., Little, G., Miller, R. C, Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., and Panovich, K. Soylent: a word processor with a crowd inside. In UIST, 2010.
6. Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
7. Burges, C, Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., and Hüllender, G. Learning to rank using gradient descent. In ICML, 2005.
8. Carlson, A., Betteridge, J., Wang, R. and Hruschka Jr, E., and Mitchell, T. Coupled semi-supervised learning for information extraction. In WSDM, 2010.
9. Chros, O. and Sundell, S. Digitalkoot: Making old archives accessible using crowdsourcing. In Human Computation, 2011.
10. Clayton, M. J. Delphi: a technique to harness expert opinion for critical decision-making tasks in education. Educational Psychology, 17(4):373-386, 1997.
11. Deng, J., Dong, W., Socher, R.. Li, L.-J., Li. K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In IEEE CVPR, 2009.
12. Dijkstra, E. W. On the foolishness of "natural language programming". In Program Construction. 1979.
13. Dow, S., Kulkarni, A., Bunge, B., Nguyen, T., Klemmer, S., and Hartmann, B. Shepherding the crowd: managing and providing feedback to crowd workers. In CHI, 2011.
14. Fleurbaay, E. and Eveleigh, A. Crowdsourcing: Prone to error? In International Council on Archives, 2012.
15. Gao, H., Barbier, G., and Goolsby, R. Harnessing the crowdsourcing power of social media for disaster relief. IEEE Intelligent Systems, 2011.
16. Gerber, A., Green, D., and Larimer, C. Social pressure and voter turnout: Evidence from a large-scale field experiment. American Political Science Review, 2008.
17. Gneiting, T. and Raftery, A. E. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 2007.
18. Gu, L. Crowdsourcing for complex tasks: How to ensure quality output, September 2015. http://engineering.godaddy.com/crowdsourcing-forcomplex-tasks-how-to-ensure-quality-output/.
19. Gupta. A., Thies, W., Cutrell, E., and Balakrishnan, R. mclerk: enabling mobile crowdsourcing in developing regions. In SIGCHI, 2012.
20. Haas, D., Ansel, J., Gu, L., and Marcus, A. Argonaut: macrotask crowdsourcing for complex data processing. VLDB, 2015.
21. Hara, K., Le, V., and Froehlich, J. Combining crowdsourcing and google street view to identify street-level accessibility problems. In SIGCHI, 2013.
22. Harrison, D. and Rubinfeld, D. Boston housing dataset. http://www.es.toronto.edu/-delve/data/boston/bostonDetail.html. Retrieved: October 13, 2015.
23. Harrison, D. and Rubinfeld, D. L. Hedonic housing prices and the demand for clean air. Journal of environmental economics and management, 5(1):81-102, 1978.
24. Hasson, F., Keeney, S., and McKenna, H. Research guidelines for the delphi survey technique. Journal of advanced Nursing, 32(4):1008-1015, 2000.
25. Jiang, H. and Matsubara, S. Improving crowdsourcing efficiency based on division strategy. In IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology, 2012.
26. Kahneman, D. and Frederick, S. Representativeness revisited: Attribute substitution in intuitive judgment. Heuristics and biases: The psychology of intuitive judgment, 49, 2002.
27. Karger, D. R., Oh, S., and Shah, D. Iterative learning for reliable crowdsourcing systems. In NIPS, 2011.
28. Kazai, G. In search of quality in crowdsourcing for search engine evaluation. In Advances in information retrieval, 2011.
29. Kazai, G., Kamps, J., Koolen, M., and Milic-Frayling, N. Crowdsourcing for book search evaluation: impact of HIT design on comparative system ranking. In ACM SIGIR, 2011.
30. Kerr, N. and Tindale, R. Group performance and decision making. Annual Review of Psychology, 2004.
31. Khatib et al., F. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature structural & molecular biology, 2011.
32. Kozlowski, S. W. and Ilgen, D. R. Enhancing the effectiveness of work groups and teams. Psychological science in the public interest, 7(3):77-124, 2006.
33. Krosnick, J. A. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology, 5(3):213-236, 1991.
34. Lang, A. and Rio-Ross, J. Using Amazon Mechanical Turk to transcribe historical handwritten documents. The Code4Lib Journal, 2011.
35. Lasecki, W., Miller, C, and Bigham, J. Warping time for more effective real-time crowdsourcing. In SIGCHI, 2013.
36. Levin, I., Schneider, S., and Gaeth, G. All frames are not created equal: A typology and critical analysis of framing effects. Organizational behavior and human decision processes, 1998.
37. Lichman, M. UCI digits dataset (UCI machine learning repository). http://archive. ics.uci. edu/ml/datasets/Pen-Based+Recognition+of+ Handwritten+Digits, 2013. Retrieved: October 13, 2015.
38. Liu, Q., Peng, J., and Ihler, A. Variational inference for crowdsourcing. In NIPS, 2012a.
39. Liu, X., Lu, M., Ooi, B. C, Shen, Y, Wu, S., and Zhang, M. Cdas: a crowdsourcing data analytics system. In VLDB, 2012b.
40. Miller, B. and Steyvers, M. The wisdom of crowds with communication. In Conference of the Cognitive Science Society, 2011.
41. Miller, N., Resnick, R, and Zeckhauser, R. Eliciting informative feedback: The peer-prediction method. Management Science, 51(9):1359-1373, 2005.
42. Muchnik, L., Aral, S., and Taylor, S. J. Social influence bias: A randomized experiment. Science, 2013.
43. Prelec, D. A Bayesian truth serum for subjective data. Science, 306(5695):462-466, 2004.
44. Ranade, G. and Varshney, L. To crowdsource or not to crowdsource. In HCOMP workshop at AAAl, 2012.
45. Raykar, V., Yu, S., Zhao, L., Valadez, G., Florin, C, Bogoni, L., and Moy, L. Learning from crowds. J MLR, 2010.
46. Rowe, G. and Wright, G. The delphi technique as a forecasting tool: issues and analysis. International Journal of Forecasting, 15(4):353-375, 1999.
47. Savage, L. J. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):783-801, 1971.
48. Shah, N. B. and Zhou, D. Double or nothing: Multiplicative incentive mechanisms for crowdsourcing. In NIPS, 2015.
49. Shah, N. B., Zhou, D., and Peres, Y. Approval voting and incentives in crowdsourcing. In ICML, 2015.
50. Su, H., Deng, J., and Fei-Fei, L. Crowdsourcing annotations for visual object detection. In Workshops at AAAL 2012.
51. Tesser, A., Campbell, J., and Mickler, S. The role of social pressure, attention to the stimulus, and self-doubt in conformity. European Journal of Social Psychology, 1983.
52. The New Yorker. Why smart people are stupid. http://www.newyorker. com/tech/frontal-cortex/why-smart-people-are-stupid, June 2012.
53. Tversky, A. and Kahneman, D. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124-1131, 1974.
54. Von Ahn, L., Maurer, B., McMillen, C, Abraham, D., and Blum, M. reCAPTCHA: Human-based character recognition via web security measures. Science, 2008.
55. Vuurens, J., de Vries, A. P., and Eickhoff, C. How much spam can you take? An analysis of crowdsourcing results to increase accuracy. In SIGIR Workshop on Crowdsourcing for Information Retrieval, 2011.
56. Wais, P., Lingamneni, S., Cook, D., Fennell, J., Goldenberg, B., Lubarov, D., Marin, D., and Simons, H. Towards building a high-quality workforce with Mechanical Turk. NIPS workshop on computational social science and the wisdom of crowds, 2010.
57. Zhou, D., Liu, Q., Platt, J. C, Meek, C, and Shah, N. B. Regularized minimax conditional entropy for crowdsourcing. arXiv preprint arXiv:1503.07240, 2015.