Comparing person- And process-centric strategies for obtaining quality data on amazon mechanical turk

Conference on Human Factors in Computing Systems - Proceedings

Volumn 2015-April, Issue , 2015, Pages 1345-1354

  Mitra, Tanushree ^a   Hutto, C J ^a   Gilbert, Eric ^a  

^a GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Crowd sourcing; Experimentation; Human computation; Mechanical turk; Micro task; Qualitative coding

Indexed keywords

HUMAN COMPUTER INTERACTION; HUMAN ENGINEERING;

CROWD SOURCING; EXPERIMENTATION; HUMAN COMPUTATION; MECHANICAL TURKS; MICROTASKS; QUALITATIVE CODING;

QUALITY CONTROL;

EID: 84951200321     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2702123.2702553     Document Type: Conference Paper

References (45)

1. Andre, P., Kittur, A., and Dow, S.P. Crowd synthesis: Extracting categories and clusters from complex data. CSCW 2014, 989-998.
2. Berinsky, A.J., Huber, G.A., and Lenz, G.S. Evaluating Online Labor Markets for Experimental Research: Amazon. com's Mechanical Turk. Political Analysis 20, 3 2012, 351-368.
3. Blei, D., Ng, A., and Jordan, M. Latent dirichlet allocation. J. of Machine Learning Res.3, 2003, 993-1022.
4. Castillo, C., Mendoza, M., and Poblete, B. Information credibility on twitter. CHI 2011, 675-684.
5. Chang, J., Boyd-Graber, J., Gerrish, S., Wang, C., and Blei, D. Reading Tea Leaves: How Humans Interpret Topic Models. NIPS 2009, 288-296.
6. Crump, M.J.C., McDonnell, J.V., and Gureckis, T.M. Evaluating Amazon's Mechanical Turk as a Tool for Experimental Beh. Res. PLoS ONE 8, 3 2013.
7. Dalal, N. and Triggs, B. Histograms of oriented gradients for human detection. CVPR 2005, 886-893.
8. Downs, J.S., Holbrook, M.B., Sheng, S., and Cranor, L.F. Are your participants gaming the system?: Screening mechanical turk workers. CHI 2010, 2399-2402.
9. Gilbert, E. What if we ask a different question?: Social inferences create product ratings faster. CHI 2014.
10. Hart, S.G. and Staveland, L.E. Development of NASATLX (Task Load Index): Results of empirical and theoretical research. Adv. in psych. 52, 1988, 139-183.
11. Heer, J. and Bostock, M. Crowdsourcing graphical perception: Using mechanical turk to assess visualization design. CHI 2010, 203-212.
12. Huang, S.-W. and Fu, W.-T. Don't hide in the crowd!: Increasing social transparency between peer workers improves crowdsourcing outcomes. CHI 2013.
13. Hutto, C.J. and Gilbert, E. VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. ICWSM 2014, 216-255.
14. Ipeirotis, P.G. Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads17, 2 2010, 16-21.
15. Ipeirotis, P.G., Provost, F., and Wang, J. Quality management on Amazon Mechanical Turk. HCOMP 2010.
16. Kazai, G., Kamps, J., and Milic-Frayling, N. Worker types and personality traits in crowdsourcing relevance labels. CIKM 2011, 1941-1944.
17. Kittur, A., Chi, E.H., and Suh, B. Crowdsourcing user studies with Mechanical Turk. CHI 2008, 453-456.
18. Kriplean, T., Bonnar, C., Borning, A., Kinney, B., and Gill, B. Integrating on-demand fact-checking with public dialogue. CSCW 2014, 1188-1199.
19. Lasecki, W.S., Gordon, M., Koutra, D., Jung, M.F., Dow, S.P., and Bigham, J.P. Glance: rapidly coding behavioral video with the crowd. UIST 2014, 551-562.
20. Lau, J.H., Collier, N., and Baldwin, T. On-line Trend Analysis with Topic Models: #twitter Trends Detection Topic Model Online. COLING 2012, 1519-1534.
21. Little, G., Chilton, L.B., Goldman, M., and Miller, R.C. Exploring iterative and parallel human computation processes. CHI 2010, 68-76.
22. MacDonald, P.L. and Gardner, R.C. Type I error rate comparisons of post hoc procedures for I j Chi-Square tables. Ed. and Psych. Meas. 60, 5 2000, 735-754.
23. Mason, W. and Suri, S. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods 44, 1 2012, 1-23.
24. Mason, W. and Watts, D.J. Financial incentives and the performance of crowds. ACM SIGKDD Explorations Newsletter 11, 2 (2010), 100-108.
25. Morris, M.R., Counts, S., Roseway, A., Hoff, A., and Schwarz, J. Tweeting is believing?: Understanding microblog credibility perceptions. CSCW 2012, 441-450.
26. Naaman, M., Boase, J., and Lai, C.-H. Is it really about me?: Message content in social awareness streams. CSCW 2010, 189-192.
27. Paolacci, G., Chandler, J., and Ipeirotis, P.G. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making 5, 5 2010, 411-419.
28. Papageorgiou, C. and Poggio, T. A trainable system for object detection. Int. J. of Computer Vision 38, 1 2000.
29. Peer, E., Vosgerau, J., and Acquisti, A. Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 2013, 1-9.
30. Prelec, D. A Bayesian truth serum for subjective data. Science 306, 5695, 2004, 462-466.
31. Qazvinian, V., Rosengren, E., Radev, D.R., and Mei, Q. Rumor has it: Identifying misinformation in microblogs. EMNLP 2011, 1589-1599.
32. Rashtchian, C., Young, P., Hodosh, M., and Hockenmaier, J. Collecting image annotations using Amazon's Mechanical Turk. In Proc. NAACL HLT Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, 139-147, 2010.
33. Rosenthal, R. and Rubin, D.B. Multiple contrasts and ordered Bonferroni procedures. J. Ed. Psy. 76, 6 1984.
34. Saldana, J. The Coding Manual for Qualitative Researchers. SAGE Publications, 2009.
35. Shaw, A.D., Horton, J.J., and Chen, D.L. Designing incentives for inexpert human raters. CHI 2011.
36. Sheng, V.S., Provost, F., and Ipeirotis, P.G. Get another label? improving data quality and data mining using multiple, noisy labelers. KDD 2008, 614-622.
37. Sheshadri, A. and Lease, M. SQUARE: A Benchmark for Research on Computing Crowd Consensus. HCOMP 2013, 156-164.
38. Snow, R., O'Connor, B., Jurafsky, D., and Ng, A.Y. Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. EMNLP 2008.
39. Soni, S., Mitra, T., Gilbert, E., and Eisenstein, J. Modeling Factuality Judgments in Social Media Text. In Proc. ACL, 415-420, 2014.
40. Sorokin, A. and Forsyth, D. Utility data annotation with Amazon Mechanical Turk. CVPRW 2008, 1-8.
41. Sun, Y.-A., Roy, S., and Little, G. Beyond Independent Agreement: A Tournament Selection Approach for Quality Assurance of Human Computation Tasks. Human Computation, 2011.
42. Surowiecki, J. The Wisdom of Crowds. Anchor Books, New York, NY, 2004.
43. Wang, Y.-C., Kraut, R., and Levine, J.M. To stay or leave?: The relationship of emotional and informational support to commitment in online health support groups. CSCW 2012, 833-842.
44. Willett, W., Heer, J., and Agrawala, M. Strategies for crowdsourcing social data analysis. CHI 2012.
45. Zhao, W.X., Jiang, J., Weng, J., et al. Comparing twitter and traditional media using topic models. Advances in Information Retrieval. Springer, 2011, 338-349.