Exploring crowd consistency in a mechanical Turk survey

Proceedings - 3rd International Workshop on CrowdSourcing in Software Engineering, CSI-SE 2016

Volumn , Issue , 2016, Pages 8-14

  Sun, Peng ^a   Stolee, Kathryn T ^a  

^a North Carolina State University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CROWDSOURCING; SOFTWARE ENGINEERING; SURVEYS;

AMAZON'S MECHANICAL TURKS; EVALUATING SOFTWARE; MECHANICAL TURKS; WORKERS';

COMPUTER SOFTWARE;

EID: 84973497272     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2897659.2897662     Document Type: Conference Paper

References (21)

1. R. A. Cochran, L. D'Antoni, B. Livshits, D. Molnar, and M. Veanes. Program boosting: Program synthesis via crowd-sourcing. In Proceedings of the 42Nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL '15, pages 677-688, New York, NY, USA, 2015. ACM.
2. E. Dolstra, R. Vliegendhart, and J. Pouwelse. Crowdsourcing gui tests. In Software Testing, Verification and Validation (ICST), 2013 IEEE Sixth International Conference on, pages 332-341, March 2013.
3. J. S. Downs, M. B. Holbrook, S. Sheng, and L. F. Cranor. Are your participants gaming the system?: screening Mechanical Turk workers. In Proceedings of the 28th international conference on Human factors in computing systems, 2010.
4. A. Finnerty, P. Kucherbaev, S. Tranquillini, and G. Convertino. Keep it simple: Reward and task design in crowdsourcing. In Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI, CHItaly '13, pages 14:1-14:4, New York, NY, USA, 2013. ACM.
5. Z. P. Fry, B. Landau, and W. Weimer. A human study of patch maintainability. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, ISSTA 2012, pages 177-187, New York, NY, USA, 2012. ACM.
6. Z. P. Fry and W. Weimer. A human study of fault localization accuracy. In Proceedings of the 2010 IEEE International Conference on Software Maintenance, ICSM '10, pages 1-10, Washington, DC, USA, 2010. IEEE Computer Society.
7. O. S. GÃşmez, N. Juristo, and S. Vegas. Understanding replication of experiments in software engineering: A classification. Information and Software Technology, 56(8):1033-1048, 2014.
8. S.-H. Kim, H. Yun, and J. S. Yi. How to filter out random clickers in a crowdsourcing-based study? In Proceedings of the 2012 BELIV Workshop: Beyond Time and Errors-Novel Evaluation Methods for Visualization, BELIV '12, pages 15:1-15:7, New York, NY, USA, 2012. ACM.
9. A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08, pages 453-456, New York, NY, USA, 2008. ACM.
10. T. D. LaToza, W. B. Towne, C. M. Adriano, and A. van der Hoek. Microtask programming: Building software with a crowd. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST '14, pages 43-54, New York, NY, USA, 2014. ACM.
11. W. Mason and D. J. Watts. Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter, 11(2):100-108, 2010.
12. M. Nebeling, M. Speicher, and M. C. Norrie. Crowdstudy: General toolkit for crowdsourced evaluation of web interfaces. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS '13, pages 255-264, New York, NY, USA, 2013. ACM.
13. G. Paolacci, J. Chandler, and P. G. Ipeirotis. Running experiments on amazon mechanical turk. Judgment and Decision making, 5(5):411-419, 2010.
14. J. Rogstadius, V. Kostakos, A. Kittur, B. Smus, J. Laredo, and M. Vukovic. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In ICWSM, 2011.
15. J. Ross, L. Irani, M. S. Silberman, A. Zaldivar, and B. Tomlinson. Who are the crowdworkers?: shifting demographics in Mechanical Turk. In Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems, 2010.
16. T. W. Schiller and M. D. Ernst. Reducing the barriers to writing verified specifications. SIGPLAN Not., 47(10):95-112, Oct. 2012.
17. N. Stewart, C. Ungemach, A. J. Harris, D. M. Bartels, B. R. Newell, G. Paolacci, J. Chandler, et al. The average laboratory samples a population of 7,300 amazon mechanical turk workers. Judgment and Decision Making, 10(5):479-491, 2015.
18. K. T. Stolee and S. Elbaum. Exploring the use of crowdsourcing to support empirical studies in software engineering. In International Symposium on Empirical Software Engineering and Measurement, 2010.
19. K. T. Stolee and S. Elbaum. Refactoring pipe-like mashups for end-user programmers. In International Conference on Software Engineering, 2011.
20. K. T. Stolee, S. Elbaum, and D. Dobos. Solving the search for source code. ACM Trans. Softw. Eng. Methodol., 23(3):26:1-26:45, June 2014.
21. M. Yin, Y. Chen, and Y.-A. Sun. Monetary interventions in crowdsourcing task switching. In Second AAAI Conference on Human Computation and Crowdsourcing, 2014.