Toward a learning science for complex crowdsourcing tasks

Conference on Human Factors in Computing Systems - Proceedings

Volumn , Issue , 2016, Pages 2623-2634

  Doroudi, Shayan ^a   Kamar, Ece ^b   Brunskill, Emma ^a   Horvitz, Eric ^b  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

Crowdsourcing; Education; Peer review; Web search; Worked examples; Worker training

Indexed keywords

EDUCATION; HUMAN COMPUTER INTERACTION; HUMAN ENGINEERING; INFORMATION RETRIEVAL; WEBSITES;

DOMAIN EXPERTISE; LEARNING SCIENCE; PEER REVIEW; TRAINING STRATEGY; WEB SEARCHES; WORKED EXAMPLES;

CROWDSOURCING;

EID: 85014764511     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2858036.2858268     Document Type: Conference Paper

References (31)

1. Anne Aula and Daniel M Russell. 2008. Complex and exploratory web search. In Information Seeking Support Systems Workshop (ISSS 2008), Chapel Hill, NC, USA.
2. Scott Bateman, Jaime Teevan, and Ryen W White. 2012. The search dashboard: how reflection and comparison impact search behavior. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1785-1794.
3. Michael S Bernstein, Greg Little, Robert C Miller, Björn Hartmann, Mark S Ackerman, David R Karger, David Crowell, and Katrina Panovich. 2010. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 313-322.
4. Paul Boekhout, Tamara Gog, Margje WJ Wiel, Dorien Gerards-Last, and Jacques Geraets. 2010. Example-based learning: Effects of model expertise in relation to student expertise. British Journal of Educational Psychology 80, 4(2010), 557-566.
5. Justin Cheng, Jaime Teevan, Shamsi T Iqbal, and Michael S Bernstein. 2015. Break it down: A comparison of macro- and microtasks. In Proceedings of CHI.
6. Laura B Cohen. 2001. 10 tips for teaching how to search the Web. American Libraries (2001), 44-46.
7. Mira Dontcheva, Robert R Morris, Joel R Brandt, and Elizabeth M Gerber. 2014. Combining crowdsourcing and learning to improve engagement and performance. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 3379-3388.
8. Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 1013-1022.
9. Juan M Fernández-Luna, Juan F Huete, Andrew MacFarlane, and Efthimis N Efthimiadis. 2009. Teaching and learning in information retrieval. Information Retrieval 12, 2(2009), 201-226.
10. Morgan Harvey, Claudia Hauff, and David Elsweiler. 2015. Learning by Example: training users with high-quality query suggestions. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 133-142.
11. Pamela J Hinds, Michael Patterson, and Jeffrey Pfeffer. 2001. Bothered by abstraction: the effect of expertise on knowledge transfer and subsequent novice performance. Journal of applied psychology 86, 6(2001), 1232.
12. Slava Kalyuga, Paul Chandler, Juhani Tuovinen, and John Sweller. 2001. When problem solving is superior to studying worked examples. Journal of educational psychology 93, 3(2001), 579.
13. Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 1301-1318.
14. Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E Kraut. 2011. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 43-52.
15. Avraham N Kluger and Angelo DeNisi. 1996. The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological bulletin 119, 2(1996), 254.
16. Andreas Lachner and Matthias Nückles. 2015. Bothered by abstractness or engaged by cohesion? Experts explanations enhance novices deep-learning. Journal of Experimental Psychology: Applied 21, 1(2015), 101.
17. Ard W Lazonder. 2003. Principles for designing web searching instruction. Education and Information Technologies 8, 2(2003), 179-193.
18. Wendy Lucas and Heikki Topi. 2004. Training for Web search: Will it get you in shape? Journal of the American Society for Information Science and Technology 55, 13(2004), 1183-1198.
19. Tanushree Mitra, CJ Hutto, and Eric Gilbert. 2015. Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1345-1354.
20. Neema Moraveji, Daniel Russell, Jacob Bien, and David Mease. 2011. Measuring improvement in user search performance resulting from optimal search tips. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval. ACM, 355-364.
21. Fleurie Nievelstein, Tamara Van Gog, Gijs Van Dijck, and Henny PA Boshuizen. 2013. The worked example and expertise reversal effect in less structured tasks: Learning to reason about legal cases. Contemporary Educational Psychology 38, 2(2013), 118-125.
22. David Oleson, Alexander Sorokin, Greg P Laughlin, Vaughn Hester, John Le, and Lukas Biewald. 2011. Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing. Human computation 11, 11 (2011).
23. Ron JCM Salden, Kenneth R Koedinger, Alexander Renkl, Vincent Aleven, and Bruce M McLaren. 2010. Accounting for beneficial effects of worked examples in tutored problem solving. Educational Psychology Review 22, 4(2010), 379-392.
24. Adish Singla, Ilija Bogunovic, Gabor Bartok, Amin Karbasi, and Andreas Krause. 2014. Near-Optimally Teaching the Crowd to Classify. In Proceedings of the 31st International Conference on Machine Learning (ICML-14). 154-162.
25. John Sweller and Graham A Cooper. 1985. The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction 2, 1(1985), 59-89.
26. Andrew Thatcher. 2008. Web search strategies: The influence of Web experience and task type. Information Processing & Management 44, 3(2008), 1308-1329.
27. Kurt VanLehn. 1996. Cognitive skill acquisition. Annual review of psychology 47, 1(1996), 513-539.
28. Henry M Walker and Kevin Engel. 2006. Research exercises: immersion experiences to promote information literacy. Journal of Computing Sciences in Colleges 21, 4(2006), 61-68.
29. Wesley Willett, Jeffrey Heer, and Maneesh Agrawala. 2012. Strategies for crowdsourcing social data analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 227-236.
30. Haoqi Zhang, Edith Law, Rob Miller, Krzysztof Gajos, David Parkes, and Eric Horvitz. 2012. Human computation tasks with global constraints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 217-226.
31. Haiyi Zhu, Steven P Dow, Robert E Kraut, and Aniket Kittur. 2014. Reviewing versus doing: Learning and performance in crowd assessment. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 1445-1455.