And now for something completely different: Improving crowdsourcing workflows with micro-diversions

CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing

Volumn , Issue , 2015, Pages 628-638

  Dai, Peng ^a   Rzeszotarski, Jeffrey M ^b   Paritosh, Praveen ^a   Chi, Ed H ^a  

^a GOOGLE INC   (United States)

^b Carnegie Mellon University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER SUPPORTED COOPERATIVE WORK; CROWDSOURCING; INTERACTIVE COMPUTER SYSTEMS; NATURAL LANGUAGE PROCESSING SYSTEMS; PROBLEM SOLVING; PRODUCTIVITY;

AMAZON'S MECHANICAL TURKS; COMPUTATIONAL RESOURCES; CROWDSOURCING PLATFORMS; IMAGE LABELING; LONG SEQUENCES; NATURAL LANGUAGE PROCESSING; QUALITY OF WORK; RETENTION RATE;

GROUPWARE;

EID: 84968909143     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2675133.2675260     Document Type: Conference Paper

References (43)

1. Piotr D. Adamczyk and Brian P. Bailey. If not now, when?: The effects of interruption at different moments within task execution. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 271-278, 2004.
2. James Bo Begole and John C. Tang. Incorporating human and machine interpretation of unavailability and rhythm awareness into the design of collaborative applications. Human-Computer Interaction, 22(1-2):7-45, 2007.
3. Michael S. Bernstein, Greg Little, Robert C. Miller, Bjorn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. Soylent: A word processor with a crowd inside. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology, UIST, 2010.
4. Kurk Bollacker, Colin Evans, Praveen Paritosh, Time Sturge Tim, and Jamie Taylor. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, pages 1247-1250, 2008.
5. Chris Callison-Burch. Fast, cheap, and creative: Evaluating translation quality using amazon's mechanical turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 286-295, 2009.
6. Jonathan S. A. Carriere, J. Allan Cheyne, and Daniel Smilek. Everyday attention lapses and memory failures: The affective consequences of mindlessness. Consciousness and Cognition, 17:835-847, 2008.
7. Jonathan S. A. Carriere, J. Allan Cheyne, and Daniel Smilek. Boredom: An emotional experience distinct from apathy, anhedonia, or depression. Social and Clinical Psychology, 30(6):647-666, 2011.
8. Dana Chandler and Adam Kapelner. Breaking monotony with meaning: Motivation in crowdsourcing markets. CoRR, abs/1210.0962, 2012.
9. Lydia B. Chilton, John J. Horton, Robert C. Miller, and Shiri SAzenkot. Task search in a human computation market. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP '10, pages 1-9, 2010.
10. Mihaly Csikszentmihalyi. Creativity: Flow and the Psychology of Discovery and Invention. HarperCollins, 1997.
11. Mihaly Csikszentmihalyi. Beyond Boredom and Anxiety: Experiencing Flow in Work and Play. Jossey-Bass, 2000.
12. Edward Cutrell, Mary Czerwinski, and Eric Horvitz. Notification, disruption, and memory: Effects of messaging interruptions on memory and performance. In INTERACT, pages 263-269, 2001.
13. Mary Czerwinski, Edward Cutrell, and Eric Horvitz. Instant messaging and interruption: Inuence of task type on performance, 2000.
14. Peng Dai, Christopher H. Lin, Mausam, and Daniel S. Weld. POMDP-based control of workows for crowdsourcing. Artificial Intelligence, 202:52-85, 2013.
15. Peng Dai, Mausam, and Daniel S. Weld. Decision-Theoretic control of crowd-sourced workows. In AAAI, 2010.
16. Peng Dai, Mausam, and Daniel S. Weld. Artificial intelligence for artificial artificial intelligence. In AAAI, 2011.
17. Pinar Donmez, Jaime G. Carbonell, and Je G. Schneider. A probabilistic framework to learn from multiple annotators with time-varying accuracy. In SDM, pages 826-837, 2010.
18. Robert A. Henning, Steven L Sauter, Gavriel Salvendy, and Edward F. Krieg. Microbreak length, performance, and stress in a data entry task. Ergonomics, 32(7):855-864, 1989.
19. John Joseph Horton. Employer expectations, peer effects and productivity: Evidence from a series of field experiments. CoRR, abs/1008.2437, 2010.
20. John Joseph Horton and Lydia B. Chilton. The labor economics of paid crowdsourcing. CoRR, 2010.
21. Eric Huang, Haoqi Zhang, David C. Parkes, Krzysztof Z. Gajos, and Yiling Chen. Toward automatic task design: A progress report. In Proceedings of the ACM SIGKDD Workshop on Human Computation, pages 77-85, 2010.
22. Shamsi T. Iqbal and Brian P. Bailey. Effects of intelligent notification management on users and their tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08, pages 93-102, 2008.
23. Shamsi T. Iqbal and Eric Horvitz. Disruption and recovery of computing tasks: Field study, analysis, and directions. In CHI, pages 677-686, 2007.
24. Daniel Kahneman. Thinking, Fast and Slow. Macmillan, 2011.
25. Aniket Kittur, Ed H. Chi, and Bongwon Suh. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 453-456, 2008.
26. Aniket Kittur, Susheel Khamkar, Paul Andre, and Robert Kraut. Crowdweaver: Visually managing complex crowd work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW, pages 1033-1036, 2012.
27. Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pages 43-52, 2011.
28. Aniket Kittur, Bongwon Suh, Bryan A. Pendleton, and Ed H. Chi. He says, she says: Conict and coordination in wikipedia. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI, pages 453-462, 2007.
29. Shailesh Kochhar, Stefano Mazzocchi, and Praveen Paritosh. The anatomy of a large-scale human computation engine. In Proceedings of the ACM SIGKDD Workshop on Human Computation, pages 10-17, 2010.
30. John G. Kreifeldt and M. E. Mccarthy. Interruption as a test of the user-computer interface. In Proceedings of the 17th Annual Conference on Manual Control, pages 655-667, 1981.
31. Gerald P. Krueger. Sustained work, fatigue, sleep loss and performance: A review of the issues. Work & Stress, 3:129-141, 1989.
32. Chris J. Lintott, Kevin Schawinski, Kate L, Daniel Thomas, M. Jordan Raddick, Robert C. Nichol, Dan Andreescu, Phil Murray, and Jan Van Den Berg. Galaxy zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey, " monthly notices of the royal. Astronomical Society, 2008.
33. Andrew Mao, Ece Kamar, and Eric Horvitz. Why stop now? predicting worker engagement in online crowdsourcing. In HCOMP, 2013.
34. Gloria Mark, Daniela Gudith, and Ulrich Klocke. The cost of interrupted work: More speed and stress. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08, pages 107-110, 2008.
35. Nathalie Pattyn, Xavier Neyt, David Henderickx, and Eric Soetens. Psychophysiological investigation of vigilance decrement: Boredom or cognitive fatigue? Physiology & Behavior, 93:369-378, 2008.
36. Layne P. Perelli. Fatigue Stressors in Simulated Long-duration Flight: Effects on Performance, Information Processing, Subjective Fatigue, and Physiological Cost. USAF School of Aerospace Medicine, Aerospace Medical Division (AFSC), 1980.
37. George Robertson, Eric Horvitz, Mary Czerwinski, Patrick Baudisch, Dugald R. Hutchings, Brain Meyers, Daniel Robbins, and Greg Smith. Scalable fabric: Flexible task management. In Proceedings of the Working Conference on Advanced Visual Interfaces, pages 85-89, 2004.
38. Jeffrey M. Rzeszotarski and Aniket Kittur. Instrumenting the crowd: Using implicit behavioral measures to predict task performance. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST, pages 13-22, 2011.
39. Cheri Speier, Joseph S. Valacich, and Iris Vessey. The effects of task interruption and information presentation on individual decision making. In ICIS, pages 21-36, 1997.
40. Digger U. Verson, 2012. www.diggercomic.com.
41. Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier R. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, pages 2035-2043, 2009.
42. Glenn Wylie and Alan Allport. Task switching and the measurement of switch costs". Psychological Research, 63:212-233, 2000.
43. Lixiu Yu and Jeffrey V. Nickerson. Cooks or cobblers?: Crowd creativity through combination. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1393-1402, 2011.