Measuring crowdsourcing effort with error-time curves

Conference on Human Factors in Computing Systems - Proceedings

Volumn 2015-April, Issue , 2015, Pages 1365-1374

  Cheng, Justin ^a   Teevan, Jaime ^b   Bernstein, Michael S ^a  

^a STANFORD UNIVERSITY   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

Crowdsourcing; Inicrotasks; Task effort

Indexed keywords

COSTS; HUMAN COMPUTER INTERACTION; HUMAN ENGINEERING;

CROWDSOURCING; DATA DRIVEN; EFFECTIVE MEASURES; INICROTASKS; MINIMUM TIME; OBJECTIVE MEASURE; TASK EFFORTS; TIME CURVES;

ERRORS;

EID: 84951038767     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2702123.2702145     Document Type: Conference Paper

References (34)

1. Japanese-English Bilingual Corpus of Wikipedia's Kyoto Articles. http://goo.gl/4SfnUi.
2. Alwin, D. F., and Krosnick, J. A. The measurement of values in surveys: A comparison of ratings and rankings. Public Opinion Quarterly (1985).
3. Anderson, J. R., Matessa, M., and Lebiere, C. ACT-R: A theory of higher level cognition and its relation to visual attention. HCI (1997).
4. Bernstein, M. S., et al. Soylent: A word processor with a crowd inside. In UIST (2010).
5. Czerwinski, M., Horvitz, E., and Cutrell, E. Subjective duration assessment: An implicit probe for software usability. In IHM-HCI (2001).
6. Dai, P., et al. Decision-theoretic control of crowd-sourced workflows. In AAAI (2010).
7. DeLeeuw, K. E., and Mayer, R. E. A comparison of three measures of cognitive load: Evidence for separable measures of intrinsic, extraneous, and germane load. J. Educ. Psychol. (2008).
8. Faradani, S., Hartmann, B., and Ipeirotis, P. G. What's the right price? pricing tasks for finishing on time. Human Computation (2011).
9. Gevins, A., and Smith, M. E. Neurophysiological measures of cognitive workload during human-computer interaction. Theor. Issues. Ergon. (2003).
10. Hart, S. G., and Staveland, L. E. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Adv. Psychol. (1988).
11. Herlocker, J., Konstan, J. A., and Riedl, J. An empirical analysis of design choices in neighborhood-based collaborative filtering algorithms. Inform. Retrieval (2002).
12. Hinds, P. J. The curse of expertise: The effects of expertise and debiasing methods on prediction of novice performance. J. Exp. Psychol.-Appl. (1999).
13. Impara, J. C., and Plake, B. S. Teachers' ability to estimate item difficulty: A test of the assumptions in the angoff standard setting method. J. Educ. Meas. (1998).
14. Ipeirotis, P. G. Mechanical Turk, low wages, and the market for lemons. http://goo.gl/uoXA6Y.
15. Ipeirotis, P. G. Analyzing the Amazon Mechanical Turk marketplace. XRDS (2010).
16. Ipeirotis, P. G., Provost, F., and Wang, J. Quality management on Amazon Mechanical Turk. In KDD (2010).
17. Iyengar, S. S., and Lepper, M. R. When choice is demotivating: Can one desire too much of a good thing? J. Pers. Soc. Psychol. (2000).
18. Joglekar, M., Garcia-Molina, H., and Parameswaran, A. Evaluating the crowd with confidence. In KDD (2013).
19. Kahneman, D. Thinking, fast and slow. 2011.
20. Kittur, A., et al. The future of crowd work. In CSCW (2013).
21. Kruger, J., and Dunning, D. Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. J. Pers. Soc. Psychol. (1999).
22. Lohman, D. F. The effect of speed-accuracy tradeoff on sex differences in mental rotation. Percept. Psychophys. (1986).
23. Mao, A., et al. Volunteering versus work for pay: Incentives and tradeoffs in crowdsourcing. In HCOMP (2013).
24. Mason, W., and Watts, D. J. Financial incentives and the performance of crowds. SigKDD Explorations (2010).
25. McCracken, J., and Aldrich, T. Analyses of selected LHX mission functions: Implications for operator workload and system automation goals. Tech. rep., 1984.
26. Pew, R. W. The speed-accuracy operating characteristic. Acta Psychol. (1969).
27. Robinson, P. Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Appl. Linguist. (2001).
28. Rzeszotarski, J. M., et al. Inserting micro-breaks into crowdsourcing workflows. In HCOMP (2013).
29. Rzeszotarski, J. M., and Kittur, A. Instrumenting the crowd: Using implicit behavioral measures to predict task performance. In UIST (2011).
30. Toomim, M., Kriplean, T., Pörtner, C., and Landay, J. Utility of human-computer interactions: Toward a science of preference measurement. In CHI (2011).
31. Welinder, P., et al. The multidimensional wisdom of crowds. In NIPS (2010).
32. Wickelgren, W. A. Speed-accuracy tradeoff and information processing dynamics. Acta Psychol. (1977).
33. Wickens, C. D. Processing resources in attention, dual task performance, and workload assessment. 1981.
34. Yao, B., et al. Human action recognition by learning bases of action attributes and parts. In ICCV (2011).