It is about Time: Time Aware Quality Management for Interactive Systems with Humans in the Loop

Conference on Human Factors in Computing Systems - Proceedings

Volumn 2013-April, Issue , 2013, Pages 163-168

  Krause, Markus ^a   Porzel, Robert ^b  

^a LEIBNIZ UNIVERSITY HANNOVER   (Germany)

^b UNIVERSITY OF BREMEN   (Germany)

Author keywords

Crowd sourcing; Human computation; Interactive Systems; Quality Management

Indexed keywords

HUMAN COMPUTER INTERACTION; HUMAN ENGINEERING;

COMPUTATIONAL OVERHEADS; DATA QUALITY; DATA RELIABILITY; HUMAN COMPUTATION; INTERACTIVE SYSTEM; INTERACTIVE WORKFLOW; INTERACTIVITY;

QUALITY MANAGEMENT;

EID: 84928035645     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2468356.2468386     Document Type: Conference Paper

References (11)

1. Bigham, J.P., Jayant, C., Ji, H., et al. VizWiz: Nearly Real-time Answers to Visual Questions. (2010).
2. Cohen, J. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 1 (1960), 37.
3. Dawid, A.P. and Skene, A.M. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics 28, (1979), 20–28.
4. Donmez, P., Carbonell, J., and Schneider, J. A Probabilistic Framework to Learn from Multiple Annotators with Time-Varying Accuracy. .
5. Ipeirotis, P.G., Provost, F., and Wang, J. Quality management on amazon mechanical turk. HComp’10 Proceedings of the ACM SIGKDD Workshop on Human Computation, ACM Press (2010), 0–3.
6. Kamar, E., Hacker, S., and Horvitz, E. Combining Human and Machine Intelligence in. Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012), (2012).
7. Krause, M., Takhtamysheva, A., Wittstock, M., and Malaka, R. Frontiers of a Paradigm - Exploring Human Computation with Digital Games. HComp’10 Proceedings of the ACM SIGKDD Workshop on Human Computation, ACM Press (2010), 4.
8. Law, E. and Ahn, L. von. Input-agreement: a new mechanism for collecting data using human computation games. CHI’09 Proceedings of the 27th international conference on Human factors in computing systems, ACM Press (2009), 1197–1206.
9. Provost, F.J. and Kohavi, R. On Applied Research in Machine Learning. Applied Research in Machine Learning 30, 1-6 (1998), 127–132.
10. Sheng, V.S., Provost, F., and Ipeirotis, P.G. Get Another Label ? Improving Data Quality and Data Mining Using Multiple Noisy Labelers. Proceedings of the 14th ACM International Conference on Knowledge Discovery and Data Mining (2008), 614-622.
11. Whitehill, J., Ruvolo, P., Wu, T., Bergsma, J., and Movellan, J. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. 1, 1–9.