Toward crowdsourcing micro-level behavior annotations: The challenges of interface, training, and generalization

International Conference on Intelligent User Interfaces, Proceedings IUI

Volumn , Issue , 2014, Pages 37-46

  Park, Sunghyun ^a   Shoemark, Philippa ^a   Morency, Louis Philippe ^a  

^a University of Southern California ^*  (United States)

Author keywords

Behavior annotations; Crowdsourcing; Inter rater reliability; Micro level annotations; Training crowd workers

Indexed keywords

BEHAVIOR ANNOTATIONS; CROWDSOURCING; CROWDSOURCING PLATFORMS; HUMAN BEHAVIOR ANALYSIS; INTER-RATER RELIABILITIES; MICRO-LEVEL ANNOTATIONS; OBSERVATIONAL STUDY; RELIABILITY IMPROVEMENT;

E-LEARNING; TOOLS;

BEHAVIORAL RESEARCH;

EID: 84897748991     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2557500.2557512     Document Type: Conference Paper

References (32)

1. Allwood, J., Cerrato, L., Dybkjaer, L., Jokinen, K., Navarretta, C., and Paggio, P. The MUMIN multimodal coding scheme. Proc. Workshop on Multimodal Corpora and Annotation 2004.
2. Bernstein, M., Brandt, J., Miller, R., and Karger, D. Crowds in two seconds: Enabling realtime crowd-powered interfaces. Proc. UIST 2011, 33-42.
3. Biel, J. I. and Gatica-Perez, D. The good, the bad, and the angry: Analyzing crowdsourced impressions of vloggers. Proc. ICWSM 2012, 407-410.
4. Dasiopoulou, S., Giannakidou, E., Litos, G., Malasioti, P., and Kompatsiaris, Y. A survey of semantic image and video annotation tools. Knowledge-Driven Multimedia Information Extraction and Ontology Evolution, Lecture Notes in Computer Science 6050 (2011), 196-239.
5. Downs, J. S., Holbrook, M. B., Sheng, S., and Cranor, L. F. Are your participants gaming the system?: Screening mechanical turk workers. Proc. CHI 2010, 2399-2402.
6. Gao, Q. and Vogel, S. Consensus versus expertise: A case study of word alignment with Mechanical Turk. Proc. CSLDAMT 2010, 30-34.
7. Hsueh, P. Y., Melville, P., and Sindhwani, V. Data quality from crowdsourcing: A study of annotation selection criteria. Proc. ALLNP 2009, 27-35.
8. Kang, S., Gratch, J., Sidner, C., Artstein, R., Huang, L., Morency, L. Towards building a virtual counselor: Modeling nonverbal behavior during intimate self-disclosure. Proc. AAMAS 2012, 63-70.
9. Kim, J., Nguyen, P., Weir, S., Guo, P., Miller, R., and Gajos, K. Crowdsourcing step-by-step information extraction to enhance existing how-to videos. Proc. CHI 2014.
10. Krippendorff, K. Content Analysis: An Introduction to Its Methodology. Sage, Beverly Hills, CA, 2004.
11. Krippendorff, K. On the reliability of unitizing contiguous data. Sociological Methodology 25 (1995), 47-76.
12. Le, J., Edmonds, A., Hester, V., and Biewald, L. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. Proc. SIGIR CSE 2010, 21-26.
13. Lew, M. S., Sebe, N., Djeraba, C., and Jain, R. Content-based multimedia information retrieval: State of the art and challenges. ACM T. Multim. Comput. 2, 1 (2006), 1-19.
14. Marge, M., Banerjee, S., and Rudnicky, A. I. Using the Amazon Mechanical Turk for transcription of spoken language. Proc. ICASSP 2010, 5270-5273.
15. Mason, W. and Suri, S. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 44, 1 (2012), 1-23.
16. McKeown, G., Valstar, M.F., Cowie, R., and Pantic, M. The SEMAINE corpus of emotionally coloured character interactions. Proc. ICME 2010, 1079-1084.
17. Novotney, S. and Callison-Burch, C. Cheap, fast and good enough: Automatic speech recognition with non-expert transcription. Proc. HLT 2010, 207-215.
18. Nowak, S. and Ruger, S. How reliable are annotations via crowdsourcing: A study about inter-annotator agreement for multi-label image annotation. Proc. MIR 2010, 557-566.
19. Pantic, M., Pentland, A., Nijholt, A., and Huang, T. Human computing and machine understanding of human behavior: A survey. Proc. ICMI 2006, 239-248.
20. Park, S., Mohammadi, G., Artstein, R., and Morency, L.-P. Crowdsourcing micro-level multimedia annotations: The challenge of evaluation and interface. Proc. CrowdMM 2012, 29-34.
21. Poppe, R. A survey on vision-based human action recognition. Image Vision Comput. 28, 6 (2010), 976-990.
22. Quinn, A. J. and Bederson, B. B. Human computation: A survey and taxonomy of a growing field. Proc. CHI 2011, 1403-1412.
23. Rashtchian, C., Young, P., Hodosh, M., and Hockenmaier, J. Collecting image annotations using Amazon's Mechanical Turk. Proc. CSLDAMT 2010, 139-147.
24. Riek, L., O'Connor, M., and Robinson, P. Guess what? A game for affective annotation of video using crowd sourcing. Proc. ACII 2011, 277-285.
25. Ross, J., Irani, L., Silberman, M., Zaldivar, A., and Tomlinson, B. Who are the crowdworkers?: Shifting demographics in Mechanical Turk. Proc. Ext. Abstracts CHI 2010, 2863-2872.
26. Sheng, V. S., Provost, F., and Ipeirotis, P. G. Get another label? Improving data quality and data mining using multiple, noisy labelers. Proc. KDD 2008, 614-622.
27. Snow, R., O'Connor, B., Jurafsky, D. and Ng, A. Y. Cheap and fast but is it good?: Evaluating non-expert annotations for natural language tasks. Proc. EMNLP 2008, 254-263.
28. Spiro, I., Taylor, G., Williams, G., and Bregler, C. Hands by hand: Crowd-sourced motion tracking for gesture annotation. Proc. CVPRW 2010, 17-24.
29. Vondrick, C., Ramanan, D., and Patterson, D. Efficiently scaling up video annotation with crowdsourced marketplaces. Computer Vision - ECCV 2010 6314, 610-623.
30. Wu, S. Y., Thawonmas, R., and Chen, K. T. Video summarization via crowdsourcing. Proc. Ext. Abstracts CHI 2011, 1531-1536.
31. Yuen, M. C., King, I., and Leung, K. S. A survey of crowdsourcing systems. Proc. SocialComp 2011, 766-773.
32. Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., and Horvitz, E. Human computation tasks with global constraints. Proc. CHI 2012, 217-226