Crowdsourcing for quantifying transcripts: An exploratory study

Evaluation and Program Planning

Volumn 54, Issue , 2016, Pages 63-73

  Azzam, Tarek ^a   Harman, Elena ^a  

^a CLAREMONT GRADUATE UNIVERSITY   (United States)

Author keywords

Crowdsourcing; Mechanical Turk; MTurk; Qualitative analysis; Stability; Transcript coding; Transcript rating

Indexed keywords

COMPUTER SIMULATION; NUMERICAL MODEL; QUALITATIVE ANALYSIS;

ADULT; CROWDSOURCING; FEMALE; HUMAN; INTERNET; INTERVIEW; MALE; MEASUREMENT ACCURACY; METHODOLOGY; PROGRAM EVALUATION; QUALITATIVE RESEARCH; SOCIOECONOMICS; STANDARDS;

ADULT; CROWDSOURCING; DATA ACCURACY; FEMALE; HUMANS; INTERNET; INTERVIEWS AS TOPIC; MALE; PROGRAM EVALUATION; QUALITATIVE RESEARCH; RESEARCH DESIGN; SOCIOECONOMIC FACTORS;

EID: 84945308805     PISSN: 01497189     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.evalprogplan.2015.09.002     Document Type: Article

References (16)

1. Behrend T.S., Sharek D.J., Meade A.W., Wiebe E.N. The viability of crowdsourcing for survey research. Behavior Research Methods 2011, 43(3):800-813.
2. Berinsky A.J., Huber G.A., Lenz G.S. Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis 2012, 20(3):351-368.
3. Buhrmester M.D., Kwang T., Gosling S.D. Amazon's mechanical Turk: A new source of inexpensive, yet high-quality, data?. Perspectives on Psychological Science 2011, 6(1):3-5.
4. Chandler D., Kapelner A. Breaking monotony with meaning: Motivation in crowdsourcing markets 2010, University of Chicago, Becker Center on Chicago Price Theory, Chicago, IL.
5. Greene J. Mixed methods in social inquiry 2007, Jossey-Bass, San Francisco.
6. Hayes A.F., Krippendorff K. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures 2007, 1(1):77-89.
7. Horton J., Rand D., Zeckhauser R. The online laboratory: Conducting experiments in a real labor market. Experimental Economics 2010, 4:399-442.
8. Mason W., Suri S. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 2012, 44(1):1-23.
9. Nespor J. Anonymity and place in qualitative inquiry. Qualitative Inquiry 2000, 6(4):546-569.
10. Paolacci G., Chandler J., Ipeirotis P. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making 2010, 5(5):411-419.
11. Harman, E., & Azzam, T. (Revise & Resubmit). Validating program theories: A new role for qualitative data using crowdsourcing.
12. Ross J., Irani L., Silberman M.S., Zaldivar A., Tomlinson B. Who are the crowdworkers? Shifting demographics in Mechanical Turk. CHI EA'10: Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems 2010, 2863-2872. ACM, New York, NY.
13. Saal F.E., Downey R.G., Lahey M.A. Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin 1980, 88(2):413-428.
14. Sandelowski M., Voils C.I., Knafl G. On quantitizing. Journal of Mixed Methods Research 2009, 3(3):208-222.
15. Surowiecki J. The wisdom of crowds 2004, Anchor Books, New York, NY.
16. Teddlie C., Tashakkori A. A general typology in research designs featuring mixed methods. Research in the Schools 2006, 13:12-28.