Finding a Comparison Group: Is Online Crowdsourcing a Viable Option?

American Journal of Evaluation

Volumn 34, Issue 3, 2013, Pages 372-384

  Azzam, Tarek ^a   Jacobson, Miriam R ^a  

^a CLAREMONT GRADUATE UNIVERSITY   (United States)

Author keywords

crowdsourcing; evaluation methods; experimental design; matched comparison group; quasi experimental design

Indexed keywords

EID: 84880674389     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214013490223     Document Type: Article

References (28)

1. Azzam T.,Christie C.Using public databases to study relative program impact.Canadian Journal of Program Evaluation. 2006;22:57-68.
2. Behrend T. S.,Sharek D. J.,Meade A. W.,Wiebe E. N.The viability of crowdsourcing for survey research.Behavior Research Methods. 2011;43:800-813.
3. Berinsky A. J.,Huber G. A.,Lenz G. S.Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk.Political Analysis. 2012;20:351-368.
4. Bloom H. S.,Michalopoulos C.,Carolyn J. H.Learning more from social experiments. Bloom H. S., ed. New York, NY: Russel Sage Foundation; 2005:173-235.
5. Buhrmester M. D.,Kwang T.,Gosling S. D.Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data?.Perspectives on Psychological Science. 2011;6:3-5.
6. Chandler D.,Kapelner A.Breaking monotony with meaning: Motivation in crowdsourcing markets. Chicago, IL: University of Chicago, Becker Center on Chicago Price Theory; 2010:.
7. ChatmanS. P. (2007). A common factor solution for UCEUS upper-division Core items(JAD.5.11.07). Berkeley, CA: Center for Studies in Higher Education. Retrieved fromhttp://cshe.berkeley.edu/publications/docs/Chatman.TechReport.10.29.09.pdf.
8. Crano W.,Brewer M.,Lac A.Principles and methods of social research. 2001). Principles and methods of social research (2nd ed.). New York, NY: Psychology Press.New York, NY: Psychology Press; 2001:.
9. Cook T. D.,Shadish W. R.,Wong V. C.Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons.Journal of Policy Analysis and Management. 2008;27:724-750.
10. Dehejia R. H.,Wahba S.Causal effect in non-experimental studies: Reevaluating the evaluation of training programs.Journal of American Statistical Association. 1999;94:1053-1062.
11. FriessS. (2007, November). Online fossett searchers ask, was it worth it?Wired Magazine. Retrieved fromhttp://www.wired.com/techbiz/it/news/2007/11/fossett_search.
12. Gottfried A. E.Children's academic intrinsic motivation inventory. Lutz, FL: Psychological Assessment Resources; 1986:.
13. Guo S.,Fraser M. W.Propensity score analysis: Statistical methods and applications. Thousand Oaks, CA: Sage; 2010:.
14. Horton J.,Chilton L.Proceedings of the 11th ACM conference on electronic commerce. New York, NY: ACM; 2010:209-218.
15. Horton J.,Rand D.,Zeckhauser R.The online laboratory: Conducting experiments in a real labor market.Experimental Economics. 2010;4:399-442.
16. HoweJ. (2006, June). The rise of crowdsourcing. Wired Magazine. Retrieved fromhttp://www.wired.com/wired/archive/14.06/crowds.html.
17. Kraut R.,Olson J.,Banaji M.,Bruckman A.,Cohen J.,Couper M.Psychological research online: Report of Board of Scientific Affairs' Advisory Group on the Conduct of Research on the Internet.American Psychologist. 2004;59:105.
18. Kuh G. D.The national survey of student engagement.New Directions for Institutional Research. 2009;141:5-20.
19. Marsh H. W.Simulation study of non-equivalent group-matching and regression discontinuity designs: Evaluation of gifted and talented programs.Journal of Experimental Education. 1998;66:163-192.
20. Paolacci G.,Chandler J.,Ipeirotis P. I.Running experiments on Amazon mechanical turk.Judgment and Decision Making. 2010;5:411-419.
21. Rosenbaum P.,Rubin D.The central role of the propensity score in observational studies for causal effects.Biometrika. 1983;70:41-55.
22. Ross J.,Irani L.,Silberman M. S.,Zaldivar A.,Tomlinson B.CHI EA '10: Proceedings of the 28th of the international conference extended abstracts on human factors in computing systems. New York, NY: ACM; 2010:2863-2872.
23. Scriven M.Contemporary thinking about causation in evaluation: A dialogue with Tom Cook and Michael Scriven.American Journal of Evaluation. 2010;31:105-117.
24. Shadish W.,Cook T.,Campbell D.Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin; 2002:.
25. Shadish W. R.,Clark M. H.,Steiner P. M.Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments.Journal of the American Statistical Association. 2008;103:1334-1344.
26. Smith H.Matching with multiple controls to estimate treatment effects in observational studies.Sociological Methodology. 1997;27:325-353.
27. Stuart E. A.Matching methods for causal inference: A review and a look forward.Statistical Science. 2010;25:1-21.
28. Winchester S.The meaning of everything: The story of the Oxford English Dictionary. New York, NY: Oxford University Press; 2003:.