Challenges and dilemmas in implementing random assignment in educational research

American Journal of Evaluation

Volumn 32, Issue 1, 2011, Pages 29-49

  Ong Dean, Colin ^a   Hofstetter, Carolyn Huie ^a   Strick, Betsy R ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

external validity; literacy education; random assignment; RCTs; research methods

Indexed keywords

EID: 79551618140     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214010376532     Document Type: Article

References (26)

1. American Evaluation Association. (2003). American evaluation association response to U.S. Department of Education notice of proposed priority. Federal Register, RIN 1890-ZA00, November 4, 2003.
2. Berliner, D.C. (2002). Educational research: The hardest science of all. Educational Researcher, 31, 18-20.
3. Bloom, H.S. (1984). Accounting for no-shows in experimental evaluation designs. Evaluation Review, 8, 225-246.
4. Bloom, H.S., Bos, J.M., & Lee, S.W. (1999). Using cluster random assignment to measure program impacts- Statistical implications for the evaluation of education programs. Evaluation Review, 23, 445-469.
5. Burtless, G. (2002). Randomized field trials for policy evaluation: Why not in education? In F. Mosteller & R. Boruch (Eds.), Evidence matters: Randomized trials in educational research (pp. 179-197). Washington, DC: Brookings Institution Press.
6. Campbell, D.T., & Stanley, J.C. (1963). Experimental and quasi-experimental designs for research. Boston, MA: Houghton Mifflin.
7. Cartwright, N. (2007). Are RCTs the gold standard? Biosocieties, 2, 11-20.
8. Cook, T.D. (1993). A quasi-sampling theory of the generalization of causal relationships. In L. Sechrest & A. G. Scott (Eds.), Understanding causes and generalizing about them (pp. 39-82). San Francisco, CA: Jossey-Bass.
9. Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago, IL: Rand-McNally.
10. Cook, T.D., & Payne, M.R. (2001). Objecting to the objections to using random assignment in educational research. In F. Mosteller & R. Boruch (Eds.), Evidence matters (pp. 150-177). Washington, DC: Brookings Institution Press.
11. Cook, T.D., Scriven, M., Coryn, C.L.S., & Evergreen, S. D. H. (2010). Contemporary thinking about causation in evaluation: A dialogue with tom cook and Michael scriven. American Journal of Evaluation, 31, 105-117.
12. Cook, T.D., & Shadish, W.R. (1994). Social experiments-Some developments over the past 15 years. Annual Review of Psychology, 45, 545-580.
13. Cronbach, L.J. (1982). Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass.
14. Cronbach, L.J., Ambron, S.R., Dornbusch, S.M., Hess, R.D., Hornik, R.D., Phillips, D.C.,. . . Weiner, S.S. (1980). Toward reform of program evaluation: Aims, methods, and institutional arrangements. San Francisco, CA: Jossey-Bass.
15. Erickson, F., & Gutierrez, K. (2002). Culture, rigor, and science in educational research. Educational Researcher, 31, 21-24.
16. Glass, G.V. (2000). Meta-analysis at 25. Retrieved March 1, 2010, from http://glass.ed.asu.edu/gene/papers/meta25.html.
17. Glazerman, S., Levy, D.M., & Myers, D. (2003). Nonexperimental versus experimental estimates of earnings impacts. Annals of the American Academy of Political and Social Science, 589, 63-93.
18. Goldkamp, J.S. (2008). Missing the target and missing the point: "Successful" random assignment but misleading results. Journal of Experimental Criminology, 4, 83-115.
19. Guba, E.G., & Lincoln, Y.S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.
20. Gueron, J.M. (2002). The politics of random assignment: Implementing studies and impacting policy. In F. Mosteller & R. Boruch (Eds.), Evidence matters: Randomized trials in education research (pp. 15-49). Washington, DC: Brookings Institution Press.
21. Heckman, J., & Hotz, V.J. (1988). Are classical experiments necessary for evaluating the impact of manpower training programs? A critical assessment. Industrial Relations Research Association, 40th Annual Proceedings, 291-302.
22. Jenney, B., & Lohr, S. (2009). Experimental designs for multiple-level responses, with application to a large-scale educational intervention. Annals of Applied Statistics, 3, 691-709.
23. Lalonde, R.J. (1986). Evaluating the econometric evaluations of training-programs with experimental-data. American Economic Review, 76, 604-620.
24. Raudenbush, S.W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173-185.
25. Shadish, W.R., Clark, M.H., & Steiner, P.M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments. Journal of the American Statistical Association, 103, 1334-1343.
26. Wolff, N. (2000). Using randomized controlled trials to evaluate socially complex services: Problems, challenges and recommendations. The Journal of Mental Health Policy and Economics, 3, 97-109.