A randomized experiment comparing random and cutoff-based assignment

Psychological Methods

Volumn 16, Issue 2, 2011, Pages 179-191

  Shadish, William R ^a   Galindo, Rodolfo ^a   Wong, Vivian C ^b   Steiner, Peter M ^b   Cook, Thomas D ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b NORTHWESTERN UNIVERSITY   (United States)

Author keywords

Cutoff based assignment; Randomized experiment; Regression discontinuity design

Indexed keywords

ADOLESCENT; ADULT; ARTICLE; CLINICAL TRIAL; COMPARATIVE STUDY; COMPUTER GRAPHICS; CONTROLLED CLINICAL TRIAL; CONTROLLED STUDY; EDUCATION; EPIDEMIOLOGY; FEMALE; HUMAN; INTERNET; LINGUISTICS; MALE; MATHEMATICS; METHODOLOGY; RANDOMIZED CONTROLLED TRIAL; SAMPLE SIZE; STATISTICAL ANALYSIS; STATISTICAL MODEL; STATISTICS; TEACHING;

ADOLESCENT; BIAS (EPIDEMIOLOGY); COMPUTER GRAPHICS; COMPUTER-ASSISTED INSTRUCTION; DATA INTERPRETATION, STATISTICAL; FEMALE; HUMANS; INTERNET; MALE; MATHEMATICS; MODELS, STATISTICAL; RESEARCH DESIGN; SAMPLE SIZE; VOCABULARY; YOUNG ADULT;

EID: 79958811489     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0023345     Document Type: Article

References (38)

1. Aiken L.S., West S.G., Schwalm D.E., Carroll J., Hsuing S. Comparison of a randomized and two quasi-experimental designs in a single outcome evaluation: Efficacy of a university-level remedial writing program. Evaluation Review 1998, 22:207-244. doi: 10.1177/0193841X9802200203.
2. Angrist J.D., Pischke J.-S. Mostly harmless econometrics. An empiricist's companion 2009, Princeton University Press, Princeton, NJ.
3. Berk R., Barnes G., Ahlman L., Kurtz E. When second best is good enough: A comparison between a true experiment and a regression discontinuity quasi-experiment. Journal of Experimental Criminology 2010, 6:191-208. doi: 10.1007/s11292-010-9095-3.
4. Berk R.A., Rauma D. Capitalizing on nonrandom assignment to treatment: A regression discontinuity evaluation of a crime control program. Journal of the American Statistical Association 1983, 78:21-27.
5. Black D., Galdo J., Smith J.A. Evaluating the regression discontinuity design using experimental data 2007, Retrieved from economics.uwo.ca/newsletter/misc/2009/smith_mar25.pdf.
6. Black D., Galdo J., Smith J.C. Evaluating the regression discontinuity design using experimental data 2005, Retrieved from http://www.personal.ceu.hu/departs/personal/Gabor_Kezdi/Program-Evaluation/Black-Galdo-Smith-2005-RegressionDiscontinuity.pdf.
7. Bloom, H. S., Kemple, J. J., Gamse, B., & Jacob, R. (April, 2005). Using regression discontinuity analysis to measure the impacts of Reading First. Paper presented at annual conference of the American Educational Research Association, Montreal, Canada.
8. Buddelmeyer H., Skoufias E. An evaluation of the performance of regression discontinuity design on PROGRESA. (World Bank Policy Research Working Paper No. 3386; IZA Discussion Paper No. 827) 2004, Retrieved from http://ssrn.com/abstract=434600.
9. Cook T.D. " Waiting for life to arrive": A history of the regression-discontinuity design in psychology statistics and economics. Journal of Econometrics 2008, 142:636-654. doi: 10.1016/j.jeconom.2007.05.002.
10. Cook T.D., Shadish W.R., Wong V.C. Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management 2008, 27:724-750. doi: 10.1002/pam.20375.
11. Cook T.D., Wong V.C. Empirical tests of the validity of the regression-discontinuity design. Annales d'Economie et de Statistique 2008, 91/92:127-150.
12. Vocabulary test II (V-2). Kit of factor referenced cognitive tests 1962, Educational Testing Service, Educational Testing Services, Princeton, NJ.
13. Arithmetic aptitude test (RG-1). Kit of factor referenced cognitive tests 1993, Educational Testing Service, Educational Testing Services, Princeton, NJ.
14. Fan J., Gijbels I. Local polynomial modeling and its implications 1996, Chapman & Hall, London, England.
15. Glazerman S., Levy D.M., Myers D. Nonexperimental versus experimental estimates of earnings impacts. The Annals of the American Academy of Political and Social Science 2003, 589:63-93. doi: 10.1177/0002716203254879.
16. Goldberger A.S. Selection bias in evaluating treatment effects: The case of interaction [Discussion paper] 1972, Institute for Research on Poverty, University of Wisconsin, Madison, Madison, WI.
17. Goldberger A.S. Selection bias in evaluating treatment effects: Some formal illustrations. Modeling and evaluating treatment effects in economics 2008, 1-31. JAI Press, Amsterdam, the Netherlands, doi: 10.1016/S0731-9053(07)00001-1. T. Fomby, R.C. Hill, D.L. Millimet, J.A. Smith, E.J. Vytlacil (Eds.).
18. Göritz A.S., Birnbaum M.H. Generic HTML form processor: A versatile PHP script to save web-collected data into a MySQL database. Behavior Research Methods 2005, 37:703-710. doi: 10.3758/BF03192743.
19. Hahn J., Todd P., van der Klaauw W. Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica 2001, 69:201-209. doi: 10.1111/1468-0262.00183.
20. Hastie T., Tibshirani R. Generalized additive models 1990, Chapman & Hall, Boca Raton, LA.
21. Imbens, G., & Kalyanaraman, K. (2010). Optimal bandwidth choice for the regression discontinuity estimator. Unpublished manuscript.
22. Imbens G., Lemieux T. Regression discontinuity designs. Journal of Econometrics 2008, 142:615-635. doi: 10.1016/j.jeconom.2007.05.001.
23. Lee D.S., Card D. Regression discontinuity inference with specification error. Journal of Econometrics 2008, 142:655-674. doi: 10.1016/j.jeconom.2007.05.003.
24. Luellen J.K., Shadish W.R., Clark M.H. Propensity scores: An introduction and experimental test. Evaluation Review 2005, 29:530-558. doi: 10.1177/0193841X05275596.
25. Peikes D.N., Moreno L., Orzol S.M. Propensity score matching: A note of caution for evaluators of social programs. The American Statistician 2008, 62:222-231. doi: 10.1198/000313008X332016.
26. Pohl S., Steiner P.M., Eisermann J., Soellner R., Cook T.D. Unbiased causal inference from an observational study: Results of a within-study comparison. Educational Evaluation and Policy Analysis 2009, 31:463-479. doi: 10.3102/0162373709343964.
27. Rauma D., Berk R.A. Remuneration and recidivism: The long-term impact of unemployment compensation on ex-offenders. Journal of Quantitative Criminology 1987, 3:3-27.
28. Rescorla L., Rosnthal A.S. Growth in standardized ability and achievement test scores from 3rd to 10th grade. Journal of Educational Psychology 2004, 96:85-96. doi: 10.1037/0022-0663.96.1.85.
29. Rubin D.B. Assignment to treatment group on the basis of a covariate. Journal of Educational Statistics 1977, 2:1-26. doi: 10.2307/1164933.
30. Rubin D.B. Comment: The design and analysis of gold standard randomized experiments. Journal of the American Statistical Association 2008, 103:1350-1353. doi: 10.1198/016214508000001011.
31. Schochet P. Statistical power for regression discontinuity design in educational evaluations. Journal of Educational and Behavioral Statistics 2009, 34:238-266. doi: 10.3102/1076998609332748.
32. Shadish W.R., Clark M.H., Steiner P.M. Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association 2008, 103:1334-1344. doi: 10.1198/016214508000000733.
33. Shadish W.R., Cook T.D. The renaissance of field experimentation in evaluating interventions. Annual Review of Psychology 2009, 60:607-629. doi: 10.1146/annurev.psych.60.110707.163544.
34. Shadish W.R., Cook T.D., Campbell D.T. Experimental and quasi-experimental designs for generalized causal inference 2002, Houghton Mifflin, Boston, MA.
35. Shadish W.R., Luellen J.K., Clark M.H. Propensity scores and quasi-experimentation. Strengthening research methodology: Psychological measurement and evaluation 2006, 143-157. American Psychological Association, Washington, DC, doi: 10.1037/11384-008. R.R. Bootzin, P. McKnight (Eds.).
36. Thistlethwaite D.L., Campbell D.T. Regression-discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology 1960, 51:309-317. doi: 10.1037/h0044319.
37. Wilde E.T., Hollister R. How close is close enough? Testing nonexperimental estimates of impact against experimental estimates of impact with education test scores as outcomes. Journal of Policy Analysis and Management 2007, 26:455-477. doi: 10.1002/pam.20262.
38. Wood S.N. Generalized additive models. An introduction with R 2006, Chapman & Hall/CRC, Boca Raton, LA.