Pretest measures of the study outcome and the elimination of selection bias: Evidence from three within study comparisons

Prevention Science

Volumn 19, Issue 3, 2018, Pages 274-283

  Hallberg, Kelly ^a   Cook, Thomas D ^b,c   Steiner, Peter M ^d   Clark, M H ^e  

^a UNIVERSITY OF CHICAGO   (United States)

^b MATHEMATICA POLICY RESEARCH   (United States)

^c NORTHWESTERN UNIVERSITY   (United States)

^d UNIVERSITY OF WISCONSIN MADISON   (United States)

^e UNIVERSITY OF CENTRAL FLORIDA   (United States)

Author keywords

Causal inference; Propensity score matching; Randomized experiment; Within study comparison

Indexed keywords

ACHIEVEMENT; ARTICLE; CASE REPORT; CLINICAL ARTICLE; EDUCATION; HUMAN; HUMAN EXPERIMENT; OBSERVATIONAL STUDY; OUTCOME ASSESSMENT; PROPENSITY SCORE; SELECTION BIAS; THEORETICAL STUDY; BENCHMARKING; INFORMATION PROCESSING; RANDOMIZATION;

BENCHMARKING; DATASETS AS TOPIC; HUMANS; OBSERVATIONAL STUDIES AS TOPIC; PROPENSITY SCORE; RANDOM ALLOCATION; SELECTION BIAS;

EID: 85053528056     PISSN: 13894986     EISSN: 15736695     Source Type: Journal    
DOI: 10.1007/s11121-016-0732-6     Document Type: Article

References (30)

1. Alexander, K. L., Entwisle, R. D., & Dauber, S. L. (2003). On the success of failure: A reassessment of the effects of retention in the primary school grades. Cambridge: New York.
2. Ashenfelter, O. (1978). Estimating the effect of training programs on earnings. Review of Economics and Statistics, 67, 47–57.
3. Bifulco, R. (2012). Can nonrandomized estimates replicate estimates based on random assignment in evaluations of school choice? A within-study comparison. Journal of Policy Analysis and Management, 31, 729–751.
4. Bloom, H., Michalopoulos, C., & Hill, C. (2005). Using experiments to assess nonexperimental comparison-group methods for measuring program effects. In H. Bloom (Ed.), Learning more from social experiments. New York: Russell Sage.
5. Campbell, D. T. (1957). Factors relevant to the validity of experiments in social setting. Psychological Bulletin, 54, 297–312.
6. Campbell, D. T., & Boruch, R. F. (1975). Making the case for randomized assignment to treatments by considering the alternatives. In C. A. Bennett & A. A. Lumsdaine (Eds.), Evaluation and experiments: Some critical issues in assessing social programs. New York: Academic.
7. Campbell, D.T., & Erlebacher, A. E. (1970). How regression artifacts can mistakenly make compensatory education programs look harmful. In J. Hellmuth (Ed.), The disadvantaged child: Vol. 3, Compensatory education: A national debate. New York: Brunner/ Mazel.
8. Campbell, D. T., & Stanley, J. (1963). Experimental and quasiexperimental designs for research. Boston, MA: Houghton Mifflin Company.
9. Cook, T. D., Shadish, W. J., & Wong, V. C. (2008). Three conditions under which observational studies produce the same results as experiments. Journal of Policy Analysis and Management, 27, 724–750.
10. Cronbach, L. (1982). Desigining evaluations of educational and social programs. San Francisco, CA: Jossey-Bass Publishers.
11. Demirtas, H., & Hedeker, D. (2011). A practical way for computing approximate upper and lower correlation bounds. The American Statistician, 65, 2.
12. Elwert, F. & Winship, C. (2014). Endogenous selection bias: The problem of conditioning on a collider variable. The Annual Review of Sociology.
13. Glazerman, S., Levy, D., & Myers, D. (2003). Nonexperimental versus experimental estimates of earnings impacts. The Annals of the American Academy, 589, 63–91.
14. Hong, G., & Raudenbush, S.W. (2005). Effects of kindergarten retention policy on children’s cognitive growth in reading and mathematics. Education Evaluation and Policy Analysis, 27, 205–224.
15. Hong, G., & Raudenbush, S. W. (2006). Evaluation kindergarten retention: A case study for causal inference for multilevel observational data. Journal of the American Statistical Association, 101, 901–910.
16. Jackson, G. B. (1975). The research evidence on the effects of grade retention. Review of Educational Research, 45, 613–635.
17. Konstantopoulos, S., Miller, S., & van der Ploeg, A. (2013). The impact of Indiana’s interim assessments on methematics and reading. Educational Evaluation and Policy Analysis, 35, 481–499.
18. LaLonde, R. (1986). Evaluating the econometric evalautions of training programs with experimental data. Annual Economic Review, 76, 604–20.
19. Pearl, J. (2009). The structural theory of causation. In P. McKay Illari, F. Russo, & J.Williamson (Eds.), Causality in the sciences (pp. 1–30). Oxford: Clarendon.
20. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
21. Rubin, D. B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services & Outcomes Research Methodology, 2, 169–188.
22. Rubin, D. B., & Thomas, N. (1996). Characterizing the effect of using linear propensity score methods with normal distributions. Biometrika, 79, 797–809.
23. Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association, 103, 1334–1343.
24. Shadish, W. R., Cook, T., & Campbell, D. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin Company.
25. Smith, J., & Todd, P. (2005). Does matching overcome LaLonde’s critique of nonexperimental estimators? Journal of Econometrics, 305–353.
26. St Clair, T., Cook, T.D., & Hallberg, K. (2014). Examining the internal validity and statistical precision of the comparative interrupted time series design by comparison with a randomized experiment. American Journal of Evaluation.
27. Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15, 250–67.
28. Steiner, P. M., Cook, T. D., & Shadish, W. R. (2011). On the importance of reliable covariate measurement in selection bias adjustments using propensity scores. Journal of Educational and Behavioral Statistics, 36, 213.
29. Wong, V., Valentine, J.C. & Miller-Bains, K. (2016). Empirical performance of covariates in education observational studies. Journal of Research on Educational Effectiveness.
30. Wooldridge, J.M. (2009). Should instrumental variables be used as matching variables? Working paper.