How bias reduction is affected by covariate choice, unreliability, and mode of data analysis: Results from two types of within-study comparisons

Multivariate Behavioral Research

Volumn 44, Issue 6, 2009, Pages 828-847

  Cook, Thomas D ^a   Steiner, Peter M ^a   Pohl, Steffi ^b  

^a NORTHWESTERN UNIVERSITY   (United States)

^b FRIEDRICH SCHILLER UNIVERSITY JENA   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77951656304     PISSN: 00273171     EISSN: None     Source Type: Journal    
DOI: 10.1080/00273170903333673     Document Type: Review

References (37)

1. Angrist, J., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91, 444-472.
2. Campbell, D. T., & Erlebacher, A. E. (1970). How regression artifacts can mistakenly make compensatory education programs look harmful. In J. Hellmuth (Ed.), The disadvantaged child: Vol. 3. Compensatory education: A national debate (pp. 185-210). New York: Brunner/Mazel.
3. Cochran, W. G., & Rubin, D. B. (1973). Controlling bias in observational studies: A review. Sankhya: The Indian Journal of Statistics, 35, 417-466.
4. Cook, T. D. (2008). Waiting for life to arrive: A history of the regression-discontinuity design in psychology, statistics and economics. Journal of Econometrics, 142, 636-654.
5. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand-McNally.
6. Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which experiments and observational studies often produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724-750.
7. Cook, T. D., Wong, V. C., Taylor, J., Gandhi, A., Kendziora, K., Choi, K., et al. (2008). Impacts of school improvement status on students with disabilities: Technical work group materials. Washington, DC: American Institutes for Research.
8. Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.
9. Diaz, J. J., & Handa, S. (2006). An assessment of propensity score matching as a nonexperimental impact estimator. The Journal of Human Resources, 41, 319-345.
10. Glazerman, S., Levy, D. M., & Myers, D. (2003). Nonexperimental versus experimental estimates of earnings impacts. Annals of the American Academy, 589, 63-93.
11. Goldberger, A. S. (1972). Structural equation methods in the social sciences. Econometrica, 40, 979-1001.
12. Hahn, J., Todd, P., & van der Klaauw, W. (2001). Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica, 69, 201-209.
13. Heckman, J. J. (2000). Causal parameters and policy analysis in economics: A twentieth century retrospective. The Quarterly Journal of Economics, 115, 45-97.
14. Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlations for planning group-randomized experiments in education. Educational Evaluation and Policy Analysis, 29, 60-87.
15. Hong, G., & Raudenbush, S. W. (2006). Evaluating kindergarten retention policy: A case study of causal inference for multilevel observational data. Journal of the American Statistical Association, 101, 901-910.
16. Imbens, G., & Lemieux, T. (2008). Regression discontinuity designs: A guide to practice Journal of Econometrics, 142, 615-635.
17. Kang, J., & Shafer, J. L. (2007). Demystifying double robustness: a comparison of alternative strategies for estimating population means from incomplete data. Statistical Science, 26, 523-539.
18. LaLonde, R. (1986). Evaluating the econometric evaluations of training with experimental data. American Economic Review, 76, 604-620.
19. Lord, F. M. (1960). Large-scale covariance analysis when the control variable is fallible. Journal of the American Statistical Association, 55, 307-331.
20. Lunceford, J. K., & Davidian, M. (2004). Stratification and weighting via propensity score in estimation of causal treatment effects: A comparative study. Statistical Medicine, 23, 2937-2960.
21. Morgan, S. L., & Winship C. (2007). Counterfactuals and causal inference: Methods and principles for social research. Cambridge, UK: Cambridge University Press.
22. Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D. (in press). Unbiased causal inference from an observational study: Results of a within-study comparison. Educational Evaluation and Policy Analysis.
23. Robins, J. M., & Rotnitzky, A. (1995). Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 90, 122-129.
24. Rosenbaum, P. R. (2002). Observational Studies (2nd ed.). New York: Springer.
25. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55.
26. Rubin, D. B. (2006). Matched sampling for causal effects. Cambridge, UK: Cambridge University Press.
27. Rubin, D. B. (2008). For objective causal inference, design trumps analysis. The Annals of Applied Statistics, 2, 808-840.
28. Rubin, D. B., & Thomas, N. (1996). Matching using estimated propensity scores: Relating theory to practice. Biometrics, 52, 249-264.
29. Rubin, D. B., & Thomas, N. (2000). Combining propensity score matching with additional adjustments for prognostic covariates. Journal of the American Statistical Association, 95, 573-585.
30. Schafer, J. L., & Kang, J. (2008). Average causal effects from non-randomized studies: A practical guide and simulated example. Psychological Methods, 13, 279-313.
31. Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association, 103, 1334-1344.
32. Shah, B. R., Laupacis, A., Hux, J. E., & Austin, P. C. (2005). Propensity score methods gave similar results to traditional regression modeling in observational studies: A systematic review. Journal of Clinical Epidemiology, 58, 550-559.
33. Steiner, P. M., Cook, T. D., & Shadish, W. R. (2009). On the importance of reliable covariate measurement in selection bias adjustments using propensity scores. Manuscript submitted for publication.
34. Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2009). The importance of covariate selection in controlling for selection bias in observational studies. Manuscript submitted for publication.
35. Stuart E. A., & Rubin, D. B. (2007). Best practices in quasi-experimental designs:Matching methods for causal inference. In J. Osborne (Ed.), Best practices in quantitative methods (chap. 11, pp. 155-176). Thousand Oaks, CA: Sage.
36. Stürmer, T., Joshi, M., Glynn, R. J., Avorn, J., Rothman, K. J., & Schneeweiss, S. (2006). A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. Journal of Clinical Epidemiology, 59, 437-447.
37. West, S. G., & Thoemmes, F. (in press). Campbell's and Rubin's perspectives on causal inference. Psychological Methods