The importance of covariate selection in controlling for selection bias in observational studies

Psychological Methods

Volumn 15, Issue 3, 2010, Pages 250-267

  Steiner, Peter M ^a,b   Cook, Thomas D ^a   Shadish, William R ^c   Clark, M H ^d  

^a NORTHWESTERN UNIVERSITY   (United States)

^b INSTITUTE FOR ADVANCED STUDIES   (Austria)

^c UNIVERSITY OF CALIFORNIA   (United States)

^d SOUTHERN ILLINOIS UNIVERSITY   (United States)

Author keywords

Hidden bias; Propensity score; Selection bias; Strong ignorability; Within study comparison

Indexed keywords

ARTICLE; BEHAVIORAL RESEARCH; EDUCATION; EPIDEMIOLOGY; HUMAN; MATHEMATICS; METHODOLOGY; PATIENT SELECTION; PROPENSITY SCORE; RANDOMIZATION; STATISTICAL ANALYSIS; STATISTICAL MODEL; STATISTICS;

BEHAVIORAL RESEARCH; CAUSALITY; DATA INTERPRETATION, STATISTICAL; EPIDEMIOLOGIC METHODS; HUMANS; MATHEMATICS; MODELS, STATISTICAL; PATIENT SELECTION; PROPENSITY SCORE; RANDOM ALLOCATION; SELECTION BIAS;

EID: 77956791074     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0018719     Document Type: Article

References (58)

1. Agodini R., Dynarski M. Are experiments the only option? A look at dropout prevention programs. The Review of Economics and Statistics 2004, 86:180-194.
2. Aiken L.S., West S.G., Schwalm D.E., Carroll J., Hsuing S. Comparison of a randomized and two quasi-experimental designs in a single outcome evaluation: Efficacy of a university-level remedial writing program. Evaluation Review 1998, 22:207-244.
3. Bang H., Robins J.M. Doubly robust estimation in missing data and causal inference models. Biometrics 2005, 61:962-972.
4. Beck A.T., Beck R.W. Screening depressed patients in family practice: A rapid technic. Postgraduate Medicine 1972, 51:81-85.
5. Black D., Galdo J., Smith J.A. Evaluating the regression discontinuity design using experimental data 2007, Available from http://economics.uwo.ca/newsletter/misc/2009/smith_mar25.pdf.
6. Bloom H.S., Michalopoulos C., Hill C.J., Lei Y. Can nonexperimental comparison group methods match the findings from a random assignment evaluation of mandatory welfare-to-work programs? 2002, Manpower Demonstration Research Corporation, Washington, DC.
7. Buddelmeyer H., Skoufias E. An evaluation of the performance of regression discontinuity design on PROGRESA (World Bank Policy Research Working Paper No. 3386; IZA Discussion Paper No. 827) 2004, Available from http://ssrn.com/abstract=434600.
8. Cochran W.G. The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics 1968, 24:295-313.
9. Cook T.D. "Waiting for life to arrive": A history of the regression-discontinuity design in psychology, statistics and economics. Journal of Econometrics 2008, 142:636-654.
10. Cook T.D., Shadish W.R., Wong V.C. Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management 2008, 27:724-750.
11. Cook T.D., Steiner P.M., Pohl S. Assessing how bias reduction is influenced by covariate choice, unreliability, and data analytic mode: An analysis of different kinds of within-study comparisons in different substantive domains. Multivariate Behavioral Research 2009, 44:828-847.
12. Dehejia R., Wahba S. Causal effects in non-experimental studies: Re-evaluating the evaluation of training programs. Journal of the American Statistical Association 1999, 94:1053-1062.
13. Diaz J.J., Handa S. An assessment of propensity score matching as a nonexperimental impact estimator. The Journal of Human Resources 2006, 41:319-345.
14. Dorans N.J., Lyu C.F., Pommerich M., Houston W.M. Concordance between ACT assessment and recentered SAT I sum scores. College and University 1997, 73:24-33.
15. Vocabulary Test II (V-2): Kit of factor referenced cognitive tests 1962, Educational Testing Service, Author, Princeton, NJ.
16. Arithmetic Aptitude Test (RG-1): Kit of factor referenced cognitive tests 1993, Educational Testing Service, Author, Princeton, NJ.
17. Faust M.W., Ashcraft M.H., Fleck D.E. Mathematics anxiety effects in simple and complex addition. Mathematical Cognition 1996, 2:25-62.
18. Glazerman S., Levy D.M., Myers D. Nonexperimental versus experimental estimates of earnings impacts. The Annals of the American Academy 2003, 589:63-93.
19. Goldberg L.R. The development of markers for the Big-Five factor structure. Psychological Assessment 1992, 4:26-42.
20. Goldberger A.S. Selection bias in evaluating treatment effects: Some formal illustrations (Discussion Paper No. 123) 1972, Institute for Research on Poverty, University of Wisconsin-Madison, Madison, WI.
21. Heckman J.J., Ichimura H., Smith J.C., Todd P. Characterizing selection bias. Econometrica 1998, 66:1017-1098.
22. Heckman J.J., Navarro-Lozano S. Using matching, instrumental variables, and control functions to estimate economic choice models. The Review of Economics and Statistics 2004, 86:30-57.
23. Hill J.L., Reiter J.P., Zanutto E.L. A comparison of experimental and observational data analyses. Applied Bayesian modeling and causal inference from incomplete data perspectives 2004, 51-60. Wiley, New York, NY. A. Gelman, X.-L. Meng (Eds.).
24. Hirano K., Imbens G.W. Estimation of causal effects using propensity score weighting: An application to data on right heart catheterization. Health Services & Outcomes Research Methodology 2001, 2:259-278.
25. Holland P.W. Statistics and causal inference. Journal of the American Statistical Association 1986, 81:945-970.
26. Hong G., Hong Y. Reading instruction time and homogeneous grouping in kindergarten: An application of marginal mean weighting through stratification. Educational Evaluation and Policy Analysis 2009, 31:54-81.
27. Hong G., Raudenbush S.W. Evaluating kindergarten retention policy: A case study of causal inference for multilevel observational data. Journal of the American Statistical Association 2006, 101:901-910.
28. Hotz V.J., Imbens G.W., Mortimer J.H. Predicting the efficacy of future training programs using past experiences (NBER Technical Working Paper 238) 1999, National Bureau of Economic Research, Cambridge, MA.
29. Kang J., Schafer J.L. Demystifying double robustness: A comparison of alternative strategies for estimating population means from incomplete data. Statistical Science 2007, 26:523-539.
30. Kaplan D. An extension of the propensity score adjustment method for the analysis of group differences in MIMIC models. Multivariate Behavioral Research 1999, 34:467-492.
31. Lunceford J.K., Davidian M. Stratification and weighting via propensity score in estimation of causal treatment effects: A comparative study. Statistical Medicine 2004, 23:2937-2960.
32. Morgan S.L., Winship C. Counterfactuals and causal inference: Methods and principles for social research 2007, Cambridge University Press, Cambridge, England.
33. Pearl J. Causality: Models, reasoning, and inference 2009, Cambridge University Press, Cambridge, England. 2nd ed.
34. Pohl S., Steiner P.M., Eisermann J., Soellner R., Cook T.D. Unbiased causal inference from an observational study: Results of a within-study comparison. Educational Evaluation and Policy Analysis 2009, 31:463-479.
35. R: A language and environment for statistical computing 2007, R Development Core Team, R Foundation for Statistical Computing, Vienna, Austria, Available from http://www.R-project.org.
36. Robins J.M., Rotnitzky A. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association 1995, 90:122-129.
37. Robins J.M., Rotnitzky A. Comment on "Inference for semiparametric models: Some questions and answers" by Bickel and Kwon. Statistica Sinica 2001, 11:920-936.
38. Rosenbaum P.R. From association to causation in observational studies: The role of tests of strongly ignorable treatment assignment. Journal of the American Statistical Association 1984, 79:41-48.
39. Rosenbaum P.R. Dropping out of high school in the United States: An observational study. Journal of Educational Statistics 1986, 11:207-224.
40. Rosenbaum P.R. Observational studies 2002, Springer-Verlag, New York, NY. 2nd ed.
41. Rosenbaum P.R., Rubin D.B. The central role of the propensity score in observational studies for causal effects. Biometrika 1983, 70:41-55.
42. Rosenbaum P.R., Rubin D.B. Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association 1984, 79:516-524.
43. Rubin D.B. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 1974, 66:688-701.
44. Rubin D.B. Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services and Outcomes Research Methodology 2001, 2:169-188.
45. Rubin D.B. Matched sampling for causal effects 2006, Cambridge University Press, Cambridge, England.
46. Rubin D.B. The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials. Statistics in Medicine 2007, 26:20-36.
47. Rubin D.B. The design and analysis of gold standard randomized experiments. Discussion of "Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment" by W. R. Shadish, M. H. Clark, and P. M. Steiner. Journal of the American Statistical Association 2008, 103:1350-1353.
48. Rubin D.B. For objective causal inference, design trumps analysis. The Annals of Applied Statistics 2008, 2:808-840.
49. Rubin D.B., Thomas N. Matching using estimated propensity scores: Relating theory to practice. Biometrics 1996, 52:249-264.
50. Schafer J.L., Graham J.W. Missing data: Our view of the state of the art. Psychological Methods 2002, 7:147-177.
51. Schafer J.L., Kang J. Average causal effects from non-randomized studies: A practical guide and simulated example. Psychological Methods 2008, 13:279-313.
52. Shadish W.R., Clark M.H., Steiner P.M. Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association 2008, 103:1334-1343.
53. Shadish W.R., Cook T.D., Campbell D.T. Experimental and quasi-experimental designs for generalized causal inference 2002, Houghton-Mifflin, Boston, MA.
54. Shah B.R., Laupacis A., Hux J.E., Austin P.C. Propensity score methods gave similar results to traditional regression modeling in observational studies: A systematic review. Journal of Clinical Epidemiology 2005, 58:550-559.
55. Steiner, P. M., Cook, T. D., & Shadish, W. R. (in press). On the importance of reliable covariate measurement in selection bias adjustments using propensity scores. Journal of Educational and Behavioral Statistics.
56. Steyer R. Analyzing individual and average causal effects via structural equation models. Methodology 2005, 1:39-64.
57. Steyer R., Gabler S., von Davier A.A., Nachtigall C. Causal regression models: II. Unconfoundedness and causal unbiasedness. Methods of Psychological Research Online 2000, 5:55-87.
58. Stürmer T., Joshi M., Glynn R.J., Avorn J., Rothman K.J., Schneeweiss S. A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. Journal of Clinical Epidemiology 2006, 59:437-447.