Average Causal Effects From Nonrandomized Studies: A Practical Guide and Simulated Example

Psychological Methods

Volumn 13, Issue 4, 2008, Pages 279-313

  Schafer, Joseph L ^a   Kang, Joseph ^b  

^a PENNSYLVANIA STATE UNIVERSITY   (United States)

^b NORTHWESTERN UNIVERSITY   (United States)

Author keywords

nonequivalent control group design; propensity scores; Rubin's causal model

Indexed keywords

ADOLESCENT; ARTICLE; CLINICAL TRIAL; EMOTION; EPIDEMIOLOGY; FEMALE; HUMAN; LOW CALORY DIET; OUTCOME ASSESSMENT; PERSONALITY TEST; PSYCHOLOGICAL ASPECT; PSYCHOLOGY; PSYCHOMETRY; REGRESSION ANALYSIS; STATISTICAL MODEL; STATISTICS;

ADOLESCENT; BIAS (EPIDEMIOLOGY); CAUSALITY; CLINICAL TRIALS AS TOPIC; DIET, REDUCING; EMOTIONS; FEMALE; HUMANS; MODELS, STATISTICAL; OUTCOME ASSESSMENT (HEALTH CARE); PERSONALITY INVENTORY; PSYCHOLOGY, EXPERIMENTAL; PSYCHOMETRICS; REGRESSION ANALYSIS;

EID: 57749105474     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0014268     Document Type: Article

References (138)

1. Agresti A. Categorical data analysis. 2nd ed. (2002), Wiley, New York
2. Aiken L.S., and West S.G. Multiple regression: Testing and interpreting interactions (1991), Sage, Newbury Park, CA
3. Albert J.H., and Chib S. Bayesian analysis of binary and polychotomous response data. Journal of the American Statistical Association 88 (1993) 669-679
4. Allison P.D. Missing data (2001), Sage, Thousand Oaks, CA
5. Angrist J.D., Imbens G., and Rubin D.B. Identification of causal effects using instrumental variables. Journal of the American Statistical Association 91 (1996) 444-455
6. Bang H., and Robins J.M. Doubly robust estimation in missing data and causal inference models. Biometrics 61 (2005) 962-972
7. Berman N.C., and Greenhouse S.W. Comment: Adjusting for demographic covariates by the analysis of covariance [with rejoinder]. Journal of Clinical and Experimental Neuropsychology 14 (1992) 981-985
8. Breiman L., Friedman J.H., Olshen R.A., and Stone C.M. Classification and regression trees (1984), CRC Press, Boca Raton, FL
9. Carpenter J., Kenward M., and Vansteelandt S. A comparison of multiple imputation and inverse probability weighting for analyses with missing data. Journal of the Royal Statistical Society, Series A 169 (2006) 571-584
10. Cassel C.M., Särndal C.E., and Wretman J.H. Some results on generalized difference estimation and generalized regression estimation for finite populations. Biometrika 63 (1976) 615-620
11. Cassel C.M., Särndal C.E., and Wretman J.H. Foundations of inference in survey sampling (1977), Wiley, New York
12. Chambers R.L., and Skinner C.J. Analysis of survey data (2003), Wiley, New York
13. Cleveland W.S., and Devlin S.J. Locally-weighted regression: An approach to regression analysis by local fitting. Journal of the American Statistical Association 83 (1988) 596-610
14. Cochran W.G. The planning of observational studies of human populations. Journal of the Royal Statistical Society, Series A 128 (1965) 234-265
15. Cochran W.G. The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics 24 (1968) 205-213
16. Cochran W.G. Sampling techniques. 2nd ed. (1977), Wiley, New York
17. Cochran W.G. Planning and analysis of observational studies (1983), Wiley, New York
18. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. (1988), Erlbaum, Hillsdale, NJ
19. Cohen J., Cohen P., West S.G., and Aiken L.S. Applied multiple regression/correlation analysis for the behavioral sciences. 3rd ed. (2003), Erlbaum, Mahwah, NJ
20. Collins L.M., Schafer J.L., and Kam C.M. A comparison of inclusive and restrictive missing-data strategies in modern missing-data procedures. Psychological Methods 6 (2001) 330-351
21. Cook T., and Campbell D. Quasi-experimentation: Design and analysis issues for field settings (1979), Houghton-Mifflin, Boston
22. Cornfield J., Haenszel W., Hammond E., Lilienfeld A., Shimkin M., and Wynder E. Smoking and lung cancer: Recent evidence and a discussion of some questions. Journal of the National Cancer Institute 22 (1959) 173-203
23. D'Agostino Jr. R.B. Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Statistics in Medicine 17 (1998) 2265-2281
24. Davidian M., Tsiatis A.A., and Leon S. Semiparametric estimation of treatment effect in a pretest-posttest study without missing data. Statistical Science 20 (2005) 261-301
25. Dawid A.P. Causal inference without counterfactuals (with discussion). Journal of the American Statistical Association 95 (2000) 407-448
26. Dehejia R.H., and Wahba S. Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association 94 (1999) 1053-1062
27. DeMaris A. Explained variance in logistic regression. Sociological Methods and Research 31 (2002) 27-74
28. Dominici F., Zeger S.L., Parmigiani G., Katz J., and Christian P. Estimating percentile-specific treatment effects in counterfactual models: A case-study of micronutrient supplementation, birth weight and infant mortality. Applied Statistics 55 (2006) 261-280
29. DuMouchel W.H., and Duncan G.J. Using sample survey weights in multiple regression analyses of stratified samples. Journal of the American Statistical Association 78 (1983) 535-543
30. Elliott M.R., and Little R.J.A. Model-based alternatives to trimming survey weights. Journal of Official Statistics 16 (2000) 191-209
31. Gelman A., Carlin J.B., Stern H.S., and Rubin D.B. Bayesian data analysis. 2nd ed. (2004), Chapman and Hall, London
32. Gelman A., and Hill J. Data analysis using regression and multilevel/hierarchical models (2007), Cambridge University Press, New York
33. Gelman A., and Meng X.L. Applied Bayesian modeling and causal inference from incomplete-data perspectives (2004), Wiley, New York
34. Gerner B., and Wilson P.H. The relationship between friendship factors and adolescent girls' body image concern, body dissatisfaction, and restrained eating. International Journal of Eating Disorders 37 (2005) 313-320
35. Green M.W., and Rogers P.J. Impaired cognitive function during spontaneous dieting. Psychological Medicine 25 (1995) 1003-1010
36. Greenland S., Pearl J., and Robins J. Causal diagrams for epidemiologic research. Epidemiology 10 (1999) 37-48
37. Haavelmo T. The probability approach in econometrics. Econometrica 15 (1944) 413-419
38. Hardin J.W., and Hilbe J.M. Generalized linear models (2003), Chapman & Hall/CRC Press, New York
39. Haviland A., Nagin D.S., and Rosenbaum P.R. Combining propensity score matching and group-based trajectory analysis in an observational study. Psychological Methods 12 (2007) 247-267
40. Heatherton T.F., and Polivy J. Chronic dieting and eating disorders: A spiral model. In: Crowther J.H., Tennenbaum D.L., Hobfoll S.E., and Stephens M.A.P. (Eds). The etiology of bulimia nervosa: The individual and family context (1992), Hemisphere Publishing, London 133-148
41. Heckman J.J., Smith J., and Clements N. Making the most out of programme evaluations and social experiments: Accounting for heterogeneity in programme impacts. Review of Economic Studies 64 (1997) 487-535
42. Hernan M., Brumback B., and Robins J.M. Marginal structural models and causal inference in epidemiology. Epidemiology 11 5 (2000) 550-560
43. Hill A.J. Does dieting make you fat?. British Journal of Nutrition 92 Suppl. 1 (2004) S15-S18
44. Hinkley D. Transformation diagnostics for linear models. Biometrika 72 (1985) 487-496
45. Hirano K., and Imbens G. Estimation of causal effects using propensity score weighting: An application to data on right heart catherization. Health Services and Outcome Research Methodology 2 (2001) 259-278
46. Ho D.E., Imai K., King G., and Stuart E.A. MatchIt: Nonparametric preprocessing for parametric causal inference (2004). http://www.gking.harvard.edu Statistical software package for R, available from http://www.gking.harvard.edu
47. Höfler M. Causal inference based on counterfactuals. BMC Medical Research Methodology 5 (2005) 28
48. Holland P.W. Statistics and causal inference. Journal of the American Statistical Association 81 (1986) 945-970
49. Hong G., and Raudenbush S.W. Effects of kindergarten retention policy on children's cognitive growth in reading and mathematics. Educational Evaluation and Policy Analysis 27 (2005) 205-224
50. Horvitz D.G., and Thompson D.J. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association 47 (1952) 663-685
51. Hosmer D.W., and Lemeshow S. Applied logistic regression. 2nd ed. (2003), Wiley, New York
52. Hsu L.K. Epidemiology of the eating disorders. Psychiatric Clinics of North America 19 (1996) 681-700
53. Huitema B. Analysis of covariance and alternatives (1980), Wiley, New York
54. Jaccard J., and Wan C.K. LISREL approaches to interaction effects in multiple regression (1996), Sage, Thousand Oaks, CA
55. Johnson F., and Wardle J. Dietary restraint, body dissatisfaction, and psychological distress: A prospective analysis. Journal of Abnormal Psychology 115 (2005) 119-125
56. Kang J.D.Y., and Schafer J.L. Demystifying double robustness: A comparison of alternative strategies for estimating population means from incomplete data. Statistical Science 26 (2007) 523-539
57. Katz D.L. Competing dietary claims for weight loss: Finding the forest through truculent trees. Annual Review of Public Health 26 (2005) 61-88
58. King G. The dangers of extreme counterfactuals. Political Analysis 14 (2006) 131-159
59. King G., and Zeng L. Improving forecasts of state failure. World Politics 53 (2002) 623-658
60. King G., and Zeng L. When can history be our guide? The pitfalls of counterfactual inference. International Studies Quarterly 51 (2006) 183-210
61. Kovacs M., Obrosky D.S., and Sherrill J. Developmental changes in the phenomenology of depression in girls compared to boys from childhood onward. Journal of Affective Disorders 74 (2003) 33-48
62. Kutner M.H., Nachtsheim C.J., Neter J., and Li W. Applied linear statistical models. 4th ed. (2005), McGraw-Hill, New York
63. Lalonde R. Evaluating the econometric evaluations of training programs with experimental data. American Economic Review 76 (1986) 604-620
64. Leuven E., and Sianesi B. PSMATCH2: Stata module to perform full Mahalanobis and propensity score matching, common support graphing, and covariate imbalance testing (2003) Statistical Software Components S432001, Boston College Department of Economics, Boston College
65. Levine M.P., Smolak L., and Hayden H. The relation of sociocultural factors to eating attitudes and behaviors among middle school girls. Journal of Early Adolescence 14 (1994) 471-490
66. Liang K.Y., and Zeger S.L. Longitudinal data analysis using generalized linear models. Biometrika 73 (1986) 13-22
67. Little R.J.A., and An H. Robust likelihood-based analysis of multivariate data with missing values. Statistica Sinica 14 (2004) 949-968
68. Little R.J.A., An H., Johanns J., and Giordani B. A comparison of subset selection and analysis of covariance for the adjustment of confounders. Psychological Methods 5 (2000) 459-476
69. Little R.J.A., and Rubin D.B. Statistical analysis with missing data. 2nd ed. (2002), Wiley, New York
70. Liu C. Robit regression: A simple robust alternative to logistic and probit regression. In: Gelman A., and Meng X.L. (Eds). Applied Bayesian modeling and causal inference from incomplete-data perspectives (2004), Wiley, New York 227-238
71. Lohr S. Sampling: Design and analysis (1999), Duxbury Press, Pacific Grove, CA
72. Luellen J.K., Shadish W.R., and Clark M.H. Propensity scores: An introduction and experimental test. Evaluation Review 29 (2005) 530-558
73. Lunceford J.K., and Davidian M. Stratification and weighting via the propensity score in estimation of causal treatment effects: A comparative study. Statistics in Medicine 23 (2004) 2937-2960
74. Manski C.F. Comment: Choice as an alternative to control in observational studies. Statistical Science 14 (1999) 279-281
75. McCaffrey D.F., Ridgeway G., and Morral A.R. Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods 9 (2004) 403-425
76. Menard S. Applied logistic regression analysis. 2nd ed. (2002), Sage, Thousand Oaks, CA
77. Miller G.A., and Chapman J.P. Misunderstanding analysis of covariance. Journal of Abnormal Psychology 110 (2001) 40-48
78. Ming K., and Rosenbaum P.R. Substantial gains in bias reduction from matching with a variable number of controls. Biometrics 56 (2000) 118-124
79. Morgan S.L., and Winship C. Counterfactuals and causal inference: Methods and principles for social research (2007), Cambridge University Press, New York
80. Neumark-Sztainer D., and Hannan P.J. Weight-related behaviors among adolescent girls and boys: Results from a national survey. Archives of Pediatrics and Adolescent Medicine 154 (2000) 569-577
81. Newey W.K., and McFadden D. Large sample estimation and hypothesis testing. In: Griliches Z. (Ed). Handbook of econometrics 4 (1994), Elsevier, Amsterdam 2111-2245
82. Neyman J. On the application of probability theory to agricultural experiments: Essays on Principles, Section 9. Translated in Statistical Science 5 (1990) 465-480 (Original work published 1923)
83. Parsons L.J. Using SAS software to perform a case-control match on propensity score in an observational study. Proceedings of the Twenty-Fifth Annual SAS Users Group International Conference, Paper 225-25 (2000), SAS Institute, Cary, NC
84. Patton G.C., Selzer R., Coffey C., Carlin J.B., and Wolfe R. Onset of adolescent eating disorders: Population based cohort study over 3 years. British Medical Journal 318 (1999) 765-768
85. Pearl J. Causality: Models, reasoning and inference (2000), Cambridge University Press, Cambridge, England
86. Pfefferman D. The role of sampling weights when modeling survey data. International Statistical Review 61 (1993) 317-337
87. Pfefferman D. The use of sampling weights for survey data analysis. Statistical Methods in Medical Research 5 (1996) 239-261
88. Pratt J.W., and Schlaifer R. On the interpretation and observation of laws. Journal of Econometrics 39 (1988) 23-52
89. Pregibon D. Resistant fits for some commonly used logistic models with medical applications. Biometrics 38 (1982) 485-498
90. Resnick M.D., Bearman P.S., Blum R.W., Bauman K.E., Harris K.M., Jones J., et al. Protecting adolescents from harm: Findings from the National Longitudinal Study on Adolescent Health. Journal of the American Medical Association 278 (1997) 823-832
91. Robins J. Marginal structural models versus structural nested models as tools for causal inference. In: Halloran M.E., and Berry D. (Eds). Statistical models in epidemiology: The environment and clinical trials (1999), Springer-Verlag, New York 95-134
92. Robins J.M., Hernan M., and Brumback B. Marginal structural models and causal inference in epidemiology. Epidemiology 11 (2000) 550-560
93. Robins J.M., and Rotnitzky A. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association 90 (1995) 122-129
94. Robins J.M., Rotnitzky A., and Zhao L.P. Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association 89 (1994) 846-866
95. Robins J.M., Rotnitzky A., and Zhao L.P. Analysis of semiparametric regression models for repeated outcomes in the presence of missing data. Journal of the American Statistical Association 90 (1995) 106-121
96. Rosen J.C., Tacy B., and Howell D. Life stress, psychological symptoms and weight reducing behavior in adolescent girls: A prospective analysis. International Journal of Eating Disorders 9 (1990) 17-26
97. Rosenbaum P.R. The consequences of adjustment for a concomitant variable that has been affected by the treatment. Journal of the Royal Statistical Society 147 (1984) 656-666
98. Rosenbaum P.R. Observational studies. 2nd ed. (2002), Springer, New York
99. Rosenbaum P.R., and Rubin D.B. The central role of the propensity score in observational studies for causal effects. Biometrika 70 (1983) 41-55
100. Rosenbaum P.R., and Rubin D.B. Reducing bias in observational studies using sub-classification on the propensity score. Journal of the American Statistical Association 79 (1984) 516-524
101. Rosenbaum P.R., and Rubin D.B. Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. American Statistician 39 (1985) 33-38
102. Rotnitzky A., Robins J.M., and Scharfstein D.O. Semiparametric regression for repeated outcomes with ignorable nonresponse. Journal of the American Statistical Association 93 (1998) 1321-1339
103. Rubin D.B. The use of matched sampling and regression adjustment to remove bias in observational studies. Biometrics 29 (1973) 185-203
104. Rubin D.B. Characterizing the estimation of parameters in incomplete data problems. Journal of the American Statistical Association 69 (1974) 467-474
105. Rubin D.B. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 66 (1974) 688-701
106. Rubin D.B. Inference and missing data. Biometrika 63 (1976) 581-592
107. Rubin D.B. Bayesian inference for causal effects: The role of randomization. Annals of Statistics 6 (1978) 34-58
108. Rubin D.B. Using multivariate matched sampling and regression adjustment to control bias in observational studies. Journal of the American Statistical Association 74 (1979) 318-328
109. Rubin D.B. Discussion of "randomization analysis of experimental data in the Fisher randomization test" by D. Basu. Journal of the American Statistical Association 75 (1980) 591-593
110. Rubin D.B. William G. Cochran's contributions to the design, analysis and evaluation of observational studies. In: Rao P.S.R.S., and Sedransk J. (Eds). W. G. Cochran's impact on statistics (1984), Wiley, New York 37-69
111. Rubin D.B. Multiple imputation for nonresponse in surveys (1987), Wiley, New York
112. Rubin D.B. Neyman (1923) and causal inference in experiments and observational studies. Statistical Science 5 (1990) 472-480
113. Rubin D.B. Estimating causal effects from large data sets using propensity scores. Annals of Internal Medicine 127 (1997) 757-763
114. Rubin D.B. Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services and Outcomes Research Methodology 2 (2001) 169-188
115. Rubin D.B. Causal effects via potential outcomes. Scandinavian Journal of Statistics 31 (2004) 161-170
116. Rubin D.B. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association 100 (2005) 322-331
117. Rubin D.B. Matched sampling for causal effects (2006), Cambridge University Press, New York
118. Rubin D.B., and Thomas N. Matching using estimated propensity scores: Relating theory to practice. Biometrics 52 (1996) 249-264
119. Rubin D.B., and Thomas N. Combining propensity score matching with additional adjustments for prognostic covariates. Journal of the American Statistical Association 95 (2000) 573-585
120. Särndal C.-E., Swensson B., and Wretman J. Model assisted survey sampling (1992), Springer, New York
121. Schafer J.L., and Graham J.W. Missing data: Our view of the state of the art. Psychological Methods 7 (2002) 147-177
122. Schafer J.L., and Schenker N. Inference with imputed conditional means. Journal of the American Statistical Association 95 (2000) 144-154
123. Scharfstein D.O., Rotnitzky A., and Robins J.M. Adjusting for nonignorable dropout using semiparametric nonresponse models. Journal of the American Statistical Association 94 (1999) 1096-1120 (with Rejoinder, 1135-1146)
124. Shadish, W. R., Clark, M. H., & Steiner, P. M. (in press). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association
125. Sobel M.E. Causal inference in the social and behavioral sciences. In: Arminger G., Clogg C.C., and Sobel M.E. (Eds). Handbook of statistical modeling for the social and behavioral sciences (1995), Plenum, New York 1-38
126. Steyer R. Analyzing individual and average causal effects via structural equation models. Methodology 1 (2005) 39-54
127. Stice E., Mazotti L., Krebs M., and Martin S. Predictors of adolescent dieting behaviors: A longitudinal study. Psychology of Addictive Behaviors 12 (1998) 195-205
128. Stice E., Shaw H., and Nemeroff C. Dual pathway model of bulimia nervosa: Longitudinal support for dietary restraint and affect-regulation mechanisms. Journal of Social and Clinical Psychology 17 (1998) 129-149
129. Udry J.R. The National Longitudinal Study of Adolescent Health (Add Health), Waves I and II, 1994-1996; Wave III, 2001-2002 (machine-readable data file and documentation) (2003), Carolina Population Center, University of North Carolina at Chapel Hill, Chapel Hill, NC
130. van der Laan M.J., and Robins J.M. Unified methods for censored longitudinal data and causality (2003), Springer, New York
131. Vartivarian S., and Little R.J.A. On the formation of weighting adjustment cells for unit nonresponse. Proceedings of the Survey Research Methods Section, American Statistical Association (2002) 3553-3558
132. Warren C., and Cooper P.J. Psychological effects of dieting. British Journal of Clinical Psychology 27 (1988) 269-270
133. West S.G., Biesanz J.C., and Pitts S.C. Causal inference and generalization in field settings: Experimental and quasi-experimental designs. In: Reis H.T., and Judd C.M. (Eds). Handbook of research methods in social and personality psychology (2000), Cambridge University Press, New York 40-84
134. White H. A heteroscedasticity-consistent covariance matrix estimator and a direct test for heteroscedasticity. Econometrica 48 (1980) 817-838
135. Wilson G.T. Relation of dieting and voluntary weight loss to psychological functioning and binge eating. Annals of Internal Medicine 119 (1993) 727-730
136. Winship C., and Morgan S.L. The estimation of causal effects from observational data. Annual Review of Sociology 25 (1999) 650-707
137. Winship C., and Sobel M.E. Causal inference in sociological studies. In: Hardy M. (Ed). Handbook of data analysis (2004), Sage, Thousand Oaks, CA 481-504
138. Zanutto E. A comparison of propensity score and linear regression analysis of complex survey data. Journal of Data Science 4 (2006) 67-91