The use of propensity scores to assess the generalizability of results from randomized trials

Journal of the Royal Statistical Society. Series A: Statistics in Society

Volumn 174, Issue 2, 2011, Pages 369-386

  Stuart, Elizabeth A ^a   Cole, Stephen R ^b   Bradshaw, Catherine P ^a   Leaf, Philip J ^a  

^a JOHNS HOPKINS BLOOMBERG SCHOOL OF PUBLIC HEALTH   (United States)

^b UNIVERSITY OF NORTH CAROLINA   (United States)

Author keywords

'Positive behavioral interventions and supports'; Causal inference; External validity; Research synthesis

Indexed keywords

EID: 79952586276     PISSN: 09641998     EISSN: 1467985X     Source Type: Journal    
DOI: 10.1111/j.1467-985X.2010.00673.x     Document Type: Article

References (61)

1. Barrett, S., Bradshaw, C. and Lewis-Palmer, T. (2008) Maryland state-wide PBIS initiative: systems, evaluation, and next steps. J. Pos. Behav. Intervens, 10, 105-114.
2. Bradshaw, C. P., Koth, C. W., Thornton, L. A. and Leaf, P. J. (2009) Altering school climate through school-wise Positive Behavioral Interventions and Supports: findings from a group-randomized effectiveness trial. Prevn Sci., 10, 100-115.
3. Bradshaw, C., Mitchell, M. and Leaf, P. (2010) Examining the effects of Schoolwide Positive Behavioral Interventions and Supports on student outcomes: results from a randomized controlled effectiveness trial in elementary schools. J. Pos. Behav. Intervens, 12, 133-148.
4. Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, R. J., Avorn, J. and Sturmer, T. (2006) Variable selection for propensity score models. Am. J. Epidem., 163, 1149-1156.
5. Brown, C. H., Wang, W., Kellam, S. G., Muthen, B., Petras, H., Toyinbo, P., Poduska, J., Ialongo, N., Wyman, P. A., Chamberlain, P., Sloboda, Z., MacKinnon, D. P., Windham, A. and the Prevention Science Methodology Group (2008) Methods for testing theory and evaluating impact in randomized field trials: intent-to-treat analyses for integrating the perspectives of person, place, and timeDrug Alc. Depend., 95, S74-S104.
6. Campbell, D. T. (1957) Factors relevant to the validity of experiments in social settings. Psychol. Bull., 54, 297-312.
7. Cochran, W. G. and Rubin, D. B. (1973) Controlling bias in observational studies: a review. Sankhya A, 35, 417-446.
8. Cole, S. R. and Stuart, E. A. (2010) Generalizing evidence from randomized clinical trials to target populations: the ACTG-320 trial. Am. J. Epidem., 172, 107-115.
9. Cook, T. D. (2007) Evidence-based practice: where do we stand? In Proc. 20th A. Res. Conf. System of Care for Children's Mental Health: Expanding the Research Base. Tampa: University of South Florida. (Available from .)
10. Couper, M. P. and Miller, P. V. (2008) Web survey methods: introduction. Publ. Opin. Q., 72, 831-835.
11. Eddy, D., Hasselblad, V. and Shachter, R. (1992) Meta-analysis by the Confidence Profile Method: the Statistical Synthesis of Evidence. New York: Academic Press.
12. Flay, B. R. (1986) Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Prevn Med., 15, 451-474.
13. Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., Moscicki, E. K., Schinke, S. and Valentine, J. (2005) Standards of evidence: criteria for efficacy, effectiveness and dissemination. Prevn Sci., 6, 151-175.
14. Frangakis, C. E. (2009) The calibration of treatment effects from clinical trials to target populations. Clin. Trials, 6, 136-140.
15. Glasgow, R. E., Nelson, C. C., Strycker, L. A. and King, D. E. (2006) Using RE-AIM metrics to evaulate diabetes self-management support interventions. Am. J. Prev. Med., 30, 67-73.
16. Greenhouse, J. B., Kaizar, E. E., Kelleher, K., Seltman, H. and Gardner, W. (2008) Generalizing from clinical trial data: a case study of the risk of suicidality among pediatric antidepressant users. Statist. Med., 27, 1801-1813.
17. Grodstein, F., Clarkson, T. and Manson, J. (2003) Understanding the divergent data on post-menopausal hormone therapy. New Engl. J. Med., 348, 645-650.
18. Haneuse, S., Schildcrout, J., Crane, P., Sonnen, J., Breitner, J. and Larson, E. (2009) Adjustment for selection bias in observational studies with application to the analysis of autopsy data. Neuroepidemiology, 32, 229-239.
19. Hansen, B. B. (2004) Full matching in an observational study of coaching for the SAT. J. Am. Statist. Ass., 99, 609-618.
20. Hansen, B. B. (2008) The prognostic analogue of the propensity score. Biometrika, 95, 481-488.
21. Hedges, L. V. and Olkin, I. (1985) Statistical Methods for Meta-analysis. Burlington: Academic Press.
22. Hernán, M., Alonso, A., Logan, R., Grodstein, F., Michels, K., Willett, W., Manson, J. and Robins, J. (2008) Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease (with discussion). Epidemiology, 19, 766-779.
23. Ho, D. E., Imai, K., King, G. and Stuart, E. A. (2007) Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Polit. Anal., 15, 199-236.
24. Hong, G. and Raudenbush, S. W. (2006) Evaluating kindergarten retention policy: a case study of causal inference for multilevel observational data. J. Am. Statist. Ass., 101, 901-910.
25. Horner, R., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. and Esperanza, J. (2009) A randomized, wait-list controlled effectiveness trial assessing school-wide Positive Behavior Support in elementary schools. J. Pos. Behav. Intervens, 11, 133-144.
26. Horvitz, D. and Thompson, D. (1952) A generalization of sampling without replacement from a finite universe. J. Am. Statist. Ass., 47, 663-685.
27. Imai, K., King, G. and Stuart, E. A. (2008) Misunderstandings between experimentalists and observationalists about causal inference. J. R. Statist. Soc. A, 171, 481-502.
28. Insel, T. R. (2006) Beyond efficacy: the STAR*D trial. Am. J. Psychiatr., 163, 5-7.
29. Institute of Medicine (2006) Improving the Quality of Health Care for Mental and Substance-use Conditions. Washington, DC: National Academies Press.
30. Kalton, G. and Flores-Cervantes, I. (2003) Weighting methods. J. Off. Statist., 19, 81-97.
31. Kam, C., Greenberg, M. and Walls, C. (2003) Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prevn Sci., 1, 55-63.
32. Kang, J. D. and Schafer, J. L. (2007) Demystifying double robustness: a comparison of alternative strategies for estimating a population mean from incomplete data. Statist. Sci., 22, 523-539.
33. Koth, C., Bradshaw, C. and Leaf, P. (2009) Teacher Observation of Classroom Adaptation-Checklist (TOCA-C): development and factor structure. Measmnt Evaln Counsel. Devlpmnt, 42, 15-30.
34. Lunceford, J. K. and Davidian, M. (2004) Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study. Statist. Med., 23, 2937-2960.
35. Mamdani, M. M., Sykora, K., Li, P., Normand, S.-L. T., Streiner, D. L., Austin, P. C., Rochon, P. A. and Anderson, G. M. (2005) Reader's guide to critical appraisal of cohort studies: 2, assessing potential for confounding. Br. Med. J., 330, 960-962.
36. National Institute of Mental Health (1999) Bridging science and service: a report by the NIMH Council's clinical treatment and services research workgroup. Technical Report. National Institute of Mental Health, Bethesda. (Available from .)
37. Pan, Q. and Schaubel, D. E. (2009) Evaluating bias correction in weighted proportional hazards regression. Liftime Data Anal., 15, 120-146.
38. Peto, R., Collins, R. and Gray, R. (1995) Large-scale randomized evidence: large, simple trials and overviews of trials. J. Clin. Epidem., 48, 23-40.
39. Prevost, T. C., Abrams, K. R. and Jones, D. R. (2000) Hierarchical models in generalized synthesis of evidence: an example based on studies of breast cancer screening. Statist. Med., 19, 3359-3376.
40. Rosenbaum, P. R. (1991) A characterization of optimal designs for observational studies. J. R. Statist. Soc. B, 53, 597-610.
41. Rosenbaum, P. R. (2002) Observational Studies, 2nd edn. New York: Springer.
42. Rosenbaum, P. R. and Rubin, D. B. (1983a) The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41-55.
43. Rosenbaum, P. R. and Rubin, D. B. (1983b) Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. J. R. Statist. Soc. B, 45, 212-218.
44. Rothwell, P. M. (2005a) External validity of randomised controlled trials: ''To whom do the results of this trial apply?''. Lancet, 365, 82-93.
45. Rothwell, P. M. (2005b) Subgroup analysis in randomised controlled trials: importance, indications, and interpretation. Lancet, 365, 176-186.
46. Rubin, D. B. (1973) The use of matched sampling and regression adjustment to remove bias in observational studies. Biometrics, 29, 185-203.
47. Rubin, D. B. (1976) Inference and missing data (with discussion). Biometrika, 63, 581-592.
48. Rubin, D. B. (1977) Assignment to treatment group on the basis of a covariate. J. Educ. Statist., 2, 1-26.
49. Rubin, D. B. (2001) Using propensity scores to help design observational studies: application to the tobacco litigation. Hlth Serv. Outcms Res. Methodol., 2, 169-188.
50. Rubin, D. B. (2008) For objective causal inference, design trumps analysis. Ann. Appl. Statist., 2, 808-840.
51. Rubin, D. B. and Thomas, N. (1992) Characterizing the effect of matching using linear propensity score methods with normal distributions. Biometrika, 79, 797-809.
52. Rubin, D. B. and Thomas, N. (2000) Combining propensity score matching with additional adjustments for prognostic covariates. J. Am. Statist. Ass., 95, 573-585.
53. Shadish, W. R., Cook, T. D. and Campbell, D. T. (2002) Experimental and Quasi-experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.
54. Stuart, E. A. (2010) Matching methods for causal inference: a review and a look forward. Statist. Sci., 25, 1-21.
55. Stuart, E. A. and Green, K. M. (2008) Using full matching to estimate causal effects in non-experimental studies: examining the relationship between adolescent marijuana use and adult outcomes. Devlpmntl Psychol., 44, 395-406.
56. Sugai, G. and Horner, R. (2006) A promising approach for expanding and sustaining school-wide positive behavior support. School Psychol. Rev., 35, 245-259.
57. Sutton, A. J. and Higgins, J. P. (2008) Recent developments in meta-analysis. Statist. Med., 27, 625-650.
58. US Department of Education (2009) The impacts of regular Upward Bound on postsecondary outcomes seven to nine years after scheduled high school graduation. Technical Report. Office of Planning, Evaluation, and Policy Development, Policy and Program Studies Service, Washington DC.
59. US Department of Health and Human Services (2010) Head start impact study final report. Technical Report. Administration for Children and Families, Washington DC.
60. Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J. and Drazen, J. M. (2007) Statistics in medicine-reporting of subgroup analyses in clinical trials. New Engl. J. Med., 357, 2189-2194.
61. Weisberg, H., Hayden, V. and Pontes, V. (2009) Selection criteria and generalizability within the counterfactual framework: explaining the paradox of antidepressant-induced suicidality? Clin. Trials, 6, 109-118.