Assessing the Generalizability of Randomized Trial Results to Target Populations

Prevention Science

Volumn 16, Issue 3, 2015, Pages 475-485

  Stuart, Elizabeth A ^a   Bradshaw, Catherine P ^b   Leaf, Philip J ^a  

^a JOHNS HOPKINS BLOOMBERG SCHOOL OF PUBLIC HEALTH   (United States)

^b UNIVERSITY OF VIRGINIA   (United States)

Author keywords

Horvitz Thompson weighting; Positive behavioral interventions and supports (PBIS); Propensity scores

Indexed keywords

CHILD; CHILD BEHAVIOR DISORDERS; EVIDENCE BASED PRACTICE; HEALTH PROMOTION; HUMAN; MARYLAND; ORGANIZATION AND MANAGEMENT; PROPENSITY SCORE; RANDOMIZED CONTROLLED TRIAL (TOPIC); SCHOOL HEALTH SERVICE; SOCIAL BEHAVIOR;

CHILD; CHILD BEHAVIOR DISORDERS; EVIDENCE-BASED PRACTICE; HEALTH PROMOTION; HUMANS; MARYLAND; PROPENSITY SCORE; RANDOMIZED CONTROLLED TRIALS AS TOPIC; SCHOOL HEALTH SERVICES; SOCIAL BEHAVIOR;

EID: 84925488414     PISSN: 13894986     EISSN: 15736695     Source Type: Journal    
DOI: 10.1007/s11121-014-0513-z     Document Type: Article

References (50)

1. Bradshaw, C. P., Koth, C. W., Thornton, L. A., & Leaf, P. J. (2009). Altering school climate through school-wide positive behavioral interventions and supports: Findings from a group-randomized effectiveness trial. Prevention Science, 10, 100–115.
2. Bradshaw, C. P., Waasdorp, T. E., & Leaf, P. J. (2012). Effects of school-wide positive behavioral interventions and supports on child behavior problems. Pediatrics, 130, 1136–1145.
3. Braslow, J. T., Duan, N., Starks, S. L., Polo, A., Bromley, E., & Wells, K. B. (2005). Generalizability of studies on mental health treatment and outcomes, 1981–1996. Psychiatric Services, 56, 1261–1268.
4. Brown, C. H., Wang, W., & Sandler, I. (2008). Examining how context changes intervention impact: The use of effect sizes in multilevel mixture meta-analysis. Child Development Perspectives, 2, 198–205.
5. Cole, S. R., & Stuart, E. A. (2010). Generalizing evidence from randomized clinical trials to target populations: The ACTG-320 trial. American Journal of Epidemiology, 172, 107–115.
6. Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., et al. (2005). Standards of evidence: Criteria for efficacy, effectiveness, and dissemination. Prevention Science, 6, 151–175.
7. Frangakis, C. E. (2009). The calibration of treatment effects from clinical trials to target populations. Clinical Trials, 6, 136–140.
8. Green, L. W., & Glasgow, R. E. (2006). Evaluating the relevance, generalization, and applicability of research: Issues in external validation and translation methodology. Evaluation & the Health Professions, 29, 126–153.
9. Hansen, B. B. (2008). The prognostic analogue of the propensity score. Biometrika, 95, 481–488.
10. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando: Academic.
11. Holt, D., & Smith, T. M. F. (1979). Post stratification. Journal of the Royal Statistical Society, Series A, 142, 33–46.
12. Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., et al. (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11, 133–144.
13. Horvitz, D., & Thompson, D. (1952). A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47, 663–685.
14. Humphreys, K., Weingardt, K. R., & Harris, A. H. S. (2007). Influence of subject eligibility criteria on compliance with national institutes of health guidelines for inclusion of women, minorities, and children in treatment research. Alcoholism: Clinical and Experimental Research, 31, 988–995.
15. Imai, K., King, G., & Stuart, E. A. (2008). Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society, Series A, 171, 481–502.
16. Insel, T. R. (2006). Beyond efficacy: The STAR*D trial. American Journal of Psychiatry, 163, 5–7.
17. Kang, J. D., & Schafer, J. L. (2007). Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science, 22, 523–539.
18. Koth, C. W., Bradshaw, C. P., & Leaf, P. J. (2009). Teacher observation of classroom adaptation-checklist: Development and factor structure. Measurement and Evaluation in Counseling and Development, 42, 15–30.
19. Murray, D. M. (1998). Design and analysis of group-randomized trials. New York: Oxford.
20. Nature. (2010). Editorial: Putting gender on the agenda. Nature, 465, 665.
21. O’Muircheartaigh, C., & Hedges, L. V. (2014). Generalizing from unrepresentative experiments: A stratified propensity score approach. Journal of the Royal Statistical Society: Series C: Applied Statistics. doi:10.1111/rssc.12037. Early view online.
22. Olsen, R., Bell, S., Orr, L., & Stuart, E. A. (2013). External validity in policy evaluations that choose sites purposively. Journal of Policy Analysis and Management, 32, 107–121.
23. Pan, Q., & Schaubel, D. E. (2009). Evaluating bias correction in weighted proportional hazards regression. Lifetime Data Analysis, 15, 120–146.
24. Pas, E., Bradshaw, C. P., & Mitchell, M. M. (2011). Examining the validity of office discipline referrals as an indicator of student behavior problems. Psychology in the Schools, 48, 541–555.
25. Pressler, T. R., & Kaizar, E. E. (2013). The use of propensity scores and observational data to estimate randomized controlled trial generalizability bias. Statistics in Medicine. doi:10.1002/sim.5802.
26. Prevost, T. C., Abrams, K. R., & Jones, D. R. (2000). Hierarchical models in generalized synthesis of evidence: An example based on studies of breast cancer screening. Statistics in Medicine, 19, 3359–3376.
27. R Core Team. (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from the R project website: http://www.R-project.org/.
28. Rabe-Hesketh, S., Skrondal, A., & Pickles, A. (2004). Generalized multilevel structural equation modelling. Psychometrika, 69, 167–190.
29. Raudenbush, S. W., Bryk, A. S., Cheong, Y. F., Congdon, R. T., Jr., & du Toit, M. (2011). Hierarchical linear and nonlinear modeling (HLM7). Lincolnwood: Scientific Software International, Inc.
30. Rosenbaum, P. R. (1987). Model-based direct adjustment. Journal of the American Statistical Association, 82, 387–394.
31. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
32. Rothwell, P. M. (2005). External validity of randomised controlled trials: “To whom do the results of this trial apply?”. Lancet, 365, 82–93.
33. Rubin, D. B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services & Outcomes Research Methodology, 2, 169–188.
34. Schochet, P. Z., Burghardt, J., & McConnell, S. (2008). Does job corps work? impact findings from the national job corps study. American Economic Review, 98, 1864–86.
35. Shadish, W. R. (1995). The logic of generalization: Five principles common to experiments and ethnographies. American Journal of Community Psychology, 23, 419–428.
36. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin Company.
37. StataCorp. (2011). Stata Statistical Software: Release 12. College Station, TX: StataCorp LP.
38. Stirman, S. W., Derubeis, R. J., Crits-Christoph, P., & Rothman, A. (2005). Can the randomized controlled trial literature generalize to nonrandomized patients? Journal of Consulting and Clinical Psychology, 73, 127–35. PMID: 15709839.
39. Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25, 1–21.
40. Stuart, E. A., Cole, S., Bradshaw, C. P., & Leaf, P. J. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society, Series A, 174, 369–386.
41. Sugai, G., Horner, R., & Gresham, F. (2001). Behaviorally effective school environments. In M. Shinn, G. Stoner, & H. Walker (Eds.), Interventions for academic and behavior problems: Preventive and remedial approaches (pp. 315–350). Silver Spring: National Association of School Psychiatrists.
42. Sutton, A. J., & Higgins, J. P. (2008). Recent developments in meta-analysis. Statistics in Medicine, 27, 625–650.
43. Tipton, E. (2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38, 239–266.
44. Tipton, E., Hedges, L. V., Vaden-Kiernan, M., Borman, G. D., Sullivan, K., & Caverly, S. (2014). Sample selection in randomized experiments: A new method using propensity score stratified sampling. Journal of Research on Educational Effectiveness, 7, 114–135.
45. Turner, R. M., Spiegelhalter, D. J., Smith, G. C. S., & Thompson, S. G. (2009). Bias modelling in evidence synthesis. Journal of the Royal Statistical Society, Series A, 172, 21–47.
46. U.S. Department of Education. (2009). The impacts of regular upward bound on postsecondary outcomes seven to nine years after scheduled high school graduation. Washington: Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service.
47. U.S. Department of Health and Human Services. (2010). Head start impact study final report. Washington: Office of Planning, Evaluation and Policy Development, Administration for Children and Families, Policy and Program Studies Service.
48. Waasdorp, T. E., Bradshaw, C. P., & Leaf, P. J. (2012). The impact of schoolwide positive behavioral interventions and supports on bullying and peer rejection. Archives of Pediatrics and Adolescent Medicine, 166, 149–156.
49. Westen, D. I., Stirman, S. W., & DeRubeis, R. J. (2006). Are research patients and clinical trials representative of clinical practice? In J. C. Norcross, L. E. Beutler, & R. F. Levant (Eds.), Evidence-based practices in mental health: Debate and dialogue on the fundamental questions (pp. 161–189). Washington: American Psychological Association.
50. Wisniewski, S., Rush, A., Nierenberg, A., Gaynes, B., Warden, D., Luther, J., et al. (2009). Can phase III trial results of antidepresseant medications be generalized to clinical practice? a STAR*D report. American Journal of Psychiatry, 166, 599–607.