External Validity: The Next Step for Systematic Reviews?

Evaluation Review

Volumn 41, Issue 4, 2017, Pages 283-325

  Avellar, Sarah A ^a   Thomas, Jaime ^a   Kleinman, Rebecca ^a   Sama Miller, Emily ^a   Woodruff, Sara E ^a   Coughlin, Rebecca ^a   Westbrook, T’Pring R ^b  

^a MATHEMATICA POLICY RESEARCH   (United States)

^b Annie E Casey Foundation   (United States)

Author keywords

external validity; systematic reviews

Indexed keywords

ADOLESCENT; FEMALE; HOME VISIT; HUMAN; MALE; PREGNANCY; REPRODUCIBILITY; YOUNG ADULT;

ADOLESCENT; FEMALE; HOUSE CALLS; HUMANS; MALE; PREGNANCY; REPRODUCIBILITY OF RESULTS; SYSTEMATIC REVIEWS AS TOPIC; YOUNG ADULT;

EID: 85025429265     PISSN: 0193841X     EISSN: 15523926     Source Type: Journal    
DOI: 10.1177/0193841X16665199     Document Type: Article

References (44)

1. Amato P. R., (2014). Does social and economic disadvantage moderate the effects of relationship education on unwed couples? An analysis of data from the 15-month Building Strong Families evaluation. Journal of Marriage and Family, 63, 343–355.
2. Antman E. M., Lau J., Kupelnick B., Mosteller F., Chalmers T. C., (1992). Comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts treatments for myocardial infarction. Journal of the American Medical Association, 268, 240–248.
3. Belsky J., Pluess M., (2011). Differential susceptibility to long-term effects of quality of child care on externalizing behavior in adolescence. International Journal of Behavioral Development, 36, 2–10.
4. Bero L., Rennie D., (1995). The Cochrane Collaboration: Preparing, maintaining, and disseminating systematic reviews of the effects of health care. Journal of the American Medical Association, 274, 1935–1938.
5. Bitler M. P., Gelbach J. B., Hoynes H. W., (2006). What mean impacts miss: Distributional effects of welfare reform experiments. American Economic Review, 96, 988–1012.
6. Bitler M. P., Gelbach J. B., Hoynes H. W. (2012). Can variation in subgroups’ average treatment effects explain treatment effect heterogeneity? Evidence from a social experience (NBER Working Paper Series, Working paper 20142). Retrieved from www.nber.org/papers/w20142.pdf
7. Blase K., Fixsen D., (2013). Core intervention components: Identifying and operationalizing what makes programs work (ASPE Research Brief). Washington, DC: U.S. Department of Health and Human Services.
8. Bonell C., Oakley A., Hargreaves J., Strange V., Rees R., (2006). Assessment of generalisability in trials of health interventions: Suggested framework and systematic review. British Medical Journal, 333, 346–349.
9. Burwell S. M., Muñoz C., Holdren J., Krueger A. (2013). Next steps in the evidence and innovation agenda (OMB Memorandum No. M-13-17). Retrieved from www.whitehouse.gov/sites/default/files/omb/memoranda/2013/m-13-17.pdf
10. Chen H. T., (2010). The bottom-up approach to integrative validity: A new perspective for program evaluation. Evaluation and Program Planning, 33, 205–214.
11. Cook D. J., Mulrow C. D., Haynes R. B., (1997). Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine, 126, 376–380.
12. Davies P., (2000). The relevance of systematic reviews to educational policy and practice. Oxford Review of Education, 26, 365–378.
13. Dishion T. J., McCord J., Poulin F., (1999). When interventions hard: Peer groups and problem behavior. American Psychologist, 54, 755–764.
14. Durlak J. A., DuPre E. P., (2008). Implementation matters: A review of research on the influence of implementation of program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.
15. Eldridge S., Ashby D., Bennett C., Wakelin M., Feder G., (2008). Internal and external validity of cluster randomised trials: Systematic review of recent trials. British Medical Journal, 336, 876–880.
16. Green L. W., Glasgow R. E., (2006). Evaluating the relevance, generalization, and applicability of research issues in external validation and translation methodology. Evaluation & the Health Professions, 29, 126–153.
17. Horne C., (in press). Assessing and strengthening evidence-based program registries’ usefulness for social service program replication and adaptation. Evaluation Review
18. Imbens G. W., (2010). An economist’s perspective on Shadish (2010) and West and Thoemmes (2010). Psychological Methods, 15, 47–55.
19. Klerman J., (in press). Rethinking external validity in official literature reviews. Evaluation Review.
20. Lipsey M. W., Wilson D. B., (2001). Practical meta-analysis. Applied Social Research Methods Series (Vol. 49). Thousand Oaks, CA: Sage.
21. Montgomery P., Grant S., Hopewell S., Macdonald G., Moher D., Michie S., Mayo-Wilson E., (2013). Protocol for CONSORT-SPI: An extension for social and psychological interventions. Implementation Science, 8. doi:10.1186/1748-5908-8-99
22. Mulrow C. D., (1994). Rationale for systematic reviews. British Medical Journal, 309, 597.
23. Olsen R. B., Orr L. L., Bell S. H., Stuart E. A., (2013). External validity in policy evaluations that choose sites purposively. Journal of Policy Analysis and Management, 32, 107–121.
24. Paulsell D., Del Grosso P., Supplee L., (2014). Supporting replication and scale-up of evidence-based home visiting programs: Assessing the implementation knowledge base. American Journal of Public Health, 104, 1624–1632.
25. Pawson R., Greenhalgh T., Harvey G., Walshe K., (2005). Realist review—A new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy, 10, 21–34.
26. Pearson M., Coomber R., (2009). The challenge of external validity in policy-relevant systematic reviews: A case study from the field of substance misuse. Addiction, 105, 136–145.
27. Petticrew M., (2003). Why certain systematic reviews reach uncertain conclusions. British Medical Journal, 326, 756–758.
28. Pluess M., Belsky J., (2009). Differential susceptibility to rearing experience: The case of childcare. The Journal of Child Psychology and Psychiatry, 50, 396–404.
29. Post P. N., de Beer H., Guyatt G. H., (2013). How to generalize efficacy results of randomized trials: Recommendations based on a systematic review of possible approaches. Journal of Evaluation in Clinical Practice, 19, 638–643.
30. Rothwell P. M., (2005). External validity of randomized controlled trials: To whom do the results of this trial apply? Lancet, 365, 82–93.
31. Schochet P. Z., (2009). An approach for addressing the multiple testing problem in social policy impact evaluations. Evaluation Review, 33, 539–567.
32. Schulz K. F., Altman D. G., Moher D., (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. BMC Medicine, 8, 18.
33. Schünemann H. J., Tugwell P., Reeves B. C., Akl E. A., Santesso N., Spencer F. A., …Helfand M. (2013). Non-randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions. Research Synthesis Methods, 4, 49–62.
34. Shadish W. R., Cook T. D., Campbell D. T., (2002). Experimental and quasi-experimental designs for generalized causal inference. New York, NY: Houghton Mifflin Company.
35. Shen C., Jeong J., Li X., Chen P. S., Buxton A., (2013). Treatment benefits and treatment harm rate to characterize heterogeneity in treatment effect. Biometrics, 69, 724–731.
36. Steckler A., McLeroy K. R., (2008). The importance of external validity. American Journal of Public Health, 98, 9–10. doi:10.2105/AJPH.2007.126847
37. Stuart E. A., Cole S. R., Bradshaw C. P., Leaf P. J., (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society, 174, 369–386.
38. The Cochrane Collaboration. (2013). History [Internet web page]. Retrieved from http://www.cochrane.org/about-us/history
39. Tipton E., (2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38, 239–266.
40. Tipton E., Hedges L. V., Vaden-Kiernan M., Borman G. D., Sullivan K., Caverly S., (2014). Sample selection in randomized experiments: A new method using propensity score stratified sampling. Journal of Research on Educational Effectiveness, 7, 114–135.
41. Tranfield D., Denyer D., Smart P., (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14, 207–222.
42. Wasserstein R. L., Lazar N. A., (2016). The ASA’s statement on p-values: Context, process, and purpose. The American Statistician, 70, 129–133.
43. Westbrook T. R., Avellar S. A., Seftor N. (in press). Reviewing the reviews: Examining similarities and differences between federally funded evidence reviews. Evaluation Review.
44. Zorzela L., Golder S., Liu Y., Pilkington K., Hartling L., Joffe A., …Vohra S., (2014). Quality of reporting in systematic reviews of adverse events: Systematic review. British Medical Journal, 348. Retrieved from http://www.bmj.com/content/348/bmj.f7668