Interaction of Theory and Practice to Assess External Validity

Evaluation Review

Volumn 41, Issue 5, 2017, Pages 436-471

  Leviton, Laura C ^a   Trujillo, Mathew D ^a  

^a ROBERT WOOD JOHNSON FOUNDATION   (United States)

Author keywords

methodological development; outcome evaluation (other than economic evaluation); program implementation; program theory; real world dissemination

Indexed keywords

DECISION MAKING; EVALUATION STUDY; EVIDENCE BASED PRACTICE; MANAGEMENT; REPRODUCIBILITY; THEORETICAL MODEL;

DECISION MAKING; EVALUATION STUDIES AS TOPIC; EVIDENCE-BASED PRACTICE; MODELS, THEORETICAL; POLICY MAKING; REPRODUCIBILITY OF RESULTS;

EID: 85036543667     PISSN: 0193841X     EISSN: 15523926     Source Type: Journal    
DOI: 10.1177/0193841X15625289     Document Type: Article

References (100)

1. Avellar S. A., Kleinman R., Sama-Miller E.. (In Press). External validity: The next step for systematic reviews? Evaluation Review.
2. Babitsch B., Gohl D., von Lengerke T., (2012). Re-revisiting Andersen’s behavioral model of health services use: A systematic review of studies from 1998–2011. GMS Psycho-Social-Medicine, 9. doi:http://doi.org/10.3205/psm000089
3. Backer T. E., (2001). Finding the balance: Program fidelity and adaptation in substance abuse prevention: A state of-the-art review. Retrieved from http://modelprograms.samhsa.gov/pdfs/Finding-Balance1.pdf
4. Baumeister R. F., Leary M. R., (1995). The need to belong: Desire for interpersonal attachments as a fundamental human motivation. Psychological Bulletin, 117, 497–529.
5. Berwick D., (2003). Disseminating innovation in health care. Journal of the American Medical Association, 289, 1969–1975.
6. Bishop D. C., Pankratz M. M., Hansen W. B., Albritton J., Albritton L., Strack J., (2014). Measuring fidelity and adaptation reliability of an instrument for school-based prevention programs. Evaluation & the Health Professions, 37, 231–257.
7. Blakely C. H., Mayer J. P., Gottschalk R. G., Schmitt N., Davidson W. S., Roitman D. B., (1987). The fidelity adaptation debate: Implications for the implementation of public sector social programs. American Journal of Community Psychology, 15, 253–268.
8. Bryk A. S., Yeager D. S., Hausman H., Muhich J., Dolle J. R., Grunow A., Gomez L., (2013, 6). Improvement research carried out through networked communities: Accelerating learning about practices that support more productive student mindsets. A white paper prepared for the White House meeting on “Excellence in Education: The Importance of Academic Mindsets.” Stanford, CA: Carnegie Foundation for the Advancement of Teaching. Retrieved from http://www.carnegiefoundation.org/resources/publications/improvement-research-carried-networked-communities-accelerating-learning-practices-support-productive-student-mindsets/
9. Campbell D. T., Stanley J. C., (1966). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally.
10. Canadian Institutes of Health Research. (2014). About knowledge translation. Ottawa, Canada. Retrieved from http://www.cihr-irsc.gc.ca/e/29418.html
11. Castro F. G., Barrera M., Jr., Holleran Steiker L. K., (2010). Issues and challenges in the design of culturally adapted evidence-based interventions. Annual Review of Clinical Psychology, 6, 213–239.
12. Center for Mental Health in Schools at University of California, Los Angeles. (2008). Conduct and behavior problems: Intervention and resources for school aged youth. Los Angeles, CA: Author. Retrieved from http://smhp.psych.ucla.edu
13. Centers for Disease Control and Prevention. (2014). Compendium of evidence-based interventions and best practices for HIV prevention. Retrieved from http://www.cdc.gov/hiv/prevention/research/compendium/index.html
14. Chang H., (2012, 10 11). Scientific pluralism and the mission of history and philosophy of science. [YouTube] Inaugural lecture, University of Cambridge. Retrieved from www.youtube.com/watch?v=zGUsIf9qYw8
15. Chen H. T., (2010). The bottom-up approach to integrative validity: A new perspective for program evaluation. Evaluation and Program Planning, 33, 205–214.
16. Chen H. T., Rossi P. H., (1992). Using theory to improve program and policy evaluations. Westport, CT: Greenwood.
17. Chorpita B., Becker K., Daleiden E., (2007). Understanding the common elements of evidence-based practice: Misconceptions and clinical examples. Journal of the American Academy of Child and Adolescent Psychiatry, 46, 647–652.
18. Collins L. M., (2013). Optimizing family interventions: The multiphase optimization strategy (MOST). In McHale S., McHale P., Amato P., Booth A., (Eds.), Emerging methods in family research (pp. 231–244). New York, NY: Springer.
19. Copi I. M., Cohen C., Flage D. E., (2007). Essentials of logic (2nd ed.). Upper Saddle River, NJ: Pearson Education.
20. Coyle S. L., Needle R. H., Normand J., (1998). Outreach-based HIV prevention for injecting drug users: A review of published outcome data. Public Health Reports, 113, 19–30.
21. Cronbach L. J., Meehl P. E., (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.
22. Cronbach L. J., Shapiro K., (1982). Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass.
23. Davidoff F., Dixon-Woods M., Leviton L., Michie S., (2015). Demystifying theory and its use in improvement. BMJ Quality and Safety, 24, 228–238. doi:10.1136/bmjqs-2014-003627
24. Del Grosso P., Kleinman R., Mraz Esposito A., Sama-Miller E., Paulsell D., (2014). Assessing the evidence of effectiveness of home visiting program models implemented in tribal communities. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
25. Dreyfus H. L., Dreyfus S. E., (1988). Mind over machine: The power of human intuition and expertise in the era of the computer. New York, NY: Free Press.
26. Durlak J. A., DuPre E. P., (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.
27. Dusenbury L., Brannigan R., Hansen W. B., Walsh J., Falco M., (2005). Quality of implementation: Developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research, 20, 308–313.
28. Eccles M. P., Mittman B. S., (2006). Welcome to Implementation Science. Implementation Science, 1. doi:10.1186/1748-5908-1 -1
29. Elliott D. S., Mihalic S., (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5, 47–53.
30. Embry D. D., Biglan A., (2008). Evidence-based kernels: Fundamental units of behavioral influence. Clinical Child and Family Psychological Review, 11, 75–113.
31. Emshoff J. G., Blakely C., Gottschalk R., Mayer J., Davidson W. S., Erickson S., (1987). Innovation in education and criminal justice: Measuring fidelity of implementation and program effectiveness. Educational Evaluation and Policy Analysis, 9, 300–311.
32. Epstein D., Klerman J. A., (2013). When is a program ready for rigorous impact evaluation? The role of a falsifiable logic model. Evaluation Review, 36, 375–401.
33. Fishbein M., Ajzen I., (2010). Predicting and changing behavior: The Reasoned Action approach. New York, NY: Psychology Press.
34. Fixsen D., (2015). Our approach. National Implementation Research Network. Retrieved from http://nirn.fpg.unc.edu/about-nirn/our-approach
35. Flay B. R., Berkowitz M. W., Bier M. C., & The Social and Character Development Research Consortium. (2009). Elementary school-based programs theorized to support social development, prevent violence, and promote positive school climate: Description and hypothesized mechanisms of change. Journal of Research in Character Education, 7, 21–50.
36. Foshee V. A., Bauman K. E., Ennett S. T., Linder G. F., Benefield T. S., Suchindran C., (2004). Assessing the long-term effects of the Safe Dates program and a booster in preventing and reducing adolescent dating violence victimization and perpetration. American Journal of Public Health, 94, 619–624.
37. Foshee V. A., Dixon K. S., Ennett S. T., Moracco K. E., Bowling J. M., Chang L., Moss J. L., (2015). The process of adapting a universal dating abuse prevention program to adolescents exposed to domestic violence. Journal of Interpersonal Violence, 30, 2151–2173.
38. Foshee V. A., Reyes H. L. M., Ennett S. T., Cance J. D., Bauman K. E., Bowling J. B., (2012). Assessing the effects of Families for Safe Dates, a family-based teen dating abuse prevention program. Journal of Adolescent Health, 51, 349–356.
39. Freire K. E., Perkinson L., Morrel-Samuels S., Zimmerman M. A., (2015). Three Cs of translating evidence-based programs for youth and families to practice settings. New Directions for Child and Adolescent Development, 2015, 25–39.
40. Glasgow R. E., Vogt T. M., Boles S. M., (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89, 1323–1327.
41. Griner D., Smith T. B., (2006). Culturally adapted mental health interventions: A metaanalytic review. Psychotherapy: Theory, Research, Practice, Training, 43, 531–548.
42. Hawe P., Shiell A., Riley T., (2004). Complex interventions: how “out of control” can a randomised controlled trial be? British Medical Journal, 328, 1561–1563.
43. Hausmann L. R., Schofield J. W., Woods R. L., (2007). Sense of belonging as a predictor of intentions to persist among African American and White first-year college students. Research in Higher Education, 48, 803–839.
44. Hill L. G., Maucione K., Hood B. K., (2007). A focused approach to assessing program fidelity. Prevention Science, 8, 25–34.
45. Hoffmann T., Glasziou P., Boutron I., Milne R., Perera R., Moher D., …Michie S., (2014). Better reporting of interventions: Template for intervention description and replication (TIDieR) checklist and guide. British Medical Journal, 348, g1687.
46. Horne C. S., (In Press). Assessing and strengthening evidence-based program registries’ usefulness for social service program replication and adaptation. Evaluation Review.
47. Hovland C. I., Janis I. L., Kelley H. H., (1953). Communication and persuasion. New Haven, CT: Yale University Press.
48. Hulleman C., Cordray D. S., (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research Educational Effectiveness, 2, 88–110.
49. Institute for Healthcare Improvement. (2015). About us: The science of improvement. Retrieved from http://www.ihi.org/about/Pages/ScienceofImprovement.aspx
50. International Initiative for Impact Evaluation. (2015). Evidence matters. Retrieved from http://blogs.3ieimpact.org/evidence-matters/
51. Jagers R. J., Sydnor K., Mouttapa M., Flay B. R., (2007). Protective factors associated with preadolescent violence: Preliminary work on a cultural model. American Journal of Community Psychology, 40, 138–145.
52. Johnson K., (2009). State-based home visiting: Strengthening programs through state leadership. New York, NY: National Center for Children in Poverty. Retrieved from http://www.nccp.org/publications/pdf/text_862.pdf
53. Lattimore D., Griffin S. F., Wilcox S., Rheaume C., Dowdy D., Leviton L. C., Ory M. G., (2010). Understanding the challenges and adaptations made by community organizations for translation of evidence-based behavior change physical activity interventions: A qualitative study. American Journal of Health Promotion, 24, 427–434.
54. Leung C., Sanders M. R., Leung S., Mak R., Lau J., (2003). An outcome evaluation of the implementation of the Triple P—Positive Parenting Program in Hong Kong. Family Process, 42, 531–544.
55. Leviton L. C., (2015). Evaluation practice and theory: Up and down the ladder of abstraction. American Journal of Evaluation, 36, 238–242.
56. Leviton L. C., Guinan M. E., (2003). HIV prevention and the evaluation of public health programs. In Valdiserri R. O., (Ed.), Dawning answers: How the HIV/AIDS epidemic has helped to strengthen public health (pp. 155–176). Oxford, England: Oxford University Press.
57. Lipsey M. W., (1993). Theory as method: Small theories of treatments. New Directions for Program Evaluation, 1993, 5–38.
58. Lipsey M. W., (2009). The primary factors that characterize effective interventions with juvenile offenders: A meta-analytic overview. Victims and Offenders, 4, 124–147.
59. March J. G., (1994). A primer on decision making: How decisions happen. New York, NY: Free Press.
60. Michie S., West R., Campbell R., Brown J., Gainforth H., (2014). An ABC of behavior change theories. London, England: Silverback.
61. Morrel-Samuels S., Hutchison P., Perkinson L., Bostic B., Zimmerman M., (2014). Selecting, implementing and adapting Youth Empowerment Solutions. Ann Arbor: University of Michigan.
62. Naylor M. D., Feldman P. H., Keating S., Koren M. J., Kurtzman E. T., Maccoy M. C., Krakauer R., (2009). Translating research into practice: Transitional care for older adults. Journal of Evaluation in Clinical Practice, 15, 1164–1170.
63. Nowak C., Heinrichs N., (2008). A comprehensive meta-analysis of Triple P—Positive Parenting Program using hierarchical linear modeling: Effectiveness and moderating variables. Clinical Child and Family Psychology Review, 11, 114–144.
64. Ostrom E., (1990). Governing the commons: The evolution of institutions for collective action. Cambridge, England: Cambridge University Press.
65. Øvretveit J., Leviton L. C., Parry G., (2011). Increasing the generalisability of improvement research with an improvement replication programme. BMJ Quality and Safety, 20, i87–i91.
66. Patton M. Q., (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York, NY: Guilford.
67. Pawson R., Tilley N., (1997). Realistic evaluation. Thousand Oaks, CA: Sage.
68. Peters R. H., Wexler H. K., (2005). Substance abuse treatment for adults in the criminal justice system: A treatment improvement protocol (DHHS Publication No. (SMA) 05-4056). Rockville, MD: Substance Abuse and Mental Health Services Administration.
69. Perkinson L., (2012). Environmental scan of adaptation guidance (Unpublished report). Atlanta, GA: CDC Foundation.
70. Pressman J. L., Wildavsky A. B., (1984). Implementation. Berkeley: University of California Press.
71. Prinz R. J., Sanders M. R., Shapiro C. J., Whitaker D. J., Lutzker J. R., (2009). Population-based prevention of child maltreatment: The U.S. Triple P system population trial. Prevention Science, 10, 1–12.
72. Prochaska J. O., Norcross J. C., (2009). Systems of psychotherapy: A transtheoretical analysis (7th ed.). Belmont, CA: Wadsworth.
73. Ringwalt C. L., Ennett S., Johnson R., Rohrbach L. A., Simons-Rudolph A., Vincus A., (2003). Factors associated with fidelity to substance use prevention curriculum guides in nation’s middle schools. Health Education & Behavior, 30, 375–391.
74. Robinson T. N., Matheson D. M., Kraemer H. C., Wilson D. M., Obarzanek E., Thompson N. S., …Killen J. D., (2010). A randomized controlled trial of culturally-tailored dance and reducing screen time to prevent weight gain in low-income African-American girls: Stanford GEMS. Archives of Pediatrics & Adolescent Medicine, 164, 995–1004.
75. Rogers E., (2003). Diffusion of innovations (5th ed.). New York, NY: Free Press.
76. Rohrbach L. A., Grana R., Sussman S., Valente T. W., (2006). Type II translation: Transporting prevention interventions from research to real-world settings. Evaluation and the Health Professions, 29, 302–333.
77. Rossi P. H., (1987). The Iron Law of evaluation and other metallic rules. Research in Social Problems and Public Policy, 4, 3–20.
78. Rossi P. H., Lipsey M. W., Freeman H. E., (2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage.
79. Rotheram-Borus M. J., Swendeman D., Becker K. D., (2014). Adapting evidence-based interventions using a common theory, practices and principles. Journal of Clinical Child and Adolescent Psychology, 43, 229–243.
80. Rotheram-Borus M. J., Swendeman D., Chorpita B., (2012). Disruptive innovations for designing and diffusing evidence-based interventions. American Psychologist, 67, 463–476.
81. Schön D. A., (1983). The reflective practitioner: How professionals think in action. New York, NY: Basic Books.
82. Schouten L. M., Hulscher M. E., van Everdingen J. J., Huijsman R., Grol R. P., (2008). Evidence for the impact of quality improvement collaboratives: Systematic review. British Medical Journal, 336, 1491–1494.
83. Schuh R. G., Leviton L. C., (2006). A framework to assess the development and capacity of nonprofit agencies. Evaluation and Program Planning, 29, 171–179.
84. Shadish W. R., Cook T. D., Campbell D. T., (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
85. Shadish W. R., Cook T. D., Leviton L. C., (1991). Foundations of program evaluation: Theorists and their theories. Newbury Park, CA: Sage.
86. Stake R. E., Schwandt T. A., (2006). On discerning quality in evaluation. In Shaw I., Greene J. C., Mark M., (Eds.), The SAGE handbook of evaluation (pp. 404–418). Thousand Oaks, CA: Sage.
87. Steele C. M., Spencer S. J., Aronson J., (2002). Contending with group image: The psychology of stereotype and social identity threat. Advances in Experimental Social Psychology, 34, 379–440.
88. Stuart E. A., Rhodes A., (In Press). Assessing the external validity of randomized trial results: A case study in the difficulties of finding sufficient data. Evaluation Review.
89. Tipton E., Hallberg K., Hedges L. V., Chan W., (In Press). Implications of small samples for generalization: Adjustments and rules of thumb. Evaluation Review.
90. Tipton E., Peck L.R., (In Press). A design-based approach to improve external validity in welfare policy evaluations. Evaluation Review.
91. Walton G. M., Cohen G. L., (2007). A question of belonging: Race, social fit, and achievement. Journal of Personality and Social Psychology, 92, 82–96.
92. Walton G. M., Cohen G. L., (2011). A brief social-belonging intervention improves academic and health outcomes of minority students. Science, 331, 1447–1451.
93. Walton G. M., Cohen G. L., Garcia J., Apfel N., Master A., (2012). A brief intervention to buttress middle school students’ sense of social-belonging: Effects by race and gender. Unpublished manuscript, Stanford University, Palo Alto, CA.
94. Walton G. M., Logel C., Peach J. M., Spencer S. J., Zanna M. P., (2015). Two brief interventions to mitigate a “chilly climate” transform women’s experience, relationships, and achievement in engineering. Journal of Educational Psychology, 107, 468.
95. Weiss C. H., (1997). Evaluation: Methods for studying programs and policies (2nd ed.). Englewood Cliffs, NJ: Prentice Hall.
96. Wenger E., McDermott R., Snyder W., (2002). Cultivating communities of practice: A guide to managing knowledge. Cambridge, MA: Harvard Business School Press.
97. Whittingham K., Sofronoff K., Sheffield J., Sanders M. R., (2008). Stepping Stones Triple P: An RCT of a parenting program with parents of a child diagnosed with an Autism Spectrum Disorder. Journal of Abnormal Child Psychology, 37, 469–480.
98. Wilcox S., Dowda M., Leviton L. C., Bartlett-Prescott J., Bazzarre T., Campbell-Voytal K., …Wegley S.. (2008). The active for life initiative: Final results of translating two evidence-based physical activity programs into practice. American Journal of Preventive Medicine, 35, 340–351.
99. Wilson D. B., Lipsey M. W., (2001). The role of method in treatment effectiveness research: Evidence from meta-analysis. Psychological Methods, 6, 413–429.
100. Yeaton W. H., Sechrest L., (1981). Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49, 156–167.