Development, inter-rater reliability and feasibility of a checklist to assess implementation (Ch-IMP) in systematic reviews: The case of provider-based prevention and treatment programs targeting children and youth

BMC Medical Research Methodology

Volumn 15, Issue 1, 2015, Pages

  Cargo, Margaret ^a   Stankov, Ivana ^a   Thomas, James ^b   Saini, Michael ^c   Rogers, Patricia ^d   Mayo Wilson, Evan ^e   Hannes, Karin ^f  

^a UNIVERSITY OF SOUTH AUSTRALIA   (Australia)

^b UNIVERSITY OF LONDON   (United Kingdom)

^c UNIVERSITY OF TORONTO   (Canada)

^d RMIT UNIVERSITY   (Australia)

^e JOHNS HOPKINS BLOOMBERG SCHOOL OF PUBLIC HEALTH   (United States)

^f UNIVERSITY OF LEUVEN   (Belgium)

Author keywords

Checklist development; Children and youth; Complex interventions; Implementation; Process evaluation; Provider based interventions; Systematic reviews; Theory driven reviews

Indexed keywords

ADOLESCENT; CHECKLIST; CHILD; FEASIBILITY STUDY; FEMALE; HUMAN; INFANT; LITERATURE; MALE; META ANALYSIS (TOPIC); NEWBORN; PRESCHOOL CHILD; REPRODUCIBILITY; YOUNG ADULT;

ADOLESCENT; CHECKLIST; CHILD; CHILD, PRESCHOOL; FEASIBILITY STUDIES; FEMALE; HUMANS; INFANT; INFANT, NEWBORN; MALE; META-ANALYSIS AS TOPIC; REPRODUCIBILITY OF RESULTS; REVIEW LITERATURE AS TOPIC; YOUNG ADULT;

EID: 84940862312     PISSN: None     EISSN: 14712288     Source Type: Journal    
DOI: 10.1186/s12874-015-0037-7     Document Type: Review

References (74)

1. Pawson R. Realistic evaluation. London: Sage Publication Ltd; 1997.
2. Dishman RK, Motl RW, Saunders R, Felton G, Ward DS, Dowda M, et al. Self-efficacy partially mediates the effect of a school-based physical-activity intervention among adolescent girls. Prev Med. 2004;38(5):628-36.
3. Lehto R, Maatta S, Lehto E, Ray C, Te Velde S, Lien N, et al. The PRO GREENS intervention in Finnish schoolchildren-the degree of implementation affects both mediators and the intake of fruits and vegetables. Br J Nutr. 2014;112(7):1185-94.
4. van Nassau F, Singh A, Cerin E, Salmon J, van Mechelen W, Brug J, et al. The Dutch Obesity Intervention in Teenagers (DOiT) cluster controlled implementation trial: intervention effects and mediators and moderators of adiposity and energy balance-related behaviours. Int J Behav Nutr Phys Act. 2014;11(1):158.
5. Rossi PH, Lipsey MW, Freeman HE. Evaluation. A systematic approach. 7th ed. Thousand Oaks: Sage Publications; 2004.
6. Suchman E. Evaluative research. New York: Russell Sage Foundation; 1967.
7. Weiss CH. Evaluation research. Methods of assessing program effectiveness. Englewood Cliffs, NJ: Prentice-Hall; 1972.
8. Baranowski T, Stables G. Process evaluations of the 5-a-day projects. Health Educ Behav. 2000;27(2):157-66.
9. Harachi TW, Abbott RD, Catalano RF, Haggerty KP, Fleming C. Opening the black box: using process evaluation measures to assess implementation and theory building. Am J Community Psychol. 1999;27(5):715-35.
10. Moore G, Audrey S, Barker M, Bond L, Bonell C, Cooper C, et al. Process evaluation in complex public health intervention studies: the need for guidance. J Epidemiol Community Health. 2014;68(2):101-2.
11. Murta SG, Sanderson K, Oldenburg B. Process evaluation in occupational stress management programs: a systematic review. Am J Health Promot. 2007;21(4):248-54.
12. Steckler A, Linnan L, editors. Process evaluation for public health interventions and research. San Francisco: Jossey-Bass; 2002.
13. Funnell SC, Rogers PJ. Purposeful program theory: effective use of theories of change and logic models. San Francisco, CA: John Wiley/Jossey-Bass; 2011.
14. Anderson LM, Petticrew M, Rehfuess E, Armstrong R, Ueffing E, Baker P, et al. Using logic models to capture complexity in systematic reviews. Research Synthesis Methods. 2011;2(1):33-42.
15. Grant S, Mayo-Wilson E, Hopewell S, Macdonald G, Moher D, Montgomery P. New guidelines are needed to improve the reporting of trials in addiction sciences. Addiction. 2013;108(9):1687-8.
16. Grant SP, Mayo-Wilson E, Melendez-Torres GJ, Montgomery P. Reporting quality of social and psychological intervention trials: a systematic review of reporting guidelines and trial publications. PLoS One. 2013;8(5):e65442.
17. Montgomery P, Mayo-Wilson E, Hopewell S, Macdonald G, Moher D, Grant S. Developing a reporting guideline for social and psychological intervention trials. Am J Public Health. 2013;103(10):1741-6.
18. Naleppa MJ, Cagle JG. Treatment fidelity in social work intervention research: a review of published studies. Res Soc Work Pract. 2010;20:674-81.
19. Roen K, Arai L, Roberts H, Popay J. Extending systematic reviews to include evidence on implementation: methodological work on a review of community-based initiatives to prevent injuries. Soc Sci Med. 2006;63(4):1060-71.
20. Armstrong RWE, Jackson N, Oliver S, Popay J, Shepherd J, Petticrew M, et al. Guidelines for systematic reviews of health promotion and public health interventions. Version 2. Australia: Melbourne University; 2007.
21. Arai L, Roen K, Roberts H, Popay J. It might work in Oklahoma but will it work in Oakhampton? Context and implementation in the effectiveness literature on domestic smoke detectors. Inj Prev. 2005;11(3):148-51.
22. Montgomery P, Grant S, Hopewell S, Macdonald G, Moher D, Michie S, et al. Protocol for CONSORT-SPI: an extension for social and psychological interventions. Implement Sci. 2013;8(1):99.
23. Montgomery P, Underhill K, Gardner F, Operario D, Mayo-Wilson E. The Oxford Implementation Index: a new tool for incorporating implementation data into systematic reviews and meta-analyses. J Clin Epidemiol. 2013;66(8):874-82.
24. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348.
25. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655.
26. Metz A, Albers B. What does it take? How federal initiatives can support the implementation of evidence-based programs to improve outcomes for adolescents. J Adolesc Health. 2014;54(3 Suppl):S92-6.
27. Institute of Medicine. Accelerating progress in obesity prevention: solving the weight of the nation. Washington DC: National Academies Press; 2012.
28. Margolis AL, Roper AY. Practical experience from the office of adolescent health's large scale implementation of an evidence-based teen pregnancy prevention program. J Adolesc Health. 2014;54(3 Suppl):S10-4.
29. Cross WF, West JC. Examining implementer fidelity: conceptualizing and measuring adherence and competence. J Child Serv. 2011;6(1):18-33.
30. Chen H-T. Practical program evaluation. Assessing and improving planning, implementation and effectiveness. Thousand Oaks, CA: Sage Publications; 2005.
31. Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2:40.
32. Century J, Rudnick M, Freeman C. A framework for measuring fidelity of implementation: a foundation for shared language and accumulation of knowledge. Am J Eval. 2010;31(2):199-218.
33. Mowbray CT, Holter MC, Teague GB, Bybee D. Fidelity criteria: development, measurement, and validation. Am J Eval. 2003;24(3):315-40.
34. Dariotis JK, Bumbarger BK, Duncan L, Greenberg MT. How do implementation efforts relate to program adherence? Examining the role of organizational, implementer, and program factors. J Community Psychol. 2008;36(6):744-60.
35. Littell J, Forsythe B, Popa M, Austin S. Multisystemic therapy for social, emotional, and behavioral problems in youth aged 10-17. In: Campbell Systematic Reviews. 2005.
36. Resnicow K, Baranowski T, Ahluwalia JS, Braithwaite RL. Cultural sensitivity in public health: defined and demystified. Ethn Dis. 1999;9(1):10-21.
37. Lasker RD, Weiss ES, Miller R. Partnership synergy: a practical framework for studying and strengthening the collaborative advantage. Milbank Q. 2001;79(2):179-205. III-IV.
38. Dane AV, Schneider BH. Program integrity in primary and early secondary prevention: are implementation effects out of control? Clin Psychol Rev. 1998;18(1):23-45.
39. Rogers P. Using programme theory to evaluate complicated and complex aspects of interventions. Eval Health Prof. 2000;14(1):29-48.
40. Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2003;18(2):237-56.
41. Moncher FJ, Prinz RJ. Treatment fidelity in outcome studies. Clin Psychol Rev. 1991;11:247-66.
42. Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract. 2005;6(2):134-47.
43. Eisenstein EL, Lobach DL, Montgomery P, Kawamoto K, Anstrom KJ. Evaluating implementation fidelity in health information technology interventions. In: American Medical Informatics Association 2007 Conference Proceedings Biomedical and Health Informatics: from Foundations to Applications and Policy: 2007; 10-14 November, Chicago, Illinois. 2007. p. 211-5.
44. Greenberg MT, Domitrovich CE, Graczyk P, Zins J. Report to the Center for Mental Health Services. A conceptual model for the implementation of school-based preventive interventions: Implications for research, practice and policy. 2001.
45. Weiss CH. Which links in which theories shall we evaluate? New Dir Eval. 2000;87:35-45.
46. Lipsey MA. What can you build with thousands of bricks? Musings on the cumulation of knowledge in program evaluation. N Dir Eval. 1997;76(Winter):7-23.
47. Becker BJ. Examining theoretical models through research synthesis. The benefits of model-driven meta-analysis. Eval Health Prof. 2001;24(2):190-217.
48. Von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61:344-9.
49. Armstrong R, Waters E, Moore L, Riggs E, Cuervo LG, Lumbiganon P, et al. Improving the reporting of public health intervention research: advancing TREND and CONSORT. J Public Health (Oxf). 2008;30(1):103-9.
50. Des Jarlais DC, Lyles C, Crepaz N. The TREND group: improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. Am J Public Health. 2004;94:361-6.
51. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMC Medicine. 2010;8:18.
52. Glasziou P, Meats E, Heneghan C, Shepperd S. What is missing from descriptions of treatment in trials and reviews? BMJ. 2008;336:1472.
53. Cochrane Health Promotion and Public Health Field Victoria Health Promotion Foundation: Systematic reviews of health promotion and public health interventions. Sydney; No date.
54. Higgins J, Green S, editors. Cochrane handbook for systematic reviews of interventions. Great Britain: Antony Rowe Ltd, Chippenham, Wiltshire; 2008.
55. Promoting health after sifting the evidence: tools http://www.eppi.ioe.ac.uk/cms/Default.aspx?tabid=2370&langu
56. Society for Prevention Research: Standards of evidence. Criteria for efficacy, effectiveness and dissemination. http://www.preventionresearch.org/StandardsofEvidencebook.pdf; No date.
57. Zaza S, Wright-De Agüero LK, Briss PA, Truman BI, Hopkins DP. Data collection instrument and procedure for systematic reviews in the guide to community preventive services. Am J Prev Med. 2000;18(1S):44-74.
58. Thomas JBJ, Graziosi S. EPPI-Reviewer 4.0: software for research synthesis. In: EPPI-Centre Software. London: Social Science Research Unit, Institute of Education; 2010.
59. Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360-3.
60. Abramson JH. PAIRSetc manual version 3.32. 2013.
61. Cicchetti D, Sparrow SS. Developing criteria for establishing inter-rater reliability of specific items: applications to assessment of adaptive behavior. Am J Ment Defic. 1981;86:127-37.
62. Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005;85(3):257-68.
63. Public Health Resource Unit CASP. Critical appraisal tool for qualitative studies. 2009.
64. Pearson A, Field J, Jordan Z. Appendix 2: critical appraisal tools. Evidence-Based Clinical Practice in Nursing and Health Care: Assimilating research, experience and expertise. 2009;177-182.
65. Effective Practice and Organisation of Care (EPOC): EPOC data collection checklist. EPOC resources for review authors. 2009. Available at: http://epoc.cochrane.org/epoc-specific-resources-review-authors.
66. Farrington D, Ttofi M. School-based programs to reduce bullying and victimization. In: Campbell Systematic Reviews. 2009.
67. Petrosino A, Buehler J, Turpin-Petrosino C. Scared straight and other juvenile awareness programs for preventing juvenile deliquency. In: Campbell Systematic Reviews. 2007.
68. Coren E, Barlow J. Individual and group based parenting for improving psychosocial outcomes for teenage parents and their children. In: Campbell Systematic Reviews. 2004.
69. Wilson SJ, Lipsey MW. The effects of school-based social information processing interventions on aggressive behavior: part I: universal programs. In: Campbell Systematic Reviews. 2006.
70. Wilson SJ, Lipsey MW. The effects of school-based social information processing interventions on aggressive behavior: part II: selected/indicated pull-out programs. In: Campbell Systematic Reviews. 2008.
71. Kristjansson E, Farmer AP, Greenhalgh T, Janzen L, Krasevec J, MacDonald B, et al. School feeding for improving the physical and psychosocial health of disadvantaged students. In: Campbell Systematic Reviews. 2007.
72. Mayo-Wilson E, Dennis J, Montgomery P. Personal assistance for children and adolescents (0-18) with intellectual impairments. In: Campbell Systematic Reviews. 2008.
73. Armelius B-A, Andreassen TH. Cognitive-behavioral treatment for antisocial behavior in youth in residential treatment. In: Campbell Systematic Reviews. 2007.
74. Winokur M, Holtan A, Valentine D. Kinship care for the safety, permanency, and well-being of children removed from the home for maltreatment. In: Campbell Systematic Reviews. 2009.