Assessing organizational implementation context in the education sector: Confirmatory factor analysis of measures of implementation leadership, climate, and citizenship

Implementation Science

Volumn 13, Issue 1, 2018, Pages

  Lyon, Aaron R ^a   Cook, Clayton R ^b   Brown, Eric C ^c   Locke, Jill ^a   Davis, Chayna ^a   Ehrhart, Mark ^d   Aarons, Gregory A ^e,f  

^a UNIVERSITY OF WASHINGTON   (United States)

^b UNIVERSITY OF MINNESOTA   (United States)

^c UNIVERSITY OF MIAMI MILLER SCHOOL OF MEDICINE   (United States)

^d UNIVERSITY OF CENTRAL FLORIDA   (United States)

^e UNIVERSITY OF CALIFORNIA   (United States)

^f CHILD AND ADOLESCENT SERVICES RESEARCH CENTER   (United States)

Author keywords

Education; Implementation citizenship; Implementation climate; Implementation leadership; Reliability; Schools; Structural validity

Indexed keywords

ADULT; CHILD; CLIMATE; EVIDENCE BASED PRACTICE; FACTOR ANALYSIS; FEMALE; HUMAN; LEADERSHIP; MALE; ORGANIZATION; ORGANIZATION AND MANAGEMENT; ORGANIZATIONAL CULTURE; REPRODUCIBILITY; YOUNG ADULT;

ADULT; CHILD; CLIMATE; EVIDENCE-BASED PRACTICE; FACTOR ANALYSIS, STATISTICAL; FEMALE; HUMANS; LEADERSHIP; MALE; ORGANIZATIONAL CULTURE; ORGANIZATIONAL INNOVATION; REPRODUCIBILITY OF RESULTS; YOUNG ADULT;

EID: 85040461028     PISSN: None     EISSN: 17485908     Source Type: Journal    
DOI: 10.1186/s13012-017-0705-6     Document Type: Article

References (85)

1. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Admin Pol Ment Health. 2011;38:4-23.
2. Beidas RS, Kendall PC. Training therapists in evidence-based practice: a critical review of studies from a systems-contextual perspective. Clin Psychol Sci Pract. 2010;17:1-30.
3. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.
4. Glisson C. Assessing and changing organizational culture and climate for effective services. Res Soc Work Pract. 2007;17:736-47.
5. Lau R, Stevenson F, Ong BN, Dziedzic K, Treweek S, Eldridge S, et al. Achieving change in primary care-causes of the evidence to practice gap: systematic reviews of reviews. Implement Sci. 2016;11:40.
6. Diebold C, Miller G, Gensheimer L, Mondschein E, Ohmart H. Building an intervention: a theoretical and practical infrastructure for planning, implementing, and evaluating a metropolitan-wide school-to-career initiative. J Educ Psychol Consult. 2000;11:147-72.
7. Gonzalez JE, Nelson JR, Gutkin TB, Shwery CS. Teacher resistance to school-based consultation with school psychologists: a survey of teacher perceptions. J Emot Behav Disord. 2004;12:30-7.
8. Odom SL, Cox AW, Brock ME, Others. Implementation science, professional development, and autism spectrum disorders. Except Child. 2013;79:233-51.
9. Cummings GG, Estabrooks CA, Midodzi WK, Wallin L, Hayduk L. Influence of organizational characteristics and context on research utilization. Nurs Res. 2007;56:S24-39.
10. Henggeler SW, Chapman JE, Rowland MD, Halliday-Boykins CA, Randall J, Shackelford J, et al. Statewide adoption and initial implementation of contingency management for substance-abusing adolescents. J Consult Clin Psychol. 2008;76:556.
11. Scott SD, Estabrooks CA, Allen M, Pollock C. A context of uncertainty: how context shapes nurses' research utilization behaviors. Qual Health Res. 2008;18:347-57.
12. McCarthy P, Kerman B. Inside the belly of the beast: how bad systems trump good programs. Adm Policy Ment Health Ment Health Serv Res. 2010;37:167-72.
13. Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:22.
14. Ehrhart MG, Schneider B, Macey WH. Organizational climate and culture: an introduction to theory, research, and practice. Routledge; 2013. https://books.google.com/books?hl=en&lr=&id=KuMkAgAAQBAJ&oi=fnd&pg=PP1&dq=Organizational+climate+and+culture:+An+introduction+to+theory,+research,+and+practice.+Routledge.&ots=86sm9Sbehz&sig=T5EIoeG39fDVeO6pTvPkRp5yIsQ.
15. Aarons GA, Ehrhart MG, Farahnak LR, Sklar M. Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation. Annu Rev Public Health. 2014;35:255-74.
16. Blase J, Blase J. Effective instructional leadership: teachers' perspectives on how principals promote teaching and learning in schools. J Educ Adm. 2000;38:130-41.
17. Goldring E, Huff J, May H, Camburn E. School context and individual characteristics: what influences principal practice? J Educ Adm. 2008;46:332-52.
18. Schein EH. Organizational culture and leadership. John Wiley & Sons; 2010. https://books.google.com/books?hl=en&lr=&id=DlGhlT34jCUC&oi=fnd&pg=PR9&dq=Organizational+culture+and+leadership+(Vol.+2).2010&ots=-cv35nTgEW&sig=HQAutAdW3cKwucCANkkUOfwkPlM.
19. Hong Y, Liao H, Hu J, Jiang K. Missing link in the service profit chain: a meta-analytic review of the antecedents, consequences, and moderators of service climate. J Appl Psychol. 2013;98:237-67.
20. Ehrhart MG, Aarons GA, Torres EM, Finn NK, Roesch SC. Implementation in child welfare: leadership, climate, attitudes, and citizenship behavior. Anaheim, CA: Poster presented at the 31st annual conference of the Society for Industrial and Organizational Psychology; 2016.
21. Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). Implement Sci. 2014;9:157.
22. Jacobs SR, Weiner BJ, Bunger AC. Context matters: measuring implementation climate among individuals and groups. Implement Sci. 2014;9:46.
23. Schneider B, Ehrhart MG, Macey WH. Organizational climate and culture. Annu Rev Psychol. 2013;64:361-88.
24. Organ DW, Podsakoff PM, MacKenzie SB. Organizational citizenship behavior: its nature, antecedents, and consequences. Beverly Hills: Sage Publications; 2005.
25. Ehrhart MG, Aarons GA, Farahnak LR. Going above and beyond for implementation: the development and validity testing of the Implementation Citizenship Behavior Scale (ICBS). Implement Sci. 2015;10:65.
26. Glasgow RE, Riley WT. Pragmatic measures: what they are and why we need them. Am J Prev Med. 2013;45:237-43.
27. Aarons GA, Ehrhart MG, Farahnak LR, Hurlburt MS. Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implement Sci. 2015;10:11.
28. Finn NK, Torres EM, Ehrhart MG, Roesch SC, Aarons GA. Cross-validation of the Implementation Leadership Scale (ILS) in child welfare service organizations. Child Maltreat. 2016;21:250-5.
29. McKleroy VS, Galbraith JS, Cummings B, Jones P, Harshbarger C, Collins C, et al. Adapting evidence-based behavioral interventions for new settings and target populations. AIDS Educ Prev. 2006;18(4 Suppl A):59-73.
30. Bernal G, Bonilla J, Bellido C. Ecological validity and cultural sensitivity for outcome research: issues for the cultural adaptation and development of psychosocial treatments with Hispanics. J Abnorm Child Psychol. 1995;23:67-82.
31. Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Saf Health Care. 1998;7:149-58.
32. Costello EJ, He J, Sampson NA, Kessler RC, Merikangas KR. Services for adolescents with psychiatric disorders: 12-month data from the National Comorbidity Survey-Adolescent. Psychiatr Serv. 2014;65:359-66.
33. Farmer EMZ, Burns BJ, Phillips SD, Angold A, Costello EJ. Pathways into and through mental health services for children and adolescents. PS. 2003;54:60-6.
34. Langer DA, Wood JJ, Wood PA, Garland AF, Landsverk J, Hough RL. Mental health service use in schools and non-school-based outpatient settings: comparing predictors of service use. Sch Ment Heal. 2015;7:161-73.
35. Lyon AR, Ludwig KA, Vander Stoep A, Gudmundsen G, McCauley E. Patterns and predictors of mental healthcare utilization in schools and other service sectors among adolescents at risk for depression. Sch Ment Heal. 2013;5:155-65.
36. Merikangas KR, He J, Burstein M, Swendsen J, Avenevoli S, Case B, et al. Service utilization for lifetime mental disorders in U.S. adolescents: results of the National Comorbidity Survey-Adolescent Supplement (NCS-A). J Am Acad Child Adolesc Psychiatry. 2011;50:32-45.
37. Blount A, Schoenbaum M, Kathol R, Rollman BL, Thomas M, O'Donohue W, et al. The economics of behavioral health services in medical settings: a summary of the evidence. Prof Psychol. 2007;38:290-7.
38. Bruns EJ, Duong MT, Lyon AR, Pullmann MD, Cook CR, Cheney D, et al. Fostering SMART partnerships to develop an effective continuum of behavioral health services and supports in schools. Am J Orthopsychiatry. 2016;86:156-70.
39. Owens JS, Lyon AR, Brandt NE, Warner CM, Nadeem E, Spiel C, et al. Implementation science in school mental health: key constructs in a developing research agenda. Sch Ment Heal. 2014;6:99-111.
40. Forman SG, Olin SS, Hoagwood KE, Crowe M, Saka N. Evidence-based interventions in schools: developers' views of implementation barriers and facilitators. Sch Ment Heal. 2009;1:26.
41. Rones M, Hoagwood K. School-based mental health services: a research review. Clin Child Fam Psychol Rev. 2000;3:223-41.
42. Gottfredson DC, Gottfredson GD. Quality of school-based prevention programs: results from a national survey. J Res Crime Delinq. 2002;39:3-35.
43. Domitrovich CE, Bradshaw CP, Poduska JM, Hoagwood K, Buckley JA, Olin S, et al. Maximizing the implementation quality of evidence-based preventive interventions in schools: a conceptual framework. Adv School Ment Health Promot. 2008;1:6-28.
44. Fallon LM, Collier-Meek MA, Sanetti LMH, Feinberg AB, Kratochwill TR. Implementation planning to promote parents' treatment integrity of behavioral interventions for children with autism. J Educ Psychol Consult. 2016;26:87-109.
45. Elias MJ, O'Brien U, Weisberg P. Transformative leadership for social-emotional learning. Princ Leadersh. 2006;7:10-3.
46. Louis KS, Murphy J, Smylie M. Caring leadership in schools: findings from exploratory analyses. Educ Adm Q. 2016;52:310-48.
47. Wahlstrom KL, Louis KS. How teachers experience principal leadership: the roles of professional community, trust, efficacy, and shared responsibility. Educ Adm Q. 2008;44:458-95.
48. Elliott SN, Clifford M. Principal assessment: leadership behaviors known to influence schools and the learning of all students (document no. LS-5). Retrieved from University of Florida, Collaboration for Effective Educator, Development, Accountability, and Reform Center website: http://ceedar.education.ufl.edu/ls-5_final_09-26-14/.
49. Murphy JF, Goldring EB, Cravens XC, Elliott SN, Porter AC. The Vanderbilt assessment of leadership in education: measuring learning-centered leadership. J of East China Normal University. 2007;29:1-10. https://www.researchgate.net/profile/Joseph_Murphy6/publication/254300402_The_Vanderbilt_Assessment_of_Leadership_in_Education_Measuring_Learning-Centered_Leadership/links/0deec53bed3af323c1000000.pdf.
50. Hallinger P. Principal instructional management rating scale. Sarasota, FL: Leading Development Associates; 1990.
51. Hallinger P. Leadership for learning: lessons from 40 years of empirical research. J Educ Admin. 2011;49:125-42.
52. National School Climate Council. The school climate challenge: narrowing the gap between school climate research and school climate policy, practice guidelines and teacher education policy. 2007. http://www.schoolclimate.org.
53. Furlong MJ, You S, Renshaw TL, Smith DC, O'Malley MD. Preliminary development and validation of the social and emotional health survey for secondary school students. Soc Indic Res. 2014;117:1011-32.
54. Bogler R, Somech A. Influence of teacher empowerment on teachers' organizational commitment, professional commitment and organizational citizenship behavior in schools. Teach Teach Educ. 2004;20:277-89.
55. DiPaola MF, Tarter CJ, Hoy WK. Measuring organizational citizenship in schools: the OCB scale. Educat Leadership Reform. 2005;4:319-41.
56. Haider S, Fernandez-Ortiz A, de Pablos Heredero C. Organizational citizenship behavior and implementation of evidence-based practice: moderating role of senior management's support. Health Sys. 2017;6:226-41.
57. Mashburn AJ, Downer JT, Rivers SE, Brackett MA, Martinez A. Improving the power of an efficacy study of a social and emotional learning program: application of generalizability theory to the measurement of classroom-level outcomes. Prev Sci. 2014;15:146-55.
58. Hambleton RK. Guidelines for adapting educational and psychological tests. 1996. http://eric.ed.gov/?id=ED399291.
59. Hambleton RK, Merenda PF, Spielberger CD. Adapting educational and psychological tests for cross-cultural assessment. Psychology Press; 2004. https://books.google.com/books?hl=en&lr=&id=po15AgAAQBAJ&oi=fnd&pg=PP1&dq=Adapting+educational+and+psychological+tests+for+cross-cultural+assessment&ots=yw4xiz3Tqn&sig=gkT46GXzc4Kig5g5LXp3QegJu0M.
60. Rea LM, Parker RA. Designing and conducting survey research: a comprehensive guide. John Wiley & Sons; 2014. https://books.google.com/books?hl=en&lr=&id=Ub8BBAAAQBAJ&oi=fnd&pg=PA201&dq=Designing+and+Conducting+Survey+Research:+A+Comprehensive+Guide&ots=iwJyv1FoyI&sig=gQGdbCZJOETFWY0_IbGA7SvPXEM.
61. Muthén B, Asparouhov T. Latent variable analysis with categorical outcomes: multiple-group and growth modeling in Mplus. Mplus Web Notes. 2002;4:1-22.
62. Muthén LK, Muthén BO. Mplus Version 7 user's guide. Los Angeles, CA: Muthén & Muthén; 2006.
63. Bentler PM. Comparative fit indexes in structural models. Psychol Bull. 1990;107:238.
64. Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model Multidiscip J. 1999;6:1-55.
65. Tucker LR, Lewis C. A reliability coefficient for maximum likelihood factor analysis. Psychometrika. 1973;38:1-10.
66. Browne MW, Cudeck R. Alternative ways of assessing model fit. In: Bollen KA, Long JS, editors. Testing Structral Equation Models. Newbury Park, CA: SAGE; 1993. p. 136-62. https://books.google.com/books?hl=en&lr=&id=FvIxxeYDLx4C&oi=fnd&pg=PA136&dq=Alternative+ways+of+assessing+model+fit.+&ots=_L2FG2SzzL&sig=uqUu5DK-Kx7SrkZnavjjbkZ5gvI.
67. Tabachnick BG, Fidell LS. Using multivariate statistics. 5th ed. Needham Heights, MA, USA: Allyn & Bacon, Inc.; 2007.
68. Hallinger P, Heck RH. Reassessing the principal's role in school effectiveness: a review of empirical research, 1980-1995. Educ Adm Q. 1996;32:5-44.
69. Aarons GA, Ehrhart MG, Moullin JC, Torres EM, Green AE. Testing the leadership and organizational change for implementation (LOCI) intervention in substance abuse treatment: a cluster randomized trial study protocol. Implement Sci. 2017;12:29.
70. Bickman L, Goldring E, De Andrade AR, Breda C, Goff P. Improving principal leadership through feedback and coaching. Society for Research on Educational Effectiveness. 2012. http://eric.ed.gov/?id=ED530123.
71. Smylie MA, Murphy J, Louis KS. Caring school leadership: a multidisciplinary, cross-occupational model. Am J Educ. 2016;123:1-35.
72. Johnson B, Stevens JJ. Student achievement and elementary teachers' perceptions of school climate. Learn Environ Res. 2006;9:111-22.
73. Thapa A, Cohen J, Guffey S, Higgins-D'Alessandro A. A review of school climate research. Rev Educ Res. 2013;83:357-85.
74. Steele JL, Murnane RJ, Willett JB. Do financial incentives help low-performing schools attract and keep academically talented teachers? Evidence from California. J Policy Anal Manage. 2010;29:451-78.
75. Balch R, Springer MG. Performance pay, test scores, and student learning objectives. Econ Educ Rev. 2015;44:114-25.
76. Weiner BJ, Belden CM, Bergmire DM, Johnston M. The meaning and measurement of implementation climate. Implement Sci. 2011;6:78.
77. Stoll L, Bolam R, McMahon A, Wallace M, Thomas S. Professional learning communities: a review of the literature. J Educ Change. 2006;7:221-58.
78. Vescio V, Ross D, Adams A. A review of research on the impact of professional learning communities on teaching practice and student learning. Teach Teach Educ. 2008;24:80-91.
79. Atkins MS, Frazier SL, Leathers SJ, Graczyk PA, Talbott E, Jakobsons L, et al. Teacher key opinion leaders and mental health consultation in low-income urban schools. J Consult Clin Psychol. 2008;76:905-8.
80. Ginsberg A, Abrahamson E. Champions of change and strategic shifts: the role of internal and external change advocates. J Manag Stud. 1991;28:173-90.
81. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Admin Pol Ment Health. 2011;38:65-76.
82. Wolf EJ, Harrington KM, Clark SL, Miller MW. Sample size requirements for structural equation models: an evaluation of power, bias, and solution propriety. Educ Psychol Meas. 2013;73:913-34.
83. National Center on Response to Intervention. Essential components of RTI: a closer look at response to intervention. Washington, DC: U.S. Department of Education, Office of Special Education Programs, National Center on Response to Intervention: National Center on Response to Intervention; 2010.
84. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implement Sci. 2015;10:21.
85. Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the expert recommendations for implementing change (ERIC) study. Implement Sci. 2015;10:109.