Strategies for Improving Power in School-Randomized Studies of Professional Development

Evaluation Review

Volumn 37, Issue 6, 2013, Pages 520-554

  Kelcey, Ben ^a   Phelps, Geoffrey ^b  

^a UNIVERSITY OF CINCINNATI   (United States)

^b EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

cluster randomized trials; covariance adjustment; design of experiments; group randomized designs; power; precision; pretest posttest; professional development; teacher development; teacher knowledge

Indexed keywords

EID: 84903180470     PISSN: 0193841X     EISSN: 15523926     Source Type: Journal    
DOI: 10.1177/0193841X14528906     Document Type: Article

References (62)

1. Ball D. L.,Bass H.Proceedings of the 2002 Annual Meeting of the Canadian Mathematics Education Study Group. Davis B.Simmt E., ed. Edmonton, Alberta: CMESG/GDEDM; 2003:3-14.
2. Ball D. L.,Thames M.,Phelps G."Content Knowledge for Teaching: What Makes It Special?".Journal of Teacher Education. 2008;59:389-407.
3. Barrett N.,Butler J. S.,Toma E. F."Do Less Effective Teachers Choose Professional Development Does It Matter?".Evaluation Review. 2012;36:346-74.
4. Bloom H.Learning More from Social Experiments: Evolving Analytic Approaches. Bloom H. S., ed. New York: Russell Sage Foundation; 2005:115-72.
5. Bloom H.,Hill C.,Black A.,Lipsey M."Performance Trajectories and Performance Gaps as Achievement Effect-size Benchmarks for Educational Interventions.".Journal of Research on Educational Effectiveness. 2008;1:289-328.
6. Borko H."Professional Development and Teacher Learning: Mapping the Terrain.".Educational Researcher. 2004;33:3-15.
7. Bos C. S.,Mather N.,Narr R. F.,Babur N."Interactive, Collaborative Professional Development in Early Literacy Instruction: Supporting the Balancing Act.".Learning Disabilities Research & Practice. 1999;14:227-38.
8. Butler D.,Lauscher H.,Jarvis-Selinger S.,Beckingham B."Collaboration and Self-regulation in Teachers' Professional Development.".Teaching and Teacher Education. 2004;20:435-55.
9. Carlisle J.,Kelcey B.,Rowan B.,Phelps G."Teachers' Knowledge about Early Reading: Effects on Students' Gains in Reading Achievement.".Journal for Research on Educational Effectiveness. 2011;4:289-321.
10. Chen H.,Rossi P."Evaluating with Sense: The Theory-driven Approach.".Evaluation Review. 1983;7:283-302.
11. Cohen J.Statistical Power Analysis for the Behavioral Sciences. 1988. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum.Hillsdale, NJ: Lawrence Erlbaum; 1988:.
12. Cook T.,Campbell D.Quasi-experimentation. New York: Rand McNally; 1979:.
13. Cook T.,Hunt H.,Murphy R."Comer's School Development Program in Chicago: A Theory-based Evaluation.".American Educational Research Journal. 2000;37:535-97.
14. Cunningham A. E.,Perry K. E.,Stanovich K. E.,Stanovich P. J."Disciplinary Knowledge of K-3 Teachers and Their Knowledge Calibration in the Domain of Early Literacy.".Annals of Dyslexia. 2004;54:139-67.
15. Darling-Hammond L.,Wei R. C.,Andree A.,Richardson N.,Orpanos S."State of the Profession: Study Measures Status of Professional Development.".Journal of Staff Development. 2009;30:42-50.
16. Desimone L."Improving Impact Studies of Teachers' Professional Development: Toward Better Conceptualizations and Measures.".Educational Researcher. 2009;38:181-99.
17. Donner A.,Koval J. J."Design Considerations in the Estimation of Intraclass Correlation.".Annals of Human Genetics. 1982;46:271-77.
18. DuFour R.,Eaker R.,R DuFour.Closing the knowing-doing gap. On Common Ground: The Power of Professional Learning Communities. IN: The National Education Service. 2005;:.
19. Eaker R.,DuFour R.,DuFour R.Getting Started: Reculturing Schools to Become Professional Learning Communities. Bloomington, IN: National Education Services; 2002:.
20. Franke M.,Carpenter T.,Levi L.,Fennema E."Capturing Teachers' Generative Change: A Follow-up Study of Professional Development in Mathematics.".American Educational Research Journal. 2001;38:653-89.
21. Garet M. S.,Porter A. C.,Desimone L. M.,Birman B.,Yoon K. S."What Makes Professional Development Effective? Analysis of a National Sample of Teachers.".American Educational Research Journal. 2001;38:915-45.
22. Garet M. S.,Wayne A. J.,Stancavage F.,Taylor J.,Eaton M.,Walters K.,Song M.,Brown S.,Hurlburt S.,Zhu P.,Sepanik S.,Doolittle F.Middle School Mathematics Professional Development Impact Study: Findings after the Second Year of Implementation. Washington, DC: National Center for Education Evaluation and Regional Assistance; 2011:.
23. Garner C. L.,Raudenbush S. W."Neighborhood Effects on Educational Attainment: A Multilevel Analysis.".Sociology of Education. 1991;64:251-62.
24. Hedges L.,Hedberg E."Intraclass Correlation Values for Planning Group-randomized Trials in Education.".Educational Evaluation and Policy Analysis. 2007;29:60-87.
25. Hill H. C.,Ball D. L."Learning Mathematics for Teaching: Results from California's Mathematics Professional Development Institutes.".Journal of Research in Mathematics Education. 2004;35:330-51.
26. Hill C.,Bloom H.,Black A.,Lipsey M."Empirical Benchmarks for Interpreting Effect Sizes in Research.".Child Development Perspectives. 2008;2:172-77.
27. Hill H. C.,Rowan B.,Ball D."Effects of Teachers' Mathematical Knowledge for Teaching on Student Achievement.".American Educational Research Journal. 2005;42:371-406.
28. Hill H. C.,Schilling S.,Ball D. L."Developing Measures of Teachers' Mathematics Knowledge for Teaching.".Elementary School Journal. 2004;105:11-30.
29. Hill H. C.,Umland K.,Litke E.,Kapitula L. R."Teacher Quality and Quality Teaching: Examining the Relationship of a Teacher Assessment to Practice.".American Journal of Education. 2012;118:489-519.
30. Hindin A.,Morocco-Cobb C.,Arwen-Mott E.,Mata-Aguilar C."More Than Just a Group: Teacher Collaboration and Learning in the Workplace.".Teachers and Teaching: Theory and practice. 2007;13:349-76.
31. Hinkley D."Bootstrap Methods.".Journal of the Royal Statistical Society B. 1988;50:321-37.
32. Holland P."Statistics and Causal Inference.".Journal of the American Statistical Associations. 1986;81:945-60.
33. Ingersoll R."Deprofessionalizing the Teaching Profession: The Problem of Out-of-field Teaching.".Educational Horizons. 2001;80:28-31.
34. Research Grants Request for Applications for Awards Beginning in Fiscal Year 2013: CFDA Number 84.305A. Washington, DC: U.S. Department of Education, Institute of Education Sciences; 2012:.
35. Jacob R.,Zhu P.,Bloom H."New Empirical Evidence for the Design of Group Randomized Trials in Education.".Journal of Research on Educational Effectiveness. 2010;3:157-98.
36. Jerald C."All Talk, No Action: Putting an End to Out-of-field Teaching.".The Education Trust. 2002;9:1-14.
37. Kain J.,Singleton K."Equality of Educational Opportunity Revisited.".New England Economic Review. 1996;:.
38. Keaton P.Documentation to the NCES Common Core of Data Local Education Agency Universe Survey Preliminary Directory File: School Year 2011-12 (NCES 2013-322). Washington, DC: U.S. Department of Education, National Center for Education Statistics; 2012:.
39. Kelcey B."Assessing the Effects of Teachers' Reading Knowledge on Students' Achievement Using Multilevel Propensity Score Stratification.".Educational Evaluation & Policy Analysis. 2011;33:458-82.
40. Kelcey B.,Phelps G."Considerations for Designing Group Randomized Trials of Professional Development with Teacher Knowledge Outcomes.".Educational Evaluation and Policy Analysis. 2013;35:370-90.
41. Kennedy M. M.Form and Substance in In-service Teacher Education (Research Monograph No. 13). Arlington, VA: National Science Foundation; 1998:.
42. Krei M. S."Intensifying the Barriers-The Problem of Inequitable Teacher Allocation in Low-income Urban Schools.".Urban Education. 1998;33:71-94.
43. Lipsey M. W.Design Sensitivity: Statistical Power for Experimental Research. Newbury Park, CA: Sage; 1990:.
44. Livesay M. E.,Moore C. A.,Stankay R. J.,Waters M. J.,Waff D.,Gentile C. A."Collaborative Learning Communities: Building Leadership in a High School English Department.".English Journal. 2005;95:16-18.
45. "Gathering Feedback for Teaching: Combining High-quality Observations with Student Surveys and Achievement Gains.". 2012;:.
46. Nye B.,Konstantopoulous S.,Hedges L. V."How Large are Teacher Effects?".Education Evaluation and Policy Analysis. 2004;26:237-57.
47. RaudenbushS.2003. Designing Field Trials of Educational Interventions. Accessed April 4, 2009.http://www.ssicentral.com/hlm/techdocs/DRDC.pdf.
48. Raudenbush S. W."Statistical Analysis and Optimal Design for Cluster Randomized Trials.".Psychological Methods. 1997;2:173-85.
49. Raudenbush S. W.,Bryk A. S.Hierarchical Linear Models: Applications and Data Analysis Methods. 2002. Hierarchical Linear Models: Applications and Data Analysis Methods. 2nd ed. Thousand Oaks, CA: Sage.Thousand Oaks, CA: Sage; 2002:.
50. Raudenbush S. W.,Martinez A.,Spybrook J."Strategies for Improving Precision in Group-randomized Experiments.".Educational Evaluation and Policy Analysis. 2007;29:5-29.
51. Rhoads C."The Implication of "Contamination" for Experimental Design in Education.".Journal of Educational and Behavioral Statistics. 2011;36:76-104.
52. Robbins P.,Alvy H. B.The Principal's Companion: Strategies and Hints to Make the Job Easier. Thousand Oaks, CA: Corwin; 2003:.
53. Rossi A.,Lipsey M.,Freeman H.Evaluation: A Systematic Approach. 2004. Evaluation: A Systematic Approach. 7th ed. Thousand Oaks, CA: Sage.Thousand Oaks, CA: Sage; 2004:.
54. Schmoker M.On Common Ground. Eaker R.DuFour R., ed. Bloomington, IN: National Education Services; 2005:135-54.
55. Scribner J.,Sawyer R.,Watson S.,Myers V."Teacher Teams and Distributed Leadership: A Study of Group Discourse and Collaboration.".Educational Administration Quarterly. 2007;43:67-100.
56. Smith P. S.,Banilower E. B."Measuring Teachers' Knowledge for Teaching Force and Motion Concepts.". 2006;:.
57. Spear-Swerling L.,Brucker P. O."Teacher-education Students' Reading Abilities and Their Knowledge about Word Structure.".Teacher Education and Special Education. 2006;29:116-26.
58. Spybrook J.,Raudenbush S. W."An Examination of the Precision and Technical Accuracy of the First Wave of Group Randomized Trials Funded by the Institute of Education Sciences.".Educational Evaluation and Policy Analysis. 2009;31:298-318.
59. Teaching at Risk: A Call to Action. New York: The Teaching Commission, The CUNY Graduate Center; 2004:.
60. van der Leeden R.,Busing F. M. T. A.,Meijer E.1997:.
61. Wayne A. J.,Yoon K. S.,Zhu P.,Cronen S.,Garet M. S."Experimenting with Teacher Professional Development: Motives and Methods.".Educational Researcher. 2008;37:469-79.
62. Yoon K. S.,Duncan T.,Lee S. W.-Y.,Scarloss B.,Shapley K.Reviewing the Evidence On How Teacher Professional Development Affects Student Achievement (Issues & Answers Report, No. 033). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest; 2007:.