Critical issues and common pitfalls in designing and conducting impact studies in education: Lessons learned from the what works clearinghouse (phase i)

Educational Evaluation and Policy Analysis

Volumn 32, Issue 3, 2010, Pages 351-371

  Song, Mengli ^a   Herman, Rebecca ^a  

^a AMERICAN INSTITUTES FOR RESEARCH   (United States)

Author keywords

experimental design; program evaluation; research methodology

Indexed keywords

EID: 77958475338     PISSN: 01623737     EISSN: 19351062     Source Type: Journal    
DOI: 10.3102/0162373710373389     Document Type: Article

References (71)

1. Aitkin, M., & Longford, N. (1986). Statistical modeling issues in school effectiveness studies (with discussion). Journal of the Royal Statistical Society, A(149), 1-43.
2. Altman, D.G., Schulz, K.F., Moher, D., Egger, M., Davidoff, F., Elbourne, D.,et al.(2001). The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Annals of Internal Medicine, 134(8), 663-694.
3. American Educational Research Association. (2006). Standards for reporting on empirical social science research in AERA publications. Educational Researcher, 35(6), 33-40.
4. American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
5. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B (Methodological), 57(1), 289-300.
6. Benjamini, Y., Hochberg, Y., & Kling, Y. (1993). False discovery rate control in pairwise comparisons (Working Paper 93-2). Tel Aviv: Department of Statistics and O.R., Tel Aviv University.
7. Benjamini, Y., Hochberg, Y., & Kling, Y. (1997). False discovery rate control in multiple hypothesis testing using dependent test statistics (Research Paper 97-1). Tel Aviv: Department of Statistics and O.R., Tel Aviv University.
8. Bloom, H.S. (1995). Minimum detectable effects: A simple way to report the statistical power. Evaluation Review, 19(5), 547-556.
9. Bloom, H.S. (2005). Randomizing groups to evaluate place-based programs. In H. S. Bloom (Ed.), Learning more from social experiments: Evolving analytic approaches (pp. 115-172). New York: Russell Sage Foundation.
10. Bloom, H.S., Michalopoulos, C., & Hill, C.J. (2005). Using experiments to assess nonexperimental comparison-group methods for measuring program effects. In H. S. Bloom (Ed.), Learning more from social experiments (pp. 173-235). New York: Russell Sage Foundation.
11. Bloom, H.S., Richburg-Hayes, L., & Black, A.R. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29(1), 30-59.
12. Bonferroni, C.E. (1935). Il calcolo delle assicurazioni su gruppi di teste. In Studi in Onore del Professore Salvatore Ortu Carboni (pp. 13-60). Rome: Bardi.
13. Coalition for Evidence-Based Policy. (2005). Reporting the results of your study: A user-friendly guide for evaluators of educational programs and practices. Retrieved January 2, 2009, from http://ies.ed.gov/ncee/wwc/pdf/guide_SRF.pdf.
14. Cochran, W.G., & Rubin, D.B. (1973). Controlling bias in observational studies: A review. Sankhya, 35, 417-446.
15. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
16. Cook, T.D., Shadish, W.R., & Wong, V.C. (2008). Three conditions under which experiment and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 724-750.
17. Cortina, J.M. (1993). What is coefficient alpha? An examination of theory and application. Journal of Applied Psychology, 78(1), 98-104.
18. Curran-Everett, D. (2000). Multiple comparisons: Philosophies and illustrations. American Journal of Physiology. Regulatory, Integrative and Comparative Physiology, 279, R1-R8.
19. Donner, A., & Klar, N. (2000). Design and analysis of cluster randomized trials in health research. London: Arnold.
20. Drake, C. (1993). Effects of misspecification of the propensity score on estimators of treatment effect. Biometrics, 49, 1231-1236.
21. Dunnett, C. (1955). A multiple comparisons procedure for comparing several treatments with a control. Journal of the American Statistical Association, 50, 1096-1121.
22. Flay, B.R. (1986). Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Preventive Medicine, 15, 451-474.
23. Flay, B.R., & Collins, L.M. (2005). Historical review of school-based randomized trials for evaluating problem behavior prevention programs. The Annals of the American Academy of Political and Social Science, 599, 147-175.
24. Fraker, T., & Maynard, R. (1987). The adequacy of comparison group designs for evaluations of employment-related programs. Journal of Human Resources, 22(2), 194-227.
25. Glazerman, S., Levy, D.M., & Myers, D. (2003). Nonexperimental versus experimental estimates of earnings impacts. The Annals of the American Academy of Political and Social Science, 589, 63-93.
26. Gu, X.S., & Rosenbaum, P.R. (1993). Comparison of multivariate matching methods: Structures, distances, and algorithms. Journal of Computational and Graphic Statistics, 2(4), 405-420.
27. Gueron, J.M. (2002). The politics of random assignment: Implementing studies and impacting policy. In F. Mosteller and R. Boruch (Eds.), Evidence matters: Randomized trials in education research (pp. 15-49). Washington, DC: Brookings Institution Press.
28. Heckman, J.J., Ichimura, H., & Todd, P. (1997). Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. Review of Economic Studies, 64, 605-654.
29. Hedges, L.V. (2007). Correcting a significance test for clustering. Journal of Educational and Behavioral Statistics, 32(2), 151-179.
30. Hill, J.L., Waldfogel, J., Brooks-Gunn, J., & Han, W.J. (2005). Maternal employment and child development: A fresh look using newer methods. Developmental Psychology, 41(6), 833-850.
31. Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75, 800-803.
32. Hong, G., & Raudenbush, S.W. (2006). Evaluating kindergarten retention policy: A case study of causal inference for multilevel observation data. Journal of American Statistical Association, 101(475), 901-910.
33. Lenth, R.V. (2006-2009). Java Applets for Power and Sample Size [Computer software]. Retrieved February 13, 2010, from http://www.stat.uiowa.edu/~rlenth/Power.
34. Lipsey, M.W., & Wilson, D.B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
35. Little, R.J., & Rubin, D.B. (2001). Causal effects in clinical and epidemiological studies via potential outcomes: Concepts and analytical approaches. Annual Review of Public Health, 21, 121-145.
36. Moher, D., Schulz, K.F., & Altman, D. (2001). The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials. Journal of the American Medical Association, 285(15), 1987-1991.
37. Murray, D.M. (1998). Design and analysis of group-randomized trials (Vol. 27). New York: Oxford University Press.
38. Peduzzi, P., Concato, J., Kemper, E., Holford, T.R., & Feinstein, A. (1996). A simulation of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 99, 1373-1379.
39. Raudenbush, S.W. (1997). Statistical analysis and optimal design in cluster randomized trials. Psychological Methods, 2(2), 173-185.
40. Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). Thousand Oaks, CA: Sage.
41. Raudenbush, S.W., Martinez, A., & Spybrook, J. (2007). Strategies for improving precision in group-randomized experiments. Educational Evaluation and Policy Analysis, 29(1), 5-29.
42. Rosenbaum, P.R. (1999). Choice as an alternative to control in observational studies. Statistical Science, 14(3), 259-301.
43. Rosenbaum, P.R., & Rubin, D.B. (1983). The central role of propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55.
44. Rosenbaum, P.R., & Rubin, D.B. (1985). Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician, 39(1), 33-38.
45. Rubin, D.B. (1973). The use of matched sampling and regression adjustment to remove bias in observational studies. Biometrics, 29, 185-203.
46. Rubin, D.B. (1974). Estimating causal effects of treatments in randomized and non-randomized studies. Journal of Educational Psychology, 66, 688-701.
47. Rubin, D.B. (1979). Using multivariate matched sampling and regression adjustment to control bias in observational studies. Journal of the American Statistical Association, 74, 318-328.
48. Rubin, D.B. (1980). Bias reduction using Mahalanobis metric matching. Biometrics, 36, 293-298.
49. Rubin, D.B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services & Outcomes Research Methodology, 2, 169-188.
50. Rubin, D.B., & Thomas, N. (2000). Combining propensity score matching with additional adjustments for prognostic covariates. Journal of the American Statistical Association, 95, 573-585.
51. Scheffe, H. (1953). A method for judging all contrasts in the analysis of variance. Biometrika, 40, 87-104.
52. Schochet, P.Z. (2008a). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33(1), 62-87.
53. Schochet, P.Z. (2008b). Technical methods report: Guidelines for multiple testing in impact evaluations (NCEE 2008-4018). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved December 15, 2008, from http://ies.ed.gov/ncee/pdf/20084018.pdf.
54. Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin Company.
55. Slavin, R.E. (2008). What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37(1), 5-14.
56. Snijders, T., & Bosker, R. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.
57. Song, M. (2009). What Works Clearinghouse (Phase I) computation tools. Washington, DC: American Institutes for Research. Retrieved April 19, 2010, from http://www.air.org/focus-area/education/index.cfm?fa=viewContent&content_id=749&id=1.
58. Song, M., & Herman, R. (2009). A practical guide on designing and conducting rigorous impact studies in education: Lessons learned from the What Works Clearinghouse (Phase I). Washington, DC: American Institutes for Research. Retrieved April 19, 2010, from http://www.air.org/focus-area/education/index.cfm?fa=viewContent&content_id=753.
59. Spybrook, J., Raudenbush, S.W., Congdon, R., & Martinez, A. (2009). Optimal design for longitudinal and multilevel research: Documentation for the Optimal Design software (Version 2.0). Retrieved April 15, 2009, from http://sitemaker.umich.edu/group-based/files/od-manual-v200-20090722.pdf.
60. Starfield, B. (1977). Efficacy and effectiveness of primary medical care for children. In Harvard Child Health Project, children's medical care needs and treatment: Report of the Harvard Child Health Project. Cambridge, MA: Ballinger.
61. Stuart, E.A., & Rubin, D.B. (2008). Best practices in quasi-experimental designs: Matching methods for causal inference. In J. Osborne (Ed.), Best practices in quantitative social science (pp. 155-176). Thousand Oaks, CA: Sage.
62. Tukey, J. (1949). Comparing individual means in the analysis of variance. Biometrika, 5, 99-114.
63. Valentine, J.C. (2009). Judging the quality of primary research for research synthesis. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 129-146). New York: Russell Sage Foundation.
64. Valentine, J.C., & Cooper, H. (2003). Effect size substantive interpretation guidelines: Issues in the interpretation of effect sizes. Washington, DC: U.S. Department of Education. Retrieved December 15, 2008, from http://ies.ed.gov/ncee/wwc/pdf/essig.pdf.
65. Van Dusen, L., & Worthen, B. (1994). The impact of integrated learning system implementation on student outcomes: Implications for research and evaluation. International Journal of Educational Research, 21, 13-24.
66. Varnell, S.P., Murray, D.M., & Baker, W.L. (2001). An evaluation of analysis options for the one-group-per-condition design: Can any of the alternatives overcome the problems inherent in this design? Evaluation Review, 25(4), 440-453.
67. What Works Clearinghouse. (2006a). WWC study review standards. Washington, DC: Author.
68. What Works Clearinghouse. (2006b). Tutorial on Mismatch Between Unit of Assignment and Unit of Analysis. Washington, DC: Author. Retrieved July 15, 2008, from http://ies.ed.gov/ncee/wwc/references/iDocViewer/Doc.aspx?docId=20&tocId=7.
69. What Works Clearinghouse. (2007). Technical details of WWC-conducted computations. Washington, DC: Author. Retrieved July 15, 2008, from http://ies.ed.gov/ncee/wwc/pdf/conducted_computations.pdf.
70. What Works Clearinghouse. (2008). WWC procedures and Version 2 standards handbook. Washington, DC: Author. Retrieved December 15, 2008, from http://ies.ed.gov/ncee/wwc/pdf/wwc_procedures_v2_standards_handbook.pdf.
71. Williams, V.S.L., Jones, L.V., & Tukey, J.W. (1999). Controlling error in multiple comparisons, with examples from state-to-state differences in educational achievement. Journal of Educational and Behavioral Statistics, 24(1), 42-69.