PowerUp!: A Tool for Calculating Minimum Detectable Effect Sizes and Minimum Required Sample Sizes for Experimental and Quasi-Experimental Design Studies

Journal of Research on Educational Effectiveness

Volumn 6, Issue 1, 2013, Pages 24-67

  Dong, Nianbo ^a   Maynard, Rebecca ^b  

^a VANDERBILT UNIVERSITY   (United States)

^b UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

minimum detectable effect size; multilevel experimental; Power analysis; quasi experimental designs

Indexed keywords

EID: 84872413608     PISSN: 19345747     EISSN: 19345739     Source Type: Journal    
DOI: 10.1080/19345747.2012.673143     Document Type: Article

References (45)

1. Angrist, J. D. and Lavy, V. 1999. Using Maimonides' rule to estimate the effect of class size on scholastic achievement. The Quarterly Journal of Economics, 114: 533-575.
2. Bloom, H. S. 1995. Minimum detectable effects: A simple way to report the statistical power of experimental designs. Evaluation Review, 19: 547-556.
3. Bloom, H. S. 1999. Estimating program impacts on student achievement using "short" interrupted time series, New York, NY: Manpower Demonstration Research Corporation.
4. Bloom, H. S. 2001. Measuring the impacts of whole-school reforms: Methodological lessons from an evaluation of Accelerated Schools, New York, NY: Manpower Demonstration Research Corporation. Retrieved from http://www.mdrc.org/sites/default/files/full_439.pdf
5. Bloom, H. S. 2003. Using short interrupted time-series analysis to measure the impacts of whole school reforms: With applications to a study of accelerated schools. Evaluation Review, 27: 3-49.
6. Bloom, H. S. 2005. "Randomizing groups to evaluate place-based programs". In Learning more from social experiments: Evolving analytic approaches (pp. 115-172), Edited by: Bloom, H. S. New York, NY: Russell Sage Foundation.
7. Bloom, H. S. 2006. The core analytics of randomized experiments for social research, New York, NY: Manpower Demonstration Research Corporation. (MDRC Working Papers on Research Methodology) Available from http://www.mdrc.org/publications/437/full.pdf
8. Bloom, H. S. 2009. Modern regression discontinuity analysis, New York, NY: Manpower Demonstration Research Corporation. (MDRC Working Papers on Research Methodology) Available from http://www.mdrc.org/publications/539/abstract.html
9. Bloom, H. S., Hill, C. J., Black, A. R. and Lipsey, M. W. 2008. Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. Journal of Research on Educational Effectiveness, 1: 289-328.
10. Bloom, H. S., Richburg-Hayes, L. and Black, A. R. 2007. Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29: 30-59.
11. Borman, G., Slavin, R. E., Cheung, A., Chamberlain, A., Madden, N. A. and Chambers, B. 2007. Final reading outcomes of the national randomized field trial of Success for All. American Educational Research Journal, 44: 701-731.
12. Boruch, R. F. 1997. Randomized experiments for planning and evaluation: A practical guide, Thousand Oaks, CA: Sage.
13. Cook, T. D., Hunt, H. D. and Murphy, R. F. 2000. Comer's school development program in Chicago: A theory-based evaluation. American Educational Research Journal, 37: 535-597.
14. Cook, T. D. and Wong, V. C. 2007. Empirical tests of the validity of regression discontinuity designs, Chicago, IL: Northwestern University.
15. Gamse, B. C., Bloom, H. S., Kemple, J. J., Jacob, R. T., Boulay, B., Bozzi, L. and Zhu, P. 2008. Reading First impact study: Interim report, Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Educational Sciences, U.S. Department of Education. (NCES 2008-4016)
16. Gleason, P., Clark, M., Tuttle, C. and Dwoyer, E. 2010. The evaluation of charter school impacts: Final report, Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences. (NCEE 2101-4010)
17. Hedges, L. V. and Hedberg, E. 2007. Interclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29: 60-87.
18. Hedges, L. V. and Rhoads, C. 2010. Statistical power analysis in education research, Washington, DC: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education. (NCSER 2010-3006) Available from http://ies.ed.gov/ncser/
19. Hill, C. J., Bloom, H. S., Black, A. R. and Lipsey, M. W. 2007. Empirical benchmarks for interpreting effect sizes in educational research, New York, NY: Manpower Demonstration Research Corporation. (MDRC Working Papers on Research Methodology) Retrieved from http://www.mdrc.org/publications/459/full.pdf
20. Imbens, G. and Lemieux, T. 2008. Regression discontinuity designs: A guide to practice. Journal of Econometrics, 142: 615-635.
21. Jacob, B. and Lefgren, L. 2004. Remedial education and student achievement: A regression-discontinuity analysis. Review of Economics and Statistics, 86: 226-244.
22. Kemple, J. J., Herlihy, C. M. and Smith, T. J. 2005. Making progress toward graduation: Evidence from the Talent Development High School Model, New York, NY: Manpower Demonstration Research Corporation.
23. Kirk, R. E. 1995. Experimental design: Procedures for the behavioral sciences, Pacific Grove, CA: Brooks/Cole.
24. Konstantopoulos, S. 2008a. The power of the test for treatment effects in three-level block randomized designs. Journal of Research on Educational Effectiveness, 1: 265-288.
25. Konstantopoulos, S. 2008b. The power of the test for treatment effects in three-level cluster randomized designs. Journal of Research on Educational Effectiveness, 1: 66-88.
26. Konstantopoulos, S. 2009. Using power tables to compute statistical power in multilevel experimental designs. Practical Assessment, Research & Evaluation, 14(10) Retrieved from http://pareonline.net/getvn.asp?v=14&n=10
27. Matsudaira, J. D. 2008. Mandatory summer school and student achievement. Journal of Econometrics, 142: 829-850.
28. Murnane, R. and Willett, J. 2010. Methods matter: Improving causal inference in educational and social science research, New York, NY: Oxford University Press.
29. Murray, D. 1998. Design and analysis of group-randomized trials, Oxford, UK: Oxford University Press.
30. Nye, B., Hedges, L. V. and Konstantopoulos, S. 1999. The long-term effects of small classes: A five-year follow-up of the Tennessee class-size experiment. Educational Evaluation and Policy Analysis, 21: 127-142.
31. Orr, L. 1998. Social experiments evaluating public programs with experimental methods, London, UK: Sage.
32. Quint, J. C., Bloom, H. S., Black, A. R., Stephens, L. and Akey, T. M. 2005. The challenge of scaling up educational reform: findings and lessons from First Things First, New York: Manpower Development Research Corporation.
33. Quint, J. C., Sepanik, S. and Smith, J. K. 2008. Using student data to improve teaching and learning: Findings from an evaluation of the Formative Assessment of Student Thinking in Reading (FAST-R) program in Boston elementary schools, New York, NY: Manpower Demonstration Research Corporation.
34. Raudenbush, S. W. 1997. Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2: 173-185.
35. Raudenbush, S. W. and Liu, X. 2000. Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5: 199-213.
36. Raudenbush, S. W., Martinez, A. and Spybrook, J. 2007. Strategies for improving precision in group-randomized experiments. Education Evaluation and Policy Analysis, 29(1): 5-29.
37. Spybrook, J. 2007. Examining the experimental designs and statistical power of group randomized trials funded by the Institute of Education Sciences (Unpublished doctoral dissertation). Ann Arbor, MI: University of Michigan
38. Spybrook, J., Raudenbush, S. W., Congdon, R. and Martinez, A. 2009. Optimal design version 2.0 [Computer software]
39. Schochet, P. Z. 2008a. Statistical power for randomized assignment evaluation of education programs. Journal of Educational and Behavioral Statistics, 33(1): 62-87.
40. Schochet, P. Z. 2008b. Technical methods report: Statistical power for regression discontinuity designs in education evaluations, Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences. (NCEE 2008-4026)
41. Schochet, P., Cook, T., Deke, J., Imbens, G., Lockwood, J. R., Porter, J. and Smith, J. 2010. Standards for regression discontinuity designs Retrieved from What Works Clearinghouse website:http://ies.ed.gov/ncee/wwc/pdf/wwc_rd.pdf
42. Shadish, W. R., Cook, T. D. and Campbell, D. T. 2002. Experimental and quasi-experimental designs for generalized causal inference, Boston, MA: Houghton Mifflin.
43. Thistelwaite, D. and Campbell, D. T. 1960. Regression-discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology, 51: 309-317.
44. Trochim, W. 1984. Research design for program evaluation: The regression-discontinuity approach, New York, NY: Sage.
45. U.S. Department of Education, What Works Clearinghouse. 2008. Procedures and standards handbook (Version 2), Washington, DC: Author. December