A Review of Statistical Methods for Generalizing From Evaluations of Educational Interventions

Educational Researcher

Volumn 47, Issue 8, 2018, Pages 516-524

  Tipton, Elizabeth ^a   Olsen, Robert B ^b  

^a Columbia University ^*  (United States)

^b GEORGE WASHINGTON UNIVERSITY   (United States)

Author keywords

educational policy; evaluation; experimental design; experimental research; external validity; generalizability; multisite studies; policy; program evaluation; propensity scores; research methodology; sampling; statistics

Indexed keywords

EID: 85049025729     PISSN: 0013189X     EISSN: 1935102X     Source Type: Journal    
DOI: 10.3102/0013189X18781522     Document Type: Review

References (31)

1. Bell S. H., Olsen R. B., Orr L. L., Stuart E. A., (2016). Estimates of external validity bias when impact evaluations select sites nonrandomly. Educational Evaluation and Policy Analysis, 38(2), 318–335
2. Chan W., (2017). Partially identified treatment effects for generalizability. Journal of Research on Educational Effectiveness, 10(3), 646–669
3. Chipman H. A., George E. I., McCulloch R. E., (2007). Bayesian ensemble learning. In Schölkopf B., Platt J. C., Hoffman T., (Eds.), Advances in Neural Information Processing Systems 19 (NIPS 2006) (pp. 265–272). Retrieved from https://papers.nips.cc/book/advances-in-neural-information-processing-systems-19-2006
4. Chipman H. A., George E. I., McCulloch R. E., (2010). BART: Bayesian additive regression trees. The Annals of Applied Statistics, 4(1), 266–298
5. Fellers L., (2017). Developing an approach to determine generalizability: A review of efficacy and effectiveness trials funded by the Institute of Education Sciences (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses. (Accession Order No. 10256121)
6. Horvitz D. G., Thompson D. J., (1952). A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260), 663–685
7. Kern H. L., Stuart E. A., Hill J., Green D. P., (2016). Assessing methods for generalizing experimental impact estimates to target populations. Journal of Research on Educational Effectiveness, 9(1), 103–127
8. Lee B. K., Lessler J., Stuart E. A., (2011). Weight trimming and propensity score weighting. PloS One, 6(3), e18174
9. Lohr S., (1999). Sampling: Design and analysis. Boston, MA: Duxbury Press
10. Nguyen T. Q., Ebnesajjad C., Cole S. R., Stuart E. A., (2017). Sensitivity analysis for an unobserved moderator in RCT-to-target-population generalization of treatment effects. The Annals of Applied Statistics, 11(1), 225–247
11. Olsen R. B., Orr L. L., (2016). On the “where” of social experiments: Selecting more representative samples to inform policy. New Directions for Evaluation, 2016(152), 61–71
12. Olsen R. B., Orr L. L., Bell S. H., Stuart E. A., (2013). External validity in policy evaluations that choose sites purposively. Journal of Policy Analysis and Management, 32(1), 107–121
13. O’Muircheartaigh C., Hedges L. V. (2014). Generalizing from unrepresentative experiments: a stratified propensity score approach. Journal of the Royal Statistical Society: Series C (Applied Statistics), 63(2), 195–210
14. Rosenbaum P. R., Rubin D. B., (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55
15. Shadish W., Cook T. D., Campbell D. T., (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin
16. Stuart E. A., (2010). Matching methods for causal inference: A review and a look forward. Statistical Science: A Review Journal of the Institute of Mathematical Statistics, 25(1), 1
17. Stuart E. A., Bell S. H., Ebnesajjad C., Olsen R. B., Orr L. L., (2017). Characteristics of school districts that participate in rigorous national educational evaluations. Journal of Research on Educational Effectiveness, 10(1), 168–206
18. Stuart E. A., Cole S. R., Bradshaw C. P., Leaf P. J., (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174(2), 369–386
19. Stuart E. A., Rhodes A., (2017). Generalizing treatment effect estimates from sample to population: A case study in the difficulties of finding sufficient data. Evaluation Review, 41(4), 357–388
20. Tipton E., (2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38, 239–266
21. Tipton E., (2014a). How generalizable is your experiment? An index for comparing experimental samples and populations. Journal of Educational and Behavioral Statistics, 39(6), 478–501
22. Tipton E., (2014b). Stratified sampling using cluster analysis: A sample selection strategy for improved generalizations from experiments. Evaluation Review, 37(2), 109–139
23. Tipton E., Fellers L., Caverly S., Vaden-Kiernan M., Borman G., Sullivan K., Ruiz de Castilla V., (2016). Site selection in experiments: An assessment of site recruitment and generalizability in two scale-up studies. Journal of Research on Educational Effectiveness, 9(Suppl 1), 209–228
24. Tipton E., Hallberg K., Hedges L. V., Chan W., (2017). Implications of small samples for generalization: Adjustments and rules of thumb. Evaluation Review, 41(5), 472–505
25. Tipton E., Hedges L., Vaden-Kiernan M., Borman G., Sullivan K., Caverly S., (2014). Sample selection in randomized experiments: A new method using propensity score stratified sampling. Journal of Research on Educational Effectiveness, 7(1), 114–135
26. Tipton E., Matlen B., (2018). The development and implementation of a strategic recruitment plan for generalizability: A case study. Working paper
27. Tipton E., Miller K., (2016). The Generalizer: A webtool for improving the generalizability of results from experiments. Retrieved from http://www.thegeneralizer.org
28. Tipton E., Yeager D., Schneider B., Iachan R., (In press). Designing probability samples to identify sources of treatment effect heterogeneity. In Lavrakas P. J., (Ed.), Experimental methods in survey research: Techniques that combine random sampling with random assignment. New York, NY: Wiley
29. Weiss M. J., Bloom H. S., Brock T., (2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33(3), 778–808
30. Weiss M. J., Bloom H. S., Verbitsky-Savitz N., Gupta H., Vigil A. E., Cullinan D. N., (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876
31. What Works Clearinghouse. (2017). What Works Clearinghouse: Standards handbook (Version 4.0). Washington, DC: U.S. Department of Education