Implications of Small Samples for Generalization: Adjustments and Rules of Thumb

Evaluation Review

Volumn 41, Issue 5, 2017, Pages 472-505

  Tipton, Elizabeth ^a   Hallberg, Kelly ^b   Hedges, Larry V ^c   Chan, Wendy ^d  

^a Columbia University ^*  (United States)

^b Department of Public Health Sciences   (United States)

^c NORTHWESTERN UNIVERSITY   (United States)

^d UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

content area; education; methodological development

Indexed keywords

ACADEMIC SUCCESS; ADOLESCENT; CHILD; EVALUATION STUDY; HUMAN; INDIANA; MANAGEMENT; PRESCHOOL CHILD; PROPENSITY SCORE; RANDOMIZED CONTROLLED TRIAL (TOPIC); SAMPLE SIZE; STATISTICAL MODEL;

ACADEMIC SUCCESS; ADOLESCENT; CHILD; CHILD, PRESCHOOL; EVALUATION STUDIES AS TOPIC; HUMANS; INDIANA; MODELS, STATISTICAL; POLICY MAKING; PROPENSITY SCORE; RANDOMIZED CONTROLLED TRIALS AS TOPIC; SAMPLE SIZE;

EID: 85036585924     PISSN: 0193841X     EISSN: 15523926     Source Type: Journal    
DOI: 10.1177/0193841X16655665     Document Type: Article

References (29)

1. Blom G., (1958). Statistical estimates and transformed beta variables. New York, NY: Wiley.
2. Cole S. R., Hernán M. A., (2008). Constructing inverse probability weights for marginal structural models. American Journal of Epidemiology, 168, 656–664.
3. Guo S., Fraser M. W., (2014). Propensity score analysis: Statistical methods and applications (Vol. 11). Thousand Oaks, CA: Sage.
4. Imai K., King G., Stuart E. A., (2008). Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society, Series A, 171, 481–502.
5. Johnson N. L., Kotz S., (1971). Distributions in statistics: Continuous univariate distributions (Vol. 1, Wiley Series in Probability and Mathematical Statistics). New York, NY: Wiley.
6. Konstantopoulos S., Miller S., van der Ploeg A., (2013). The impact of Indiana’s system of interim assessments on mathematics and reading achievement. Educational Evaluation and Policy Analysis, 35, 481–499.
7. Little R. J. A., (1986). Survey nonresponse adjustments. International Statistical Review / Revue Internationale de Statistique, 54, 139–157
8. Normand S. L. T., Landrum M. B., Guadagnoli E., Ayanian J. Z., Ryan T. J., Cleary P. D., McNeil B. J., (2001). Validating recommendations for coronary angiography following an acute myocardial infarction in the elderly: A matched analysis using propensity scores. Journal of Clinical Epidemiology, 54, 387–398.
9. Olsen R. B., Orr L. L., Bell S. H., Stuart E. A., (2013). External validity in policy evaluations that choose sites purposively. Journal of Policy Analysis and Management, 32, 107–121.
10. O’Muircheartaigh C., Hedges L. V., (2014). Generalizing from unrepresentative experiments: A stratified propensity score approach. Journal of the Royal Statistical Society: Series C (Applied Statistics), 63, 195–210.
11. Orr L. L., (2015). 2014 Rossi award lecture: Beyond internal validity. Evaluation Review, 39, 167–178. doi:10.1177/0193841X15573659
12. R Core Team. (2014). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/
13. Rosenbaum P. R., Rubin D. B., (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
14. Rosenbaum P. R., Rubin D. B., (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79, 516–524.
15. Rubin D. B., (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701.
16. Rubin D. B., (1978). Bayesian inference for causal effects: The role of randomization. Annals of Statistics, 6, 34–58.
17. Rubin D. B., (1980). Discussion of “randomization analysis of experimental data in the Fisher randomization test” by D. Basu. Journal of the American Statistical Association, 74, 318–328.
18. Rubin D. B., (1990). Neyman (1923) and causal inference in experiments and observational studies. Statistical Science, 5, 472–480.
19. Rubin D. B., (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services & Outcomes Research Methodology, 2, 169–188.
20. Schochet P. Z., Puma M., Deke J., (2014). Understanding variation in treatment effects in education impact evaluations: An overview of quantitative methods (NCEE 2014–4017). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development. Retrieved from http://ies.ed.gov/ncee/edlabs
21. Silverman B. W., (1986). Density estimation for statistics and data analysis (Vol. 26). Boca Raton, FL: CRC.
22. Steiner P. M., Cook T. D., Shadish W. R., Clark M. H., (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15, 250–267.
23. Stuart E. A., (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25, 1–21.
24. Stuart E. A., Bradshaw C. P., Leaf P. J., (2014). Assessing the generalizability of randomized trial results to target populations. Prevention Science, 16, 1–11.
25. Stuart E. A., Cole S. R., Bradshaw C. P., Leaf P. J., (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society, Series A, 2, 369–386.
26. Tipton E., (2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38, 239–266.
27. Tipton E., (2014a). How generalizable is your experiment? Comparing a sample and population through a generalizability index. Journal of Educational and Behavioral Statistics, 39, 478–501.
28. Tipton E., (2014b). Stratified sampling using cluster analysis: A sample selection strategy for improved generalizations from experiments. Evaluation Review, 37, 109–139.
29. Tipton E., Hedges L. V., Vaden-Kiernan M., Borman G. D., Sullivan K., Caverly S., (2014). Sample selection in randomized experiments: A new method using propensity score stratified sampling. Journal of Research on Educational Effectiveness, 7, 114–135.