Sample Selection in Randomized Experiments: A New Method Using Propensity Score Stratified Sampling

Journal of Research on Educational Effectiveness

Volumn 7, Issue 1, 2014, Pages 114-135

  Tipton, Elizabeth ^a   Hedges, Larry ^b   Vaden Kiernan, Michael ^c   Borman, Geoffrey ^d   Sullivan, Kate ^c   Caverly, Sarah ^c  

^a Och Spine at New York Presbyterian Hospitals   (United States)

^b NORTHWESTERN UNIVERSITY   (United States)

^c SEDL   (United States)

^d UNIVERSITY OF WISCONSIN MADISON   (United States)

Author keywords

Experimental design; recruitment; sampling; scale up; stratified

Indexed keywords

EID: 84892380430     PISSN: 19345747     EISSN: 19345739     Source Type: Journal    
DOI: 10.1080/19345747.2013.831154     Document Type: Article

References (40)

1. Borman, G. D., Dowling, N. M. and Schneck, C. 2008. A multi-site cluster randomized field trial of Open Court Reading. Educational Evaluation and Policy Analysis, 30: 389 - 407.
2. Borman, G. D., Slavin, R. E., Cheung, A., Chamberlain, A. M., Madden, N. A. and Chambers, B. 2005. Success for All: First-year results from the National Randomized Field Trial. Educational Evaluation and Policy Analysis, 27: 1 - 22.
3. Boruch, R. F. 1996. Randomized experiments for planning and evaluation: A practical guide (Applied Social Research Methods Series, Vol. 44), Thousand Oaks, CA: Sage.
4. Cochran, W. G. 1968. The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics, 24: 295 - 313.
5. Cook, T. D. 1993. " A quasi-sampling theory of the generalization of causal relationships ". In Understanding causes and generalizing about them: New directions for program evaluation, Edited by: Sechrest, L. and Scott, A. G. 39 - 82. San Francisco, CA: Jossey-Bass.
6. Cornfield, J. and Tukey, J. W. 1956. Average values of mean squares in factorials. The Annals of Mathematical Statistics, 27: 907 - 949.
7. Deming, W. E. and Stephan, F. F. 1940. On a least squares adjustment of a sampled frequency table when the expected marginal totals are known. The Annals of Mathematical Statistics, 11: 427 - 444.
8. Duncan, G. J. 2008. When to promote, and when to avoid, a population perspective. Demography, 45: 763 - 784.
9. Fienberg, S. E. and Tanur, J. M. 1987. Experimental and sampling structures: Parallels diverging and meeting. International Statistical Review, 55: 75 - 96.
10. Groves, R., Dillman, D., Eltinge, J. and Little, R. J. A. 2002. Survey nonresponse, New York, NY: Wiley.
11. Groves, R. M., Fowler, F. J., Couper, M. P., Lepkowski, J. M., Singer, E. and Tourangeau, R. 2009. Survey methodology (2nd ed.), Hoboken, NJ: Wiley.
12. Hedges, L. V. and O'Muircheartaigh, C. A. 2011. Improving generalizations from designed experiments (Working paper) Evanston, IL Northwestern University
13. Imai, K., King, G. and Stuart, E. A. 2008. Misunderstandings among experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society, Series A (Statistics in Society), 171: 481 - 502.
14. Institute for Education Sciences. 2011. Funding opportunities: Search funded research and contracts. Retrieved from http://ies.ed.gov/funding/grantsearch
15. Kalton, G. 1968. Standardization: A technique to control for extraneous variables. Applied Statistics, 17: 118 - 136.
16. Kish, L. 1987. Statistical design for research, New York, NY: Wiley. (Wiley Series in Probability and Mathematical Statistics)
17. Kitagawa, E. M. 1964. Standardized comparisons in population research. Demography, 1: 296 - 315.
18. Kruskal, W. H. and Mosteller, F. 1979. Representative sampling. III. The current statistical literature. International Statistical Review, 47: 245 - 265.
19. Little, R. J. A. and Rubin, D. B. 2002. Statistical analysis with missing data, New York, NY: Wiley. (Wiley Series in Probability and Mathematical Statistics)
20. Lohr, S. 1999. Sampling: Design and analysis., New York, NY: Duxbury.
21. National Center for Education Statistics. 2011. Common core of data. Retrieved from http://nces.ed.gov/ccd
22. National Reading Panel. 2000. Report of the National Reading Panel: Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction., Washington, DC: National Institute of Child Health and Human Development.
23. Neyman, J. 1934. On the two different aspects of the representative method: The method of stratified sampling and of purposive selection. Journal of the Royal Statistical Society, 97: 558 - 625.
24. Oaxaca, R. 1973. Male-female wage differentials in urban labor markets. International Economic Review, 14: 693 - 709.
25. Olsen, R. B., Orr, L. L., Bell, S. H. and Stuart, E. A. 2012. External validity in policy evaluations that choose sites purposively. Journal of Policy Analysis and Management., Advance online publication. doi:10.1002/pam.21660
26. Raudenbush, S. W. and Bryk, A. S. 2002. Hierarchical linear models (2nd ed.), Thousand Oaks, CA: Sage.
27. Rosenbaum, P. R. and Rubin, D. B. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika, 70: 41 - 55. doi: 10.1093/biomet/70.1.41
28. Rosenbaum, P. R. and Rubin, D. B. 1984. Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79: 516 - 524. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/20199225
29. Rosenberg, P. 1962. Test factor standardization as a method of interpretation. Social Forces, 41: 53 - 61.
30. Rubin, D. B. 1976. Inference and missing data. Biometrika, 63: 581 - 592.
31. Schneider, B. and McDonald, S. K. 2007. Scale-up in education: Ideas in principle (Vol. 1), Lanham, MD: Rowman & Littlefield.
32. Shadish, W. R., Cook, T. D. and Campbell, D. T. 2002. Experimental and quasi-experimental designs for generalized causal inference., Boston, MA: Houghton-Mifflin.
33. Slavin, R. E. and Lake, C. 2007. Effective programs in elementary mathematics: A best- evidence synthesis., Baltimore, MD: Johns Hopkins University.
34. Spybrook, J. and Raudenbush, S. W. 2009. An examination of the precision and technical adequacy of the first wave of group randomized trials funded by the Institute of Education Sciences. Educational Evaluation and Policy Analysis, 31: 298 - 318.
35. Stuart, E. A., Cole, S. R., Bradshaw, C. P. and Leaf, P. J. 2011. The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society, Series A, Part 2: 369 - 386.
36. Tipton, E. 2013. Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38: 239 - 266.
37. Tipton, E. in press. Stratified sampling using cluster analysis: A balanced-sampling strategy for improved generalizations from experiments. Evaluation review
38. Valliant, R., Dorfman, A. H. and Royall, R. M. 2000. Finite population sampling and inference: A prediction approach, New York, NY: Wiley. (Wiley Series in Probability and Statistics)
39. What Works Clearinghouse. 2007. Elementary school math. Retrieved from http://ies.ed.gov/ncee/wwc/reports/elementary_math/topic/references.asp
40. What Works Clearinghouse. 2010. Intervention: Everyday mathematics. Retrieved from http://ies.ed.gov/ncee/wwc/reports/elementary_math/eday_math/