Use of Quasi-Experimental Research Designs in Education Research: Growth, Promise, and Challenges

Review of Research in Education

Volumn 44, Issue 1, 2020, Pages 218-243

  Gopalan, Maithreyi ^a   Rosinger, Kelly ^a   Ahn, Jee Bin ^a  

^a PENNSYLVANIA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85083765256     PISSN: 0091732X     EISSN: 19351038     Source Type: Journal    
DOI: 10.3102/0091732X20903302     Document Type: Article

References (87)

1. Angrist J., Pischke J.-S., (2010). The credibility revolution in empirical economics: How better research design is taking the con out of econometrics. Journal of Economic Perspectives, 24(2), 3–30. https://doi.org/10.3386/w15794
2. Barnett W. S., Jung K., Friedman-Krauss A., Frede E. C., Nores M., Hustedt J. T., Howes C., Daniel-Echols M., (2018). State prekindergarten effects on early learning at kindergarten entry: An analysis of eight state programs. AERA Open, 4(2), Article 2332858418766291. https://doi.org/10.1177/2332858418766291
3. Belfield C., Bailey T., (2017). The labor market returns to sub-baccalaureate college: A review (Center for Analysis of Postsecondary Education and Employment Working Paper). https://ccrc.tc.columbia.edu/media/k2/attachments/labor-market-returns-sub-baccalaureate-college-review.pdf
4. Bettinger E., (2010). Instrumental variables. In McGaw B., Peterson P., Baker E., (Eds.), International encyclopedia of education (3rd ed., Vol. 8). Elsevier.
5. Callahan M. K., Meehan K., Shaw K. M., (2017). Impact of OBF on student outcomes: Tennessee and Indiana. Research for Action.
6. Campbell D. T., (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54(4), 297–312. http://dx.doi.org/10.1037/h0040950
7. Candelaria C. A., Shores K. A., (2019). Court-ordered finance reforms in the adequacy era: Heterogeneous causal effects and sensitivity. Education Finance and Policy, 14(1), 31–60. https://doi.org/10.1162/EDFP_a_00236
8. Carneiro P., Ginja R., (2014). Long-term impacts of compensatory preschool on health and behavior: Evidence from Head Start. American Economic Journal: Economic Policy, 6(4), 135–173. https://doi.org/10.1257/pol.6.4.135
9. Cellini S. R., Ferreira F., Rothstein J., (2010). The value of school facility investments: Evidence from a dynamic regression discontinuity design. Quarterly Journal of Economics, 125(1), 215–261. https://doi.org/10.1162/qjec.2010.125.1.215
10. Chingos M., Whitehurst G., (2011). Class size: What research says and what it means for state policy. Brookings Institution. https://www.brookings.edu/research/class-size-what-research-says-and-what-it-means-for-state-policy/
11. Cohodes S., (2018, February 1). Charter schools and the achievement gap. https://futureofchildren.princeton.edu/news/charter-schools-and-achievement-gap
12. Coleman J., Campbell E., Hobson C., McPartland J., Mood A., Weinfield F., York R., (1966). Equality of educational opportunity. U.S. Department of Health, Education, and Welfare, Office of Education. https://eric.ed.gov/?id=Ed012275
13. Conklin K. D., Snyder M., Stanley J., Boelscher S., (2016). Rowing together: Aligning state and federal investments in talent to common outcomes. https://www.ecs.org/wp-content/uploads/ECS_FundingReports_HCM_F.pdf
14. Cook T. D., (2008). “Waiting for life to arrive”: A history of the regression-discontinuity design in psychology, statistics and economics. Journal of Econometrics, 142(2), 636–654. https://doi.org/10.1016/j.jeconom.2007.05.002
15. Currie J., Thomas D., (1995). Does Head Start make a difference? American Economic Review, 85(3), 341–364. https://www.jstor.org/stable/2118178
16. Curs B. R., Harper C. E., (2012). Financial aid and first-year collegiate GPA: A regression discontinuity approach. The Review of Higher Education, 35(4), 627–649. https://doi.org/10.1353/rhe.2012.0040
17. Deaton A., Cartwright N., (2018). Understanding and misunderstanding randomized controlled trials. Social Science & Medicine, 210, 2–21. https://doi.org/10.1016/j.socscimed.2017.12.005
18. Deming D., (2009). Early childhood intervention and life-cycle skill development: Evidence from Head Start. American Economic Journal: Applied Economics, 1(3), 111–134. https://doi.org/10.1257/app.1.3.111
19. Dhuey E., (2011). Who benefits from kindergarten? Evidence from the introduction of state subsidization. Educational Evaluation and Policy Analysis, 33(1), 3–22. https://doi.org/10.3102/0162373711398125
20. Dougherty K. J., Jones S. M., Lahr H., Pheatt L., Natow R. S., Reddy V., (2016). Performance funding for higher education. Johns Hopkins University Press.
21. Duncan G. J., Engel M., Claessens A., Dowsett C. J., (2014). Replication and robustness in developmental research. Developmental Psychology, 50(11), 2417–2425. https://doi.org/10.1037/a0037996
22. Duncan G. J., Magnuson K., (2013). Investing in preschool programs. Journal of Economic Perspectives, 27(2), 109–132. https://doi.org/10.1257/jep.27.2.109
23. Dynarski S., (2002). The behavioral and distributional implications of aid for college. American Economic Review, 92(2), 279–285. https://doi.org/10.1257/000282802320189401
24. Fain P., (2018). As California goes? https://www.insidehighered.com/news/2018/06/12/calif-finalizes-performance-funding-formula-its-community-colleges
25. Friedman-Krauss A. H., Barnett W. S., Weisenfeld G. G., Kasmin R., DiCrecchio N., Horowitz M., (2018). The state of preschool 2017. National Institute for Early Education Research.
26. Gándara D., Rutherford A., (2017). Mitigating unintended impacts? The effects of premiums for underserved populations in performance-funding policies for higher education. Research in Higher Education, 59(6), 681–703. https://doi.org/10.1007/s11162-017-9483-x
27. Garces E., Thomas D., Currie J., (2002). Longer-term effects of Head Start. American Economic Review, 92(4), 999–1012. https://doi.org/10.1257/00028280260344560
28. Gelman A., Imbens G., (2019). Why high-order polynomials should not be used in regression discontinuity designs. Journal of Business & Economic Statistics, 37(3), 447–456. https://doi.org/10.1080/07350015.2017.1366909
29. Goodman-Bacon A., (2018). Difference-in-differences with variation in treatment timing (Working Paper No. 25018). https://doi.org/10.3386/w25018
30. Gormley W. T., Jr. Gayer T., Phillips D., Dawson B., (2005). The effects of universal pre-K on cognitive development. Developmental Psychology, 41(6), 872–884. https://doi.org/10.1037/0012-1649.41.6.872
31. Hanushek E. A., (1997). Assessing the effects of school resources on student performance: An update. Educational Evaluation and Policy Analysis, 19(2), 141–164. https://doi.org/10.3102/01623737019002141
32. Hanushek E. A., (2003). The failure of input-based schooling policies. Economic Journal, 113(485), F64–F98. https://doi.org/10.1111/1468-0297.00099
33. Heckman J. J., (2006). Skill formation and the economics of investing in disadvantaged children. Science, 312(5782), 1900–1902. https://doi.org/10.1126/science.1128898
34. Hedges L. V., (2018). Challenges in building usable knowledge in education. Journal of Research on Educational Effectiveness, 11(1), 1–21. https://doi.org/10.1080/19345747.2017.1375583
35. Hedges L. V., Laine R. D., Greenwald R., (1994). An exchange: Part I. Does money matter? A meta-analysis of studies of the effects of differential school inputs on student outcomes. Educational Researcher, 23(3), 5–14. https://doi.org/10.3102/0013189X023003005
36. Hillman N. W., Hicklin Fryar A., Crespín-Trujillo V., (2018). Evaluating the impact of performance funding in Ohio and Tennessee. American Educational Research Journal, 55(1), 144–170.
37. Hillman N. W., Tandberg D. A., Fryar A. H., (2015). Evaluating the impacts of “new” performance funding in higher education. Educational Evaluation and Policy Analysis, 37(4), 501–519. https://doi.org/10.3102/0162373714560224
38. Hillman N. W., Tandberg D. A., Gross J. P., (2014). Performance funding in higher education: Do financial incentives impact college completions? Journal of Higher Education, 85(6), 826–857. https://doi.org/10.1353/jhe.2014.0031
39. Howell P. L., Miller B. B., (1997). Sources of funding for schools. Future of Children, 7(3), 39–50. https://doi.org/10.2307/1602444
40. Jackson C. K., (2018). Does school spending matter? The new literature on an old question. Paper presented at the Fall 2018 Bronfenbrenner Center for Translational Research Conference. https://works.bepress.com/c_kirabo_jackson/38/
41. Jackson C. K., Johnson R. C., Persico C., (2016). The effects of school spending on educational and economic outcomes: Evidence from school finance reforms. Quarterly Journal of Economics, 131(1), 157–218. https://doi.org/10.1093/qje/qjv036
42. Jenkins J. M., Farkas G., Duncan G. J., Burchinal M., Vandell D. L., (2016). Head Start at Ages 3 and 4 versus Head Start followed by state pre-K: Which is more effective? Educational Evaluation and Policy Analysis, 38(1), 88–112. https://doi.org/10.3102/0162373715587965
43. Johnson R., Jackson C. K., (2019). Reducing inequality through dynamic complementarity: Evidence from Head Start and public school spending. American Economic Journal: Economic Policy, 11(4), 310–349. https://doi.org/10.3386/w23489
44. Kelchen R., (2018). Do performance-based funding policies affect underrepresented student enrollment? Journal of Higher Education, 89(5), 702–727. https://doi.org/10.1080/00221546.2018.1434282
45. Kelchen R., (2019). Exploring the relationship between performance-based funding design and underrepresented student enrollment at community colleges. Community College Review, 47(4), 382–405. https://doi.org/10.1177/0091552119865611
46. Kelchen R., Rosinger K. O., Ortagus J. C., (2019). How to create and use state-level policy data sets in education research. AERA Open, 5(3), Article 2332858419873619. https://doi.org/10.1177/2332858419873619
47. Kelchen R., Stedrak L. J., (2016). Does performance-based funding affect colleges’ financial priorities? Journal of Education Finance, 41(3), 302–321. https://doi.org/10.1353/jef.2016.0006
48. Lafortune J., Rothstein J., Schanzenbach D. W., (2018). School finance reform and the distribution of student achievement. American Economic Journal: Applied Economics, 10(2), 1–26. https://doi.org/10.1257/app.20160567
49. Lemons C. J., Fuchs D., Gilbert J. K., Fuchs L. S., (2014). Evidence-based practices in a changing world: Reconsidering the counterfactual in education research. Educational Researcher, 43(5), 242–252. https://doi.org/10.3102/0013189X14539189
50. Li A. Y., Kennedy A. I., (2018). Performance funding policy effects on community college outcomes: Are short-term certificates on the rise? Community College Review, 46(1), 3–39.
51. Lipsey M. W., Weiland C., Yoshikawa H., Wilson S. J., Hofer K. G., (2015). The prekindergarten age-cutoff regression-discontinuity design: Methodological issues and implications for application. Educational Evaluation and Policy Analysis, 37(3), 296–313. https://doi.org/10.3102/0162373714547266
52. Loeb S., Dynarski S., McFarland D., Morris P., Reardon S., Reber S., (2017). Descriptive analysis in education: A guide for researchers (No. NCEE 2017-4023; p. 53). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.
53. Magnuson K. A., Ruhm C., Waldfogel J., (2007). Does prekindergarten improve school preparation and performance? Economics of Education Review, 26(1), 33–51. https://doi.org/10.1016/j.econedurev.2005.09.008
54. Martorell P., Stange K., McFarlin I., (2016). Investing in schools: Capital spending, facility conditions, and student achievement. Journal of Public Economics, 140, 13–29. https://doi.org/10.1016/j.jpubeco.2016.05.002
55. McCoy D. C., Yoshikawa H., Ziol-Guest K. M., Duncan G. J., Schindler H. S., Magnuson K., Yang R., Koepp A., Shonkoff J. P., (2017). Impacts of early childhood education on medium- and long-term educational outcomes. Educational Researcher, 46(8), 474–487. https://doi.org/10.3102/0013189X17737739
56. McKenzie D., (2018, September 4). Have descriptive development papers been crowded out by impact evaluations? World Bank Blogs. https://blogs.worldbank.org/impactevaluations/have-descriptive-development-papers-been-crowded-out-impact-evaluations
57. Morris P. A., Connors M., Friedman-Krauss A., McCoy D. C., Weiland C., Feller A., Page L., Bloom H., Yoshikawa H., (2018). New findings on impact variation from the Head Start impact study: Informing the scale-up of early childhood programs. AERA Open, 4(2), Article 2332858418769287. https://doi.org/10.1177/2332858418769287
58. National Research Council & Institute of Medicine Committee on Integrating the Science of Early Childhood Development. (2000). From neurons to neighborhoods: The science of early childhood development (Shonkoff J. P., Phillips D. A., Eds.). National Academies Press. http://www.ncbi.nlm.nih.gov/books/NBK225557/
59. National Student Clearinghouse. (2018). Completing college. https://nscresearchcenter.org/signaturereport16/
60. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716–aac4716. https://doi.org/10.1126/science.aac4716
61. Panhans M. T., Singleton J. D., (2017). The empirical economist’s toolkit from models to methods. History of Political Economy, 49(Supplement), 127–157. https://doi.org/10.1215/00182702-4166299
62. Phillips D. A., Lipsey M. W., Dodge K. A., Haskins R., Bassok D., Burchinal M. R., Duncan G. J., Dynarski M., Magnuson K. A., Weiland C., (2017). Puzzling it out: The current state of scientific knowledge on pre-kindergarten effects: A consensus statement (p. 16). Brookings Institution.
63. Plümper T., Troeger V. E., Neumayer E., (2019). Case selection and causal inferences in qualitative comparative research. PLOS ONE, 14(7). https://doi.org/10.1371/journal.pone.0219727
64. Polikoff M., Conaway C., (2018). Getting beyond ‘did it work?’: Proposing a new approach to integrate research and policy [The Brookings Institution]. Brookings Brown Center Chalkboard. https://www.brookings.edu/blog/brown-center-chalkboard/2018/09/25/getting-beyond-did-it-work-proposing-a-new-approach-to-integrate-research-and-policy/
65. Pritchett L., (2018). Lant Pritchett Talk: “The debate about RCTs in development is over. We won. They lost.” Retrieved September 11, 2018, from http://www.nyudri.org/events-index/2018/2/22/lant-pritchett-talk-the-debate-about-rcts-in-development-is-over-we-won-they-lost
66. Reback R., (2010). Noninstructional spending improves noncognitive outcomes: Discontinuity evidence from a unique elementary school counselor financing system. Education Finance and Policy, 5(2), 105–137. https://doi.org/10.1162/edfp.2010.5.2.5201
67. Ruhm C. J., (2018). Shackling the identification police? (Working Paper No. 25320). https://doi.org/10.3386/w25320
68. Rutherford A., Rabovsky T., (2014). Evaluating impacts of performance funding policies on student outcomes in higher education. Annals of the American Academy of Political and Social Science, 655(1), 185–208. https://doi.org/10.1177/0002716214541048
69. Schanzenbach D. W., (2012). Limitations of experiments in education research. Education Finance and Policy, 7(2), 219–232. https://doi.org/10.1162/EDFP_a_00063
70. Schudde L., (2018). Heterogeneous effects in education: The promise and challenge of incorporating intersectionality into quantitative methodological approaches. Review of Research in Education, 42(1), 72–92. https://doi.org/10.3102/0091732X18759040
71. Singer J. D., (2018). Even more challenges in building usable knowledge in education. Journal of Research on Educational Effectiveness, 11(1), 22–24. https://doi.org/10.1080/19345747.2017.1402397
72. Stange K., (2015). Differential pricing in undergraduate education: Effects on degree production by field. Journal of Policy Analysis and Management, 34(1), 107–135. https://doi.org/10.1002/pam.21803
73. Stuart E. A., (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25(1), 1–21. https://doi.org/10.1214/09-STS313
74. Stuart E. A., Bell S. H., Ebnesajjad C., Olsen R. B., Orr L. L., (2017). Characteristics of school districts that participate in rigorous national educational evaluations. Journal of Research on Educational Effectiveness, 10(1), 168–206. https://doi.org/10.1080/19345747.2016.1205160
75. Tandberg D. A., Hillman N. W., (2014). State higher education performance funding: Data, outcomes, and policy implications. Journal of Education Finance, 39(3), 222–243.
76. Tandberg D. A., Hillman N., Barakat M., (2014). State higher education performance funding for community colleges: Diverse effects and policy implications. Teachers College Record, 116(12), 1–31.
77. Thistlethwaite D. L., Campbell D. T., (1960). Regression-discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology, 51(6), 309–317. https://doi.org/10.1037/h0044319
78. Thompson O., (2017). Head Start’s long-run impact: Evidence from the program’s introduction. Journal of Human Resources, 0216-7735r1. https://doi.org/10.3368/jhr.53.4.0216.7735R1
79. Tipton E., (2014). How generalizable is your experiment? An index for comparing experimental samples and populations. Journal of Educational and Behavioral Statistics, 39(6), 478–501. https://doi.org/10.3102/1076998614558486
80. Tipton E., Olsen R. B., (2018). A review of statistical methods for generalizing from evaluations of educational interventions. Educational Researcher, 47(8), 516–524. https://doi.org/10.3102/0013189X18781522
81. Umbricht M. R., Fernandez F., Ortagus J. C., (2017). An examination of the (un) intended consequences of performance funding in higher education. Educational Policy, 31(5), 643–673. https://doi.org/10.1177/0895904815614398
82. U.S. Department of Education. (2017). Standards handbook (Version 4.0). What Works Clearinghouse. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_standards_handbook_v4.pdf
83. van Huizen T., Plantenga J., (2018). Do children benefit from universal early childhood education and care? A meta-analysis of evidence from natural experiments. Economics of Education Review, 66, 206–222. https://doi.org/10.1016/j.econedurev.2018.08.001
84. Weiland C., Yoshikawa H., (2013). Impacts of a prekindergarten program on children’s mathematics, language, literacy, executive function, and emotional skills. Child Development, 84(6), 2112–2130. https://doi.org/10.1111/cdev.12099
85. Wing C., Bello-Gomez R. A., (2018). Regression discontinuity and beyond: Options for studying external validity in an internally valid design. American Journal of Evaluation, 39(1), 91–108. https://doi.org/10.1177/1098214017736155
86. Wing C., Simon K., Bello-Gomez R. A., (2018). Designing difference in difference studies: Best practices for public health policy research. Annual Review of Public Health, 39(1), 453–469. https://doi.org/10.1146/annurev-publhealth-040617-013507
87. Wong V. C., Wing C., Martin D., Krishnamachari A., (2018). Did states use implementation discretion to reduce the stringency of NCLB? Evidence from a database of state regulations. Educational Researcher, 47(1), 9–33. https://doi.org/10.3102/0013189X17743230