What Would It Take to Change an Inference? Using Rubin's Causal Model to Interpret the Robustness of Causal Inferences

Educational Evaluation and Policy Analysis

Volumn 35, Issue 4, 2013, Pages 437-460

  Frank, Kenneth A ^a   Maroulis, Spiro J ^b   Duong, Minh Q ^c   Kelcey, Benjamin M ^d  

^a MICHIGAN STATE UNIVERSITY   (United States)

^b ARIZONA STATE UNIVERSITY   (United States)

^c Pacific Metrics Corporation   (United States)

^d UNIVERSITY OF CINCINNATI   (United States)

Author keywords

causal inference; observational studies; Rubin's causal model; sensitivity analysis

Indexed keywords

EID: 84886053476     PISSN: 01623737     EISSN: 19351062     Source Type: Journal    
DOI: 10.3102/0162373713493129     Document Type: Article

References (107)

1. Abbott A.The causal devolution.Sociological Methods & Research. 1998;27:148-181.
2. Alexander K. L.Response to Shepard, Smith and Marion.Psychology in Schools. 1998;9:410-417.
3. Alexander K. L.,Entwisle D. R.,Dauber S. L.On the success of failure: A reassessment of the effects of retention in the primary school grades. Cambridge, UK: Cambridge University Press; 2003:.
4. Alexander K. L.,Pallas A. M.Private schools and public policy: New evidence on cognitive achievement in public and private schools.Sociology of Education. 1983;56:170-182.
5. AltonjiJ. G.ConleyT.ElderT.TaberC. (2010). Methods for using selection on observed variables to address selection on unobserved variables. Retrieved from https://www.msu.edu/~telder/.
6. Altonji J. G.,Elder T.,Taber C.An evaluation of instrumental variable strategies for estimating the effects of Catholic schooling.Journal of Human Resources. 2005;40:791-821.
7. An Brian P.The impact of dual enrollment on college degree attainment: Do low-SES students benefit?.Educational Evaluation and Policy Analysis. 2013;35:57-75.
8. Becker H. H.Whose side are we on?.Social Problems. 1967;14:239-247.
9. Behn R. D.,Vaupel J. W.Quick analysis for busy decision makers. New York, NY: Basic Books; 1982:.
10. Bogatz G. A.,Ball S.The impact of "sesame street" on children's first school experience. Princeton, NJ: Educational Testing Service; 1972:.
11. Borman G. D.,Dowling N. M.,Schneck C.A multi-site cluster randomized field trial of open court reading.Educational Evaluation and Policy Analysis. 2008;30:389-407.
12. Bozick R.,Dalton B.Balancing career and technical education with academic coursework: The consequences for mathematics achievement in high school.Educational Evaluation and Policy Analysis. 2013;35:123-138.
13. Brian P.The impact of dual enrollment on college degree attainment: Do low-SES students benefit?.Educational Evaluation and Policy Analysis. 2013;35:57-75.
14. Brennan R. L.The conventional wisdom about group mean scores.Journal of Educational Measurement. 1995;32:385-396.
15. Bulterman-Bos J. A.Will a clinical approach make education research more relevant for practice?.Educational Researcher. 2008;37:412-420.
16. Burkam D. T.,LoGerfo L.,Ready D.,Lee V. E.The differential effects of repeating kindergarten.Journal of Education for Students Placed at Risk. 2007;12:103-136.
17. CampbellD. T. (1976, December). Assessing the impact of planned social change. The Public Affairs Center, Dartmouth College, Hanover New Hampshire, USA. Retrieved from https://www.globalhivmeinfo.org/CapacityBuilding/Occasional%20Papers/08%20Assessing%20the%20Impact%20of%20Planned%20Social%20Change.pdf.
18. Carlson D.,Cowen J. M.,Fleming D. J.Life after vouchers: What happens to students who leave private schools for the traditional public sector?.Educational Evaluation and Policy Analysis. 2013;35:179-199.
19. Chubb J. E.,Moe T. M.Politics, markets, and America's schools. Washington, DC: The Brookings Institution; 1990:.
20. Clements D. H.,Sarama J.Experimental evaluation of the effects of a research-based preschool mathematics curriculum.American Educational Research Journal. 2008;45:443-494.
21. Coleman J. S.,Hoffer T.,Kilgore S.High school achievement: Public, catholic, and private schools compared. New York, NY: Basic Books; 1982:.
22. Cook T. D.Randomized experiments in educational policy research: A critical examination of the reasons the educational evaluation community has offered for not doing them.Educational Evaluation and Policy Analysis. 2002;24:175-199.
23. Cook T. D.Why have educational evaluators chosen not to do randomized experiments?.Annals of American Academy of Political and Social Science. 2003;589:114-149.
24. Cook T. D.,Campbell D. T.Quasi-experimentation: Design and analysis issues for field settings. Chicago, IL: Rand McNally; 1979:.
25. Copas J. B.,Li H. G.Inference for non-random samples.Journal of the Royal Statistical Society, Series B (Methodological). 1997;59:55-95.
26. Cronbach L. J.Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass; 1982:.
27. ECLS-K user guide. (2001).Retrieved from http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2002149.
28. Eisenhart M.,Towne L.Contestation and change in national policy on "scientifically based" education research.Educational Researcher. 2008;32:31-38.
29. Engel M.,Claessens A.,Finch M. A.Teaching students what they already know? The (Mis)Alignment between mathematics instructional content and student knowledge in kindergarten.Educational Evaluation and Policy Analysis. 2013;35:157-178.
30. Federal Register. (1998). Federal Register, 1998. 63(179). Retrieved from http://www.fda.gov/downloads/RegulatoryInformation/Guidances/UCM129505.pdf.
31. Finn J. D.,Achilles C. M.Answers and questions about class size: A statewide experiment.American Educational Research Journal. 1990;27:557-577.
32. Fisher R.,Sir A.Statistical methods for research workers. Darien, CT: Hafner; 1970:.
33. Frank K. A.Impact of a confounding variable on the inference of a regression coefficient.Sociological Methods & Research. 2000;29:147-194.
34. Frank K. A.,Min K.Indices of robustness for sample representation.Sociological Methodology. 2007;37:349-392.
35. Frank K. A.,Sykes G.,Anagnostopoulos D.,Cannata M.,Chard L.,Krause A.,McCrory R.Extended influence: National board certified teachers as help providers.Education, Evaluation and Policy Analysis. 2008;30:3-30.
36. Gastwirth J. L.,Krieger A. M.,Rosenbaum P. R.Dual and simultaneous sensitivity analysis for matched pairs.Biometrika. 1998;85:907-920.
37. Greco J.Epistemic value. Haddock A.Millar A.Pritchard D. H., ed. Oxford, UK: Oxford University Press; 2009:313-321.
38. Grigg J.,Kelly K. A.,Gamoran A.,Borman G. D.Effects of two scientific inquiry professional development interventions on teaching practice.Educational Evaluation and Policy Analysis. 2013;35:38-56.
39. Habermas J.Knowledge and human interests. Cambridge, UK: Polity Press; 1987:.
40. Harding D. J.Counterfactual models of neighborhood effects: The effect of neighborhood poverty on dropping out and teenage pregnancy.American Journal of Sociology. 2003;109:676-719.
41. Repeating a grade: Dopes it help?.Harvard Education Letter. 1986;2:1-4.
42. Heckman J.Sample selection bias as a specification error.Econometrica. 1979;47:153-161.
43. Heckman J.The scientific model of causality.Sociological Methodology. 2005;35:1-99.
44. Heckman J.,Urzua S.,Vytlacil E.Understanding instrumental variables in models with essential heterogeneity.Review of Economics and Statistics. 2006;88:389-432.
45. HedgesL.O'MuircheartaighC. (2011). Generalization from experiments. Retrieved from http://steinhardt.nyu.edu/scmsAdmin/uploads/003/585/Generalization%20from%20Experiments-Hedges.pdf.
46. Hill H. C.,Rowan B.,Ball D. L.Effects of teachers' mathematical knowledge for teaching on student achievement.American Educational Research Journal. 2005;42:371-406.
47. Hirano K.,Imbens G. W.Estimation of causal effects using propensity score weighting: An application to data on right heart catheterization.Health Services and Outcomes Research Methodology. 2001;2:259-278.
48. Holland P. W.Statistics and causal inference.Journal of the American Statistical Association. 1986;81:945-970.
49. Holland P. W.Choosing among alternative nonexperimental methods for estimating the impact of social programs: The case of manpower training: Comment.Journal of the American Statistical Association. 1989;84:875-877.
50. Holmes C. T.Flunking grades. Shepard L. A.Smith M. L., ed. New York, NY: Falmer Press; 1989:16-33.
51. Holmes C. T.,Matthews K.The Effects of nonpromotion on elementary and junior high school pupils: A meta analysis.Review of Educational Research. 1984;54:225-236.
52. Hong G.,Raudenbush S. W.Effects of kindergarten retention policy on children's cognitive growth in reading and mathematics.Educational Evaluation and Policy Analysis. 2005;27:205-224.
53. Ichino A.,Mealli F.,Nannicini T.From temporary help jobs to permanent employment: What can we learn from matching estimators and their sensitivity?.Journal of Applied Econometrics. 2008;23:305-327.
54. Imai K.,Keele L.,Yamamoto T.Identification, inference and sensitivity analysis for causal mediation effects.Statistical Science. 2010;25:51-71.
55. Jimerson S.Meta-analysis of grade retention research: Implications for practice in the 21st century.School Psychology Review. 2001;30:420-437.
56. Karweit N. L.Encyclopedia of educational research. Alkin M., ed. New York, NY: Macmillan; 1992:114-118.
57. Kuhn T.The structure of scientific revolutions. Chicago, IL: University of Chicago Press; 1962:.
58. Kvanvig J. L.The value of knowledge and the pursuit of understanding. Oxford, UK: Oxford University Press; 2003:.
59. Lin D. Y.,Psaty B. M.,Kronmal R. A.Assessing the sensitivity of regression results to unmeasured confounders in observational studies.Biometrics. 1998;54:948-963.
60. Lorence J.,Dworkin G.,Toenjes L.,Hill A.Brookings papers on education policy. Ravitch D., ed. Washington, DC: Brookings Institution Press; 2002:13-67.
61. Manski C.Nonparametric bounds on treatment effects.American Economic Review Papers and Proceedings. 1990;80:319-323.
62. Mariano L. T.,Martorell P.The academic effects of summer instruction and retention in New York City.Educational Evaluation and Policy Analysis. 2013;35:96-117.
63. Maroulis S.,Guimera R.,Petry H.,Gomez L.,Amaral L. A. N.,Wilensky U.A complex systems view of educational policy.Science. 2010;330:38-39.
64. Miller S.,Connolly P.A randomized controlled trial evaluation of time to read, a volunteer tutoring program for 8- to 9-year-olds.Educational Evaluation and Policy Analysis. 2013;35:23-37.
65. Morgan S. L.Counterfactuals, causal effect heterogeneity, and the Catholic school effect on learning.Sociology of Education. 2001;74:341-374.
66. Morgan S. L.,Winship C.Counterfactuals and causal inference: Methods and principles for social research. Cambridge, UK: Cambridge University Press; 2007:.
67. National Reading Panel. (2000). Report of the National Reading Panel. Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction (NIH Publication No.00-4769). Washington, DC: U.S. Government Printing Office.
68. Scientific research in education. Washington, DC: National Academy Press; 2002:.
69. Nomi T.The unintended consequences of an algebra-for-all policy on high-skill students: effects on instructional organization and students' academic outcomes.Educational Evaluation and Policy Analysis. 2012;34:489-505.
70. Nye B.,Konstantopoulos S.,Hedges L. V.How large are teacher effects?.Educational Evaluation and Policy Analysis. 2004;26:237-257.
71. Oakley A.Experimentation and social interventions: A forgotten but important history.British Medical Journal. 1998;317:1239-1242.
72. Olkin I.,Pratt J. W.Unbiased estimation of certain correlation coefficients.Annals of Mathematical Statistics. 1958;29:201-211.
73. PearlJ.BareinboimE. (2010, October). Transportability across studies: A formal approach. Retrieved from http://ftp.cs.ucla.edu/pub/stat_ser/r372.pdf.
74. Raudenbush S. W.Learning from attempts to improve schooling: The contribution of methodological diversity.Educational Researcher. 2005;34:25-31.
75. Reynolds A. J.Grade retention and school adjustment: An explanatory analysis.Educational Evaluation and Policy Analysis. 1992;14:101-121.
76. Robins J.,Rotnisky A.,Scharfstein D.Statistical models in epidemiology. Hallorn E., ed. New York, NY: Springer; 2000:1-95.
77. Roderick M.,Bryk A. S.,Jacobs B. A.,Easton J. Q.,Allensworth E.Ending social promotion: Results from the first two years. Chicago, IL: Consortium on Chicago School Research; 1999:.
78. Rosenbaum P. R.Dropping out of high school in the United States: An observational study.Journal of Educational Statistics. 1986;11:207-224.
79. Rosenbaum P. R.Observational studies. New York, NY: Springer; 2002:.
80. Rosenbaum P. R.,Rubin D. B.Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome.Journal of the Royal Statistical Society (Series B). 1983;45:212-218.
81. Rubin D. B.Estimating causal effects of treatments in randomized and nonrandomized studies.Journal of Educational Psychology. 1974;66:688-701.
82. Rubin D. B.Which ifs have causal answers? Discussion of Holland's "statistics and causal inference.".Journal of American Statistical Association. 1986;83:396.
83. Rubin D. B.Formal modes of statistical inference for causal effects.Journal of Statistical Planning and Inference. 1990;25:279-292.
84. Rubin D. B.Teaching statistical inference for causal effects in experiments and observational studies.Journal of Educational and Behavioral Statistics. 2004;29:343-368.
85. Saunders W. M.,Marcelletti D. J.The gap that can't go away: The Catch-22 of reclassification in monitoring the progress of English learners.Educational Evaluation and Policy Analysis. 2013;35:139-156.
86. Scharfstein D. A. I.Generalized additive selection models for the analysis of studies with potentially non-ignorable missing data.Biometrics. 2002;59:601-613.
87. SchneiderB. M.CarnoyJ.KilpatrickW. H.SchmidtShavelsonR. J. (2007). Estimating causal effects using experimental and observational designs. AERA. Retrieved from http://www.aera.net/Publications/Books/EstimatingCausalEffectsUsingExperimentaland/tabid/12625/Default.aspx.
88. Schweinhart L. J.,Barnes H. V.,Weikart D. P.Barnett W. S.,Epstein A. S.Significant benefits: The high/scope Perry preschool study through age 27. Ypsilanti, MI: High/Scope Press; 1993:.
89. Shadish W. R.,Clark M. H.,Steiner P. M.Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment.Journal of the American Statistical Association. 2008;103:1334-1344.
90. Shadish W. R.,Cook T. D.,Campbell D. T.Experimental and quasi-experimental designs for generalized causal inference. New York, NY: Houghton Mifflin; 2002:.
91. Shager H. M.,Schindler H. S.,Magnuson K. A.,Duncan G. J.,Yoshikawa H.,Hart C. M. D.Can research design explain variation in head start research results? A meta-analysis of cognitive and achievement outcomes.Educational Evaluation and Policy Analysis. 2013;35:76-95.
92. Shepard L. A.,Smith M. L.Flunking grades. New York, NY: Falmer Press; 1989:.
93. Shepard L. A.,Smith M. L.,Marion S. F.On the success of failure: A rejoinder to Alexander.Psychology in the Schools. 1998;35:404-406.
94. Slavin R. E.Perspectives on evidence-based research in education-what works? Issues in synthesizing educational program evaluations.Educational Researcher. 2008;37:5-14.
95. Sosa E.A virtue epistemology. Oxford, UK: Oxford University Press; 2007:.
96. Technical report performance assessments. Monterey, CA: TerraNova; 2001:.
97. Steiner P. M.,Cook T. D.,Shadish W. R.On the importance of reliable covariate measurement in selection bias adjustments using propensity scores.Journal of Educational and Behavioral Statistics. 2011;36:213-236.
98. Steiner P. M.,Cook T. D.,Shadish W. R.,Clark M. H.The importance of covariate selection in controlling for selection bias in observational studies.Psychological Methods. 2010;15:250-267.
99. Stephan J. L.,Rosenbaum J. E.Can high schools reduce college enrollment gaps with a new counseling model?.Educational Evaluation and Policy Analysis. 2013;35:200-219.
100. Stuart E. A.,Cole S. R.,Bradshaw C. P.,Leaf P. J.The use of propensity scores to assess the generalizability of results from randomized trials.Journal of the Royal Statistical Society, Series A. 2011;174:369-386.
101. Thorndike E. L.,Woodworth R. S.The influence of improvement in one mental function upon the efficiency of other functions.Psychological Review. 1901;8:247-261, 384-395, 553-564.
102. US Department of Education. (2002). Evidence based education. Retrieved from http://www.ed.gov/nclb/methods/whatworks/eb/edlite-index.html.
103. Trends in the well-being of America's children and youth. Washington, DC:; 2000:.
104. Wainer H.,Robinson D. H.Shaping up the practice of null hypothesis significance testing.Educational Researcher. 2003;32:22-30.
105. Wilkinson L.Statistical methods in psychology journals: Guidelines and explanations.American Psychologist. 1999;54:594-604.
106. Winship C.,Stephen L. M.The estimation of causal effects from observational data.Annual Review of Sociology. 1999;25:659-706.
107. Yuan K.,Le V.,McCaffrey D. F.,Marsh J. A.,Hamilton L. S.,Stecher B. M.,Springer M. G.Incentive pay programs do not affect teacher motivation or reported practices: Results from three randomized studies.Educational Evaluation and Policy Analysis. 2013;35:3-22.