Credible causal inference for empirical legal studies

Annual Review of Law and Social Science

Volumn 7, Issue , 2011, Pages 17-40

  Ho, Daniel E ^a   Rubin, Donald B ^b  

^a STANFORD LAW SCHOOL   (United States)

^b HARVARD UNIVERSITY   (United States)

Author keywords

matching; policy evaluation; regression discontinuity; Research design

Indexed keywords

EID: 80155126103     PISSN: 15503585     EISSN: None     Source Type: Book Series    
DOI: 10.1146/annurev-lawsocsci-102510-105423     Document Type: Review

References (141)

1. Abadie A, Drukker D, Herr JL, Imbens GW. 2001. Matching estimators. Statistical Software. http://www.economics.harvard.edu/faculty/imbens/software- imbens
2. Abadie A, Gardeazabal J. 2003. The economic costs of conflict: a case study of the Basque country. Am. Econ. Rev. 93: 113-32 (Pubitemid 38799489)
3. Abrams DS, Yoon AH. 2007. The luck of the draw: using random case assignment to investigate attorney ability. Univ. Chicago Law Rev. 74: 1145-77
4. Angrist JD. 1990. Lifetime earnings and the Vietnam era draft lottery: evidence from social security administrative records. Am. Econ. Rev. 80: 1284-86
5. Angrist JD, Imbens GW, Rubin DB. 1996. Identification of causal effects using instrumental variables (with discussion). J. Am. Stat. Assoc. 91: 444-55 (Pubitemid 126412813)
6. Angrist JD, Krueger AB. 1999. Empirical strategies in labor economics. In Handbook of Labor Economics, ed. O Ashenfelter, D Card, 3A: 1277-366. Amsterdam: Elsevier
7. Angrist JD, Lavy V. 1999. Using Maimonides rule to estimate the effect of class size on scholastic achievement. Q. J. Econ. 114: 533-75
8. Angrist JD, Pischke J-S. 2008. Mostly Harmless Econometrics: An Empiricists Companion. Princeton, NJ: Princeton Univ. Press
9. Angrist JD, Pischke J-S. 2010. The credibility revolution in empirical economics: how better research design is taking the con out of econometrics. J. Econ. Perspect. 24: 3-30
10. Autor DH, Donohue JJ III, Schwab SJ. 2004. The costs of wrongful-discharge laws: large, small, or none at all? Am. Econ. Rev. 94: 440-46
11. Ayres I. 1991. Fair driving: gender and race discrimination in retail car negotiations. Harvard Law Rev. 104: 817-72
12. Ayres I. 2008. Super Crunchers: Why Thinking-by-Numbers Is the New Way to Be Smart. New York: Bantam
13. Barnard J, Frangakis CE, Hill JL, Rubin DB. 2003. Principal stratification approach to broken randomized experiments: a case study of school choice vouchers in New York (with discussion). J. Am. Stat. Assoc. 98: 299-311
14. Bayer P, Hjalmarsson R, Pozen D. 2009. Building criminal capital behind bars: peer effects in juvenile corrections. Q. J. Econ. 124: 105-47
15. Becker SO, Ichino A. 2002. Stata programs for ATT estimation based on propensity score matching. Statistical Software. http://www.lrz.de/?sobecker/ pscore.html
16. Bench LL, Allen TD. 2003. Investigating the stigma of prison classification: an experimental design. Prison J. 83: 367-82
17. Berk RA. 2004. Regression Analysis: A Constructive Critique. Thousand Oaks, CA: Sage
18. Berk RA, de Leeuw J. 1999. An evaluation of Californias inmate classification system using a generalized regression discontinuity design. J. Am. Stat. Assoc. 94: 1045-52
19. Berk RA, LaddH, razianoH, Baek JH. 2003. A randomized experiment testing inmate classification systems. Criminol. Public Policy 2: 215-42
20. Berk RA, Newton PJ. 1985. Does arrest really deter wife battery? An effort to replicate the findings of the Minneapolis Spouse Abuse Experiment. Am. Sociol. Rev. 50: 253-62
21. Berry CR, Lee SL. 2007. The Community Reinvestment Act: a regression discontinuity analysis. Harris Sch. Work. Pap. Ser. 07.04, Univ. Chicago, Chicago, IL
22. Bertrand M, Duflo E, Mullainathan S. 2004. How much should we trust differences-in-differences estimates? Q. J. Econ. 119: 249-75 (Pubitemid 38361206)
23. Black BS, Kim W, Jang H, Park KS. 2008. How corporate governance affects firm value: evidence on channels from Korea. ECGI Fin.Work. Pap. No. 103/2005. http://ssrn.com/abstract=844744
24. Boyd CL, Epstein L, Martin AD. 2010. Untangling the causal effects of sex on judging. Am. J. Polit. Sci. 54: 389-411
25. Brady HE, McNulty JE. 2007. The costs of voting: disruption and transportation effects. Presented at Midwest Polit. Sci. Assoc., Chicago, IL, April 12. http://www.can-so.org/vote/polling-locationsand-transportation- effects.pdf
26. Bubb R. 2009. States, law, and property rights in West Africa. Work. Pap., Dep. Econ., Harvard Univ. http://isites.harvard.edu/fs/docs/icb. topic637140.files/Bubb-StatesLawProperty.pdf
27. Calif. Dep. Correct. (CDC). 2000. Operations Manual. Sacramento: State Calif.
28. Camp SD, Gaes GG. 2005. Criminogenic effects of the prison environment on inmate behavior: some experimental evidence. Crime Delinquency 51: 425-42 (Pubitemid 40855877)
29. Card D, Dobkin C, Maestas N. 2008. The impact of nearly universal insurance coverage on health care: evidence from Medicare. Am. Econ. Rev. 98: 2242-58
30. Card DE, Krueger AB. 1994. Minimum wages and employment: a case study of the fast food industry in New Jersey and Pennsylvania. Am. Econ. Rev. 84: 772-84
31. Chay KY, Greenstone M. 2005. Does air quality matter? Evidence from the housing market. J. Polit. Econ. 113: 376-424 (Pubitemid 41057128)
32. Chen MK, Shapiro JM. 2007. Do harsher prison conditions reduce recidivism? A discontinuity-based approach. Am. Law Econ. Rev. 9: 1-29
33. Clark W, Douglas WO, Thomas DS. 1930. The business failures project-a problem in methodology. Yale Law J. 39: 1013-24
34. Dehejia RH, Wahba S. 1999. Causal effects in nonexperimental studies: reevaluating the evaluation of training programs. J. Am. Stat. Assoc. 94: 1053-62
35. Dehejia RH, Wahba S. 2002. Propensity score-matching methods for nonexperimental causal studies. Rev. Econ. Stat. 84: 151-61
36. DiNardo J, Lee DS. 2004. Economic impacts of new unionization on private sector employers: 1984-2001. Q. J. Econ. 119: 1383-441 (Pubitemid 39636918)
37. Donohue JJ III, Ho DE. 2007. The impact of damage caps on malpractice claims: randomization inference with difference-in-differences. J. Empir. Legal Stud. 4: 69-102
38. Donohue JJ III, Wolfers J. 2006. Uses and abuses of empirical evidence in the death penalty debate. Stanford Law Rev. 58: 791-845
39. Douglas WO. 1950. Underhill Moore. Yale Law J. 59: 187-88
40. Eggers AC, ainmueller J. 2009. Mps for sale? Returns to office in postwar British politics. Am. Polit. Sci. Rev. 103: 513-33
41. Epstein L, Ho DE, King G, Segal JA. 2005. The Supreme Court during crisis: how war affects only non-war cases. N. Y. Univ. Law Rev. 80: 1-116 (Pubitemid 40687870)
42. Epstein L, King G. 2002. The rules of inference. Univ. Chicago Law Rev. 69: 1-133
43. Fisher RA. 1925. Statistical Methods for Research Workers. Edinburgh: Oliver & Boyd
44. Fisher RA. 1935. The Design of Experiments. Edinburgh: Oliver & Boyd
45. Frangakis CE, Rubin DB. 2002. Principal stratification in causal inference. Biometrics 58: 21-29 (Pubitemid 34194484)
46. Galiani S, Gertler P, Schargrodsky E. 2005. Water for life: the impact of the privatization of water services on child mortality. J. Polit. Econ. 113: 83-120 (Pubitemid 40427105)
47. Gelman A, Meng X-L, eds. 2004. Applied BayesianModeling and Causal Inference from Incomplete Data Perspectives. Hoboken, NJ: Wiley
48. Gerber A, Kessler DP, Meredith MN. 2008. The persuasive effects of direct mail: a regression discontinuity approach. NBER Work. Pap. 14206, Natl. Bur. Econ. Res., Cambridge, MA. http://www.nber.org/papers/w14206.pdf
49. Gerber AS, Green DP. 2000. The effects of canvassing, telephone calls, and direct mail on voter turnout: a field experiment. Am. Polit. Sci. Rev. 94: 653-63
50. Gibson JL. 2008. Challenges to the impartiality of state supreme courts: legitimacy theory and new-style judicial campaigns. Am. Polit. Sci. Rev. 102: 59-75
51. Green DP, Winik D. 2010. Using random judge assignments to estimate the effects of incarceration and probation on recidivism among drug offenders. Criminology 48: 357-87
52. Greiner DJ. 2008. Causal inference in civil rights litigation. Harvard Law Rev. 122: 533-98
53. Greiner DJ, Rubin DB. 2010. Causal effects of perceived immutable characteristics. Rev. Econ. Stat. In press
54. Grogger J, Ridgeway G. 2006. Testing for racial profiling in traffic stops from behind a veil of darkness. J. Am. Stat. Assoc. 101: 878-87 (Pubitemid 44422591)
55. Guttentag MD, Porath CL, Fraidin SN. 2008. Brandeis policeman: results from a laboratory experiment on how to prevent corporate fraud. J. Empir. Legal Stud. 5: 239-73
56. Hahn J, Todd P, Kasarda JD. 1999. Evaluating the effect of an antidiscrimination law using a regressiondiscontinuity design. NBER Work. Pap. 7131, Natl. Bur. Econ. Res., Cambridge, MA. http://www.nber.org/papers/w7131.pdf
57. Hahn J, Todd P, van der Klaauw W. 2001. Identification and estimation of treatment effects with a regressiondiscontinuity design. Econometrica 69: 201-209 (Pubitemid 33635744)
58. Hansen BB. 2004. Full matching in an observational study of coaching for the SAT. J. Am. Stat. Assoc. 99: 609-18 (Pubitemid 39332846)
59. Hansen BB, Fredrickson M. 2010. optmatch: functions for optimal matching. Statistical Software http://cran. r-project.org/web/packages/optmatch/index. html
60. Hastie TJ, Tibshirani R. 1990. Generalized Additive Models. London: Chapman Hall
61. Heckman J, Ichimura H, Smith J, Todd P. 1998a. Characterizing selection bias using experimental data. Econometrica 66: 1017-98 (Pubitemid 128467217)
62. Heckman JJ, Ichimura H, Todd P. 1998b. Matching as an econometric evaluation estimator. Rev. Econ. Stud. 65: 261-94 (Pubitemid 128467263)
63. Heckman JJ, Lalonde RJ, Smith JA. 1999. The economics and econometrics of active labor market programs. In Handbook of Labor Economics, ed. O Ashenfelter, D Card, 3A: 1865-2097. Amsterdam: Elsevier
64. Helland E, Tabarrok A. 2004. The fugitive: evidence on public versus private law enforcement from bail jumping. J. Law Econ. 47: 93-122 (Pubitemid 39080471)
65. Hirano K, Imbens GW. 2004. The propensity score with continuous treatments. In Applied BayesianModeling and Causal Inference from Incomplete-Data Perspectives, ed. A Gelman, X-L Meng, pp. 73-83. Hoboken, NJ: Wiley
66. Hirano K, Imbens GW, Rubin DB, Zhou XH. 2000. Assessing the effect of an influenza vaccine in an encouragement design. Biostatistics 1: 69-88 (Pubitemid 33213522)
67. HjalmarssonR. 2009a.Crime and expected punishment: changes in perceptions at the age of criminal majority. Am. Law Econ. Rev. 11: 209-48
68. Hjalmarsson R. 2009b. Juvenile jails: a path to the straight and narrow or to hardened criminality? J. Law Econ. 52: 779-809
69. Ho DE. 2005a. Affirmative actions affirmative actions: a reply to Sander. Yale Law J. 114: 2011-16
70. Ho DE. 2005b. Why affirmative action does not cause black students to fail the bar. Yale Law J. 114: 1997-2004 (Pubitemid 41031853)
71. Ho DE, Imai K. 2006. Randomization inference with natural experiments: an analysis of ballot effects in the 2003 California recall election. J. Am. Stat. Assoc. 101: 888-900 (Pubitemid 44422592)
72. Ho DE, Imai K, King G, Stuart EA. 2004. MatchIt: nonparametric preprocessing for parametric causal inference. Statistical Software. http://gking.harvard.edu/matchit
73. Ho DE, Imai K, King G, Stuart EA. 2007. Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Polit. Anal. 15: 199-236
74. Holland PW. 1986. Statistics and causal inference. J. Am. Stat. Assoc. 81: 945-60
75. Hopkins DJ. 2009. Language access and initiative outcomes: Did the Voting Rights Act influence support for bilingual education? CELS 2009 4th Annu. Conf. Empir. Legal Stud. Pap. http://ssrn.com/abstract=1434374.
76. Iacus SM, King G, Porro G. 2009a. Causal inference without balance checking: coarsened exact matching. Polit. Anal. In press. http://gking.harvard. edu/gking/files/cem-plus.pdf
77. Iacus SM, King G, Porro G. 2009b. CEM: coarsened exact matching software. Statistical Software. http://gking.harvard.edu/cem
78. Imai K, van Dyk DA. 2004. Causal inference with general treatment regimes: generalizing the propensity score. J. Am. Stat. Assoc. Theory Methods 99: 854-66 (Pubitemid 39332867)
79. ImbensGW. 2000. The role of the propensity score in estimating dose-response functions. Biometrika 87: 706-10
80. Imbens GW. 2004. Nonparametric estimation of average treatment effects under exogeneity: a review. Rev. Econ. Stat. 86: 4-29 (Pubitemid 38422764)
81. Imbens GW, Lemieux T. 2008. Regression discontinuity designs: a guide to practice. J. Econ. 142: 615-35 (Pubitemid 350236542)
82. Imbens GW, Rosenbaum PR. 2005. Robust, accurate confidence intervals with a weak instrument: quarter of birth and education. J. R. Stat. Soc. Ser. A 168: 109-26 (Pubitemid 40166964)
83. Joffe MM, Rosenbaum PR. 1999. Propensity scores. Am. J. Epidemiol. 150: 327-33
84. Kane TJ, Riegg SK, Staiger DO. 2006. School quality, neighborhoods, and housing prices. Am. Law Econ. Rev. 8: 183-212 (Pubitemid 44541971)
85. King G, Gakidou E, RavishankarN, oore RT, Lakin J, et al. 2007. A politically robust experimental design for public policy evaluation, with application to the Mexican universal health insurance program. J. Policy Anal. Manag. 26: 479-506
86. King G, Zeng L. 2007. When can history be our guide? The pitfalls of counterfactual inference. Int. Stud. Q. 51: 183-210 (Pubitemid 46344556)
87. Kritzer HM. 2010. The (nearly) forgotten early empirical legal research. In Oxford Handbook of Empirical Legal Research, ed. P Cane, HM Kritzer, pp. 875-900. Oxford: Oxford Univ. Press
88. Lalive R. 2008.Howdo extended benefits affect unemployment duration? A regression discontinuity approach. J. Econom. 142: 785-806 (Pubitemid 350236547)
89. Lalonde R. 1986. Evaluating the econometric evaluations of training programs. Am. Econ. Rev. 76: 604-20
90. Leamer EE. 1978. Specification Searches. New York: JohnWiley & Sons
91. Leamer EE. 1983. Lets take the con out of econometrics. Am. Econ. Rev. 73: 31-43
92. Lee DS. 2008. Randomized experiments from non-random selection in U.S. house elections. J. Econom. 142: 675-97 (Pubitemid 350245707)
93. Lee DS, Lemieux T. 2010. Regression discontinuity designs in economics. J. Econ. Lit. 48: 281-355
94. Lee DS, McCrary J. 2005. Crime, punishment, and myopia. NBER Work. Pap. 11491, Natl. Bur. Econ. Res., Cambridge, MA. http://www.nber.org/papers/w11491
95. Lemieux T, Milligan K. 2008. Incentive effects of social assistance: a regression discontinuity approach. J. Econom. 142: 807-28 (Pubitemid 350236548)
96. List JA, Margolis M, Osgood DE. 2006. Is the Endangered Species Act endangering species? NBER Work. Pap. 12777, Natl. Bur. Econ. Res., Cambridge, MA. http://www.nber.org/papers/w12777
97. Listokin Y. 2008. Management always wins the close ones. Am. Law Econ. Rev. 10: 159-84
98. Listokin Y. 2009. Corporate voting versus market price setting. Am. Law Econ. Rev. 11: 608-35
99. Litvak K. 2007. Sarbanes-Oxley and the cross-listing premium. Mich. Law Rev. 105: 1857-98
100. Ludwig J, Miller DL. 2007. Does Head Start improve childrens life chances? Evidence from a regression discontinuity design. Q. J. Econ. 122: 159-208
101. Manski CF. 1990. Nonparametric bounds on treatment effects. Am. Econ. Rev. Papers Proc. 80: 319-23
102. Manski CF. 1995. Identification Problems in the Social Sciences. Cambridge, MA: Harvard Univ. Press
103. McCrary J. 2008. Manipulation of the running variable in the regression discontinuity design: a density test. J. Econom. 142: 698-714 (Pubitemid 350236545)
104. Mocan NH, Tekin E. 2006. Catholic schools and bad behavior: a propensity score matching analysis. B.E. J. Econ. Anal. Policy 5(1): 13. http://www.bepress.com/bejeap/contributions/vol5/iss1/art13
105. Moore U, Callahan CC. 1943. Law and learning theory: a study in legal control. Yale Law J. 53: 1-136
106. Morantz AD. 2010. Coal mining safety: Do unions make a difference? Stanford Law Econ. Olin Work. Pap. No. 413. http://papers.ssrn.com/sol3/papers. cfm?abstract-id=1846700
107. Morgan SL, Winship C. 2007. Counterfactuals and Causal Inference: Methods and Principles for Social Research. Cambridge, UK: Cambridge Univ. Press
108. Pager D. 2003. The mark of a criminal record. Am. J. Sociol. 108: 937-75 (Pubitemid 37374313)
109. Papachristos AV, Meares TL, Fagan J. 2007. Attention felons: evaluating Project Safe Neighborhoods in Chicago. J. Empir. Legal Stud. 4: 223-72
110. Pearl J. 2000. Causality. New York: Cambridge Univ. Press
111. Persson T, Tabellini G. 2002. Do constitutions cause large governments? Quasi-experimental evidence. Eur. Econ. Rev. 46: 908-18 (Pubitemid 34580836)
112. Petersilia J. 2008. Californias correctional paradox of excess and deprivation. Crime Justice 37: 207-78
113. Petersilia J, Turner S, Peterson J. 1986. Prison versus Probation in California. Santa Monica, CA: RAND
114. Pfaff JF. 2010. A plea for more aggregation: the looming threat to empirical legal scholarship. SSRN Work. Pap., July 16. http://ssrn.com/abstract= 1641435
115. Qian Y. 2007. Do national patent laws stimulate domestic innovation in a global patenting environment? A cross-country analysis of pharmaceutical patent protection, 1978-2002. Rev. Econ. Stat. 89: 436-53
116. Ridgeway G. 2006. Assessing the effect of race bias in post-traffic stop outcomes using propensity scores. J. Quant. Criminol. 22: 1-29 (Pubitemid 43681760)
117. Rosenbaum PR. 2002. Observational Studies. New York: Springer. 2nd ed.
118. Rosenbaum PR, Rubin DB. 1983a. Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. J. R. Stat. Soc. Ser. B Methodol. 45: 212-18
119. Rosenbaum PR, Rubin DB. 1983b. The central role of the propensity score in observational studies for causal effects. Biometrika 70: 41-55
120. Rosenbaum PR, RubinDB. 1984. Reducing bias in observational studies using subclassification on the propensity score. J. Am. Stat. Assoc. 79: 516-24
121. Rubin DB. 1973. Matching to remove bias in observational studies. Biometrics 29: 159-83
122. Rubin DB. 1974. Estimating causal effects of treatments in randomized and nonrandomized studies. J. Educ. Psychol. 66: 688-701
123. Rubin DB. 1975. Bayesian inference for causality: the importance of randomization. Proc. Soc. Stat. Sect. Am. Stat. Assoc., pp. 233-39. Alexandria, VA: Am. Stat. Assoc.
124. Rubin DB. 1976. Multivariate matching methods that are equal percent bias reducing. I: Some examples. Biometrics 32: 109-20
125. Rubin DB. 1977. Assignment to treatment group on the basis of a covariate. J. Educ. Behav. Stat. 2: 1-26
126. Rubin DB. 1978. Bayesian inference for causal effects: the role of randomization. Ann. Stat. 6: 34-58
127. Rubin DB. 1979. Using multivariate matched sampling and regression adjustment to control bias in observational studies. J. Am. Stat. Assoc. 74: 318-28
128. Rubin DB. 2006. Matched Sampling for Causal Effects. New York: Cambridge Univ. Press
129. Rubin DB. 2008. For objective causal inference, design trumps analysis. Ann. Appl. Stat. 2: 808-40
130. Rubinfeld DL. 2010. Econometric issues in antitrust analysis. J. Inst. Theor. Econ. 166: 62-77
131. Schlegel JH. 1995. American Legal Realism and Empirical Social Science. Chapel Hill: Univ. N. C. Press
132. Sekhon JS. 2009. Opiates for the matches: matching methods for causal inference. Annu. Rev. Polit. Sci. 12: 487-508
133. Sekhon JS. 2011. Multivariate and propensity score matching software with automated balance optimization: the matching package for R. J. Stat. Softw. In press. http://sekhon.berkeley.edu/matching
134. Sobel ME. 2000. Causal inference in the social sciences. J. Am. Stat. Assoc. 95: 647-51
135. Stigler SM. 1977. Eight centuries of sampling inspection: the trial of the Pyx. J. Am. Stat. Assoc. 72: 493-500
136. Strnad J. 2007. Should legal empiricists go Bayesian? Am. Law Econ. Rev. 9: 195-303
137. Stuart EA, Rubin DB. 2008. Best practices in quasi-experimental designs. In Best Practices in Quantitative Methods, ed. JW Osborne, pp. 155-76. Thousand Oaks, CA: Sage
138. Thistlethwaite DL, Campbell DT. 1960. Regression-discontinuity analysis: an alternative to the ex post facto experiment. J. Educ. Psychol. 51: 309-17
139. Urquiola M, Verhoogen E. 2009. Class-size caps, sorting, and the regression-discontinuity design. Am. Econ. Rev. 99: 179-215
140. van der Klaauw W. 2002. Estimating the effect of financial aid offers on college enrollment: a regressiondiscontinuity approach. Int. Econ. Rev. 43: 1249-87
141. Zimring FE, Hawkins G. 1973. Deterrence: The Legal Threat in Crime Control. Chicago: Univ. Chicago Press