Designing Difference in Difference Studies: Best Practices for Public Health Policy Research

Annual Review of Public Health

Volumn 39, Issue , 2018, Pages 453-469

  Wing, Coady ^a   Simon, Kosali ^a   Bello Gomez, Ricardo A ^a  

^a INDIANA UNIVERSITY   (United States)

Author keywords

causal inference; difference in difference; policy analysis; quasi experiments; research design

Indexed keywords

APPLIED RESEARCH; CAUSALITY; CLINICAL PRACTICE; COMPARATIVE STUDY; DIFFERENCE IN DIFFERENCE; METHODOLOGY; PRIORITY JOURNAL; PUBLIC HEALTH; PUBLIC HEALTH SERVICE; PUBLIC POLICY; QUASI EXPERIMENTAL STUDY; RANDOMIZED CONTROLLED TRIAL (TOPIC); REVIEW; ROBUSTNESS CHECK; SENSITIVITY ANALYSIS; HEALTH CARE POLICY; HUMAN; MANAGEMENT; STATISTICAL ANALYSIS;

CAUSALITY; DATA INTERPRETATION, STATISTICAL; HEALTH POLICY; HUMANS; POLICY MAKING; PUBLIC HEALTH; PUBLIC POLICY; RESEARCH DESIGN;

EID: 85044844785     PISSN: 01637525     EISSN: 15452093     Source Type: Book Series    
DOI: 10.1146/annurev-publhealth-040617-013507     Document Type: Review

References (106)

1. Abadie A, Diamond A, Hainmueller J. 2010. Synthetic control methods for comparative case studies: estimating the effect of California's tobacco control program. J. Am. Stat. Assoc. 105(490):493-505
2. Almond D, Edlund L, Palme M. 2009. Chernobyl's subclinical legacy: prenatal exposure to radioactive fallout and school outcomes in Sweden. Q. J. Econ. 124(4):1729-72
3. Alpert A. 2016. The anticipatory effects of Medicare Part D on drug utilization. J. Health Econ. 49:28-45
4. Anderson DM, Hansen B, Rees DI. 2015. Medical marijuana laws and teen marijuana use. Am. Law Econ. Rev. 17(2):495-528
5. Anderson DM, Rees DI, Sabia JJ. 2014. Medical marijuana laws and suicides by gender and age. Am. J. Public Health 104(12):2369-76
6. Angrist JD, Pischke JS. 2008. Mostly Harmless Econometrics: An Empiricist's Companion. Princeton, NJ: Princeton Univ. Press
7. Angrist J, Rokkanen M. 2012. Wanna get away? RD identification away from the cutoff. NBER Work. Pap. 18662, Cambridge, MA
8. Antwi YA, Moriya AS, Simon K. 2013. Effects of federal policy to insure young adults: evidence from the 2010 Affordable Care Act's dependent-coverage mandate. Am. Econ. J. Econ. Policy 5(4):1-28
9. Atanasov V, Black B. 2016. Shock-based causal inference in corporate finance and accounting research. Crit. Finance Rev. 5:207-304
10. Athey A, Imbens GW. 2017. The state of applied econometrics: causality and policy evaluations. J. Econ. Perspect. 31(2):3-32
11. BachhuberMA, SalonerB, CunninghamCO, Barry CL. 2014. Medical cannabis laws and opioid analgesic overdose mortality in the United States, 1999-2010. JAMA Intern. Med. 174(10):1668-73
12. Baicker K, Taubman SL, Allen HL, Bernstein M, Gruber JH, Newhouse JP, et al. 2013. The Oregon experiment-effects of Medicaid on clinical outcomes. N. Engl. J. Med. 368:1713-22
13. Basu S, Meghani A, Siddiqi A. 2017. Evaluating the health impact of large-scale public policy changes: classical and novel approaches. Annu. Rev. Public Health 38:351-70
14. Bedard K, Kuhn P. 2015. Micro-marketing healthier choices: effects of personalized ordering suggestions on restaurant purchases. J. Health Econ. 39:106-22
15. Bell RM, McCaffrey DF. 2002. Bias reduction in standard errors for linear regression with multi-stage samples. Surv. Methodol. 28(2):169-81
16. Bellou A, Bhatt R. 2013. Reducing underage alcohol and tobacco use: evidence from the introduction of vertical identification cards. J. Health Econ. 32(2):353-66
17. Bertanha M, Imbens GW. 2014. External validity in fuzzy regression discontinuity designs. NBER Work. Pap. 20773, Cambridge, MA
18. Bertrand M, Duflo E, Mullainathan S. 2004. How much should we trust differences-in-differences estimates? Q. J. Econ. 119(1):249-75
19. Besley T, Case A. 2000. Unnatural experiments? Estimating the incidence of endogenous policies. Econ. J. 110(467):672-94
20. Bleakley H. 2007. Disease and development: evidence from hookworm eradication in the American South. Q. J. Econ. 122(1):73-117
21. Bitler MP, Carpenter CS. 2016. Health insurance mandates, mammography, and breast cancer diagnoses. Am. Econ. J. : Econ. Policy 8(3):39-68
22. Bradford AC, BradfordWD. 2016. Medical marijuana laws reduce prescription medication use in Medicare Part D. Health Aff. 35(7):1230-36
23. BramsonH, Jarlais DCD, Arasteh K, Nugent A, Guardino V, Feelemyer J, et al. 2015. State laws, syringe exchange, and HIV among persons who inject drugs in the United States: history and effectiveness. J. Public Health Policy 36(2):212-30
24. Brot-Goldberg ZC, Chandra A, Handel BR, Kolstad JT. 2017. What does a deductible do? The impact of cost-sharing on health care prices, quantities, and spending dynamics. Q. J. Econ. 132(3):1261-318
25. Bullinger LR. 2017. The effect of minimum wages on adolescent fertility: a nationwide analysis. Am. J. Public Health 107(3):447-52
26. Burlig F, PreonasL, Woerman M. 2017. Panel data and experimental design. Work. Pap. 277a, Energy Inst. Haas, Univ. Calif., Berkeley. https://ei. haas. berkeley. edu/research/papers/WP277Appendix. pdf
27. Bustamante AV, Mendez CA. 2014. Health care privatization in Latin America: comparing divergent privatization approaches in Chile, Colombia, and Mexico. J. Health Polit. Policy Law 39(4):841-86
28. Cameron AC, Gelbach JB, Miller DL. 2008. Bootstrap-based improvements for inference with clustered errors. Rev. Econ. Stat. 90(3):414-27
29. Cameron AC, Gelbach JB, Miller DL. 2011. Robust inference with multiway clustering. J. Bus. Econ. Stat. 29(2):238-49
30. Cameron AC, Miller DL. 2015. A practitioner's guide to cluster-robust inference. J. Hum. Resour. 50(2):317-72
31. Cantor J, Torres A, Abrams C, Elbel B. 2015. Five years later: awareness of New York City's calorie labels declined, with no changes in calories purchased. Health Aff. 34(11):1893-900
32. Card D, Shore-Sheppard L. 2004. Using discontinuous eligibility rules to identify the effects of the federal Medicaid expansion on low-income children. Rev. Econ. Stat. 86:752-66
33. Carpenter C, Cook PJ. 2008. Cigarette taxes and youth smoking: new evidence from national, state, and local Youth Risk Behavior Surveys. J. Health Econ. 27(2):287-99
34. Carpenter C, Dobkin C. 2017. The minimum legal drinking age and morbidity in the United States. Rev. Econ. Stat. 99(1):95-104
35. Cawley J, Frisvold DE. 2017. The pass-through of taxes on sugar-sweetened beverages to retail prices: the case of Berkeley, California. J. Policy Anal. Manag. 36(2):303-26
36. Chatterjee N, Shi J, García-ClosasM. 2016. Developing and evaluating polygenic risk predictionmodels for stratified disease prevention. Nat. Rev. Genet. 17:392-406
37. Cohodes SR, Grossman DS, Kleiner SA, Lovenheim MF. 2016. The effect of child health insurance access on schooling: evidence from public insurance expansions. J. Hum. Resour. 51(3):727-59
38. Conley TG, Taber CR. 2011. Inference with difference in differences with a small number of policy changes. Rev. Econ. Stat. 93(1):113-25
39. Cook PJ, Durrance CP. 2013. The virtuous tax: lifesaving and crime-prevention effects of the 1991 federal alcohol-tax increase. J. Health Econ. 32:261-67
40. Cotti C, Tefft N. 2013. Fast food prices, obesity, and the minimum wage. Econ. Hum. Biol. 11(2):134-47
41. Craig P, Katikireddi SV, Leyland A, Popham F. 2017. Natural experiments: an overview of methods, approaches, and contributions to public health intervention research. Annu. Rev. Public Health 38:39-56
42. Currie J, Gruber J. 1996. Health insurance eligibility, utilization of medical care, and child health. Q. J. Econ. 111(2):431-66
43. Currie J, Gruber J, Fischer M. 1995. Physician payments and infant mortality: evidence from Medicaid fee policy. Am. Econ. Rev. 85(2):106-11
44. Decker S. 2009. Changes in Medicaid physician fees and patterns of ambulatory care. INQUIRY: J. Health Care Org. Provis. Financ. 46(3):291-304
45. Dimick JB, Ryan AM. 2014. Methods for evaluating changes in health care policy: the difference-indifferences approach. JAMA 312(22):2401-2
46. Donald SG, Lang K. 2007. Inference with difference-in-differences and other panel data. Rev. Econ. Stat. 89(2):221-33
47. Doudchenko N, Imbens GW. 2016. Balancing, regression, difference-in-differences and synthetic control methods: a synthesis. NBERWork. Pap. 22791, Cambridge, MA
48. Finkelstein A. 2007. The aggregate effects of health insurance: evidence from the introduction of Medicare. Q. J. Econ. 122(1):1-37
49. Garthwaite C, Gross T, NotowidigdoMJ. 2014. Public health insurance, labor supply, and employment lock. Q. J. Econ. 129(2):653-96
50. GebelM, Vossemer J. 2014. The impact of employment transitions on health in Germany: a differencein-differences propensity score matching approach. Soc. Sci. Med. 108:128-36
51. Goodman P, Agnew M, McCaffrey M, Paul G, Clancy L. 2007. Effects of the Irish smoking ban on respiratory health of bar workers and air quality inDublin pubs. Am. J. Respir. Crit. Care Med. 175(8):840-45
52. Gruber J. 1994. The incidence of mandated maternity benefits. Am. Econ. Rev. 84(3):622-41
53. Gruber J, Simon K. 2008. Crowd-out 10 years later: Have recent public insurance expansions crowded out private health insurance? J. Health Econ. 27(2):201-17
54. Hackmann M, Kolstadt J, Kowalski A. 2015. Adverse selection and an individual mandate: when theory meets practice. Am. Econ. Rev. 105(3):1030-66
55. Hamersma S, Kim M. 2013. Participation and crowd out: assessing the effects of parental Medicaid expansions. J. Health Econ. 32(1):160-71
56. Hansen B, Sabia JJ, Rees DI. 2017. Have cigarette taxes lost their bite?New estimates of the relationship between cigarette taxes and youth smoking. Am. J. Health Econ. 3(1):60-75
57. Harper S, Strumpf EC, Kaufman JS. 2012. Do medical marijuana laws increase marijuana use? Replication study and extension. Ann. Epidemiol. 22(3):207-12
58. Heim B, Lin L. 2016. Does health reform lead to an increase in early retirement? Evidence from Massachusetts. ILR Rev. 70(3):704-32
59. Ho DE, Imai K, KingG, Stuart EA. 2007. Matching as nonparametric preprocessing for reducingmodel dependence in parametric causal inference. Polit. Anal. 15(3):199-236
60. Holahan J, Weil A, Wiener JM. 2003. Federalism and Health Policy. Washington, DC: Urban Inst. Press
61. Hoynes H, Schanzenbach DW, Almond D. 2016. Long-run impacts of childhood access to the safety net. Am. Econ. Rev. 106(4):903-34
62. ImbensGW, KolesarM. 2016. Robust standard errors in small samples: some practical advice. Rev. Econ. Stat. 98(4):701-12
63. Imbens GW, Rubin DB. 2015. Causal Inference in Statistics, Social, and Biomedical Sciences. Cambridge, UK: Cambridge Univ. Press
64. Kaestner R, Garret B, Chen J, Gangopadhyaya A, Fleming C. 2017. Effects of ACA Medicaid expansion on health insurance coverage and labor supply. J. Policy Anal. Manag. 36(3):608-42
65. Ketcham JD, Simon KI. 2008. Medicare Part D's effects on elderly patients' drug costs and utilization. Am. J. Manag. Care 14(11):14-22
66. Kolstad JT, Kowalski AE. 2012. The impact of health care reform on hospital and preventive care: evidence from Massachusetts. J. Public Econ. 96(11-12):909-29
67. Kowalski A. 2016. Doing more when you're running LATE: applying marginal treatment effect methods to examine treatment effect heterogeneity in experiments. NBERWork. Pap. 22363, Cambridge, MA
68. KreifN, Grieve R, Hangartner D, Turner AJ, Nikolova S, SuttonM. 2016. Examination of the synthetic control method for evaluating health policies with multiple treated units. Health Econ. 25(12):1514-28
69. Kuehnle D, Wunder C. 2016. The effects of smoking bans on self-assessed health: evidence from Germany. Health Econ. 26(3):321-37
70. Lane S, Hennes E, West T. 2016. I've got the power: how anyone can do a power analysis of any type of study using simulation. Presented at Soc. Personal. Soc. Psychol. Annu. Conv., San Diego. http://meeting. spsp. org/2016/sites/default/files/Lane%2C%20Hennes%2C%20West%20SPSP%20Power% 20Workshop%202016. pdf
71. Lemos S. 2005. Political variables as instruments for the minimum wage. B. E. J. Econ. Anal. Policy 4(1):1-33
72. Liang KY, Zeger SL. 1986. Longitudinal data analysis using generalized linear models. Biometrika 73(1):13-22
73. Madrian BC. 1994. Employment-based health insurance and job mobility: Is there evidence of job-lock? Q. J. Econ. 109(1):27-54
74. Manning WG, Newhouse JP, Duan N, Keeler EB, Leibowitz A. 1987. Health insurance and the demand for medical care: evidence from a randomized experiment. Am. Econ. Rev. 77(3):251-77
75. Marcus J, SiedlerT. 2015. Reducing binge drinking?The effect of a ban on late-night off-premise alcohol sales on alcohol-related hospital stays in Germany. J. Public Econ. 123:55-77
76. Marks MS. 2011. Minimum wages, employer-provided health insurance, and the non-discrimination law. Ind. Relat. J. Econ. Soc. 50(2):241-62
77. McClelland G, Gault S. 2017. The synthetic control method as a tool to understand state policy. Res. Rep., Urban Inst., Washington, DC
78. Miller S. 2012. The effect of insurance on emergency room visits: an analysis of the 2006 Massachusetts health reform. J. Public Econ. 96(11-12):893-908
79. Moulton BR. 1990. An illustration of a pitfall in estimating the effects of aggregate variables on micro units. Rev. Econ. Stat. 72(2):334-38
80. Muthén LK, Muthén BO. 2009. How to use aMonte Carlo study to decide on sample size and determine power. Struct. Equ. Model. : Multidiscip. J. 9(4):599-620
81. NAMSDL (Natl. Alliance Model State Drug Laws). 2017. Use of marijuana for medicinal purposes: map of state laws. NAMSDL, Manchester, IA. http://www. namsdl. org/library/CDC6A46B-F8E2-5F41-DBF6FBE02C0BC877/
82. Nathan RP. 2005. Federalism and health policy. Health Aff. 24(6):1458-66
83. Obermeyer Z, MakarM, Abujaber S. 2014. Association between theMedicare hospice benefit and health care utilization and costs for patients with poor-prognosis cancer. JAMA 312(18):1888-96
84. Paik M, Black B, Hyman D. 2013. The receding tide of medical malpractice litigation part 2: effect of damage caps. J. Empir. Legal Stud. 10:639-69
85. Paik M, Black B, Hyman DA. 2016. Damage caps and the labor supply of physicians: evidence from the third reform wave. Am. Law Econ. Rev. 18(2):463-505
86. Pimentel SD, Kelz RR, Silber JH, Rosenbaum PR. 2015. Large, sparse optimal matching with refined covariate balance in an observational study of the health outcomes produced by new surgeons. J. Am. Stat. Assoc. 110(510):515-27
87. Pustejovsky JE, Tipton E. 2016. Small sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models. J. Bus. Econ. Stat. In press. https://doi. org/10. 1080/ 07350015. 2016. 1247004
88. Raifman J, Moscoe E, Austin B, McConnell M. 2017. Difference-in-differences analysis of the association between state same-sex marriage policies and adolescent suicide attempts. JAMA Pediatr. 171(4):350-56
89. Rieger M, Wagner N, Bedi AS. 2017. Universal health coverage at the macro level: synthetic control evidence from Thailand. Soc. Sci. Med. 172:46-55
90. Rosenbaum PR. 2002. Covariance adjustment in randomized experiments and observational studies. Stat. Sci. 17(3):286-327
91. Saloner B, Feder KA, Krawczyk N. 2017. Closing the medication-assisted treatment gap for youth with opioid use disorder. JAMA Pediatr. 171(8):729-31
92. ShadishWR, Cook TD, Campbell DT. 2002. Experimental andQuasi-Experimental Designs for Generalized Causal Inference. Boston, MA: Houghton Mifflin C.o.
93. Simon D. 2016. Does early life exposure to cigarette smoke permanently harm childhood welfare? Evidence from cigarette tax hikes. Am. Econ. J. Appl. Econ. 8(4):128-59
94. Simon KI. 2005. Adverse selection in health insurance markets? Evidence from state small-group health insurance reforms. J. Public Econ. 89(9-10):1865-77
95. Simon KI, Kaestner R. 2004. Do minimum wages affect non-wage job attributes? Evidence on fringe benefits. ILR Rev. 58(1):52-70
96. SimonK, SoniA, Cawley J. 2017. The impact of health insurance on preventive care and health behaviors: evidence from the first two years of the ACA Medicaid expansions. J. Policy Anal. Manag. 36(2):390-417
97. Snow J. 1855. On the Mode of Communication of Cholera. London: John Churchill
98. Solon G, Haider SJ, Wooldridge JM. 2015. What are we weighting for? J. Hum. Resour. 50(2):301-16
99. Somers MA, Zhu P, Jacob R, Bloom H. 2013. The validity and precision of the comparative interrupted time series design and the difference-in-difference design in educational evaluation. Work. Pap. Res. Methodol., MDRC, New York. https://www. mdrc. org/sites/default/files/validity-precision-comparative- interrupted-time-series-design. pdf
100. Sommers BD, Gunja MZ, Finegold K, Musco T. 2015. Changes in self-reported insurance coverage, access to care, and health under the Affordable Care Act. JAMA 314(4):366-74
101. Sommers BD, Kronick R. 2012. The Affordable Care Act and insurance coverage for young adults. JAMA 307(9):913-14
102. Trogdon JG, Shafer PR, Shah PD, Calo WA. 2016. Are state laws granting pharmacists authority to vaccinate associated with HPV vaccination rates among adolescents? Vaccine 34(38):4514-19
103. Wehby G, Dave D, Kaestner R. 2016. Effects of the minimum wage on infant health. NBER Work. Pap. 22373, Cambridge, MA
104. Wing C, Cook TD. 2013. Strengthening the regression discontinuity design using additional design elements: a within-study comparison. J. Policy Anal. Manag. 32(4):853-77
105. Wolfe B, Scrivner S. 2005. The devil may be in the details: how the characteristics of SCHIP programs affect take-up. J. Policy Anal. Manag. 24(3):499-522
106. Xiao L, Jiang Y, Liu X, Li Y, Gan Q, Liu F. 2017. Smoking reduced in urban restaurants: the effect of Beijing Smoking Control Regulation. Tob. Control 26:e75-78