A "politically robust" experimental design for public policy evaluation, with application to the Mexican universal health insurance program

Journal of Policy Analysis and Management

Volumn 26, Issue 3, 2007, Pages 479-506

  King, Gary ^a,b   Gakidou, Emmanuela ^b   Ravishankar, Nirmala ^b   Moore, Ryan T ^b   Lakin, Jason ^b   Vargas, Manett ^c   Téllez Rojo, Martha María ^d   Ávila, Juan Eugenio Hernández ^d   Ávila, Mauricio Hernández ^e   Llamas, Héctor Hernández ^a  

^a Department of Government ^*

^b HARVARD UNIVERSITY   (United States)

^c Ministry of Health   (Mexico)

^d INSTITUTO NACIONAL DE SALUD PÚBLICA   (Mexico)

^e Salud (Ministry of Health)   (Mexico)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; GOVERNMENT; HEALTH CARE DELIVERY; HEALTH CARE POLICY; HEALTH CARE QUALITY; HEALTH INSURANCE; HUMAN; INFORMATION PROCESSING; INSURANCE; METHODOLOGY; MEXICO; ORGANIZATION AND MANAGEMENT; POLICY; POLITICS; PUBLIC HEALTH;

DATA COLLECTION; DELIVERY OF HEALTH CARE; GOVERNMENT PROGRAMS; HEALTH CARE REFORM; HEALTH POLICY; HUMANS; INSURANCE COVERAGE; INSURANCE, HEALTH; MEXICO; NATIONAL HEALTH PROGRAMS; POLITICS; PROGRAM EVALUATION; PUBLIC POLICY; RESEARCH DESIGN;

EID: 34547138712     PISSN: 02768739     EISSN: 15206688     Source Type: Journal    
DOI: 10.1002/pam.20279     Document Type: Article

References (66)

1. Adato, M., Coady, D., & Ruel, M. (2000). An operations evaluation of PROGRESA from the perspective of beneficiaries, promotoras, school directors and health staff. Final report, International Food Policy Research Institute, 2033 K Street, NW Washington, DC 20006.
2. Alesina, A., & Tabellini, G. (1990). A positive theory of fiscal deficits and government debt. The Review of Economic Studies, 57, 403-414.
3. Barnard, J., Frangakis, C. E., Hill, J. L., & Rubin, D. B. (2003). Principal stratification approach to broken randomized experiments: A case study of school choice vouchers in New York City. Journal of the American Statistical Association, 98, 299-324.
4. Behrman, J. R., & Todd, P. E. (1999). Randomness in the experimental samples of PROGRESA (education, health, and nutrition program). Research report. Washington, DC: International Food Policy Research Institute.
5. Bloom, H. S., Richburg-Hayes, L., & Black, A. (2007, in press). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis.
6. Bohm, P. (1984). Are there practicable demand-revealing mechanisms? In H. Hanusch (Ed.), Public finance and the quest for efficiency (pp. 127-139). Detroit: Wayne State University Press.
7. Boruch, R., May, H., Turner, H., Lavenberg, J., Petrosino, A., de Moya, D., Grimshaw, J., & Foley, E. (2004). Estimating the effects of interventions that are deployed in many places: Place-randomized trials. American Behavioral Scientist, 47, 608-633.
8. Boruch, R. F. (1997). Randomized experiments for planning and evaluation. Thousand Oaks, CA: Sage Publications.
9. Box, G. E., Hunger, W. G., & Hunter, J. S. (1978). Statistics for experimenters. New York: Wiley-Interscience.
10. Burtless, G. (1995). The case for randomized field trials in economic and policy research. The Journal of Economic Perspectives, 9, 63-84.
11. Camasso, M. J., Jagannathan, R., Harvey, C., & Killingsworth, M. (2003). The use of client surveys to gauge the threat of contamination in welfare reform experiments. Journal of Policy Analysis and Management, 22, 207-223.
12. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNaIIy College Publishing Company.
13. Dee, T. S., & Keys, B. J. (2004). Does merit pay reward good teachers? Evidence from a randomized experiment. Journal of Policy Analysis and Management, 23, 471-488.
14. Derthick, M. (1979). Policymaking for Social Security. Washington, DC: The Brookings Institution.
15. Donner, A., & Klar, N. (2000). Design and analysis of cluster randomization trials in health research. London: Arnold.
16. Flay, B. R., & Best, J. A. (1982). Overcoming design problems in evaluating health behavior programs. Evaluation & The Health Professions, 5, 43-69.
17. Frangakis, C. E., Rubin, D. B., & Zhou, Z.-H. (2002). Clustered encouragement designs with individual noncompliance: Bayesian inference with randomization, and application to advance directive forms. Biostatistics, 3, 147-164.
18. Franzese, R. (2002). Macroeconomic policies of developed democracies. New York: Cambridge University Press.
19. Frenk, J., Gonzalez-Pier, E. Gomez-Dantes, O., Lezana, M. A., & Knaul, F. M. (2006). Comprehensive reform to improve health system performance in Mexico. The Lancet, 368, 1524-1534.
20. Frenk, J., Sepúlveda, J., Gómez-Dantés, O., & Knaul, F. (2003). Evidence-based health policy: Three generations of reform in Mexico. The Lancet, 362, 1667-1671.
21. Gertler, P. (2006). Do conditional cash transfers improve child health? Evidence from PROGRESAS control randomized experiment. The American Economic Review: Papers and Proceedings, 94, 336-42.
22. Gertler, P. J. (2000). Final report: The impact of PROGRESA on health. International Food Policy Research Institute.
23. Glazerman, S., Mayer, D., & Decker, P. (2006). Alternative routes to teaching: The impacts of Teach for America on student achievement and other outcomes. Journal of Policy Analysis and Management, 25, 75-96.
24. Goodman, J. S., & Blum, T. C. (1996). Assessing the non-random sampling effects of subject attrition in longitudinal research. Journal of Management, 22, 627-652.
25. Green, D. P., & Gerber, A. S. (2002). Reclaiming the experimental tradition in political science. In Milner, H., & Katznelson, I. (Eds.), State of the discipline, vol. III (pp. 805-832). New York: W.W. Norton & Company, Inc..
26. Greenberg, D., & Shroder, M. (2004). The digest of social experiments (3rd ed.). Washington, DC: Urban Institute Press.
27. Greenberg, D. H., Michalopoulos, C., & Robins, P. K. (2006). Do experimental and nonexperimental evaluations give different answers about the effectiveness of government funded training programs? Journal of Policy Analysis and Management, 25, 523-552.
28. Greevy, R., Lu, B., Silver, J. H., & Rosenbaum, P. (2004). Optimal multivariate matching before randomization. Biostatistics, 5, 263-275.
29. Harrison, G. W., & List, J. A. (2004). Field experiments. Journal of Economic Literature, 42, 1009-1055.
30. Heckman, J. J. (1992). Randomization and social policy evaluation. In Manski, C. F., & Garfinkel, I. (eds.), Evaluating welfare and training programs. Boston: Harvard University Press.
31. Heckman, J. J., & Smith, J. A. (1995). Assessing the case for social experiments. The Journal of Economic Perspectives, 9, 85-110.
32. Hirano, K., Imbens, G. W., Rubin, D. B., & Zhou, X.-H. (2000). Assessing the effect of an influenza vaccine in an encouragement design. Biostatistics, 1, 69-88.
33. Ho, D., Imai, K., King, G., & Stuart, E. (2007). Matching as nonparametric preprocessing for parametric causal inference. Political analysis, http://gking.harvard.edu/files/abs/matchp-abs.shtml.
34. Holland, P. W., & Wainer, H. (Eds.) (1993). Differential item functioning. Hillside, NJ: Lawrence Erlbaum.
35. Holzer, H. J., Quigley, J. M., & Raphael, S. (2003). Public transit and the spatial distribution of minority employment: Evidence from a natural experiment. Journal of Policy Analysis and Management, 22, 415-441.
36. Honaker, J., & King, G. (2006). What to do about missing values in time series cross-section data, http://gking.harvard.edu/files/abs/pr-abs.shtml.
37. Howell, W G. (2004). Dynamic selection effects in means-tested, urban school voucher programs. Journal of Policy Analysis and Management, 23, 225-250.
38. Imai, K. (2005). Do get-out-the-vote calls reduce turnout? The importance of statistical methods for field experiments. American Political Science Review, 99, 283-300.
39. Imai, K., King, G., & Lau, O. (2006). Zelig: Everyone's statistical software. http://gking.harvard.edu/zelig.
40. Imai, K., King, G., & Stuart, E. (2007). Misunderstandings among experimentalists and observationalists: Balance test fallacies in causal inference, http://gking.harvard.edu/files/abs/matchse-abs.shtml.
41. King, G., Honaker, J., Joseph, A., & Scheve, K. (2001). Analyzing incomplete political science data: An alternative algorithm for multiple imputation. American Political Science Review, 95, 49-69. http://gking.harvard. edu/files/abs/evil-abs.shtml.
42. King, G., Murray, C. J., Salomon, J. A., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98, 191-205. http://gking.harvard. edu/files/abs/vign-abs.shtml.
43. King, G., Tomz, M., & Wittenberg, J. (2000). Making the most of statistical analyses: Improving interpretation and presentation. American Journal of Political Science, 44, 341-355. http://gking.harvard.edu/files/abs/ making-abs.shtml.
44. Kish, L. (1949). A procedure for objective respondent selection within the household. Journal of the American Statistical Association, 44, 380-387.
45. Klar, N., & Donner, A. (1997). The merits of matching in community intervention trials: A cautionary tale. Statistics in Medicine, 16, 1753-1764.
46. Klarman, M. J. (1997). Majoritarian judicial review: The entrenchment problem. The Georgetown Law Journal, 85, 491-554.
47. Kramer, M., & Shapiro, S. (1984). Scientific challenges in the application of randomized trials. Journal of the American Medical Association, 252, 2739-45.
48. Krueger, A. (1999). Experimental estimates of education production functions. Quarterly Journal of Economics, 114, 497-532.
49. Lakin, J. (2005). Letting the outsiders in: Democratization and health reform in Mexico. Paper presented at the annual meeting of the American Political Science Association, Washington DC.
50. Lewis, M. (2005). Improving efficiency and impact in health care services: Lessons from Central America. In Forgia, G. M. L. (ed.), Health systems innovation in Central America. Washington, DC: The World Bank.
51. Murray, C. J., & Evans, D. B. (Eds.) (2003). Health systems performance assessment: Debates, methods and empiricism. Geneva: World Health Organization.
52. Murray, D. M. (1998). Design and analysis of group-randomized trials. New York: Oxford University Press.
53. Nickerson, D. W. (2005). Scalable protocols offer efficient design for field experiments. Political Analysis, 13, 233-252.
54. Palmer, T., & Petrosino, A. (2003). The "experimenting agency." The California Youth Authority Research Division. Evaluation Review, 22, 228-266.
55. Posner, E. A., & Vermeule, A. (2002). Legislative entrenchment: A reappraisal. The Yale Law Journal, 111, 1665-1705.
56. Raudenbush, S. W., Martinez, A., & Spybrook, J. (2007). Strategies for improving precision in group-randomized experiments. Educational Evaluation and Policy Analysis.
57. Robins, J. M., & Rotnitzky, A. (2001). Comment on the Peter J. Bickel and Jaim Young Kwon, "Inference for semiparametric models: Some questions and an answer." Statistica Sinica, 11, 920-936.
58. Rosner, B., & Hennekens, C. H. (1978). Analytic methods in matched pair epidemiological studies. International Journal of Epidemiology, 7, 367-372.
59. Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. New York: John Wiley.
60. Sommer, A., & Zeger, S. (1991). On estimating efficacy from clinical trials. Statistics in Medicine, 10, 45-52.
61. Sterk, S. E. (2003). Retrenchment on entrenchment. The George Washington Law Review, 71, 231-254.
62. Thompson, D. F. (2005). Democracy in time: Popular sovereignty and temporal representation. Constellations, 12, (pp. 245-261).
63. Torp, H., Rauum, O., Hernaes, E., & Goldstein, H. (1993). The first Norwegian experiment. In Karsten, J., & Madsen, P. K. (eds.), Measuring labour market measures: Evaluating the effects of active labour market policies. Copenhagen, Ministry of Labour.
64. Western, B. (1995). Concepts and suggestions for robust regression analysis. American Journal of Political Science, 39, 786-817.
65. Wilde, E. T., & Hollister, R. (2007). How close is close enough? Evaluating propensity score matching using data from a class-size reduction experiment. Journal of Policy Analysis and Management, 26, 455-480.
66. Zaman, A., Rousseeuw, P. J., & Orhan, M. (2001). Econometric applications of high-break-down robust regression techniques. Economics Letters, 71, 1-8.