From sample average treatment effect to population average treatment effect on the treated: Combining experimental with observational studies to estimate population treatment effects

Journal of the Royal Statistical Society. Series A: Statistics in Society

Volumn 178, Issue 3, 2015, Pages 757-778

  Hartman, Erin ^a   Grieve, Richard ^b   Ramsahai, Roland ^c   Sekhon, Jasjeet S ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b LONDON SCHOOL OF HYGIENE AND TROPICAL MEDICINE   (United Kingdom)

^c UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

Causal inference; Cost effectiveness studies; External validity; Observational studies; Placebo tests; Randomized controlled trials

Indexed keywords

EID: 84928770171     PISSN: 09641998     EISSN: 1467985X     Source Type: Journal    
DOI: 10.1111/rssa.12094     Document Type: Article

References (63)

1. Abadie, A. and Imbens, G. W. (2008) On the failure of the bootstrap for matching estimators. Econometrica, 76, 1537-1557.
2. Allcott, H. (2011) Social norms and energy conservation. J. Publ. Econ., 95, 1082-1095.
3. Allcott, H. (2014) Site selection bias in program evaluation. Working Paper.
4. Allcott, H. and Mullainathan, S. (2012) External validity and partner selection bias. Technical Report. National Bureau of Economic Research, Cambridge.
5. Benjamini, Y. and Hochberg, Y. (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Statist. Soc. B, 57, 289-300.
6. Bickel, P. J. and Sakov, A. (2008) On the choice of m in the m out of n bootstrap and confidence bounds for extrema. Statist. Sin., 18, 967-985.
7. Breiman, L. (2001) Random forests. Mach. Learn., 45, 5-32.
8. Chipman, H. A., George, E. I. and McCulloch, R. E. (2010) Bart: Bayesian additive regression trees. Ann. Appl. Statist., 4, 266-298.
9. Chittock, D. R., Dhingra, V. K., Ronco, J. J., Russell, J. A., Forrest, D. M., Tweeddale, M. and Fenwick, J. C. (2004) Severity of illness and risk of death associated with pulmonary artery catheter use. Crit. Care Med., 32, 911-915.
10. Cole, S. R. and Frangakis, C. E. (2009) The consistency statement in causal inference: a definition or an assumption? Epidemiology, 20, 3-5.
11. Cole, S. R. and Stuart, E. A. (2010) Generalizing evidence from randomized clinical trials to target populations the actg 320 trial. Am. J. Epidem., 172, 107-115.
12. Connors, A. F., Speroff, T. S., Dawson, N. V., Thomas, C., Harrell, F. E., Wagner, D., Desbiens, N., Goldman, L., Wu, A. W., Califf, R. M., Fulkerson, W. J., Vidaillet, H., Broste, S., Bellamy, P., Lynn, J. and Knaus, W. A. (1996) The effectiveness of right heart catheterization in the initial care of critically ill patients. J. Am. Med. Ass., 276, 889-897.
13. Dalen, J. E. (2001) The pulmonary artery catheter-friend, foe, or accomplice? J. Am. Med. Ass., 286, 348-350.
14. Deaton, A. (2009) Instruments of development: randomization in the tropics, and the search for the elusive keys to economic development. Working Paper 14690. National Bureau of Economic Research, Cambridge.
15. Diamond, A. and Sekhon, J. S. (2013) Genetic matching for estimating causal effects: a general multivariate matching method for achieving balance in observational studies. Rev. Econ. Statist., 95, 932-945.
16. Finfer, S. and Delaney, A. (2006) Pulmonary artery catheters as currently used, do not benefit patients. Br. Med. J., 333, 930-931.
17. Gheorghe, A., Roberts, T. E., Ives, J. C., Fletcher, B. R. and Calvert, M. (2013) Centre selection for clinical trials and the generalisability of results: a mixed methods study. PLOS ONE, 8, no. 2, article e56560.
18. Greenhouse, J. B., Kaizar, E. E., Kelleher, K., Seltman, H. and Gardner, W. (2008) Generalizing from clinical trial data: a case study, the risk of suicidality among pediatric antidepressant users. Statist. Med., 27, 1801-1813.
19. Greenland, S. (2005) Multiple-bias modelling for analysis of observational data (with discussion). J. R. Statist. Soc. A, 168, 267-306.
20. Hansen, L. P. (1982) Large sample properties of generalized method of moments estimators. Econometrica, 50, 1029-1054.
21. Harrison, D. A., Brady, A. R. and Rowan, K. (2004) Case mix, outcome and length of stay for admissions to adult, general critical care units in England, Wales and Northern Ireland: the Intensive Care National Audit & Research Centre case mix programme database. Crit. Care, 8, 99-111.
22. Hartman, E. K. and Hidalgo, F. D. (2011) What's the alternative?: an equivalence approach to placebo and balance tests. Working Paper. Department of Political Science, University of California at Berkeley, Berkeley.
23. Harvey, S., Harrison, D. A., Singer, M., Ashcroft, J., Jones, C. M., Elbourne, D., Brampton, W., Williams, D., Young, D. and Rowan, K. (2005) An assessment of the clinical effectiveness of pulmonary artery catheters in patient management in intensive care (pac-man): a randomized controlled trial. Lancet, 366, 472-477.
24. Harvey, S., Welch, C., Harrison, D. and Singer, M. (2008) Post hoc insights from pac-man-the UK pulmonary artery catheter trial. Crit. Care, 35, 1714-1721.
25. Heckman, J. J., Ichimura, H., Smith, J. and Todd, P. (1998) Characterizing selection bias using experimental data. Econometrica, 66, 1017-1098.
26. Heckman, J. J. and Urzua, S. (2009) Comparing iv with structural models: What simple iv can and cannot identify. Discussion Paper 3980. Institute for the Study of Labor, Bonn.
27. Heckman, J. J., Urzua, S. and Vytlacil, E. (2006) Understanding instrumental variables in models with essential heterogeneity. Rev. Econ. Statist., 88, 389-432.
28. Heckman, J. J. and Vytlacil, E. (2005) Structural equations, treatment effects, and econometric policy evaluation. Econometrica, 73, 669-738.
29. Hellerstein, J. K. and Imbens, G. W. (1999) Imposing moment restrictions from auxiliary data by weighting. Rev. Econ. Statist., 81, 1-14.
30. Hoch, J. S., Briggs, A. H. and Willan, A. R. (2002) Something old, something new, something borrowed, something blue: a framework for the marriage of econometrics and cost-effectiveness analysis. Hlth Econ., 11, 415-430.
31. Hotz, V. J., Imbens, G. and Mortimer, J. H. (2005) Predicting the efficacy of future training programs using past experiences at other locations. J. Econmetr., 125, 241-270.
32. Imai, K., King, G. and Stuart, E. A. (2008) Misunderstandings between experimentalists and observationalists about causal inference. J. R. Statist. Soc. A, 171, 481-502.
33. Imai, K., Tingley, D. and Yamamoto, T. (2013) Experimental designs for identifying causal mechanisms (with discussion). J. R. Statist. Soc. A, 176, 5-51.
34. Imbens, G. (2009) Better late than nothing: Some comments on Deaton (2009) and Heckman and Urzua (2009). Working Paper 14896. National Bureau of Economic Research, Cambridge.
35. Ireland, C. T. and Kullback, S. (1968) Contingency tables with given marginals. Biometrika, 55, 179-188.
36. Jaynes, E. T. (1957) Information theory and statistical mechanics. Phys. Rev., 106, 620-630.
37. Kang, J. D. Y. and Schafer, J. L. (2007) Demystifying double robustness: a comparison of alternative strategies for estimating a population mean from incomplete data (with discussion). Statist. Sci., 22, 523-539.
38. Kish, L. (1992) Weighting for unequal Pi. J. Off. Statist., 8, 183-200.
39. Kline, B. and Tamer, E. (2011) Using observational vs. randomized controlled trial data to learn about treatment effects. Department of Economics, Northwestern University, Evanston. (Available from http://dx.doi.org/10.2139/ssrn.1810114.)
40. Kullback, S. (1997) Information Theory and Statistics. New York: Wiley.
41. Liaw, A. and Wiener, M. (2002) Classification and regression by randomforest. R News, 2, 18-22.
42. Miratrix, L. W., Sekhon, J. S. and Yu, B. (2013) Adjusting treatment effect estimates by post-stratification in randomized experiments. J. R. Statist. Soc. B, 75, 369-396.
43. Mitra, N. and Indurkhya, A. (2005) A propensity score approach to estimating the cost-effectiveness of medical therapies from observational data. Hlth. Econ., 14, 805-815.
44. Mojtabai, R. and Zivin, J. G. (2003) Effectiveness and cost-effectiveness of four treatment modalities for substance disorders: a propensity score analysis. Hlth Serv. Res., 38, 233-259.
45. National Research Council (2013) Frontiers in Massive Data Analysis. Washington DC: National Academies Press.
46. Nixon, R. M. and Thompson, S. G. (2005) Incorporating covariate adjustment, subgroup analysis and between-centre differences into cost-effectiveness evaluations. Hlth Econ., 14, 1217-1229.
47. Politis, D. N. and Romano, J. P. (1994) Large sample confidence regions based on subsamples under minimal assumptions. Ann. Statist., 22, 2031-2050.
48. Porter, K. E., Gruber, S., van der Laan, M. J. and Sekhon, J. S. (2011) The relative performance of targeted maximum likelihood estimators. Int. J. Biostatist., 7, no. 1.
49. Rothwell, P. M. (2005) External validity of randomised controlled trials: to whom do the results of this trial apply? Lancet, 365, 82-93.
50. Sakr, Y., Vincent, J.-L., Reinhart, K., Payen, D., Wiedermann, C. J., Zandstra, D. F. and Sprung, C. L. (2005) Sepsis occurrence in acutely ill patients investigators: use of the pulmonary artery catheter is not associated with worse outcome in the ICU. Chest, 128, 2722-2731.
51. Sekhon, J. S. (2009) Opiates for the matches: matching methods for causal inference. A. Rev. Politi. Sci., 12, 487-508.
52. Sekhon, J. S. (2011) Matching: multivariate and propensity score matching with automated balance search. J. Statist. Softwr., 42, 1-52.
53. Sekhon, J. S. and Grieve, R. (2012) A nonparametric matching method for covariate adjustment with application to economic evaluation (genetic matching). Hlth Econ., 21, 695-714.
54. Shadish, W. R., Cook, T. D. and Campbell, D. T. (2002) Experimental and Quasi-experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.
55. Stevens, K., McCabe, C., Jones, C., Ashcroft, J., Harvey, S. and Rowan, K. On behalf of the PAC-Man Study Collaboration (2005) The incremental cost effectiveness of withdrawing pulmonary artery catheters from routine use in critical care. Appl. Hlth Econ. Hlth Poly, 4, 257-264.
56. Stuart, E. A., Cole, S. R., Bradshaw, C. P. and Leaf, P. J. (2011) The use of propensity scores to assess the generalizability of results from randomized trials. J. R. Statist. Soc. A, 174, 369-386.
57. Tunis, S. R., Stryer, D. B. and Clancy, C. M. (2003) Practical clinical trials. J. Am. Med. Ass., 290, 1624-1632.
58. Turner, R. M., Spiegelhalter, D. J., Smith, G. and Thompson, S. G. (2009) Bias modelling in evidence synthesis. J. R. Statist. Soc. A, 172, 21-47.
59. Wellek, S. (2010) Testing Statistical Hypotheses of Equivalence and Noninferiority. Boca Roton: CRC Press.
60. Willan, A. R. and Briggs, A. H. (2006) Statistical Analysis of Cost-effectiveness Data. Hoboken: Wiley.
61. Willan, A. R., Briggs, A. H. and Hoch, J. S. (2004) Regression methods for covariate adjustment and subgroup analysis for non-censored cost-effectiveness data. Hlth Econ., 13, 461-475.
62. Willan, A. R., Chen, E. B., Cook, R. J. and Lin, D. Y. (2003) Incremental net benefit in randomized clinical trials with quality-adjusted survival. Statist. Med., 22, 353-362.
63. Willan, A. R. and Lin, D. Y. (2001) Incremental net benefit in randomized clinical trials. Statist. Med., 20, 1563-1574.