New Statistical Learning Methods for Estimating Optimal Dynamic Treatment Regimes

Journal of the American Statistical Association

Volumn 110, Issue 510, 2015, Pages 583-598

  Zhao, Ying Qi ^a   Zeng, Donglin ^b   Laber, Eric B ^c   Kosorok, Michael R ^d  

^a UNIVERSITY OF WISCONSIN SCHOOL OF MEDICINE AND PUBLIC HEALTH   (United States)

^b IL   (United States)

^c NORTH CAROLINA STATE UNIVERSITY   (United States)

^d University of North Carolina at Chapel Hill   (United States)

Author keywords

Classification; Personalized medicine; Q learning; Reinforcement learning; Risk bound; Support vector machine

Indexed keywords

EID: 84936797778     PISSN: 01621459     EISSN: 1537274X     Source Type: Journal    
DOI: 10.1080/01621459.2014.937488     Document Type: Article

References (54)

1. P.L.Bartlett,, M.I.Jordan,, J.D.McAuliffe, (2006), Convexity, Classification, and Risk Bounds, Journal of the American Statistical Association, 101, 138–156.
2. R.Bellman, (1957), Dynamic Programming, Princeton, NJ: Princeton University Press.
3. G.Blanchard,, O.Bousquet,, P.Massart, (2008), Statistical Performance of Support Vector Machines, The Annals of Statistics, 36, 489–531.
4. D.Blatt,, S.A.Murphy,, and J.Zhu, (2004), “A-Learning for Approximate Planning,” unpublished manuscript.
5. P.S.Bradley,, and O.L.Mangasarian, (1998), “Feature Selection via Concave Minimization and Support Vector Machines,” in Proceedings 15th International Conference on Machine Learning, San Francisco, CA: Morgan Kaufmann Publishers Inc.
6. B.Chakraborty,, S.Murphy,, V.Strecher, (2010), Inference for Non-Regular Parameters in Optimal Dynamic Treatment Regimes, Statistical Methods in Medical Research, 19, 317–343.
7. C.-C.Chang,, and C.-J.Lin, (2011), “LIBSVM: A Library for Support Vector Machines,” ACM Transactions on Intelligent Systems and Technology, 21–27, available at http://www.csie.ntu.edu.tw/∼cjlin/libsvm.
8. J.Cook, (2011), “Basic Properties of the Soft Maximum,” UT MD Anderson Cancer Center Department of Biostatistics Working Paper Series. Working Paper 70, available at http://biostats.bepress.com/mdandersonbiostat/paper70.
9. C.Cortes,, and V.Vapnik, (1995), “Support-Vector Networks,” Machine Learning, 20, 273–297.
10. R.Dawson,, P.Lavori, (2004), Placebo-Free Designs for Evaluating New Mental Health Treatments: The Use of Adaptive Treatment Strategies, Statistics in Medicine, 23, 3249–3262.
11. P.A.Flume,, B.P.O’Sullivan,, C.H.Goss,, J.M.Peter, J. Mogayzel, D.B.Willey-Courand,, J.Bujan,, J.Finder,, M.Lester,, L.Quittell,, R.Rosenblatt,, R.L.Vender,, L.Hazle,, K.Sabadosa,, B.Marshall, (2007), Cystic Fibrosis Pulmonary Guidelines: Chronic Medications for Maintenance of Lung Health, American Journal of Respiratory and Critical Care Medicine, 176, 957–969.
12. Y.Goldberg,, M.R.Kosorok, (2012), Q-Learning With Censored Data, The Annals of Statistics, 40, 529–560.
13. T.Hastie,, R.Tibshirani,, and J.H.Friedman, (2009), The Elements of Statistical Learning (2nd ed.), New York: Springer-Verlag Inc.
14. E.B.Laber,, M.Qian,, D.Lizotte,, W.E.Pelham,, S.Murphy, (2011), Statistical Inference in Dynamic Treatment Regimes, Revision of University of Michigan, Statistics Department of Technical Report no. 506.
15. P.W.Lavori,, R.Dawson, (2000), A Design for Testing Clinical Strategies: Biased Adaptive Within-Subject Randomization, Journal of The Royal Statistical Society, Series A, 163, 29–38.
16. (2004), Dynamic Treatment Regimes: Practical Design Considerations, Clinical Trials, 1, 9–20.
17. R.J.A.Little,, and D.B.Rubin, (2002), Statistical Analysis With Missing Data (2nd ed.), Chichester: Wiley.
18. Y.Liu,, M.Yuan, (2011), Reinforced Multicategory Support Vector Machines, Journal of the Computational and Graphical Statistics, 20, 909–919.
19. E.E.M.Moodie,, T.S.Richardson,, D.A.Stephens, (2007), Demystifying Optimal Dynamic Treatment Regimes, Biometrics, 63, 447–455.
20. S.A.Murphy, (2003), Optimal Dynamic Treatment Regimes, Journal of the Royal Statistical Society, Series B, 65, 331–366.
21. (2005a), An Experimental Design for the Development of Adaptive Treatment Strategies, Statistics in Medicine, 24, 1455–1481.
22. (2005b), A Generalization Error for Q-Learning, Journal of Machine Learning Research, 6, 1073–1097.
23. S.A.Murphy,, D.W.Oslin,, A.J.Rush,, J.Zhu,, (2007), Methodological Challenges in Constructing Effective Treatment Sequences for Chronic Psychiatric Disorders, Neuropsychopharmacology, 32, 257–262.
24. S.A.Murphy,, M.J.van der Laan,, J.M.Robins,, (2001), Marginal Mean Models for Dynamic Regimes, Journal of the American Statistical Association, 96, 1410–23.
25. I.Nahum-Shani,, M.Qian,, D.Almiral,, W.Pelham,, B.Gnagy,, G.Fabiano,, J.Waxmonsky,, J.Yu,, S.Murphy, (2012), Q-Learning: A Data Analysis Method for Constructing Adaptive Interventions, Psychological Methods, 17, 478–494.
26. L.Orellana,, A.Rotnitzky,, J.M.Robins, (2010), Dynamic Regime Marginal Structural Mean Models for Estimation of Optimal Dynamic Treatment Regimes, Part I: Main Content, The International Journal of Biostatistics, 6, 1–47.
27. J.Pineau,, M.G.Bellemare,, A.Ghizaru,, S.A.Murphy, (2007), Constructing Evidence-Based Treatment Strategies Using Methods From Computer Science, Drug and Alcohol Dependence, 88S, S52–S60.
28. M.Qian,, S.A.Murphy, (2011), Performance Guarantees for Individua-lized Treatment Rules, The Annals of Statistics, 39, 1180–1210.
29. J.M.Robins, (1986), A New Approach to Causal Inference in Mortality Studies With a Sustained Exposure Period Application to Control of the Healthy Worker Survivor Effect, Mathematical Modelling, 7, 1393–1512.
30. ——— (2004), “Optimal Structural Nested Models for Optimal Sequential Decisions,” in Proceedings of the Second Seattle Symposium on Biostatistics, eds. D. Y. Lin and P. J. Heagerty, New York:Springer, pp. 189–326.
31. P.R.Rosenbaum,, D.B.Rubin, (1983), The Central Role of the Propensity Score in Observational Studies for Causal Effects, Biometrika, 70, 41–55.
32. D.B.Rubin, (1974), Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies, Journal of the Educational Psychology, 66, 688–701.
33. ——— (1978), Bayesian Inference for Causal Effects: The Role of Randomization, The Annals of Statistics, 6, 34–58.
34. P.J.Schulte,, A.A.Tsiatis,, E.B.Laber,, and M.Davidian, (2014), “Q- and A-Learning Methods for Estimating Optimal Dynamic Treatment Regimes,” Statistical Science, in press.
35. S.M.Shortreed,, E.Laber,, D.J.Lizotte,, T.S.Stroup,, J.Pineau,, S.A.Murphy, (2011), Informing Sequential Clinical Decision-Making Through Reinforcement Learning: An Empirical Study, Machine Learning, 84, 109–136.
36. M.Socinski,, T.Stinchcombe, (2007), Duration of First-Line Chemotherapy in Advanced Non Small-Cell Lung Cancer: Less is More in the Era of Effective Subsequent Therapies, Journal of Clinical Oncology, 25, 5155–5157.
37. J.Splawa-Neyman,, D.Dabrowska,, T.Speed, (1990), On the Application of Probability Theory to Agricultural Experiments (Engl. transl. by D. M. Dabrowska and T. P. Speed), Statistical Science, 5, 465–472.
38. I.Steinwart,, C.Scovel, (2007), Fast Rates for Support Vector Machines using Gaussian Kernels, The Annals of Statistics, 35, 575–607.
39. V.Strecher,, J.McClure,, G.Alexander,, B.Chakraborty,, V.Nair,, J.Konkel,, S.Greene,, L.Collins,, C.Carlier,, C.Wiese,, R.Little,, C.Pomerleau,, O.Pomerleau, (2008), Web-Based Smoking Cessation Components and Tailoring Depth: Results of a Randomized Trial, American Journal of Preventive Medicine, 34, 373–381.
40. R.S.Sutton,, and A.G.Barto, (1998), Reinforcement Learning I: Introduction, Cambridge, MA: MIT Press.
41. P.F.Thall,, R.E.Millikan,, H.G.Sung, (2000), Evaluating Multiple Treatment Courses in Clinical Trials, Statistics in Medicine, 19, 1011–1028.
42. P.F.Thall,, H.-G.Sung,, E.H.Estey, (2002), Selecting Therapeutic Strategies Based on Efficacy and Death in Multicourse Clinical Trials, Journal of the American Statistical Association, 97, 29–39.
43. P.F.Thall,, L.H.Wooten,, C.J.Logothetis,, R.E.Millikan,, N.M.Tannir, (2007), Bayesian and Frequentist Two-Stage Treatment Strategies Based on Sequential Failure Times Subject to Interval Censoring, Statistics in Medicine, 26, 4687–4702.
44. A.B.Tsybakov, (2004), Optimal Aggregation of Classifiers in Statistical Learning, The Annals of Statistics, 32, 135–166.
45. E.Wagner,, B.Austin,, C.Davis,, M.Hindmarsh,, J.Schaefer,, A.Bonomi, (2001), Improving Chronic Illness Care: Translating Evidence Into Action, Health Affairs, 20, 64–78.
46. J.Wang,, X.Shen,, Y.Liu, (2008), Probability Estimation for Large-Margin Classifiers, Biometrika, 95, 149–167.
47. C.J.C.H.Watkins, (1989), Learning From Delayed Rewards, Ph.D. Thesis, Cambridge, UK: Kings College.
48. B.Zhang,, A.A.Tsiatis,, M.Davidian,, M.Zhang,, E.Laber, (2012a), Estimating Optimal Treatment Regimes From a Classification Perspective, Statistics, 1, 103–114.
49. B.Zhang,, A.A.Tsiatis,, E.B.Laber,, M.Davidian, (2012b), A Robust Method for Estimating Optimal Treatment Regimes, Biometrics, 68, 1010–1018.
50. (2013), Robust Estimation of Optimal Dynamic Treatment Regimes for Sequential Treatment Decisions, Biometrika, 100, 681–694.
51. Y.Q.Zhao,, M.R.Kosorok,, D.Zeng, (2009), Reinforcement Learning Design for Cancer Clinical Trials, Statistics in Medicine, 28, 3294–3315.
52. Y.Q.Zhao,, D.Zeng,, M.A.Socinski,, M.R.Kosorok, (2011), Reinforcement Learning Strategies for Clinical Trials in Nonsmall Cell Lung Cancer, Biometrics, 67, 1422–1433.
53. Y.Q.Zhao,, D.Zeng,, A.J.Rush,, M.R.Kosorok, (2012), Estimating Individualized Treatment Rules using Outcome Weighted Learning, Journal of the American Statistical Association, 107, 1106–1118.
54. J.Zhu,, S.Rosset,, T.Hastie,, and R.Tibshirani, (2004), “1-Norm Support Vector Machines,” Advances in Neural Information Processing Systems, 110, 49–56.