Q-learning with censored data

Annals of Statistics

Volumn 40, Issue 1, 2012, Pages 529-560

  Goldberg, Yair ^a   Kosorok, Michael R ^a  

^a University of North Carolina at Chapel Hill   (United States)

Author keywords

Generalization error; Q learning; Reinforcement learning; Survival analysis

Indexed keywords

EID: 84861349230     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/12-AOS968     Document Type: Article

References (43)

1. ANTHONY, M. and BARTLETT, P. L. (1999). Neural Network Learning: Theoretical Foundations. Cambridge Univ. Press, Cambridge. MR17410388
2. BELLMAN, R. (1957). Dynamic Programming. Princeton Univ. Press, Princeton, NJ. MR00904777
3. BIGANZOLI, E., BORACCHI, P., MARIANI, L. and MARUBINI, E. (1998). Feed forward neural networks for the analysis of censored survival data: A partial logistic regression approach. Stat. Med. 17 1169-1186.
4. BITOUZÉ, D., LAURENT, B. and MASSART, P. (1999). A Dvoretzky-Kiefer-Wolfowitz type inequality for the Kaplan-Meier estimator. Ann. Inst. Henri Poincaré Probab. Stat. 35 735-763. MR17257099
5. CHEN, P.-Y. and TSIATIS, A. A. (2001). Causal inference on the difference of the restricted mean lifetime between two groups. Biometrics 57 1030-1038. MR19504188
6. GOLDBERG, Y. and KOSOROK, M. R. (2012). Supplement to "Q-learning with censored data." DOI:10.1214/12-AOS968SUPP.
7. GOLDBERG, Y. and KOSOROK, M. R. (2012). Support vector regression for right censored data. Unpublished manuscript. Available at http://arxiv.org/abs/ 1202.5130.
8. KARRISON, T. G. (1997). Use of Irwin's restricted mean as an index for comparing survival in different treatment groups - interpretation and power considerations. Control Clin. Trials 18 151- 167.
9. KOSOROK, M. R. (2008). Introduction to Empirical Processes and Semiparametric Inference. Springer, New York. MR27243688
10. KRZAKOWSKI, M., RAMLAU, R., JASSEM, J., SZCZESNA, A., ZATLOUKAL, P., PAWEL, J. V., SUN, X., BENNOUNA, J., SANTORO, A., BIESMA, B., DELGADO, F. M., SALHI, Y., VAISSIERE, N., HANSEN, O., TAN, E.-H., QUOIX, E., GARRIDO, P. and DOUILLARD, J.-Y. (2010). Phase III trial comparing vinflunine with docetaxel in second-line advanced non-smallcell lung cancer previously treated with platinum-containing chemotherapy. J. Clin. Oncol. 28 2167-2173.
11. LABER, E., QIAN, M., LIZOTTE, D. J. and MURPHY, S. A. (2010). Statistical inference in dynamic treatment regimes. Available at http://arxiv.org/abs/1006. 5831.
12. LAVORI, P. W. and DAWSON, R. (2004). Dynamic treatment regimes: Practical design considerations. Clin. Trials 1 9-20.
13. LUNCEFORD, J. K., DAVIDIAN, M. and TSIATIS, A. A. (2002). Estimation of survival distributions of treatment policies in two-stage randomization designs in clinical trials. Biometrics 58 48-57. MR18910422
14. MIYAHARA, S. and WAHED, A. S. (2010). Weighted Kaplan-Meier estimators for two-stage treatment regimes. Stat. Med. 29 2581-2591. MR27569455
15. MOODIE, E. E. M., RICHARDSON, T. S. and STEPHENS, D. A. (2007). Demystifying optimal dynamic treatment regimes. Biometrics 63 447-455. MR23708033
16. MURPHY, S. A. (2003). Optimal dynamic treatment regimes. J. R. Stat. Soc. Ser. B Stat. Methodol. 65 331-366. MR19837522
17. MURPHY, S. A. (2005a). An experimental design for the development of adaptive treatment strategies. Stat. Med. 24 1455-1481. MR21376511
18. MURPHY, S. A. (2005b). A generalization error for Q-learning. J. Mach. Learn. Res. 6 1073-1097 (electronic). MR22498499
19. MURPHY, S. A., OSLIN, D. W., RUSH, A. J., ZHU, J. and MCATS (2007). Methodological challenges in constructing effective treatment sequences for chronic psychiatric disorders. Neuropsychopharmacology 32 257-262.
20. ORELLANA, L., ROTNITZKY, A. and ROBINS, J. M. (2010). Dynamic regime marginal structural mean models for estimation of optimal dynamic treatment regimes, Part I: Main content. Int. J. Biostat. 6 Art. 8, 49. MR26025511
21. ROBINS, J. M. (1999). Association, causation, and marginal structural models. Synthese 121 151- 179. MR17667766
22. ROBINS, J. M. (2004). Optimal structural nested models for optimal sequential decisions. In Proceedings of the Second Seattle Symposium in Biostatistics (D. Lin and P. J. Heagerty, eds.) 189- 326. Springer, New York. MR21294022
23. ROBINS, J., ORELLANA, L. and ROTNITZKY, A. (2008). Estimation and extrapolation of optimal treatment and testing strategies. Stat. Med. 27 4678-4721. MR25285766
24. ROBINS, J. M., ROTNITZKY, A. and ZHAO, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. J. Amer. Statist. Assoc. 89 846-866. MR12947300
25. SATTEN, G. A. and DATTA, S. (2001). The Kaplan-Meier estimator as an inverse-probability-ofcensoring weighted average. Amer. Statist. 55 207-210. MR19472666
26. SHIM, J. and HWANG, C. (2009). Support vector censored quantile regression under random censoring. Comput. Statist. Data Anal. 53 912-919. MR26570577
27. SHIVASWAMY, P. K., CHU, W. and JANSCHE, M. (2007). A support vector approach to censored targets. In Proceedings of the 7th IEEE International Conference on Data Mining (ICDM 2007), Omaha, Nebraska, USA 655-660. IEEE Computer Society.
28. STEINWART, I. and CHRISTMANN, A. (2008). Support Vector Machines. Springer, New York. MR24501033
29. STINCHCOMBE, T. E. and SOCINSKI, M. A. (2008). Considerations for second-line therapy of nonsmall cell lung cancer. Oncologist 13 28-36.
30. SUTTON, R. S. and BARTO, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA.
31. THALL, P. F., WOOTEN, L. H., LOGOTHETIS, C. J., MILLIKAN, R. E. and TANNIR, N. M. (2007). Bayesian and frequentist two-stage treatment strategies based on sequential failure times subject to interval censoring. Stat. Med. 26 4687-4702. MR24133922
32. TSITSIKLIS, J. N. and VAN ROY, B. (1996). Feature-based methods for large scale dynamic programming. Machine Learning 22 59-94.
33. VAN DER LAAN, M. J. and PETERSEN, M. L. (2007). Causal effect models for realistic individualized treatment and intention to treat rules. Int. J. Biostat. 3 Art. 3, 54. MR23068411
34. VAN DER VAART, A.W. and WELLNER, J. A. (1996). Weak Convergence and Empirical Processes: With Applications to Statistics. Springer, New York. MR13856711
35. VAPNIK, V. N. (1999). The Nature of Statistical Learning Theory, 2nd ed. Springer, New York.
36. WAHED, A. S. (2009). Estimation of survival quantiles in two-stage randomization designs. J. Statist. Plann. Inference 139 2064-2075. MR24975600
37. WAHED, A. S. and TSIATIS, A. A. (2006). Semiparametric efficient estimation of survival distributions in two-stage randomisation designs in clinical trials with censored data. Biometrika 93 163-177. MR22777488
38. WATKINS, C. J. C. H. (1989). Learning from delayed rewards. Ph.D. thesis, Cambridge Univ.
39. WATKINS, C. J. C. H. and DAYAN, P. (1992). Q-learning. Machine Learning 8 279-292.
40. WELLNER, J. A. (2007). On an exponential bound for the Kaplan-Meier estimator. Lifetime Data Anal. 13 481-496. MR23942844
41. ZHAO, Y., KOSOROK, M. R. and ZENG, D. (2009). Reinforcement learning design for cancer clinical trials. Stat. Med. 28 3294-3315. MR27502777
42. ZHAO, Y., ZENG, D., SOCINSKI, M. A. and KOSOROK, M. R. (2011). Reinforcement learning strategies for clinical trials in nonsmall cell lung cancer. Biometrics 67 1422-1433.
43. ZUCKER, D. M. (1998). Restricted mean life with covariates:Modification and extension of a useful survival analysis method. J. Amer. Statist. Assoc. 93 702-709. MR16313655