Q-Learning: Flexible Learning About Useful Utilities

Statistics in Biosciences

Volumn 6, Issue 2, 2014, Pages 223-243

  Moodie, Erica E M ^a   Dean, Nema ^b   Sun, Yue Ru ^c  

^a MCGILL UNIVERSITY   (Canada)

^b UNIVERSITY OF GLASGOW   (United Kingdom)

^c MCGILL UNIVERSITY   (Canada)

Author keywords

Adaptive treatment strategies; Discrete data; Dynamic treatment regimes; Generalized additive models; Personalized medicine; Q learning

Indexed keywords

ARTICLE; COMMUNITY PROGRAM; CONFOUNDING VARIABLE; DYNAMICS; LEARNING; OUTCOME ASSESSMENT; PROBABILITY; PROPENSITY SCORE; Q LEARNING; SAMPLE SIZE; SIMULATION;

EID: 84912121111     PISSN: 18671764     EISSN: 18671772     Source Type: Journal    
DOI: 10.1007/s12561-013-9103-z     Document Type: Article

References (33)

1. Chakraborty B (2011) Dynamic treatment regimes for managing chronic health conditions: A statistical perspective. Am J Publ Health 101(1):40–45
2. Chakraborty B, Laber EB, Zhao Y (2013) Inference for optimal dynamic treatment regimes using an adaptive m-out-of-n bootstrap scheme (submitted)
3. Chakraborty B, Moodie EEM (2013) Estimating optimal dynamic treatment regimes with shared decision rules across stages: An extension of Q-learning (submitted)
4. Chakraborty B, Murphy SA, Strecher V (2010) Inference for non-regular parameters in optimal dynamic treatment regimes. Stat Methods Med Res 19(3):317–343
5. Fava M, Rush AJ, Trivedi MH, Nierenberg AA, Thase ME, Sackeim HA, Quitkin FM, Wisniewski S, Lavori PW, Rosenbaum JF, Kupfer DJ (2003) Background and rationale for the sequenced treatment alternatives to relieve depression (STAR*D) study. Psychiatr Clin North Am 26(2):457–494
6. Golub G, Heath M, Wahba G (1979) Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21:215–224
7. Hastie T, Tibshirani R (1986) Generalized additive models. Stat Sci 1(3):297–318
8. Hastie T, Tibshirani R (1990) Generalized additive models. Chapman & Hall, London
9. Huang X, Ning J (2012) Analysis of multi-stage treatments for recurrent diseases. Stat Med 31:2805–2821
10. L, cross-validation and generalized cross-validation: Discrete index set. Ann Stat 15:958–975
11. Moodie EEM, Chakraborty B, Kramer MS (2012) Q-learning for estimating optimal dynamic treatment rules from observational data. Can J Stat 40:629–645
12. Moodie EEM, Richardson TS (2010) Estimating optimal dynamic regimes: Correcting bias under the null. Scand J Stat 37:126–146
13. Murphy SA, Oslin DW, Rush AJ, Zhu J (2007) Methodological challenges in constructing effective treatment sequences for chronic psychiatric disorders. Neuropsychopharmacology 32:257–262
14. Murphy SA (2005) A generalization error for Q-learning. J Mach Learn Res 6:1073–1097
15. Nahum-Shani I, Qian M, Almirall D, Pelham WE, Gnagy B, Fabiano GA, Waxmonsky JG, Yu J, Murphy SA (2012) Q-Learning: A data analysis method for constructing adaptive interventions. Psychol Methods 17:478–494
16. R Core Team (2012) R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna. ISBN 3-900051-07-0
17. Robins JM, Hernán MA, Brumback B (2000) Marginal structural models and causal inference in epidemiology. Epidemiology 11:550–560
18. Robins JM (2004) Optimal structural nested models for optimal sequential decisions. In: Lin DY, Heagerty P (eds) Proceedings of the second Seattle symposium on biostatistics. Springer, New York, pp 189–326
19. Rosenbaum PR, Rubin DB (1983) The central role of the propensity score in observational studies for causal effects. Biometrika 70:41–55
20. Rosthoj S, Fullwood C, Henderson R, Stewart S (2006) Estimation of optimal dynamic anticoagulation regimes from observational data: A regret-based approach. Stat Med 25:4197–4215
21. Schneider LS, Tariot PN, Lyketsos CG, Dagerman KS, Davis KL, Davis S (2001) National institute of mental health clinical antipsychotic trials of intervention effectiveness (CATIE): Alzheimer disease trial methodology. Am J Geriatr Psychiatry 9:346–360
22. Shortreed SM, Moodie EEM (2012) Estimating the optimal dynamic antipsychotic treatment regime: Evidence from the sequential-multiple assignment randomized CATIE schizophrenia study. J R Stat Soc, Ser B, Stat Methodol 61:577–599
23. Song R, Wang W, Zeng D, Kosorok MR (2013) Penalized Q-learning for dynamic treatment regimes (submitted)
24. Sutton RS, Barto AG (1998) Reinforcement learning: An introduction. MIT Press, Cambridge
25. Thall PF, Millikan RE, Sung HG (2000) Evaluating multiple treatment courses in clinical trials. Stat Med 30:1011–1128
26. Thall PF, Sung HG, Estey EH (2002) Selecting therapeutic strategies based on efficacy and death in multicourse clinical trials. J Am Stat Assoc 97(457):29–39
27. Topol E (2012) Creative destruction of medicine: How the digital revolution and personalized medicine will create better health care. Basic Books, New York
28. Wood SN (2004) Stable and efficient multiple smoothing parameter estimation for generalized additive models. J Am Stat Assoc 99(467):673–686
29. Wood SN (2006) Generalized additive models: An introduction with R. Chapman & Hall, London
30. Wood SN (2011) Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models. J R Stat Soc B 73(1):3–36
31. Xin J, Chakraborty B, Laber EB (2012) qLearn: Estimation and inference for Q-learning. R package version 1.0
32. Zhao Y, Kosorok MR, Zeng D (2009) Reinforcement learning design for cancer clinical trials. Stat Med 28:3294–3315
33. Zhao Y, Zeng D, Socinski MA, Kosorok MR (2011) Reinforcement learning strategies for clinical trials in non-small cell lung cancer. Biometrics 67(4):1422–1433