Informing sequential clinical decision-making through reinforcement learning: An empirical study

Machine Learning

Volumn 84, Issue 1-2, 2011, Pages 109-136

  Shortreed, Susan M ^a   Laber, Eric ^b   Lizotte, Daniel J ^b   Stroup, T Scott ^c   Pineau, Joelle ^a   Murphy, Susan A ^b  

^a MCGILL UNIVERSITY   (Canada)

^b UNIVERSITY OF MICHIGAN   (United States)

^c NEW YORK STATE PSYCHIATRIC INSTITUTE   (United States)

Author keywords

Fitted Q iteration; Optimal treatment policies; Policy uncertainty

Indexed keywords

CHRONIC ILLNESS; CLINICAL TRIAL; EMPIRICAL STUDIES; FITTED Q-ITERATION; FUNCTION APPROXIMATION; MISSING DATA; MULTIPLE IMPUTATION; OPTIMAL TREATMENT POLICIES; Q-FUNCTIONS; REINFORCEMENT LEARNING METHOD; TREATMENT POLICIES;

DECISION MAKING; OPTIMIZATION;

REINFORCEMENT LEARNING;

EID: 79958787689     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-010-5229-0     Document Type: Article

References (68)

1. C. E. Adams 2002 Schizophrenia trials: past, present and future Epidemiologia E Psichiatria Sociale 11 13 144 151 (Pubitemid 35277840)
2. D. W. K. Andrews 2000 Inconsistency of the bootstrap when a parameter is on the boundary of the parameter space Econometrica 68 2 399 405 1748009 1015.62044 10.1111/1468-0262.00114
3. Bagnell, A., Ng, A., & Schneider, J. (2001). Solving uncertain Markov decision problems (Tech. Rep. CMU-RI-TR-01-25). Robotics Institute, Carnegie Mellon University.
4. D. A. Berry 2006 A guide to drug discovery: Bayesian clinical trials Nature Reviews. Drug Discovery 5 27 36 10.1038/nrd1927
5. S. Biswas D. D. Liu J. J. Lee D. A. Berry 2009 Bayesian clinical trials at the University of Texas M. D. Anderson cancer center Clinical Trials 6 205 216 10.1177/1740774509104992
6. L. Breiman 1996 Bagging predictors Machine Learning 24 2 123 140 1425957 0858.68080 (Pubitemid 126724382)
7. Brunskill, E., Leffler, B. R., Li, L., Littman, M., & Roy, N. (2008). A continuous-state offset-dynamics reinforcement learner. In D. A. McAllester & P. Myllymäki (Eds.), Proceedings of 24th conference on uncertainty in artificial intelligence (UAI 2008) (pp. 53-61).
8. J. R. Carpenter M. G. Kenward I. R. White 2007 Sensitivity analysis after multiple imputation under missing at random: a weighting approach Statistical Methods in Medical Research 16 3 259 275 2371009 1122.62300 10.1177/ 0962280206075303 (Pubitemid 47018662)
9. R. Dawson P. W. Lavori 2004 Placebo-free designs for evaluating new mental health treatments: the use of adaptive strategies Statistics in Medicine 23 3249 3262 10.1002/sim.1920 (Pubitemid 39462085)
10. Dearden, R., Friedman, N., & Andre, D. (1999). Model based Bayesian exploration. In B. Kathryn, & H. P. Laskey (Eds.), Proceedings of 5th conference on uncertainty in artificial intelligence (UAI 1999) (pp. 150-159). San Mateo: Morgan Kaufmann.
11. Diggle, P., Heagerty, P., Liang, K. Y., & Zeger, S. (2002). Analysis of longitudinal data. Oxford: Oxford University Press.
12. Doshi, F., Pineau, J., & Roy, N. (2008). Reinforcement learning with limited reinforcement: using Bayes risk for active learning in POMDPs. In A. McCallum & S. Roweis (Eds.), Proceedings of the 25th annual international conference on machine learning (ICML 2008) (pp. 256-263). New York: Omnipress.
13. B. Efron 1979 Bootstrap methods: another look at the jackknife The Annals of Statistics 7 1 1 26 515681 0406.62024 10.1214/aos/1176344552
14. Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York: Chapman & Hall.
15. Y. Engel S. Mannor R. Meir 2005 Reinforcement learning with Gaussian processes L. D. Raedt S. Wrobel (eds) Proceedings of the 22nd international conference on machine learning (ICML 2005) ACM New York 201 208 10.1145/1102351.1102377 10.1145/1102351.1102377 (Pubitemid 43183334)
16. Engel, Y., Mannor, S., & Meir, R. (2005). Reinforcement learning with Gaussian processes. In L. D. Raedt & S. Wrobel (Eds.), Proceedings of the 22nd international conference on machine learning (ICML 2005) (pp. 201-208). New York: ACM. 10.1145/1102351.1102377. (Pubitemid 43183334)
17. D. Ernst P. Geurts L. Wehenkel 2005 Tree-based batch mode reinforcement learning Journal of Machine Learning Research 6 503 556 2249830
18. Ernst, D., Stan, G. B., Goncalves, J., & Wehenkel, L. (2006). Clinical data based optimal STI strategies for HIV: a reinforcement learning approach. In Proceedings of the machine learning conference of Belgium and The Netherlands (Benelearn) (pp. 65-72).
19. Fard, M. M., Pineau, J. (2009). MDPs with non-deterministic policies. In D. Koller, D. Schuurmans, Y. Bengio, & L. Bottou (Eds.), Advances in neural information processing systems (pp. 1065-1072). Cambridge: MIT Press.
20. Gelman, A., Carlin, J. B., Stern, H., & Rubin, D. B. (1995). Bayesian Data Analysis. New York: Chapman & Hall.
21. A. Gelman I. V. Mechelen G. Verbeke D. F. Heitjan M. Meulders 2005 Multiple imputation for model checking: completed-data plots with missing and latent data Biometrics 61 74 85 2135847 1077.62091 10.1111/j.0006-341X.2005. 031010.x (Pubitemid 40380966)
22. Guez, A., Vincent, R., Avoli, M., & Pineau, J. (2008). Adaptive treatment of epilepsy via batch-mode reinforcement learning. In Proceedings of the innovative applications of artificial intelligence (IAAI).
23. Hastie, T., Tibshirani, R., & Friedman, J. (2001). Elements of statistical learning. Berlin: Springer.
24. Irodova, M., & Sloan, R. H. (2005). Reinforcement learning and function approximation. In Proceeding of the twentieth national conference on artificial intelligence (AAAI) (p. 2005). American Association for Artificial Intelligence, Menlo Park.
25. L. P. Kaelbling M. L. Littman A. Moore 1996 Reinforcement learning: a survey The Journal of Artificial Intelligence Research 4 237 385
26. L. P. Kaelbling M. L. Littman A. R. Cassandra 1998 Planning and acting in partially observable stochastic domains Artificial Intelligence 101 99 134 1641530 0908.68165 10.1016/S0004-3702(98)00023-X (Pubitemid 128387390)
27. Kakade, S., Kearns, J., & Langford, J. (2003). Exploration in metric state spaces. In Proceedings of the 20th Annual International Conference on Machine Learning (ICML 2003).
28. S. R. Kay A. Flazbein L. A. Opler 1987 The positive and negative syndrome scale (PANSS) for schizophrenia Schizophrenia Bulletin 13 2 261 276 (Pubitemid 18039085)
29. Laber, E. B., Qian, M., & Murphy, S. A. (2010). Statistical inference in dynamic treatment regimes (Tech. Rep. 506). Dept. of Statistics, University of Michigan
30. M. G. Lagoudakis R. Parr 2003 Least-squares policy iteration Journal of Machine Learning Research 4 1107 1149 2125347 10.1162/jmlr.2003.4.6.1107
31. Little, R. J. A., & Rubin, D. B. (1987). Statistical analysis with missing data. New York: Wiley.
32. Lizotte, D. J., Laber, E., & Murphy, S. A. (2009) Assessing confidence in policies learned from sequential randomized trials (Tech. Rep. 481). Department of Statistics, University of Michigan.
33. Lizotte, D., Bowling, M., & Murphy, S. (2010). Efficient reinforcement learning with multiple reward functions for randomized controlled trial analysis. In Proceedings of the twenty-seventh international conference on machine learning (ICML 2010). (pp. 695-702). New York: Omnipress.
34. Mannor, S., Simester, D., Sun, P., & Tsitsiklis, J. (2007) Biases and variance in value function estimates. Management Science 53(1).
35. G. Monahan 1982 A survey of partially observable Markov decision processes Management Science 28 1 16 646904 0486.90084 10.1287/mnsc.28.1.1
36. S. M. Murphy 2003 Optimal dynamic treatment regimes Journal of the Royal Statistical Society, Series B 65 2 331 366 1983752 1065.62006 10.1111/1467-9868.00389 (Pubitemid 36607793)
37. S. A. Murphy 2005 An experimental design for the development of adaptive treatment strategies Statistics in Medicine 24 1455 1481 2137651 10.1002/sim.2022 (Pubitemid 40716347)
38. S. A. Murphy D. Oslin A. J. Rush 2007 Methodological challenges in constructing effective treatment sequences for chronic disorders Neuropsychopharmacology 32 2 257 262 10.1038/sj.npp.1301241 (Pubitemid 46106932)
39. NAP (2010). The prevention and treatment of missing data in clinical trials. The National Academies Press, Panel on Handling Missing Data in Clinical Trials. Committee on National Statistics, Division of Behavioral, Social Sciences and Education.
40. Neter, J., Kutner, M. H., Nachtsheim, C. J., & Wasserman, W. (1996). Applied linear statistical models. New York: McGraw-Hill.
41. Parr, R., Li, L., Taylor, G., Painter-Wakefield, C., & Littman, M. (2008). An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning. In A. McCallum, & S. Roweis (Eds.), Proceedings of the 25th annual international conference on machine learning (pp. 752-759). New York: Omnipress.
42. Pineau, J., Bellemare, M. G., Rush, A. J., Ghizaru, A., & Murphy, S. A. (2007). Constructing evidence-based treatment strategies using methods from computer science. Drug and Alcohol Dependence S52-S60. (Pubitemid 46546455)
43. Robins, J. M., Rotnitzky, A., & Scharfstein, D. (1999). Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models. In M. E. Halloran & D. Berry (Eds.), Statistical models in epidemiology: the environment and clinical trials (pp. 1-92). Berlin: Springer.
44. D. B. Rubin 1996 Multiple imputation after 18+ years (with discussion) Journal of the American Statistical Association 91 473 489 0869.62014 10.2307/2291635
45. A. J. Rush M. Fava S. R. Wisniewski P. W. Lavori 2004 Sequenced treatment alternatives to relieve depression (STAR*D): rational and design Controlled Clinical Trials 25 1 119 142 10.1016/S0197-2456(03)00112-0 (Pubitemid 38230109)
46. Schafer, J. L. (1997). Imputation of missing covariates under a multivariate linear mixed model (Tech. rep.). Dept. of Statistics, The Pennsylvania State University.
47. J. L. Schafer 1999 Multiple imputation: a primer Statistical Methods in Medical Research 8 1 3 15 10.1191/096228099671525676 (Pubitemid 29222784)
48. J. L. Schafer R. M. Yucel 2002 Computational strategies for multivariate linear mixed models with missing values Journal of Computational and Graphical Statistics 11 421 442 1938143 10.1198/106186002760180608
49. D. O. Scharfstein A. Rotnitzky J. M. Robins 1999 Adjusting for nonignorable drop-out using semiparametric nonresponse models Journal of the American Statistical Association 94 448 1096 1120 1731478 1072.62644 10.2307/2669923
50. J. Shao 1994 Bootstrap sample size in nonregular cases Proceedings of the American Mathematical Society 122 4 1251 1262 1227529 0820.62037 10.1090/S0002-9939-1994-1227529-8
51. Shelton, C. R. (2001). Balancing multiple sources of reward in reinforcement learning. In Advances in neural information processing systems (NIPS 2000) (pp. 1082-1088).
52. R. D. Smallwood E. J. Sondik 1973 The optimal control of partially observable Markov processes over a finite horizon Operations Research 21 1070 1088 10.1287/opre.21.5.1071
53. Strehl, A. L., & Littman, M. L. (2004). An empirical evaluation of interval Estimation for Markov decision processes. In ICTAI (pp. 128-135). Los Alamitos: IEEE Computer Society. (Pubitemid 40455005)
54. A. L. Strehl M. L. Littman 2005 A theoretical analysis of model-based interval Estimation L. D. Raedt S. Wrobel (eds) Proceedings of the 22nd international conference on Machine learning (ICML 2005 ) ACM New York 856 863 10.1145/1102351.1102459 10.1145/1102351.1102459
55. Strehl, A. L., & Littman, M. L. (2005). A theoretical analysis of model-based interval Estimation. In L. D. Raedt & S. Wrobel (Eds.), Proceedings of the 22nd international conference on Machine learning (ICML 2005 ) (pp. 856-863). New York: ACM. 10.1145/1102351.1102459.
56. Strehl, A., Li, L., Wiewiora, E., Langford, J., & Littman, M. (2006). PAC model-free reinforcement learning. In W. W. Cohen & A. Moore (Eds.), Proceedings of the 23rd annual international conference on machine learning (ICML 2006) (pp. 881-888). (Pubitemid 46966930)
57. Strens, M. J. A. (2000). A Bayesian framework for reinforcement learning. In P. Langley (Ed.), Proceedings of the seventeenth international conference on machine learning (ICML 2000) (p. 943-950). San Francisco: Morgan Kaufmann.
58. T. S. Stroup J. P. McEvoy M. S. Swartz M. J. Byerly I. D. Glick J. M. Canive M. McGee G. M. Simpson M. D. Stevens J. A. Lieberman 2003 The National Institute of Mental Health clinical antipschotic trials of intervention effectiveness (CATIE) project: schizophrenia trial design and protocol development Schizophrenia Bulletin 29 1 15 31 (Pubitemid 36871054)
59. Sutton, R. S., & Barto, A. G. (1998). Off-policy bootstrapping. In Reinforcement learning: an introduction Cambridge: MIT Press.
60. M. S. Swartz D. O. Perkins T. S. Stroup J. P. McEvoy J. M. Nieri D. D. Haal 2003 Assessing clinical and functional outcomes in the clinical antipsychotic of intervention effectiveness (CATIE) schizophrenia trial Schizophrenia Bulletin 29 1 33 43 (Pubitemid 36871055)
61. Tetreault, J., Bohus, D., & Litman, D. (2007). Estimating the reliability of MDP policies: a confidence interval approach. In Proceedings of the human language technology conference (pp. 276-283).
62. P. Thall J. Wathen 2007 Practical Bayesian adaptive randomisation in clinical trials European Journal of Cancer 43 5 859 866 10.1016/j.ejca.2007.01. 006 (Pubitemid 46366693)
63. P. F. Thall J. K. Wathan 2000 Covariate-adjusted adaptive randomization in a sarcoma trial with multistate treatments Statistics in Medicine 19 1011 1028 10.1002/(SICI)1097-0258(20000430)19:8<1011::AID-SIM414>3.0.CO;2-M (Pubitemid 30238912)
64. S. van Buuren 2007 Multiple imputation of discrete and continuous data by fully conditional specification Statistical Methods in Medical Research 16 3 219 242 2371007 1122.62382 10.1177/0962280206074463 (Pubitemid 47018660)
65. S. van Buuren J. P. L. Brand C. G. M. Groothuis-Oudshoorn D. B. Rubin 2006 Fully conditional specification in multivariate imputation Journal of Statistical Computation and Simulation 76 12 1049 1064 2307507 1144.62332 10.1080/10629360600810434 (Pubitemid 44843816)
66. T. Wang D. Lizotte M. Bowling D. Schuurmans 2005 Bayesian sparse sampling for on-line reward optimization L. D. Raedt S. Wrobel (eds) Proceedings of the 22nd international conference on machine learning (ICML 2005) ACM New York 956 963 10.1145/1102351.1102472 10.1145/1102351.1102472
67. Wang, T., Lizotte, D., Bowling, M., & Schuurmans, D. (2005). Bayesian sparse sampling for on-line reward optimization. In L. D. Raedt & S. Wrobel (Eds.), Proceedings of the 22nd international conference on machine learning (ICML 2005) (pp. 956-963). New York: ACM. 10.1145/1102351.1102472.
68. Y. Zhao M. R. Kosorok D. Zeng 2009 Reinforcement learning design for cancer clinical trials Statistics in Medicine 28 3294 3315 10.1002/sim.3720