Linear fitted-q iteration with multiple reward functions

Journal of Machine Learning Research

Volumn 13, Issue , 2012, Pages 3253-3295

  Lizotte, Daniel J ^a   Bowling, Michael ^b   Murphy, Susan A ^c  

^a UNIVERSITY OF WATERLOO   (Canada)

^b UNIVERSITY OF ALBERTA   (Canada)

^c UNIVERSITY OF MICHIGAN   (United States)

Author keywords

Decision making; Dynamic programming; Linear regression; Preference elicitation; Reinforcement learning

Indexed keywords

ARBITRARY NUMBER; CLINICAL DECISION SUPPORT; DECISION AIDS; MULTIPLE REWARDS; PREFERENCE ELICITATION; QUALITY OF LIFE; SEQUENTIAL TREATMENTS; VALUE FUNCTION APPROXIMATION; WEIGHT GAIN;

APPROXIMATION ALGORITHMS; COMPUTATIONAL GEOMETRY; DECISION MAKING; DECISION SUPPORT SYSTEMS; DYNAMIC PROGRAMMING; LINEAR REGRESSION; REINFORCEMENT LEARNING;

ITERATIVE METHODS;

EID: 84870881620     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (55)

1. D. B. Allison, J. L. Mentore, M. Heo, L. P. Chandler, J. C. Cappelleri, M. C. Infante, and P. J. Weiden. Antipsychotic-induced weight gain: A comprehensive research synthesis. American Journal of Psychiatry, 156:1686-1696, November 1999. (Pubitemid 29517559)
2. L. Barrett and S. Narayanan. Learning all optimal policies with multiple criteria. In Proceedings of the 25th International Conference on Machine Learning, pages 41-47, 2008.
3. D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming, chapter 2.1, page 12. Athena Scientific, 1996.
4. O. Bonnichsen. Elicitation of ostomy pouch preferences: a discrete-choice experiment. Patient, 4 (3):163-175, 2011.
5. C. Boutilier. A POMDP formulation of preference elicitation problems. In Proceedings of the 18th National Conference on Artificial Intelligence, pages 239-246, 2002.
6. P. Brass. On the size of higher-dimensional triangulations. In J. E. Goodman, J. Pach, and E.Welzl, editors, Combinatorial and Computational Geometry, volume 52, pages 147-152. MSRI Publications, 2005.
7. A. Castelletti, S. Galelli, M. Restelli, and R. Soncini-Sessa. Tree-based reinforcement learning for optimal water reservoir operation. Water Resources Research, 46(W06502), 2010.
8. CGAL. CGAL, Computational Geometry Algorithms Library, 2011. URL http://www.cgal.org.
9. L. P. Chew. Constrained delaunay triangulations. In Proceedings of the Third Annual Symposium on Computational geometry, SCG '87, pages 215-222, New York, NY, USA, 1987.
10. R. D. Cook and S.Weisberg. Applied Regression Including Computing and Graphics. Wiley, August 1999.
11. D. A. Cox, D. O'Shea, and J. B. Little. Ideals, Varieties and Algorithms: An Introduction to Computational Algebraic Geometry and Commutative Algebra. Springer, 1997.
12. J. A. Cramer, R. Rosenheck, W. Xu, J. Thomas, W. Henderson, and D. S. Charney. Quality of life in schizophrenia: A comparison of instruments. Schizophrenia Bulletin, 26(3):659-666, 2000.
13. C. C. Davis, M. Claudius, L. A. Palinkas, J. B. Wong, and L. K. Leslie. Putting families in the center: Family perspectives on decision making and ADHD and implications for ADHD care. Journal of Attention Disorders, Oct 2011. E-pub ahead of print.
14. M. de Berg, O. Cheong, M. van Kreveld, and M. Overmars. Computational Geometry Algorithms and Applications. Springer, 3 edition, 2008.
15. M. Ehrgott. Multicriteria Optimization, chapter 3. Springer, second edition, 2005.
16. Z. Gábor, Z. Kalmár, and C. Szepesvári. Multi-criteria reinforcement learning. In Proceedings of the 15th International Conference on Machine Learning, pages 197-205, 1998.
17. G. H. Golub and C. F. Van Loan. Matrix Computation. John Hopkins Studies in the Mathematical Sciences. Johns Hopkins University Press, 1996. ISBN 0471935271.
18. B. Grünbaum. Convex Polytopes, volume 221 of Graduate Texts in Mathematics. Springer-Verlag, 1967. ISBN 0387004246;.
19. D. W. Heinrichs, T. E. Hanlon, and W. T. C. Jr. The Quality of Life Scale: An instrument for rating the schizophrenic deficit syndrome. Schizophrenia Bulletin, 10(3):388-398, 1984.
20. L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(12):99-134, 1998. (Pubitemid 128387390)
21. S. R. Kay, A. Fiszbein, and L. A. Opfer. The Positive and Negative Syndrome Scale (PANSS) for schizophrenia. Schizophrenia Bulletin, 13(2):261-276, 1987. (Pubitemid 18039085)
22. P. W. Keller, S. Mannor, and D. Precup. Automatic basis function construction for approximate dynamic programming and reinforcement learning. In Proceedings of the 23rd International Conference on Machine learning, ICML '06, pages 449-456, 2006.
23. A. Y. Kuk, J. Li, and A. J. Rush. Recursive subsetting to identify patients in the STAR D: a method to enhance the accuracy of early prediction of treatment outcome and to inform personalized care. Journal of Clinical Psychiatry, 71(11):1502-8, November 2010.
24. E. B. Laber, M. Qian, D. J. Lizotte, and S. A. Murphy. Statistical inference in dynamic treatment regimes. Technical Report 50, Univ. of Michigan Statistics Department, 2009.
25. E. B. Laber, D. J. Lizotte, and B. Ferguson. Set-valued dynamic treatment regimes for competing outcomes. arXiv:1207.3100v2 [stat.ME], 2012.
26. D. J. Lizotte, M. Bowling, and S. A. Murphy. Efficient reinforcement learning with multiple reward functions for randomized controlled trial analysis. In J. Fürnkranz and T. Joachims, editors, Proceedings of the 27th International Conference on Machine Learning, pages 695-702, Haifa, Israel, June 2010. Omnipress.
27. S. Mannor and N. Shimkin. A geometric approach to multi-criterion reinforcement learning. Journal of Machine Learning Research, 5:325-260, 2004.
28. S. Mannor and J. N. Tsitsiklis. Mean-variance optimization in Markov decision processes. In Proceedings of the 28th International Conference on Machine Learning, pages 177-184, 2011.
29. M. S. McDonagh, K. Peterson, S. Carson, R. Fu, and S. Thakurta. Drug class review: Atypical antipsychotic drugs. Technical report, Oregon Health & Science University, July 2010. Drug Effectiveness Review Project, Update 3.
30. J. R. McKay. Treating Substance Use Disorders with Adaptive Continuing Aftercare. American Psychological Association, 2009.
31. A. L. Miller, J. A. Chiles, and J. K. C. et al. The Texas Medication Algorithm Project (TMAP) schizophrenia algorithms. Journal of Clinical Psychiatry, 60:649-657, 1999. (Pubitemid 29503644)
32. S. A. Murphy. An experimental design for the development of adaptive treatment strategies. Statistics in Medicine, 24:1455-1481, 2005. (Pubitemid 40716347)
33. S. A. Murphy, S. W. Oslin, A. J. Rush, and J. Zhu. Methodological challenges in constructing effective treatment sequences for chronic psychiatric disorders. Neuropsychopharmacology, 32: 257-262, 2007. (Pubitemid 46106932)
34. S. Natarajan and P. Tadepalli. Dynamic preferences in multi-criteria reinforcement learning. In Proceedings of the 22nd International Conference on Machine Learning, pages 601-608, 2005. (Pubitemid 43183383)
35. National Institutes of Health. Clinical Guidelines on the Identification, and Treatment of Overweight and Obesity in Adults: The Evidence Report. National Heart, Lung, and Blood Institute, Sept 1998. NIH Publication no. 98-4083.
36. A. Y. Ng and S. J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the 17th International Conference on Machine Learning, pages 663-670, 2000.
37. J. Pineau, G. Gordon, and S. Thrun. Point-based value iteration: An anytime algorithm for POMDPs. In International Joint Conference on Artificial Intelligence, pages 1025-1032, 2003.
38. J. Pineau, G. Gordon, and S. Thrun. Anytime point-based approximations for large POMDPs. Journal of Artificial Intelligence Research, 27:335-380, 2006.
39. J. Pineau,M. G. Bellemare, A. J. Rush, A. Ghizaru, and S. A.Murphy. Constructing evidence-based treatment strategies using methods from computer science. Drug and Alcohol Dependence, 88 (Suppl 2):S52-S60, May 2007. (Pubitemid 46546455)
40. P. Poupart and C. Boutilier. Value-directed compression of POMDPs. Advances in Neural Information Processing Systems, 15:1547-1554, 2002.
41. D. L. Sackett. Evidence-based medicine: What it is and what it isn't. British Medical Journal, 312 (7023):71-72, 1996.
42. S. Shortreed, E. B. Laber, D. J. Lizotte, T. S. Stroup, J. Pineau, and S. A. Murphy. Informing sequential clinical decision-making through reinforcement learning: an empirical study. Machine Learning, pages 1-28, 2010. ISSN 0885-6125.
43. T. S. Stroup, J. P. McEvoy, M. S. Swartz, M. J. Byerly, I. D. Glick, J. M. Canive, M. McGee, G. M. Simpson, M. D. Stevens, and J. A. Lieberman. The national institute of mental health clinical antipschotic trials of intervention effectiveness (CATIE) project: Schizophrenia trial design and protocol deveplopment. Schizophrenia Bulletin, 29(1):15-31, 2003. (Pubitemid 36871054)
44. B. Sturmfels. Solving Systems of Polynomial Equations. Regional conference series in mathematics. Published for the Conference Board of theMathematical Sciences by the AmericanMathematical Society, 2002. ISBN 9780821832516.
45. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
46. M. S. Swartz, D. O. Perkins, T. S. Stroup, J. P. McEvoy, J.M. Nieri, and D. D. Haal. Assessing clinical and functional outcomes in the clinical antipsychotic of intervention effectiveness (CATIE) schizophrenia trial. Schizophrenia Bulletin, 29(1):33-43, 2003. (Pubitemid 36871055)
47. C. Szepesvári. Algorithms for Reinforcement Learning. Morgan & Claypool, July 2010.
48. P. F. Thall. Some geometric methods for constructing decision criteria based on two-dimensional parameters. Journal of Statistical Planning and Inference, 138(2):516-527, 2008.
49. D. J. Uherka and A. M. Sergott. On the continuous dependence of the roots of a polynomial on its coefficients. The American Mathematical Monthly, 84(5):368-370, 1977. ISSN 0002-9890.
50. P. Vamplew, R. Dazeley, A. Berry, R. Issabekov, and E. Dekker. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine Learning, 84:51-80, 2011. ISSN 0885-6125. 10.1007/s10994-010-5232-5.
51. T. Wang, P. Poupart, M. Bowling, and D. Schuurmans. Compact, convex upper bound iteration for approximate pomdp planning. In Proceedings of the 21st National Conference on Artificial Intelligence, pages 1245-1251, 2006. (Pubitemid 44705445)
52. J. R. Weisz, B. C. Chu, and A. J. Polo. Treatment dissemination and evidence-based practice: Strengthening intervention through clinician-researcher collaboration. Clinical Psychology: Science and Practice, 11(3):300-307, 2004. ISSN 1468-2850. (Pubitemid 39136455)
53. Working Group for the Canadian Psychiatric Association and the Canadian Alliance for Research on Schizophrenia. Canadian clinical practice guidelines for the treatment of schizophrenia. Canadian Journal of Psychiatry, 43(suppl. 2):25-40S, 1998.
54. Z. Zamani, S. Sanner, and C. Fang. Symbolic dynamic programming for continuous state and action MDPs. In Proceedings of the 26th AAAI Conference on Artificial Intelligence, 2012. To appear.
55. Y. Zhao, M. R. Kosorok, and D. Zeng. Reinforcement learning design for cancer clinical trials. Statistics in Medicine, 28:3294-3315, 2009.