Bayesian active learning with basis functions

IEEE SSCI 2011: Symposium Series on Computational Intelligence - ADPRL 2011: 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning

Volumn , Issue , 2011, Pages 143-150

  Ryzhov, Ilya O ^a   Powell, Warren B ^a  

^a Princeton University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ACTIVE LEARNING; APPROXIMATE DYNAMIC PROGRAMMING; BASIS FUNCTIONS; CURSE OF DIMENSIONALITY; EXPLORATION/EXPLOITATION DILEMMAS; LINEAR COMBINATIONS; NUMERICAL EXPERIMENTS; VALUE FUNCTION APPROXIMATION;

ARTIFICIAL INTELLIGENCE; NUMERICAL METHODS; REINFORCEMENT LEARNING;

DYNAMIC PROGRAMMING;

EID: 80052219755     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ADPRL.2011.5967365     Document Type: Conference Paper

References (35)

1. M. L. Puterman, Markov Decision Processes. New York: John Wiley & Sons, 1994.
2. H. Geman and A. Roncoroni, "Understanding the Fine Structure of Electricity Prices," The Journal of Business, vol. 79, no. 3, 2006.
3. R. Sutton and A. Barto, Reinforcement Learning. Cambridge, Massachusetts: The MIT Press, 1998.
4. D. Bertsekas and J. Tsitsiklis, Neuro-Dynamic Programming. Athena Scientific, 1996.
5. J. Si, A. G. Barto, W. B. Powell, and D. Wunsch, Eds., Handbook of Learning and Approximate Dynamic Programming. New York: IEEE Press, 2004.
6. W. B. Powell, Approximate Dynamic Programming: Solving the curses of dimensionality. New York: John Wiley and Sons, 2007.
7. B. Van Roy, D. Bertsekas, Y. Lee, and J. Tsitsiklis, "A neuro-dynamic programming approach to retailer inventory management," in Proceedings of the 36th IEEE Conference on Decision and Control, vol. 4, 1997, pp. 4052-4057.
8. G. Tesauro, "Practical Issues in Temporal Difference Learning," Machine Learning, vol. 8, no. 3-4, pp. 257-277, 1992.
9. P. Sabes, "Approximating Q-values with basis function representations," in Proceedings of the Fourth Connectionist Models Summer School, Hillsdale, NJ, M. Mozer, D. Touretzky, and P. Smolensky, Eds., 1993, pp. 264-271.
10. I. Menache, S. Mannor, and N. Shimkin, "Basis function adaptation in temporal-difference reinforcement learning," Annals of Operations Research, vol. 134, no. 1, pp. 215-238, 2005. (Pubitemid 40550047)
11. R. Sutton, H. Maei, D. Precup, S. Bhatnagar, D. Silver, C. Szepesvári, and E. Wiewiora, "Fast gradient-descent methods for temporal-difference learning with linear function approximation," in Proceedings of the 26th International Conference on Machine Learning, 2009, pp. 993-1000.
12. L. P. Kaelbling, Learning in Embedded Systems. Cambridge, MA: MIT Press, 1993.
13. K. Iwata, K. Ikeda, and H. Sakai, "A new criterion using information gain for action selection strategy in reinforcement learning," IEEE Transactions on Neural Networks, vol. 15, no. 4, pp. 792-799, 2004.
14. R. Bechhofer, T. Santner, and D. Goldsman, Design and Analysis of Experiments for Statistical Selection, Screening and Multiple Comparisons. New York: John Wiley and Sons, 1995.
15. J. Gittins, Multi-Armed Bandit Allocation Indices. New York: John Wiley and Sons, 1989.
16. M. Duff and A. Barto, "Local bandit approximation for optimal learning problems," Advances in Neural Information Processing Systems, vol. 9, pp. 1019-1025, 1996.
17. R. Dearden, N. Friedman, and D. Andre, "Model-based Bayesian Exploration," in Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence, 1999, pp. 150-159.
18. M. Strens, "A Bayesian framework for reinforcement learning," in Proceedings of the 17th International Conference on Machine Learning, 2000, pp. 943-950.
19. M. Duff, "Design for an optimal probe," in Proceedings of the 20th International Conference on Machine Learning, 2003, pp. 131-138.
20. P. Poupart, N. Vlassis, J. Hoey, and K. Regan, "An analytic solution to discrete Bayesian reinforcement learning," in Proceedings of the 23rd International Conference on Machine Learning, 2006, pp. 697-704.
21. R. Dearden, N. Friedman, and S. Russell, "Bayesian Q-learning," in Proceedings of the 15th National Conference on Artificial Intelligence, 1998, pp. 761-768.
22. Y. Engel, S. Mannor, and R. Meir, "Bayes Meets Bellman: The Gaussian Process Approach to Temporal Difference Learning," in Proceedings of the 20th International Conference on Machine Learning, 2003, pp. 154-161.
23. -, "Reinforcement learning with Gaussian processes," in Proceedings of the 22nd International Conference on Machine Learning, 2005, pp. 208-215.
24. S. Gupta and K. Miescke, "Bayesian look ahead one-stage sampling allocations for selection of the best population," Journal of Statistical Planning and Inference, vol. 54, no. 2, pp. 229-244, 1996. (Pubitemid 126161097)
25. P. I. Frazier, W. B. Powell, and S. Dayanik, "A knowledge gradient policy for sequential information collection," SIAM Journal on Control and Optimization, vol. 47, no. 5, pp. 2410-2439, 2008.
26. I. O. Ryzhov and W. B. Powell, "The knowledge gradient algorithm for online subset selection," in Proceedings of the 2009 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, Nashville, TN, 2009, pp. 137-144.
27. I. O. Ryzhov, W. B. Powell, and P. I. Frazier, "The knowledge gradient algorithm for a general class of online learning problems," Submitted for publication, 2009.
28. I. O. Ryzhov, M. R. Valdez-Vivas, and W. B. Powell, "Optimal Learning of Transition Probabilities in the Two-Agent Newsvendor Problem," in Proceedings of the 2010 Winter Simulation Conference, B. Johansson, S. Jain, J. Montoya-Torres, J. Hugan, and E. Yücesan, Eds., 2010.
29. I. O. Ryzhov and W. B. Powell, "Approximate Dynamic Programming With Correlated Bayesian Beliefs," in Proceedings of the 48th Allerton Conference on Communication, Control and Computing, 2010.
30. D. Negoescu, P. Frazier, and W. Powell, "The Knowledge-Gradient Algorithm for Sequencing Experiments in Drug Discovery," INFORMS J. on Computing (to appear), 2010.
31. A. George andW. B. Powell, "Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming," Machine Learning, vol. 65, no. 1, pp. 167-198, 2006. (Pubitemid 44451197)
32. J. Scharf, Statistical signal processing. Addison-Wesley, 1991.
33. P. I. Frazier, W. B. Powell, and S. Dayanik, "The knowledge-gradient policy for correlated normal rewards," INFORMS J. on Computing, vol. 21, no. 4, pp. 599-613, 2009.
34. E. Schwartz, "The stochastic behavior of commodity prices: Implications for valuation and hedging," Journal of Finance, vol. 52, no. 3, pp. 923- 973, 1997. (Pubitemid 127344954)
35. R. Sutton, C. Szepesvári, and H. Maei, "A convergent O(n) algorithm for off-policy temporal-difference learning with linear function approximation," Advances in Neural Information Processing Systems, vol. 21, pp. 1609-1616, 2008.