Tuning continual exploration in reinforcement learning: An optimality property of the Boltzmann strategy

Neurocomputing

Volumn 71, Issue 13-15, 2008, Pages 2507-2520

  Achbany, Youssef ^a   Fouss, François ^b   Yen, Luh ^a   Pirotte, Alain ^a   Saerens, Marco ^a  

^a LOUVAIN SCHOOL OF MANAGEMENT   (Belgium)

^b Facultes Universitaires Catholiques de Mons   (Belgium)

Author keywords

Exploration and exploitation; Markov decision processes; Maximum entropy; Randomized strategy; Reinforcement learning; Shortest path problems

Indexed keywords

DIRECTED GRAPHS; DYNAMIC PROGRAMMING; ENTROPY; GLOBAL OPTIMIZATION; GRAPH THEORY; ITERATIVE METHODS; MARKOV PROCESSES; MAXIMUM ENTROPY METHODS; OPTIMIZATION; PROBABILITY DISTRIBUTIONS; REINFORCEMENT LEARNING; STOCHASTIC SYSTEMS;

EXPLORATION AND EXPLOITATION; EXPLORATION EXPLOITATIONS; GLOBAL OPTIMIZATION PROBLEMS; MARKOV DECISION PROCESSES; RANDOMIZED STRATEGY; SHORTEST PATH PROBLEM; STATIONARY ENVIRONMENTS; STOCHASTIC SHORTEST PATH PROBLEM;

NONLINEAR EQUATIONS;

COMPUTER SIMULATION; CONCEPTUAL FRAMEWORK; CONFERENCE PAPER; CONTROLLED STUDY; COST MINIMIZATION ANALYSIS; ENTROPY; ENVIRONMENTAL EXPLOITATION; EXPERIENTIAL LEARNING; INFORMATION PROCESSING; LEARNING ALGORITHM; NONLINEAR SYSTEM; PRIORITY JOURNAL; PROBABILITY; PROBLEM SOLVING; REINFORCEMENT; STATISTICAL ANALYSIS; TUNING CURVE;

EID: 56449125387     PISSN: 09252312     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neucom.2007.11.040     Document Type: Conference Paper

References (47)

1. T. Akamatsu, Cyclic flows, Markov process and stochastic traffic assignment, Transport. Res. B 30 (5) (1996) 369-386.
2. J. Bather, Decision Theory: An Introduction to Dynamic Programming and Sequential Decisions, Wiley, New York, 2000.
3. M.S. Bazaraa, H.D. Sherali, C.M. Shetty, Nonlinear Programming: Theory and Algorithms, Wiley, New York, 1993.
4. D.P. Bertsekas, Network Optimization: Continuous and Discrete Models, Athena Scientific, 1998.
5. D.P. Bertsekas, Dynamic Programming and Optimal Control, Athena Scientific, 2000.
6. D.P. Bertsekas, J. Tsitsiklis, Neuro-dynamic Programming, Athena Scientific, 1996.
7. J.A. Boyan, M.L. Littman, Packet routing in dynamically changing networks: A reinforcement learning approach, Adv. Neural Inf. Process. Syst. (NIPS) 6 (1994) 671-678.
8. S. Boyd, P. Diaconis, L. Xiao, Fastest mixing Markov chain on a graph, SIAM Rev. (2004) 667-689.
9. R.G. Brown, Smoothing, Forecasting and Prediction of Discrete Time Series, Prentice-Hall, Englewood Cliffs, NJ, 1962.
10. N. Christofides, Graph Theory: An Algorithmic Approach, Academic Press, New York, 1975.
11. T.M. Cover, J.A. Thomas, Elements of Information Theory, Wiley, New York, 1991.
12. D.P. de Farias, B.V. Roy, On the existence of fixed points for approximate value iteration and temporal-difference learning, J. Opt. Theory Appl. 105 (2000) 22-32.
13. J.-C. Delvenne, Pagerank, entropy and free energy, 2005, Manuscript submitted for publication.
14. R. Dial, A probabilistic multipath assignment model that obviates path enumeration, Transport. Res. 5 (1971) 83-111.
15. F. Fouss, A. Pirotte, J.-M. Renders, M. Saerens, Random-walk computation of similarities between nodes of a graph, with application to collaborative recommendation, IEEE Trans. Knowl. Data Eng. 19 (3) (2007) 355-369.
16. M. Guo, Y. Liu, J. Malec, A new Q-learning algorithm based on the metropolis criterion, IEEE Trans. Syst. Man Cybernet. B: Cybernet. 34 (5) (2004) 2140-2143.
17. L.P. Kaelbling, M. Littman, A. Moore, Reinforcement learning: A survey, J. Artif. Intell. Res. 4 (1996) 237-285.
18. L. Kanal, V. Kumar, Search in Artificial Intelligence, Springer, Berlin, 1988.
19. J.N. Kapur, H.K. Kesavan, Entropy Optimization Principles with Applications, Academic Press, New York, 1992.
20. J.G. Kemeny, J.L. Snell, Finite Markov Chains, Springer, Berlin, 1976.
21. M.L. Littman, Markov games as a framework for multi-agent reinforcement learning, in: Proceedings of the 11th International Conference on Machine Learning (ICML-94), 1994, pp. 157-163.
22. N. Meuleau, P. Bourgine, Exploration of multi-state environments: Local measures and back-propagation of uncertainty, Mach. Learn. 35 (1999) 117-154.
23. N. Meuleau, L. Peshkin, K. Kim, Exploration in gradient-based reinforcement learning, AI Memo 2001-003, Massachusetts Institute of Technology, 2001.
24. T.M. Mitchell, Machine Learning, McGraw-Hill Companies, 1997.
25. M.J. Osborne, An Introduction to Game Theory, Oxford University Press, Oxford, 2004.
26. R. Patrascu, D. Stacey, Adaptive exploration in reinforcement learning, in: Proceedings of the International Joint Conference on Neural Networks (IJCNN1999), 1999, pp. 2276-2281.
27. L. Peshkin, V. Savova, Reinforcement learning for adaptive routing, in: Proceedings of the International Joint Conference on Neural Networks (IJNN2002), 2002, pp. 1825-1830.
28. M. Puterman, Markov Decision Processes: Discrete Stochastic Programming, Wiley, New York, 1994.
29. H. Raiffa, Decision Analysis, Addison-Wesley, Reading, MA, 1970.
30. G. Rummery, M. Niranjan, On-line Q-learning using connectionist systems, Technical Report CUED/F-INFENG/TR 166, Cambridge University Engineering Department, 1994.
31. M. Saerens, F. Fouss, L. Yen, P. Dupont, The principal components analysis of a graph, and its relationships to spectral clustering, in: Proceedings of the 15th European Conference on Machine Learning (ECML 2004), Lecture Notes in Artificial Intelligence, vol. 3201, Springer, Berlin, 2004, pp. 371-383.
32. G. Shani, R. Brafman, S. Shimony, Adaptation for changing stochastic environments through online POMDP policy learning, in: Workshop on Reinforcement Learning in Non-Stationary Environments, ECML 2005, 2005, pp. 61-70.
33. S. Singh, R. Sutton, Reinforcement learning with replacing eligibility traces, Mach. Learn. 22 (1996) 123-158.
34. J.C. Spall, Introduction to Stochastic Search and Optimization, Wiley, New York, 2003.
35. J. Sun, S. Boyd, L. Xiao, P. Diaconis, The fastest mixing Markov process on a graph and a connection to a maximum variance unfolding problem, SIAM Rev. (2006) 681-699.
36. R.S. Sutton, A.G. Barto, Reinforcement Learning: An Introduction, The MIT Press, Cambridge, MA, 1998.
37. A. Tahbaz, A. Jadbabaie, A one-parameter family of distributed consensus algorithms with boundary: From shortest paths to mean hitting times, in: Proceedings of IEEE Conference on Decision and Control, 2006, pp. 4664-4669.
38. H.M. Taylor, S. Karlin, An Introduction to Stochastic Modeling, third ed., Academic Press, New York, 1998.
39. S. Thrun, Efficient exploration in reinforcement learning, Technical Report, School of Computer Science, Carnegie Mellon University, 1992.
40. S. Thrun, The role of exploration in learning control, in: D. White, D. Sofge (Eds.), Handbook for Intelligent Control: Neural, Fuzzy and Adaptive Approaches, Van Nostrand Reinhold, Princeton, NJ, 1992.
41. S. Thrun, W. Burgard, D. Fox, Probabilistic Robotics, MIT Press, Cambridge, 2005.
42. H.C. Tijms, A First Course in Stochastic Models, Wiley, New York, 2003.
43. E. Todorov, Linearly-solvable Markov decision problems, in: Advances in Neural Information Processing Systems, MIT Press, Cambridge, MA, 2006.
44. J. Tomlin, A new paradigm for ranking pages on the World Wide Web, in: Proceedings of the International World Wide Web Conference (WWW2003), 2003, pp. 350-355.
45. K. Verbeeck, Coordinated exploration in multi-agent reinforcement learning, Ph.D. Thesis, Vrije Universiteit Brussel, Belgium, 2004.
46. J.C. Watkins, Learning from delayed rewards, Ph.D. Thesis, King's College of Cambridge, UK, 1989.
47. J.C. Watkins, P. Dayan, Q-learning, Mach. Learn. 8 (3/4) (1992) 279-292.