Infinite-horizon policy-gradient estimation

Journal of Artificial Intelligence Research

Volumn 15, Issue , 2001, Pages 319-350

  Baxter, Jonathan ^a   Bartlett, Peter L ^b  

^a WhizBang Labs   (United States)

^b BIOwulf Technologies   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

GRADIENT-BASED APPROACHES; POLICY PARAMETERS; VALUE-FUNCTION METHODS;

ALGORITHMS; COMPUTATIONAL METHODS; MARKOV PROCESSES; MULTI AGENT SYSTEMS; PROBLEM SOLVING; RANDOM PROCESSES;

LEARNING SYSTEMS;

EID: 0013535965     PISSN: 10769757     EISSN: None     Source Type: Journal    
DOI: 10.1613/jair.806     Document Type: Article

References (49)

1. Aberdeen, D., & Baxter, J. (2001). Policy-gradient learning of controllers with internal state. Tech. rep., Australian National University.
2. Aleksandrov, V. M., Sysoyev, V. I., & Shemeneva, V. V. (1968). Stochastic optimaization. Engineering Cybernetics, 5, 11-16.
3. Baird, L., & Moore, A. (1999). Gradient descent for general reinforcement learning. In Advances in Neural Information Processing Systems 11. MIT Press.
4. Bartlett, P. L., & Baxter, J. (1999). Hebbian synaptic modifications in spiking neurons that learn. Tech. rep., Research School of Information Sciences and Engineering, Australian National University. http://csl.anu.edu.au/∼bartlett/papers/BartlettBaxter-Nov99.ps.gz.
5. Bartlett, P. L., & Baxter, J. (2001). Estimation and approximation bounds for gradient-based reinforcement learning. Journal of Computer and Systems Sciences, 62. Invited Paper: Special Issue on COLT 2000.
6. Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13, 834-846.
7. Baxter, J., Bartlett, P. L., & Weaver, L. (2001). Experiments with infinite-horizon, policy-gradient estimation. Journal of Artificial Intelligence Research. To appear.
8. Baxter, J., Tridgell, A., & Weaver, L. (2000). Learning to play chess using temporal-differences. Machine Learning, 40(3), 243-263.
9. Bertsekas, D. P., & Tsitsiklis, J. N. (1996). Neuro-Dynamic Programming. Athena Scientific.
10. Bertsekas, D. P. (1995). Dynamic Programming and Optimal Control, Vol II. Athena Scientific.
11. Breiman, L. (1966). Probability. Addison-Wesley.
12. Cao, X.-R., & Wan, Y.-W. (1998). Algorithms for Sensitivity Analysis of Markov Chains Through Potentials and Perturbation Realization. IEEE Transactions on Control Systems Technology, 6, 482-492.
13. Dudley, R. M. (1989). Real Analysis and Probability. Wadsworth & Brooks/Cole, Belmont, California.
14. Glynn, P. W. (1986). Stochastic approximation for monte-carlo optimization. In Proceedings of the 1986 Winter Simulation Conference, pp. 356-365.
15. Glynn, P. W. (1990). Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33, 75-84.
16. Glynn, P. W., & L'Ecuyer, P. (1995). Likelihood ratio gradient estimation for regenerative stochastic recursions. Advances in Applied Probability, 27, 4 (1995), 27, 1019-1053.
17. Ho, Y.-C., & Cao, X.-R. (1991). Perturbation Analysis of Discrete Event Dynamic Systems. Kluwer Academic, Boston.
18. Jaakkola, T., Singh, S. P., & Jordan, M. I. (1995). Reinforcement Learning Algorithm for Partially Observable Markov Decision Problems. In Tesauro, G., Touretzky, D., & Leen, T. (Eds.), Advances in Neural Information Processing Systems, Vol. 7. MIT Press, Cambridge, MA.
19. Kimura, H., & Kobayashi, S. (1998a). An analysis of actor/critic algorithms using eligibility traces: Reinforcement learning with imperfect value functions. In Fifteenth International Conference on Machine Learning, pp. 278-286.
20. Kimura, H., & Kobayashi, S. (1998b). Reinforcement learning for continuous action using stochastic gradient ascent. In Intelligent Autonomous Systems (IAS-5), pp. 288-295.
21. Kimura, H., Miyazaki, K., & Kobayashi, S. (1997). Reinforcement learning in POMDPs with function approximation. In Fisher, D. H. (Ed.), Proceedings of the Fourteenth International Conference on Machine Learning (ICML'97), pp. 152-160.
22. Kimura, H., Yamamura, M., & Kobayashi, S. (1995). Reinforcement learning by stochastic hill climbing on discounted reward. In Proceedings of the Twelfth International Conference on Machine Learning (ICML'95), pp. 295-303.
23. Konda, V. R., & Tsitsiklis, J. N. (2000). Actor-Critic Algorithms. In Neural Information Processing Systems 1999. MIT Press.
24. Lancaster, P., & Tismenetsky, M. (1985). The Theory of Matrices. Academic Press, San Diego, CA.
25. Marbach, P., & Tsitsiklis, J. N. (1998). Simulation-Based Optimization of Markov Reward Processes. Tech. rep., MIT.
26. Meuleau, N., Peshkin, L., Kaelbling, L. P., & Kim, K.-E. (2000). Off-policy policy search. Tech. rep., MIT Artificical Intelligence Laboratory.
27. Meuleau, N., Peshkin, L., Kim, K.-E., & Kaelbling, L. P. (1999). Learning finite-state controllers for partially observable environments. In Proceedings of the Fifteenth International Conference on Uncertainty in Artificial Intelligence.
28. Peshkin, L., Kim, K.-E., Meuleau, N., & Kaelbling, L. P. (2000). Learning to cooperate via policy search. In Proceedings of the Sixteenth International Conference on Uncertainty in Artificial Intelligence.
29. Reiman, M. I., & Weiss, A. (1986). Sensitivity analysis via likelihood ratios. In Proceedings of the 1986 Winter Simulation Conference.
30. Reiman, M. I., & Weiss, A. (1989). Sensitivity analysis for simulations via likelihood ratios. Operations Research, 37.
31. Rubinstein, R. Y. (1969). Some Problems in Monte Carlo Optimization. Ph.D. thesis.
32. Rubinstein, R. Y. (1991). How to optimize complex stochastic systems from a single sample path by the score function method. Annals of Operations Research, 27, 175-211.
33. Rubinstein, R. Y. (1992). Decomposable score function estimators for sensitivity analysis and optimization of queueing networks. Annals of Operations Research, 39, 195-229.
34. Rubinstein, R. Y., & Melamed, B. (1998). Modern Simulation and Modeling. Wiley, New York.
35. Rubinstein, R. Y., & Shapiro, A. (1993). Discrete Event Systems. Wiley, New York.
36. Samuel, A. L. (1959). Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development, 3, 210-229.
37. Shilov, G. E., & Gurevich, B. L. (1966). Integral, Measure and Derivative: A Unified Approach. Prentice-Hall, Englewood Cliffs, N.J.
38. Singh, S. P., Jaakkola, T., & Jordan, M. I. (1994). Learning Without State-Estimation in Partially Observable Markovian Decision Processes. In Proceedings of the Eleventh International Conference on Machine Learning.
39. Singh, S., & Bertsekas, D. (1997). Reinforcement learning for dynamic channel allocation in cellular telephone systems. In Advances in Neural Information Processing Systems: Proceedings of the 1996 Conference, pp. 974-980. MIT Press.
40. Smallwood, R. D., & Sondik, E. J. (1973). The optimal control of partially observable Markov decision processes over a finite horizon. Operations Research, 21, 1071-1098.
41. Sondik, E. J. (1978). The optimal control of partially observable Markov decision processes over the infinite horizon: Discounted costs. Operations Research, 26.
42. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press, Cambridge MA. ISBN 0-262-19398-1.
43. Sutton, R. S., McAllester, D., Singh, S., & Mansour, Y. (2000). Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Neural Information Processing Systems 1999. MIT Press.
44. Tao, N., Baxter, J., & Weaver, L. (2001). A multi-agent, policy-gradient approach to network routing. Tech. rep., Australian National University.
45. Tesauro, G. (1992). Practical Issues in Temporal Difference Learning. Machine Learning, 8, 257-278.
46. Tesauro, G. (1994). TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Computation, 6, 215-219.
47. Tsitsikilis, J. N., & Van-Roy, B. (1997). An Analysis of Temporal Difference Learning with Function Approximation. IEEE Transactions on Automatic Control, 42(5), 674-690.
48. Williams, R. J. (1992). Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning, 8, 229-256.
49. Zhang, W., & Dietterich, T. (1995). A reinforcement learning approach to job-shop scheduling. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1114-1120. Morgan Kaufmann.