Experiments with infinite-horizon, policy-gradient estimation

Journal of Artificial Intelligence Research

Volumn 15, Issue , 2001, Pages 351-381

  Baxter, Jonathan ^a   Bartlett, Peter L ^b   Weaver, Lex ^c  

^a WhizBang Labs   (United States)

^b BIOwulf Technologies   (United States)

^c AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

CONJUGATE-GRADIENTS; NATURAL INTERPRETATION; PARTIALLY OBSERVABLE DECISION PROCESS (POMDP);

COMPUTATIONAL METHODS; DECISION THEORY; MARKOV PROCESSES; MULTI AGENT SYSTEMS; PROBLEM SOLVING;

ALGORITHMS;

EID: 0013495368     PISSN: 10769757     EISSN: None     Source Type: Journal    
DOI: 10.1613/jair.807     Document Type: Article

References (35)

1. Aberdeen, D., & Baxter, J. (2001). Policy-gradient learning of controllers with internal state. Tech. rep., Australian National University.
2. Baird, L., & Moore, A. (1999). Gradient descent for general reinforcement learning. In Advances in Neural Information Processing Systems 11. MIT Press.
3. Bartlett, P. L., & Baxter, J. (1999). Hebbian synaptic modifications in spiking neurons that learn. Tech. rep., Research School of Information Sciences and Engineering, Australian National University. http://csl.anu.edu.au/∼bartlett/papers/BartlettBaxter-Nov99.ps.gz.
4. Bartlett, P. L., & Baxter, J. (2000a). Estimation and approximation bounds for gradient-based reinforcement learning. In Proceedings of the Thirteenth Annual Conference on Computational Learning Theory, pp. 133-141.
5. Bartlett, P. L., & Baxter, J. (2000b). Stochastic optimization of controlled partially observable markov decision processes. In Proceedings of the 39th IEEE Conference on Decision and Control (CDC00).
6. Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13, 834-846.
7. Baxter, J., & Bartlett, P. L. (2000). Reinforcement learning in POMDPs via direct gradient ascent. In Proceedings of the Seventeenth International Conference on Machine Learning.
8. Baxter, J., & Bartlett, P. L. (2001). Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research. To appear.
9. Baxter, J., Tridgell, A., & Weaver, L. (2000). Learning to play chess using temporal-differences. Machine Learning, 40(3), 243-263.
10. Bertsekas, D. P., & Tsitsiklis, J. N. (1996). Neuro-Dynamic Programming. Athena Scientific.
11. Cao, X.-R., & Chen, H.-F. (1997). Perturbation Realization, Potentials, and Sensitivity Analysis of Markov Processes. IEEE Transactions on Automatic Control, 42, 1382-1393.
12. Cao, X.-R., & Wan, Y.-W. (1998). Algorithms for Sensitivity Analysis of Markov Chains Through Potentials and Perturbation Realization. IEEE Transactions on Control Systems Technology, 6, 482-492.
13. Fine, T. L. (1999). Feedforward Neural Network Methodology. Springer, New York.
14. Fu, M. C., & Hu, J. (1994). Smooth Perturbation Derivative Estimation for Markov Chains. Operations Research Letters, 15, 241-251.
15. Glynn, P. W. (1986). Stochastic approximation for monte-carlo optimization. In Proceedings of the 1986 Winter Simulation Conference, pp. 356-365.
16. Kimura, H., & Kobayashi, S. (1998). An analysis of actor/critic algorithms using eligibility traces: Reinforcement learning with imperfect value functions. In Fifteenth International Conference on Machine Learning, pp. 278-286.
17. Kimura, H., Miyazaki, K., & Kobayashi, S. (1997). Reinforcement learning in POMDPs with function approximation. In Fisher, D. H. (Ed.), Proceedings of the Fourteenth International Conference on Machine Learning (ICML'97), pp. 152-160.
18. Kimura, H., Yamamura, M., & Kobayashi, S. (1995). Reinforcement learning by stochastic hill climbing on discounted reward. In Proceedings of the Twelfth International Conference on Machine Learning (ICML'95), pp. 295-303.
19. Konda, V. R., & Tsitsiklis, J. N. (2000). Actor-Critic Algorithms. In Neural Information Processing Systems 1999. MIT Press.
20. Marbach, P. (1998). Simulation-Based Methods for Markov Decision Processes. Ph.D. thesis, Laboratory for Information and Decision Systems, MIT.
21. Marbach, P., & Tsitsiklis, J. N. (1998). Simulation-Based Optimization of Markov Reward Processes. Tech. rep., MIT.
22. Rubinstein, R. Y., & Melamed, B. (1998). Modern Simulation and Modeling. Wiley, New York.
23. Samuel, A. L. (1959). Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development, 3, 210-229.
24. Singh, S. P., Jaakkola, T., & Jordan, M. I. (1994). Learning Without State-Estimation in Partially Observable Markovian Decision Processes. In Proceedings of the Eleventh International Conference on Machine Learning.
25. Singh, S., & Bertsekas, D. (1997). Reinforcement learning for dynamic channel allocation in cellular telephone systems. In Advances in Neural Information Processing Systems: Proceedings of the 1996 Conference, pp. 974-980. MIT Press.
26. Singh, S., Jaakkola, T., & Jordan, M. (1995). Reinforcement learning with soft state aggregation. In Tesauro, G., Touretzky, D., & Leen, T. (Eds.), Advances in Neural Information Processing Systems, Vol. 7. MIT Press, Cambridge, MA.
27. Sutton, R. (1988). Learning to Predict by the Method of Temporal Differences. Machine Learning, 3, 9-44.
28. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press, Cambridge MA. ISBN 0-262-19398-1.
29. Sutton, R. S., McAllester, D., Singh, S., & Mansour, Y. (2000). Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Neural Information Processing Systems 1999. MIT Press.
30. Tao, N., Baxter, J., & Weaver, L. (2001). A multi-agent, policy-gradient approach to network routing. Tech. rep., Australian National University.
31. Tesauro, G. (1992). Practical Issues in Temporal Difference Learning. Machine Learning, 8, 257-278.
32. Tesauro, G. (1994). TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Computation, 6, 215-219.
33. Weaver, L., & Baxter, J. (1999). Reinforcement learning from state and temporal differences. Tech. rep., Australian National University.
34. Williams, R. J. (1992). Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning, 8, 229-256.
35. Zhang, W., & Dietterich, T. (1995). A reinforcement learning approach to job-shop scheduling. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1114-1120. Morgan Kaufmann.