Empirical studies in action selection with reinforcement learning

Adaptive Behavior

Volumn 15, Issue 1, 2007, Pages 33-50

  Whiteson, Shimon ^a   Taylor, Matthew E ^a   Stone, Peter ^a  

^a University of Texas at Austin   (United States)

Author keywords

Autonomic computing; Evolutionary computation; Neural networks; Reinforcement learning; Robot soccer; Temporal difference methods

Indexed keywords

EID: 33847264400     PISSN: 10597123     EISSN: 17412633     Source Type: Journal    
DOI: 10.1177/1059712306076253     Document Type: Article

References (60)

1. Ackley, D., & Littman, M. (1991). Interactions between learning and evolution. Artificial Life II, SFI Studies in the Sciences of Complexity, 10, 487-509.
2. Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002). Finite-time analysis of the multi-armed bandit problem. Machine Learning, 47, 235-256.
3. Baird, L., & Moore, A. (1999). Gradient descent for general reinforcement learning. In M. S. Kearns, S. A. Solla, & D. A. Cohn (Eds.), Advances in neural information processing systems 11. Cambridge, MA: MIT Press.
4. Baldwin, J. M. (1896). A new factor in evolution. The American Naturalist, 30, 441-451.
5. Beielstein, T., & Markon, S. (2002). Threshold selection, hypothesis tests and DOE methods. In Proceedings of the 2002 World Congress on Evolutionary Computation (pp. 777-782). Honolulu, HI.
6. Bellman, R. E. (1956). A problem in the sequential design of experiments. Sankhya, 16, 221-229.
7. Boers, E., Borst, M., & Sprinkhuizen-Kuyper, I. (1995). Evolving artificial neural networks using the "Baldwin Effect". Technical Report TR 95-14.
8. Boyan, J. A., & Moore, A. W. (1995). Generalization in reinforcement learning: Safely approximating the value function. In G. Tesauro, D. S. Touretzky, & T. K. Leen (Eds.), Advances in neural information processing systems 7. Cambridge, MA: MIT Press.
9. Crites, R. H., & Barto, A. G. (1998). Elevator group control using multiple reinforcement learning agents. Machine Learning, 33, 235-262.
10. Downing, K. L. (2001). Reinforced genetic programming. Genetic Programming and Evolvable Machines, 2, 259-288.
11. Giraud-Carrier, C. (2000). Unifying learning with evolution through Baldwinian evolution and Lamarckism: A case study. In Proceedings of the Symposium on Computational Intelligence and Learning (CoIL-2000) (pp. 36-41). Chios, Greece.
12. Gomez, F., & Miikkulainen, R. (2002). Robust nonlinear control through neuroevolution. Technical Report AI02-292.
13. Gomez, F., & Schmidhuber, J. (2005). Co-evolving recurrent neurons learn deep memory POMDPs. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2005) (pp. 491-498). Washington, DC.
14. Hinton, G. E., & Nowlan, S. J. (1987). How learning can guide evolution. Complex Systems, 1, 495-502.
15. Holland, J. H. (1975). Adaptation in natural and artificial systems: An introductory analysis with applications to biology, control and artificial intelligence. Ann Arbor, MI: University of Michigan Press.
16. Hsu, W. H., & Gustafson, S. M. (2002). Genetic programming and multi-agent layered learning by reinforcements. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2002) (pp. 764-771). New York, NY.
17. Kaelbling, L. P. (1993). Learning in embedded systems. Cambridge, MA: MIT Press.
18. Kephart, J. O., & Chess, D. M. (2003). The vision of autonomic computing. Computer, 36, 41-50.
19. Kohl, N., & Stone, P. (2004). Machine learning for fast quadru-pedal locomotion. In Proceedings of the 19th National Conference on Artificial Intelligence (pp. 611-616). San Jose, CA.
20. Konda, V. R., & Tsitsiklis, J. N. (1999). Actor-critic algorithms. In M. S. Kearns, S. A. Solla, & D. A. Cohn (Eds.), Advances in neural information processing systems 11 (pp. 1008-1014). Cambridge, MA: MIT Press.
21. Kostiadis, K., & Hu, H. (2001). KaBaGe-RL: Kanerva-based generalisation and reinforcement learning for possession football. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2001). Maui, HI.
22. Kretchmar, R. M., & Anderson, C. W. (1997). Comparison of CMACs and radial basis functions for local function approximators in reinforcement learning. In Proceedings of the International Conference on Neural Networks. Houston, TX.
23. Lin, L.-J. (1992). Self-improving reactive agents based on reinforcement learning, planning, and teaching. Machine Learning, 8, 293-321.
24. Macready, W. G., & Wolpert, D. H. (1998). Bandit problems and the exploration/exploitation tradeoff. IEEE Transactions on Evolutionary Computation, 2, 2-22.
25. Mahadevan, S. (2005). Samuel meets Amarel: Automating value function approximation using global state space analysis. In Proceedings of the 20th National Conference on Artificial Intelligence. Pittsburgh, PA.
26. McQuesten, P., & Miikkulainen, R. (1997). Culling and teaching in neuro-evolution. In T. Bäck (Ed.), Proceedings of the 7th International Conference on Genetic Algorithms (pp. 760-767). East Lansing, MI.
27. Moriarty, D. E., & Miikkulainen, R. (1996). Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22, 11-32.
28. Moriarty, D. E., Schultz, A. C., & Grefenstette, J. J. (1999). Evolutionary algorithms for reinforcement learning. Journal of Artificial Intelligence Research, 11, 241-276.
29. Ng, A. Y., & Jordan, M. I. (2000). PEGASUS: A policy search method for large MDPs and POMDPs. In Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence (pp. 406-415). San Mateo, CA: Morgan Kaufmann Publishers.
30. Nolfi, S., Elman, J. L., & Parisi, D. (1994). Learning and evolution in neural networks. Adaptive Behavior, 2, 5-28.
31. Pollack, J. B., Blair, A. D., & Land, M. (1997). Coevolution of a backgammon player. In C. G. Langton & K. Shimohara (Eds.), Artificial Life V: Proceedings of the 5th International Workshop on the Synthesis and Simulation of Living Systems (pp. 92-98). Cambridge, MA: MIT Press.
32. Powell, M. J. D. (1987). Radial basis functions for multivariate interpolation: A review. In J. C. Mason & M. G. Cox (Eds.), Algorithms for Approximation (pp. 143-167). Oxford: Clarendon Press.
33. Prescott, T. J., Bryson, J. J., & Seth, A. K. (in press). Modelling natural action selection: An introduction to the theme issue. Philosophical Transactions of the Royal Society B: Biological Sciences.
34. Pyeatt, L. D., & Howe, A. E. (2001). Decision tree function approximation in reinforcement learning. In Proceedings of the 3rd International Symposium on Adaptive Systems: Evolutionary Computation and Probabilistic Graphical Models (pp. 70-77). Havana, Cuba.
35. Reidmiller, M. (2005). Neural fitted Q iteration - first experiences with a data efficient neural reinforcement learning method. In Proceedings of the 16th European Conference on Machine Learning (pp. 317-328). Porto, Portugal.
36. Rivest, F., & Precup, D. (2003). Combining TD-learning with cascade-correlation networks. In Proceedings of the 20th International Conference on Machine Learning (pp. 632-639). Washington, DC. Menlo Park, CA: AAAI Press.
37. Rummery, G. A., & Niranjan, M. (1994). On-line Q-learning using connectionist systems. Technical report CUED/FINFENG-RT 116, Engineering Department, Cambridge University.
38. Runarsson, T. P., & Lucas, S. M. (2005). Co-evolution versus self-play temporal difference learning for acquiring position evaluation in small-board go. IEEE Transactions on Evolutionary Computation, 9, 628-640.
39. Sherstov, A. A., & Stone, P. (2005). Function approximation via tile coding: Automating parameter choice. In J.-D. Zucker & I. Saitta (Eds.), Proceedings of the Symposium on Abstraction, Reformulation and Approximation (SARA 2005), Lecture Notes in Artificial Intelligence (Vol. 3607, pp. 194-205). Berlin: Springer-Verlag.
40. Singh, S. P., & Sutton, R. S. (1996). Reinforcement learning with replacing eligibility traces. Machine Learning, 22, 123-158.
41. Smart, W. D., & Kaelbling, L. P. (2000). Practical reinforcement learning in continuous spaces. In Proceedings of the 17th International Conference on Machine Learning (pp. 903-910). Stanford University, CA.
42. Stagge, P. (1998). Averaging efficiently in the presence of noise. In Parallel problem solving from nature (Vol. 5, pp. 188-197). Amsterdam, the Netherlands.
43. Stanley, K. O., Bryant, B. D., & Miikkulainen, R. (2003). Evolving adaptive neural networks with and without adaptive synapses. In Proceedings of the 2003 Congress on Evolutionary Computation (CEC 2003) (Vol. 4, pp. 2557-2564). Canberra, Australia.
44. Stanley, K. O., & Miikkulainen, R. (2002). Evolving neural networks through augmenting topologies. Evolutionary Computation, 10, 99-127.
45. Stanley, K. O., & Miikkulainen, R. (2004a). Competitive coevolution through evolutionary complexification. Journal of Artificial Intelligence Research, 21, 63-100.
46. Stanley, K. O., & Miikkulainen, R. (2004b). Evolving a roving eye for go. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2004). Seattle, WA.
47. Stone, P., Sutton, R. S., & Kuhlmann, G. (2005). Reinforcement learning for RoboCup-soccer keepaway. Adaptive Behavior, 13, 165-188.
48. Stone, P., Kuhlmann, G., Taylor, M. E., & Liu, Y. (2006). Keepaway soccer: From machine learning testbed to benchmark. In I. Noda, A. Jacoff, A. Bredenfeld, & Y. Takahashi (Eds.), RoboCup-2005: Robot Soccer World Cup IX (Vol. 4020, pp. 93-105). Berlin: Springer-Verlag.
49. Sutton, R. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9-44.
50. Sutton, R., McAllester, D., Singh, S., & Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. In S. A. Solla, T. K. Leen, & K.-R. Muller (Eds.), Advances in neural information processing systems (Vol. 12, pp. 1057-1063). Cambridge, MA: MIT Press.
51. Sutton, R. S. (1996). Generalization in reinforcement learning: Successful examples using sparse coarse coding. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems 8 (pp. 1038-1044). Cambridge, MA: MIT Press.
52. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
53. Taylor, M. E., Whiteson, S., & Stone, P. (2006). Comparing evolutionary and temporal difference methods in a reinforcement learning domain. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2006) (pp. 1321-1328). Seattle, WA.
54. Walsh, W. E., Tesauro, G., Kephart, J. O., & Das, R. (2004). Utility functions in autonomic systems. In Proceedings of the International Conference on Autonomic Computing (pp. 70-77). New York, NY.
55. Watkins, C. (1989). Learning from Delayed Rewards. Ph.D. Thesis, King's College, Cambridge.
56. Whiteson, S., & Stone, P. (2006a). Evolutionary function approximation for reinforcement learning. Journal of Machine Learning Research, 7, 877-917.
57. Whiteson, S., & Stone, P. (2006b). Sample-efficient evolutionary function approximation for reinforcement learning. In Proceedings of the 21st National Conference on Artificial Intelligence (AAAI 2006) (pp. 518-523). Boston, MA.
58. Whiteson, S., Kohl, N., Miikkulainen, R., & Stone, P. (2005). Evolving keepaway soccer players through task decomposition. Machine Learning, 59, 5-30.
59. Whitley, D., Dominic, S., Das, R., & Anderson, C. W. (1993). Genetic reinforcement learning for neurocontrol problems. Machine Learning, 13, 259-284.
60. Yao, X. (1999). Evolving artificial neural networks. Proceedings of the IEEE, 87, 1423-1447.