Safe exploration of state and action spaces in reinforcement learning

Journal of Artificial Intelligence Research

Volumn 45, Issue , 2012, Pages 515-564

  Garcia, Javier ^a   Fernandez, Fernando ^a  

^a UNIVERSIDAD CARLOS III DE MADRID   (Spain)

Author keywords

[No Author keywords available]

Indexed keywords

ACTION SPACES; BUSINESS MANAGEMENT; COMPLEX TRANSITIONS; CONTINUOUS STATE; EXPLORATION TECHNIQUES; HIGH-DIMENSIONAL; ROBUST BEHAVIOR; TRIAL-AND-ERROR PROCESS;

ARTIFICIAL INTELLIGENCE;

REINFORCEMENT LEARNING;

EID: 84875199879     PISSN: None     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.3761     Document Type: Article

References (62)

1. Aamodt, A., & Plaza, E. (1994). Case-Based Reasoning; Foundational Issues, Methodological Variations, and System Approaches. AI Communications, 7 (1), 39-59.
2. Abbeel, P., Coates, A., Hunter, T., & Ng, A. Y. (2008). Autonomous Autorotation of an RC Helicopter. In ISER, pp. 385-394.
3. Abbeel, P., Coates, A., & Ng, A. Y. (2010). Autonomous helicopter aerobatics through apprenticeship learning. I. J. Robotic Res., 29 (13), 1608-1639.
4. Abbott, R. G. (2008). Robocup 2007: Robot soccer world cup xi.. chap. Behavioral Cloning for Simulator Validation, pp. 329-336. Springer-Verlag, Berlin, Heidelberg.
5. Aha, D. W. (1992). Tolerating Noisy, Irrelevant and Novel Attributes in Instance-Based Learning Algorithms. International Journal Man-Machine Studies, 36 (2), 267-287.
6. Aha, D. W., & Kibler, D. (1991). Instance-based learning algorithms. In Machine Learning, pp. 37-66.
7. Anderson, C. W., Draper, B. A., & Peterson, D. A. (2000). Behavioral cloning of student pilots with modular neural networks. In Proceedings of the Seventeenth International Conference on Machine Learning, pp. 25-32. Morgan Kaufmann.
8. Argall, B., Chernova, S., Veloso, M., & Browning, B. (2009). A Survey of Robot Learning from Demonstration. Robotics and Autonomous Systems, 57 (5), 469-483.
9. Bartsch-Sprl, B., Lenz, M., & Hbner, A. (1999). Case-based reasoning: Survey and future directions.. In Puppe, F. (Ed.), XPS, Vol. 1570 of Lecture Notes in Computer Science, pp. 67-89. Springer.
10. Bianchi, R., Ros, R., & de Mántaras, R. L. (2009). Improving reinforcement learning by using case-based heuristics.. Vol. 5650, pp. 75-89. Lecture Notes in Artificial Intelligence, Springer, Lecture Notes in Artificial Intelligence, Springer.
11. Borrajo, F., Bueno, Y., de Pablo, I., Santos, B. n., Fernandez, F., Garcia, J., & Sagredo, I. (2010). SIMBA: A Simulator for Business Education and Research. Decission Support Systems, 48 (3), 498-506.
12. Boyan, J., Moore, A., & Sutton, R. (1995). Proceedings of the workshop on value function approximation, machine learning conference 1995... Technical Report CMU-CS-95- 206.
13. Chernova, S., & Veloso, M. (2007). Confidence-based policy learning from demonstration using gaussian mixture models. In Joint Conference on Autonomous Agents and Multi-Agent Systems.
14. Chernova, S., & Veloso, M. (2008). Multi-thresholded approach to demonstration selection for interactive robot learning. In Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction, HRI '08, pp. 225-232, New York, NY, USA. ACM.
15. Cichosz, P. (1995). Truncating temporal differences: On the efficient implementation of td(lambda) for reinforcement learning. Journal of Artificial Intelligence Research (JAIR), 2, 287-318.
16. Cichosz, P. (1996). Truncated temporal differences with function approximation: Successful examples using cmac. In Proceedings of the Thirteenth European Symposium on Cybernetics and Systems Research (EMCSR-96).
17. Coraluppi, S. P., & Marcus, S. I. (1999). Risk-Sensitive and Minimax Control of Discrete- Time, Finite-State Markov Decision Processes. AUTOMATICA, 35, 301-309.
18. Defourny, B., Ernst, D., & Wehenkel, L. (2008). Risk-aware decision making and dynamic programming. In NIPS 2008 Workshop on Model Uncertainty and Risk in RL.
19. Driessens, K., & Ramon, J. (2003). Relational instance based regression for relational rl. In International Conference of Machine Learning (ICML), pp. 123-130.
20. Driessens, K., & Dẑeroski, S. (2004). Integrating guidance into relational reinforcement learning. Machine Learning, 57 (3), 271-304.
21. Fernandez, F., & Isasi, P. (2008). Local feature weighting in nearest prototype classification. Neural Networks, IEEE Transactions on, 19 (1), 40-53.
22. Fernandez, F., & Borrajo, D. (2008). Two steps reinforcement learning. International Journal of Intelligent Systems, 23 (2), 213-245.
23. Floyd, M. W., & Esfandiari, B. (2010). Toward a domain-independent case-based reasoning approach for imitation: Three case studies in gaming. In Workshop on Case-Based Reasoning for Computer Games at the 18th International Conference on Case-Based Reasoning (ICCBR), pp. 55-64.
24. Floyd, M. W., Esfandiari, B., & Lam, K. (2008). A Case-Based Reasoning Approach to Imitating Robocup Players. In Proceedings of the 21st International Florida Artificial Intelligence Research Society Conference, pp. 251-256.
25. Forbes, J., & Andre, D. (2002). Representations for learning control policies. In The University of New South, pp. 7-14.
26. Gabel, T., & Riedmiller, M. (2005). Cbr for state value function approximation in reinforcement learning. In Proceedings of the 6th International Conference on Case-Based Reasoning (ICCBR 2005, pp. 206-221. Springer.
27. Geibel, P. (2001). Reinforcement Learning with Bounded Risk. In Proceedings of the 18th International Conference on Machine Learning, pp. 162-169. Morgan Kaufmann.
28. Geibel, P., & Wysotzki, F. (2005). Risk-sensitive Reinforcement Learning Applied to Control under Constraints. Journal of Artificial Intelligence Research (JAIR), 24, 81-108.
29. Hans, A., Schneegass, D., Schäfer, A. M., & Udluft, S. (2008). Safe Exploration for Reinforcement Learning. In European Symposium on Artificial Neural Network, pp. 143-148.
30. Heger, M. (1994). Consideration of Risk in Reinforcement Learning. In 11th International Conference on Machine Learning, pp. 105-111.
31. Hernández-Díaz, A. G., Coello, C. A. C., Perez, F., Caballero, R., Luque, J. M., & Santana- Quintero, L. V. (2008). Seeding the initial population of a multi-objective evolutionary algorithm using gradient-based information. In IEEE Congress on Evolutionary Computation, pp. 1617-1624. IEEE.
32. Hester, T., Quinlan, M., & Stone, P. (2011). A real-time model-based reinforcement learning architecture for robot control. Tech. rep. arXiv e-Prints 1105.1749, arXiv.
33. Hu, H., Kostiadis, K., Hunter, M., & Kalyviotis, N. (2001). Essex wizards 2001 team description. In Birk, A., Coradeschi, S., & Tadokoro, S. (Eds.), RoboCup, Vol. 2377 of Lecture Notes in Computer Science, pp. 511-514. Springer.
34. Jiang, A. X. (2004). Multiagent reinforcement learning in stochastic games with continuous action spaces..
35. Kaelbling, L., Littman, M., & Moore, A. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research (JAIR), 4, 237-285.
36. Konen, W., & Bartz-Beielstein, T. (2009). Reinforcement learning for games: failures and successes. In Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers, GECCO '09, pp. 2641- 2648, New York, NY, USA. ACM.
37. Koppejan, R., & Whiteson, S. (2009). Neuroevolutionary reinforcement learning for generalized helicopter control. In GECCO 2009: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 145-152.
38. Koppejan, R., & Whiteson, S. (2011). Neuroevolutionary reinforcement learning for generalized control of simulated helicopters. Evolutionary Intelligence, 4, 219-241.
39. Lee, J.-Y., & Lee, J.-J. (2008). Multiple Designs of Fuzzy Controllers for Car Parking Using Evolutionary Algorithm, pp. 1-6. No. May.
40. Luenberger, D. G. (1998). Investment science. Oxford University Press.
41. Mannor, S. (2004). Reinforcement learning for average reward zero-sum games. In Shawe- Taylor, J., & Singer, Y. (Eds.), COLT, Vol. 3120 of Lecture Notes in Computer Science, pp. 49-63. Springer.
42. Martin H, J., & de Lope, J. (2009). Exa: An effective algorithm for continuous actions reinforcement learning problems. In Industrial Electronics, 2009. IECON '09. 35th Annual Conference of IEEE, pp. 2063 -2068.
43. Martín H., J. A., & Lope, J. (2009). Learning Autonomous Helicopter Flight with Evolutionary Reinforcement Learning. In 12th International Conference on Computer Aided Systems Theory (EUROCAST), pp. 75-82.
44. Mihatsch, O., & Neuneier, R. (2002). Risk-Sensitive reinforcement learning. Machine Learning, 49 (2-3), 267-290.
45. Moldovan, T. M., & Abbeel, P. (2012). Safe exploration in markov decision processes. CoRR, abs/1205.4810.
46. Narendra, K. S., & Thathachar, M. A. L. (1974). Learning automata - a survey. Ieee Transactions On Systems Man And Cybernetics, SMC-4(4), 323-334.
47. Narendra, K. S., & Thathachar, M. A. L. (1989). Learning automata: an introduction. Prentice-Hall, Inc., Upper Saddle River, NJ, USA.
48. Ng, A. Y., Kim, H. J., Jordan, M. I., & Sastry, S. (2003). Autonomous Helicopter Flight via Reinforcement Learning. In Thrun, S., Saul, L. K., & Scholkopf, B. (Eds.), NIPS. MIT Press.
49. Peters, J., Tedrake, R., Roy, N., & Morimoto, J. (2010). Robot learning. In Sammut, C., & Webb, G. I. (Eds.), Encyclopedia of Machine Learning, pp. 865-869. Springer.
50. Poli, R., & Cagnoni, S. (1997). Genetic programming with user-driven selection: Experiments on the evolution of algorithms for image enhancement. In Genetic Programming 1997: Proceedings of the Second Annual Conference, pp. 269-277. Morgan Kaufmann.
51. Salkham, A., Cunningham, R., Garg, A., & Cahill, V. (2008). A collaborative reinforcement learning approach to urban tra-c control optimization. In Web Intelligence and Intelligent Agent Technology, 2008. WI-IAT '08. IEEE/WIC/ACM International Conference on, Vol. 2, pp. 560-566.
52. Santamaría, J. C., Sutton, R. S., & Ram, A. (1998). Experiments with reinforcement learning in problems with continuous state and action spaces. Adaptive Behavior, 6, 163-218.
53. Sharma, M., Holmes, M., Santamaria, J., Irani, A., Isbell, C., & Ram, A. (2007). Transfer learning in real-time strategy games using hybrid cbr/rl. In In Proceedings of the Twentieth International Joint Conference on Artificial Intelligence.
54. Siebel, N. T., & Sommer, G. (2007). Evolutionary reinforcement learning of artificial neural networks. International Journal of Hybrid Intelligent Systems, 4, 171-183.
55. Smart, W. D., & Kaelbling, L. P. (2000). Practical reinforcement learning in continuous spaces. In Artificial Intelligence, pp. 903-910. Morgan Kaufmann.
56. Smart, W. D., & Kaelbling, L. P. (2002). Effective reinforcement learning for mobile robots. In ICRA, pp. 3404-3410. IEEE.
57. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. The MIT Press.
58. Tang, J., Singh, A., Goehausen, N., & Abbeel, P. (2010). Parameterized maneuver learning for autonomous helicopter ight. In International Conference on Robotics and Automation (ICRA).
59. Taylor, M. E., Kulis, B., & Sha, F. (2011). Metric learning for reinforcement learning agents. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS).
60. Van Hasselt, H., & Wiering, M. A. (2007). Reinforcement Learning in Continuous Action Spaces. In Approximate Dynamic Programming and Reinforcement Learning, 2007. ADPRL 2007. IEEE International Symposium on, pp. 272-279.
61. Wyatt, J. (1997). Exploration and Inference in Learning from Reinforcement. University of Edinburgh.
62. Yao, X. (1999). Evolving artificial neural networks. PIEEE: Proceedings of the IEEE, 87, 1423-1447.