A behavior-based scheme using reinforcement learning for autonomous underwater vehicles

IEEE Journal of Oceanic Engineering

Volumn 30, Issue 2, 2005, Pages 416-427

  Carreras, Marc ^a   Yuh, Junku ^b   Batlle, Joan ^a   Ridao, Pere ^a  

^a UNIVERSITY OF GIRONA   (Spain)

^b NATIONAL SCIENCE FOUNDATION   (United States)

Author keywords

Autonomous underwater vehicle (AUV); Behavior based control; Neural networks; Reinforcement learning

Indexed keywords

DECISION THEORY; LEARNING ALGORITHMS; MARKOV PROCESSES; NEURAL NETWORKS; NONLINEAR CONTROL SYSTEMS; REMOTELY OPERATED VEHICLES; ROBOT LEARNING; ROBUSTNESS (CONTROL SYSTEMS);

BEHAVIOR-BASED CONTROL; BEHAVIOR-BASED SCHEME; HYBRID BEHAVIOR COORDINATION; REINFORCEMENT LEARNING;

OCEAN ENGINEERING;

EID: 27844448118     PISSN: 03649059     EISSN: None     Source Type: Journal    
DOI: 10.1109/JOE.2004.835805     Document Type: Article

References (40)

1. E. Coste-Maniere, H. H. Wang, and A. Peuch, "Control architectures: What's going on?," in Proc. US-Portugal Workshop on Undersea Robotics and Intelligent Control, Lisbon, Portugal, 1995, pp. 54-60.
2. P. Ridao, J. Batlle, J. Amat, and G. N. Roberts, "Recent trends in control architectures for autonomous underwater vehicles," Int. J. Syst. Sci., vol. 30, no. 9, pp. 1033-1056, 1999.
3. K. P. Valavanis, D. Gracanin, M. Matijasevic, R. Kolluru, and G. A. Demetriou, "Control architectures for autonomous underwater vehicles," IEEE Contr. Syst. Mag., vol. 17, no. 6, pp. 48-64, 1997.
4. P. Ridao, J. Yuh, K. Sugihara, and J. Batlle, "On AUV control architecture," in Proc. Int. Conf. Robots and Systems, Japan, 2000.
5. J. G. Bellingham and J. J. Leonard, "Task configuration with layered control," in Proc. IARP 2nd Workshop on Mobile Robots for Subsea Environments, Monterey, CA, 1994, pp. 193-302.
6. A. M. Meystel and J. S. Albus, Intelligent Systems, Architecture, Design and Control. New York: Wiley, 2002.
7. R. Arkin, Behavior-Based Robotics. Cambridge, MA: MIT Press, 1998.
8. R. C. Arkin, "Path planning for a vision-based autonomous robot," in Proc. SPIE Conf. Mobile Robots, Cambridge, MA, 1986, pp. 240-249.
9. E. Gat, "Reliable goal-directed reactive control for real-world autonomous mobile robots," Ph.D., Virginia Polytech. State Univ., Blacksburg, 1991.
10. M. Carreras, J. Batlle, P. Ridao, and G. N. Roberts, "An overview on behavior-based methods for AUV control," in MCMC2000, 5th IFAC Conf. Manoeuvring and Control of Marine Crafts, Aalborg, Denmark, 2000.
11. M. Carreras, J. Batlle, and P. Ridao, "Hybrid coordination of reinforcement learning-based behaviors for AUV control," in IEEE/RSJ Int. Conf. Intelligent Robots and Systems, HI, 2001.
12. S. Mahadevan and J. Connell, "Automatic programming of behavior-based robots using reinforcement learning," Artif. Intell., vol. 55, pp. 311-365, 1992.
13. M. Ryan and M. Pendrith, "RL-TOPs: An architecture for modularity and Re-Use in reinforcement learning," in 15th Int. Conf. Machine Learning, Madison, WI, 1998.
14. J. Shackleton and M. Gini, "Measuring the effectiveness of reinforcement learning for behavior-based robotics," Adapt. Behav., vol. 5, no. 3/4, pp. 365-390, 1997.
15. R. Sutton and A. Barto, Reinforcement Learning, an Introduction. Cambridge, MA: MIT Press, 1998.
16. Y. Takahashi and M. Asada, "Vision-guided behavior acquisition of a mobile robot by multi-layered reinforcement learning," in IEEE/RSJ Int. Conf. Intelligent Robots and Systems, 2000.
17. C. Touzet, "Neural reinforcement learning for behavior synthesis," Robot. Autonomous Syst., vol. 22, pp. 251-281, 1997.
18. D. Gachet, M. Salichs, L. Moreno, and J. Pimental, "Learning emergent tasks for an autonomous mobile robot," in Proc. Int. Conf. Intelligent Robots and Systems, Munich, Germany, 1994, pp. 290-97.
19. Z. Kalmar, C. Szepesvari, and A. Lorincz, "Module-based reinforcement learning: Experiments with a real robot," in Proc. 6th Eur. Workshop on Learning Robots, 1997.
20. P. Maes and R. Brooks, "Learning to coordinate behaviors," in Proc. 8th AAAI, 1990, pp. 796-802.
21. E. Martinson, A. Stoytchev, and R. Arkin, "Robot behavioral selection using Q_learning," in IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Lausanne, Switzerland, 2002.
22. B. Lee and R. C. Arkin, "Adaptive multi-robot behavior via learning momentum," in IEEE/RSJ Int. Conf. Intelligent Robots and Systems, 2003.
23. B. Bakker, F. Linaker, and J. Schmidhuber, "Reinforcement learning in partially observable mobile robot domains using unsupervised event extraction," in IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Lausanne, Switzerland, 2002.
24. W. T. B. Uther and M. M. Veloso, "Tree based discretization for continuous state space reinforcement learning," in Proc. 15th National Conf. Artif. Intell., 1998.
25. J. C. Santamaria, R. S. Sutton, and A. Ram, "Experiments with reinforcement learning in problems with continuous state and action spaces," Adapt. Behav., vol. 6, pp. 163.218-163.218, 1998.
26. W. D. Smart, "Making reinforcement learning work on real robots," Ph.D., Dept. Computer Science, Brown Univ., RI, 2002.
27. C. Gaskett, "Q_learning for robot control," Ph.D., Australian National Univ., 2002.
28. C. J. C. H. Watkins and P. Dayan, Q_Learning Machine Learning, 1992, vol. 8, pp. 279-292.
29. K. Baird, "Residual algorithms: Reinforcement learning with function approximation," in Machine Learning: 12th Int. Conf., San Francisco, CA, 1995.
30. G. J. Tesauro, "Practical issues in temporal difference learning," Machine Learning, vol. 8, no. 3/4, pp. 257.277-257.277, 1992.
31. S. Weaver, L. Baird, and M. Polycarpou, "An analytical framework for local feedforward networks," IEEE Trans. Neural Netw., vol. 9, 1998.
32. M. Carreras, "An overview of behavior-based robotics with simulated implementations on an underwater vehicle," Univ. Girona, Spain. Informatics and Applications Institute, Res. Rep.: IIia 00-14-RR, 2000.
33. R. A. Brooks, "Robust layered control system for a mobile robot," IEEE J. Robot. Automat., vol. RA-2, no. 1, pp. 14-23, 1986.
34. P. Maes, "Situated agents can have goals," Robot. Automat. Syst., vol. 6, pp. 49-70, 1990.
35. R. C. Arkin, "Motor schema-based mobile robot navigation," Int. J. Robotica Res., vol. 8, no. 4, pp. 92-112, 1989.
36. L. Steels, "Building agents with autonomous behavior systems," in The Artificial Route to Artificial Intelligence. Building Situated Embodied Agents. New Haven, CT: Lawrence Erlbaum, 1993.
37. S. K. Choi, J. Yuh, and G. Y. Takashige, "Development of the omni-directional intelligent navigator," IEEE Robot. Automation Mag., pp. 44-53, 1995.
38. J. Nie, J. Yuh, E. Kardash, and T. I. Fossen, "Onboard sensor-based adaptive control of small UUV's in the very shallow water," in Proc. IFAC-Control Applications in Marine Systems, Fukuoka, Japan, 1998, pp. 201-206.
39. M. Carreras, A. Tiano, A. El-Fakdi, A. Zirilli, and P. Ridao, "On the identification of non linear models of unmanncd underwater vehicles," in 1st IFAC Workshop on Guidance and Control of Underwater Vehicles GCUV '03, Wales, U.K., 2003.
40. M. Carreras, P. Ridao, R. Garcia, and T. Nicosevici, "Vision-based localization of an underwater robot in a structured environment," in IEEE Int. Conf. Robotics Automation ICRA'03, Taipei, Taiwan, 2003.