Reinforcement learning in feedback control : Challenges and benchmarks from technical process control

Machine Learning

Volumn 84, Issue 1-2, 2011, Pages 137-169

  Hafner, Roland ^a   Riedmiller, Martin ^a  

^a UNIVERSITY OF FREIBURG   (Germany)

Author keywords

Benchmarks; Feedback control; Nonlinear control; Reinforcement learning

Indexed keywords

BENCH-MARK PROBLEMS; BENCHMARKS; CLASSICAL CONTROL; CLASSICAL CONTROLLERS; CONTROL QUALITY; HIGH QUALITY; LEARNING APPROACH; LEARNING BEHAVIOR; LEARNING CONTROLLERS; LEARNING EFFORTS; LEARNING SCHEMES; LONG TERM DYNAMICS; NON LINEAR CONTROL; PERFORMANCE MEASURE; REINFORCEMENT LEARNING METHOD; SETPOINTS; TECHNICAL PROCESS;

BEHAVIORAL RESEARCH; BENCHMARKING; CONTROLLERS; FEEDBACK CONTROL; REINFORCEMENT LEARNING;

PROCESS CONTROL;

EID: 79958779459     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-011-5235-x     Document Type: Article

References (49)

1. Anderson, C., & Miller, W. (1990). Challenging control problems. In Neural networks for control (pp. 475-410).
2. C. W. Anderson D. Hittle A. Katz R. M. Kretchmar 1997 Synthesis of reinforcement learning, neural networks, and pi control applied to a simulated heating coil Journal of Artificial Intelligence in Engineering 11 4 423 431
3. Bellman, R. (1957). Dynamic programming. Princeton: Princeton Univ Press.
4. Boyan, J., & Littman, M. (1994). Packet routing in dynamically changing networks-a reinforcement learning approach. In J. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems 6.
5. Crites, R. H., & Barto, A. G. (1996). Improving elevator performance using reinforcement learning. In: Andvances in neural information processing systems 8.
6. CTM (1996). Digital Control Tutorial. University of Michigan, www.engin.umich.edu/group/ctm (online).
7. M. Deisenroth C. Rasmussen J. Peters 2009 Gaussian process dynamic programming Neurocomputing 72 7-9 1508 1524 10.1016/j.neucom.2008.12.019
8. Dullerud, G. P. F. (2000). A course in robust control theory: A convex approach. New York: Springer.
9. El-Fakdi, A., & Carreras, M. (2008). Policy gradient based reinforcement learning for real autonomous underwater cable tracking. In International conference on intelligent robots and systems, 2008. IROS 2008. IEEE/RSJ (pp. 3635-3640).
10. Farrel, J. A., & Polycarpou, M. M. (2006). Adaptive approximation based control. New York: Wiley Interscience.
11. Gabel, T., & Riedmiller, M. (2008). Adaptive reactive job-shop scheduling with reinforcement learning agents. International Journal of Information Technology and Intellifent Computing, 24(4).
12. Goodwin, G. C., & Payne, R. L. (1977). Dynamic system identification: experiment design and data analysis. New York: Academic Press.
13. Hafner, R. (2009). Dateneffiziente selbstlernende neuronale Regler. PhD thesis, University of Osnabrueck.
14. Hafner, R., & Riedmiller, M. (2007). Neural reinforcement learning controllers for a real robot application. In Proceedings of the IEEE international conference on robotics and automation (ICRA 07), Rome, Italy.
15. Jordan, M. I., & Jacobs, R. A. (1990). Learning to control an unstable system with forward modeling. In D. Touretzky (Ed.), Advances in neural information processing systems (NIPS) 2 (pp. 324-331). San Mateo: Morgan Kaufmann.
16. J. Kaloust C. Ham Z. Qu 1997 Nonlinear autopilot control design for a 2-dof helicopter model IEE Proceedings. Control Theory and Applications 144 6 612 616 0900.93226 10.1049/ip-cta:19971638 (Pubitemid 127754346)
17. Kretchmar, R. M. (2000). A synthesis of reinforcement learning and robust control theory. PhD thesis, Colorado State University, Fort Collins, CO.
18. Krishnakumar, K., & Gundy-burlet, K. (2001). Intelligent control approaches for aircraft applications (Technical report). National Aeronautics and Space Administration, Ames Research.
19. C. Kwan F. Lewis Y. Kim 1999 Robust neural network control of rigid link flexible-joint robots Asian Journal of Control 1 3 188 197 10.1111/j.1934-6093. 1999.tb00019.x
20. D. Liu H. Javaherian O. Kovalenko T. Huang 2008 Adaptive critic learning techniques for engine torque and air-fuel ratio control IEEE Transactions on Systems, Man and Cybernetics. Part B. Cybernetics 38 4 988 993 10.1109/TSMCB.2008.922019
21. Ljung, L. (1999). System identification theory for the user (2nd ed.). Upper Saddle River: PTR Prentice Hall.
22. J. J. Martinez O. Sename A. Voda 2009 Modeling and robust control of blu-ray disc servo-mechanisms Mechatronics 19 5 715 725 10.1016/j.mechatronics. 2009.02.006
23. Nelles, O. (2001). Nonlinear system identification. Berlin: Springer.
24. Ng, A. Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., Berger, E., & Liang, E. (2004). Inverted autonomous helicopter flight via reinforcement learning. In International symposium on experimental robotics.
25. Peters, J., & Schaal, S. (2006). Policy gradient methods for robotics. In Proceedings of the IEEE international conference on intelligent robotics systems (Iros 2006).
26. D. Prokhorov D. Wunsch 1997 Adaptive critic designs IEEE Transactions on Neural Networks 8 997 1007 10.1109/72.623201 (Pubitemid 127763331)
27. Riedmiller, M. (2005). Neural fitted q iteration-first experiences with a data efficient neural reinforcement learning method. In Proc. of the European conference on machine learning, ECML 2005, Porto, Portugal.
28. Riedmiller, M., & Braun, H. (1993). A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In H. Ruspini (Ed.), Proceedings of the IEEE international conference on neural networks (ICNN), San Francisco (pp. 586-591).
29. Riedmiller, M., Hafner, R., Lange, S., & Timmer, S. (2006). Clsquare-software framework for closed loop control. Available at http://ml.informatik.uni-freiburg.de/research/clsquare.
30. Riedmiller, M., Montemerlo, M., & Dahlkamp, H. (2007a). Learning to drive in 20 minutes. In Proceedings of the FBIT 2007 conference, Jeju, Korea. Berlin: Springer. Best Paper Award.
31. Riedmiller, M., Peters, J., & Schaal, S. (2007b). Evaluation of policy gradient methods and variants on the cart-pole benchmark. In Proceedings of the IEEE international symposium on approximate dynamic programming and reinforcement learning (ADPRL 07), Honolulu, USA.
32. M. Riedmiller T. Gabel R. Hafner S. Lange 2009 Reinforcement learning for robot soccer Autonomous Robots 27 1 55 74 10.1007/s10514-009-9120-4
33. Schiffmann, W., Joost, M., & Werner, R. (1993). Comparison of optimized backpropagation algorithms. In Proc. of ESANN'93, Brussels (pp. 97-104).
34. J. Sjöberg Q. Zhang L. Ljung A. Benveniste B. Deylon Y. P. Glorennec H. Hjalmarsson A. Juditsky 1995 Nonlinear black-box modeling in system identification: a unified overview Automatica 31 1691 1724 0846.93018 10.1016/0005-1098(95)00120-8
35. Slotine, J. E., & Li, W. (1991). Applied nonlinear control. New York: Prentice Hall.
36. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (adaptive computation and machine learning). Cambridge: MIT Press.
37. Szepesvari, C. (2009). Successful application of rl. Available at http://www.ualberta.ca/szepesva/RESEARCH/RLApplications.html.
38. B. Tanner A. White 2009 RL-Glue: language-independent software for reinforcement-learning experiments Journal of Machine Learning Research 10 2133 2136
39. G. Tesauro 1992 Practical issues in temporal difference learning Machine Learning 8 257 277 0772.68075
40. Tesauro, G., Chess, D. M., Walsh, W. E., Das, R., Segal, A., Whalley, I., Kephart, J. O., & White, S. R. (2004). A multi-agent systems approach to autonomic computing. In AAMAS '04: Proceedings of the third international joint conference on autonomous agents and multiagent systems (pp. 464-471). Washington: IEEE Computer Society.
41. D. M. Underwood R. R. Crawford 1991 Dynamic nonlinear modeling of a hot-water-to-air heat exchanger for control applications ASHRAE Transactions 97 1 149 155 (Pubitemid 21725993)
42. Y. Wang J. Si 2001 On-line learning control by association and reinforcement IEEE Transactions on Neural Networks 12 2 264 276 1859321 10.1109/72.914523 (Pubitemid 32371483)
43. Watkins, C. J. (1989). Learning from delayed rewards. PhD thesis, Cambridge University.
44. C. J. Watkins P. Dayan 1992 Q-learning Machine Learning 8 3 279 292 0773.68062
45. S. Whiteson B. Tanner A. White 2010 The reinforcement learning competitions The AI Magazine 31 2 81 94
46. Yang, Z.-J., & Minashima, M. (2001). Robust nonlinear control of a feedback linearizable voltage-controlled magnetic levitation system. Transactions of the Institute of Electrical Engeneers of Japan, 1203-1211.
47. Z.-J. Yang M. Tateishi 2001 Adaptive robust nonlinear control of a magnetic levitation system Automatica 37 7 1125 1131 0979.93076 10.1016/S0005-1098(01)00063-2 (Pubitemid 32498998)
48. Z.-J. Yang H. Tsubakihara S. Kanae K. Wada 2007 Robust nonlinear control of a voltage-controlled magnetic levitation system using disturbance observer Transactions of IEE of Japan 127-C 12 2118 2125
49. Z.-J. Yang K. Kunitoshi S. Kanae K. Wada 2008 Adaptive robust output feedback control of a magnetic levitation system by k-filter approach IEEE Transactions on Industrial Electronics 55 1 390 399 10.1109/TIE.2007.896488