Approximate dynamic programming strategies and their applicability for process control: A review and future directions

International Journal of Control, Automation and Systems

Volumn 2, Issue 3, 2004, Pages 263-278

  Lee, Jong Min ^a   Lee, Jay H ^a  

^a Georgia Institute of Technology   (United States)

Author keywords

Approximate dynamic programming; Function approximation; Neuro dynamic programming; Optimal control; Reinforcement learning

Indexed keywords

APPROXIMATION THEORY; AUTOMATION; FUNCTIONS; MARKOV PROCESSES; OPTIMAL CONTROL SYSTEMS; PROBLEM SOLVING; PROCESS CONTROL; SET THEORY; STRATEGIC PLANNING;

APPROXIMATE DYNAMIC PROGRAMMING; FUNCTION APPROXIMATION; NEURO-DYNAMIC PROGRAMMING (NDP); OPTIMAL CONTROL; REINFORCEMENT LEARNING;

DYNAMIC PROGRAMMING;

EID: 4544319442     PISSN: 15986446     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Review

References (107)

1. T. P. I. Ahamed, P. S. N. Rao, and P. S. Sastry, "A reinforcement learning approach to automatic generation control," Electric Power Systems Research, vol. 63, no. 1, pp. 9-26, 2002.
2. J. S. Albus, "Data storage in the cerebellar model articulation controller," Journal of Dynamic Systems, Measurement and Control, pp. 228-233, 1975.
3. J. S. Albus, "A new approach to manipulator control: The cerebellar model articulation controller (CMAC)," Journal of Dynamic Systems, Measurement and Control, pp. 220-227, 1975.
4. C. W. Anderson, "Learning to control an inverted pendulum using neural networks," IEEE Control Systems Magazine, vol. 9, no. 3, pp. 31-37, 1989.
5. C. W. Anderson, D. C. Hittle, A. D. Katz, and R. M. Kretchmar, "Synthesis of reinforcement learning, neural networks and PI control applied to a simulated heating coil," Artificial Intelligence in Engineering, vol. 11, no. 4, pp. 421-429, 1997.
6. M. Asada, S. Noda, S. Tawaratsumida, and K. Hosoda, "Purposive behavior acquisition for a real robot by vision-based reinforcement learning," Machine Learning, vol. 23, pp. 279-303, 1996.
7. K. J. Åström and A. Helmersson, "Dual control of an integrator with unknown gain," Comp. & Maths, with Appls., vol. 12A, pp. 653-662, 1986.
8. C. G. Atkeson and S. Schaal, "Robot learning from demonstration," Proc. of the Fourteenth International Conference on Machine Learning, pp. 12-20, San Francisco, CA, 1997.
9. L. Baird III, "Residual algorithms: Reinforcement learning with function approximation," Proc. of the International Conference on Machine Learning, pp. 30-37, 1995.
10. A. G. Barto, S. J. Bradtke, and S. P. Singh, "Learning to act using real-time dynamic programming," Artificial Intelligence, vol. 72, no. 1, pp. 81-138, 1995.
11. A. G. Barto, R. S. Sutton, and C. W. Anderson, "Neuronlike adaptive elements that can solve difficult learning control problems," IEEE Trans. on Systems, Man, and Cybernetics, vol. 13, no. 5, pp. 834-846, 1983.
12. R. E. Bellman, Dynamic Programming, Princeton University Press, New Jersey, 1957.
13. D. P. Bertsekas. Dynamic Programming and Optimal Control, Athena Scientic, Belmont, MA, 2nd edition, 2000.
14. D. P. Bertsekas, "Neuro-dynamic programming: An overview," In J. B. Rawlings, B. A. Ogunnaike, and J. W. Eaton, editors, Proc. of Sixth International Conference on Chemical Process Control, 2001.
15. D. P. Bertsekas and D. A. Castanon, "Adaptive aggregation for infinite horizon dynamic programming," IEEE Trans. on Automatic Control, vol. 34, no. 6, pp. 589-598, 1989.
16. D. P. Bertsekas and R. G. Gallager, Data Networks, Prentice-Hall, Englewood Cliffs, NJ, 2nd edition, 1992.
17. D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed Computation: Numerical Methods, Prentice-Hall, Englewood Cliffs, NJ, 1989.
18. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming, Athena Scientic, Belmont, Massachusetts, 1996.
19. V. Borkar, "A convex analytic approach to Markov decision processes," Probability Theory and Related Fields, vol. 78, pp. 583-602, 1988.
20. J. A. Boyan and A. W. Moore, "Generalization in reinforcement learning: safely approximating the value function," In G. Tesauro and D. Touretzky, editors, Advances in Neural Information Processing Systems, vol. 7, Morgan Kaufmann, 1995.
21. S. J. Bradtke, "Reinforcement learning applied to linear quadratic regulation," In S. J. Hanson, J. Cowan, and C. L. Giles, editors, Advances in Neural Information Processing Systems, vol, 5, Morgan Kaufmann, 1993.
22. R. Crites and A. G. Barto, "Improving elevator performance using reinforcement learning," In D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, editors, Advances in Neural Information Processing Systems, vol. 8, MIT Press, San Francisco, CA, 1996.
23. R. Crites and A. G. Barto, "Elevator group control using multiple reinforcement learning agents," Machine Learning, vol. 33, pp. 235-262, 1998.
24. P. Dayan. "The convergence of TD(X) for general λ," Machine Learning, vol. 8, pp. 341-362, 1992.
25. D. P. de Farias and B. Van Roy, "The linear programming approach to approximate dynamic programming," Operations Research, vol. 51, no. 6, pp. 850-865, 2003.
26. E. V. Denardo, "On linear programming in a Markov decision problem," Management Science, vol. 16, pp. 282-288, 1970.
27. C. G. Atkeson and J. Santamaria, "A comparison of direct and model-based reinforcement learning," Proc. of the International Conference on Robotics and Automation, 1997.
28. G. J. Gordon, "Stable function approximation in dynamic programming," Proc. of the Twelfth International Conference on Machine Learning, San Francisco, CA, pp. 261-268, 1995.
29. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer-Verlag, New York, NY, 2001.
30. A. Hordijk and L. C. M. Kallenberg, "Linear programming and Markov decision chains," Management Science, vol. 25, pp. 352-362, 1979.
31. J. C. Hoskins and D. M. Himmelblau, "Process control via artificial neural networks and reinforcement learning," Computers & Chemical Engineering, vol. 16, no. 4, pp. 241-251, 1992.
32. R. A. Howard, Dynamic Programming and Markov Processes, MIT Press, Cambridge, MA, 1960.
33. T. Jaakkola, M. I. Jordan, and S. P. Singh, "On the convergence of stochastic iterative dynamic programming algorithms," Neural Computation, vol. 6, no. 6, pp. 1185-1201, 1994.
34. L. P. Kaelbling, M. L. Littman, and A. W. Moore, "Reinforcement learning: A survey," Journal of Artificial Intelligence Research, vol. 4, pp. 237-285, 1996.
35. N. S. Kaisare, J. M. Lee, and J. H. Lee, "Simulation based strategy for nonlinear optimal control: Application to a microbial cell reactor," International Journal of Robust and Nonlinear Control, vol. 13, no. 3-4, pp. 347-363, 2002.
36. S. Koenig and R. G. Simmons, "Complexity analysis of real-time reinforcement learning," Proc. of the Eleventh National Conference on Artificial Intelligence, Menlo Park, CA, pp. 99-105, 1993.
37. V. R. Konda and J. N. Tsitsiklis, "Actor-critic algorithms," In S. A. Solla, T. K. Leen, and K.-R. Müller, editors, Advances in neural information processing systems, vol. 12, 2000.
38. B. J. A. Kröse and J. W. M. van Dam, "Adaptive state space quantisation for reinforcement learning of collision-free navigation," Proc. of the 1992 IEEE/RSJ International Conference on Intelligent Robots and Systems, Piscataway, NJ, 1992.
39. P. R. Kumar and P. P. Varaiya, Stochastic Systems: Estimation, Identification, and Adaptive Control, Prentice Hall, Englewood Cliffs, NJ, 1986.
40. J. M. Lee, N. S. Kaisare, and J. H. Lee, "Simulation-based dynamic programming strategy for improvement of control policies," AIChE Annual Meeting, San Francisco, CA, paper 43 8c, 2003.
41. J. M. Lee and J. H. Lee, "Neuro-dynamic programming approach to dual control problem," AIChE Annual Meeting, Reno, NV, paper 276e, 2001.
42. J. M. Lee and J. H. Lee, "Approximate dynamic programming based approaches for input-output data-driven control of nonlinear processes," Automatica, 2004. Submitted.
43. J. M. Lee and J. H. Lee, "Simulation-based learning of cost-to-go for control of nonlinear processes," Korean J. Chem. Eng., vol. 21, no. 2, pp. 338-344, 2004.
44. J. A. Leonard, M. A. Kramer, and L. H. Ungar, "A neural network architecture that computes its own reliability," Computers & Chemical Engineering, vol. 16, pp. 819-835, 1992.
45. L.-J. Lin, "Self-improving reactive agents based on reinforcement learning, plannin and teaching." Machine Learning, vol. 8, pp. 293-321, 1992.
46. S. Mahadevan and J. Connell, "Automatic programming of behavior-based robots using rein-forcement learning," Machine Learning, vol. 55, no. 2-3, pp. 311-365, 1992.
47. th International Conference on Machine Learning, pp. 202-210, 1997.
48. A. S. Manne, "Linear programming and sequential decisions," Management Science, vol. 6, no. 3, pp. 259-267, 1960.
49. P. Marbach and J. N. Tsitsiklis, "Simulation-based optimization of Markov reward processes," IEEE Trans. on Automatic Control, vol. 46, no. 2, pp. 191-209, 2001.
50. E. C. Martinez, "Batch process modeling for optimization using reinforcement learning," Computers & Chemical Engineering, vol. 24, pp. 1187-1193, 2000.
51. S. Miller and R. J. Williams, "Temporal difference learning: A chemical process control application," In A. F. Murray, editor, Applications of Artificial Neural Networks, Kluwer, Norwell, MA, 1995.
52. A. Moore and C. Atkeson, "The parti-game algorithm for variable resolution reinforcement learning in multidimensional state spaces. Machine Learning, vol. 21, no. 3, pp. 199-233, 1995.
53. A. W. Moore, Efficient Memory Based Robot Learning, PhD thesis, Cambridge University, October 1991.
54. A. W. Moore and C. G. Atkeson, "Prioritized sweeping: Reinforcement learning with less data and less time," Machine Learning, vol. 13, pp. 103-130, 1993.
55. M. Morari and J. H. Lee, "Model predictive control: Past, present and future," Computers & Chemical Engineering, vol. 23, pp. 667-682, 1999.
56. R. Munos, "A convergent reinforcement learning algorithm in the continuous case based on a finite difference method," Proc. of the International Joint Conference on Artificial Intelligence, 1997.
57. R. Munos, "A study of reinforcement learning in the continuous case by means of viscosity solutions," Machine Learning Journal, vol. 40, pp. 265-299, 2000.
58. R. Neuneier, "Enhancing Q-leaming for optimal asset allocation," In M. Jordan, M. Kearns, and S. Solla, editors, Advances in Neural Information Processing Systems, vol. 10, 1997.
59. D. Ormoneit and P. W. Glynn, "Kernel-based reinforcement learning in average-cost problems," IEEE Trans. on Automatic Control, vol. 47, no. 10, pp. 1624-1636, 2002.
60. D. Ormoneit and S. Sen, "Kernel-based reinforcement learning," Machine Learning, vol. 49, pp. 161-178, 2002.
61. E. Parzen, "On estimation of a probability density function and mode," Ann. Math. Statist., vol. 33, pp. 1065-1076, 1962.
62. J. Peng, Efficient Dynamic Programming-Based Learning for Control, PhD thesis, North-eastern University, Boston, MA, 1993.
63. J. Peng and R. J. Williams, "Efficient learning and planning within the Dyna framework," Adaptive Behavior, vol. 1, no. 4. pp. 437-454, 1993.
64. D. V. Prokhorov and D. C. Wunsch II, "Adaptive critic designs," IEEE Trans. on Neural Networks, vol. 8, no. 5, pp. 997-1007, September 1997.
65. M. L. Puterman, Markov Decision Processes, Wiley, New York, NY, 1994.
66. S. J. Qin and T. A. Badgwell, "A survey of industrial model predictive control technology," Control Engineering Practice, vol. 11, no. 7, pp. 733-764, 2003.
67. U. Rüde, Mathematical and Computational Techniques for Multilevel Adaptive Methods, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1993.
68. G. A. Rummery and M. Niranjan, On-line Qlearning using connectionist systems, Technical Report CUED/F-INFENG/TR 166, Engineering Department, Cambridge University, 1994.
69. P. Sabes, "Approximating Q-values with basis function representations," Proc. of the Fourth Connectionist Models Summer School, Hillsdale, NJ, 1993.
70. A. L. Samuel, "Some studies in machine learning using the game of checkers," IBM J. Res. Develop., pp. 210-229, 1959.
71. A. L. Samuel, "Some studies in machine learning using the game of checkers II - recent progress," IBM J. Res. Develop., pp. 601-617, 1967.
72. J. C. Santamaria, R. S. Sutton, and A. Ram, "Experiments with reinforcement learning in problems with continuous state and action spaces," Adaptive Behavior, vol. 6, no. 2, pp. 163-217, 1997.
73. S. Schaal, "Learning from demonstration," In M. C. Mozer, M. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems, vol. 9, pp. 1040-1046, 1997.
74. S. Schaal and C. Atkeson, "Robot juggling: An implementation of memory-based learning," IEEE Control Systems, vol. 14, no. 1, pp. 57-71, 1994.
75. N. N. Schraudolph, P. Dayan, and T. J. Sejnowski, "Temporal difference learning of position evaluation in the game of Go," In J. D. Cowan, G. Tesauro, and J. Alspector, editors, Advances in Neural Information Processing Systems, vol. 6, pp. 817-824, 1994.
76. S. Singh and D. Bertsekas, "Reinforcement learning for dynamic channel allocation in cellular telephone systems," In M. C. Mozer, M. I. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems, vol. 9, pp. 974-980, 1997.
77. S. P. Singh and R. S. Sutton, "Reinforcement learning with replacing eligibility traces," Machine Learning, vol. 22, pp. 123-158, 1996.
78. W. D. Smart and L. P. Kaelbling, "Practical reinforcement learning in continuous spaces," Proc. 17th International Conf. on Machine Learning, pp. 903-910, 2000.
79. R. Sutton, D. McAllester, S. Singh, and Y. Mansour, "Policy gradient methods for reinforce-ment learning with function approximation," In S. A. Solla, T. K. Leen, and K.-R. Muller, editors, Advances in Neural Information Processing Systems, vol. 12, pp. 1057-1063, 2000.
80. R. S. Sutton, Temporal Credit Assignment in Reinforcement Learning, PhD thesis, University of Massachusetts, Amherst, MA, 1984.
81. R. S. Sutton, "Learning to predict by the method of temporal differences," Machine Learning, vol. 3.no. 1, pp. 9-44, 1988.
82. R. S. Sutton, "Integrated architectures for learning, planning, and reacting based on approximating dynamic programming," Proc. of the Seventh International Conference on Machine Learning, Austin, TX, 1990.
83. R. S. Sutton, "Planning by incremental dynamic programming," Proc. of the Eighth International Workshop on Machine Learning, pp. 353-357, 1991.
84. R. S. Sutton, "Generalization in reinforcement learning: Successful examples using sparse coarse coding," In D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, editors, Advances in Neural Information Processing Systems, vol. 8, pp. 1038-1044, 1996.
85. R. S. Sutton and A. G. Barto, "Toward a modern theory of adaptive networks: Expectation and prediction," Psycol. Rev., vol. 88, no. 2, pp. 135-170, 1981.
86. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA, 1998.
87. M. Takeda, T. Nakamura, M. Imai, T. Ogasawara, and M. Asada, "Enhanced continuous valued Q-learning for real autonomous robots," Advanced Robotics, vol. 14, no. 5, pp. 439-442, 2000.
88. G. Tesauro, "Practical issues in temporal difference learning," Machine Learning, vol. 8, pp. 257-277, 1992.
89. G. Tesauro, "TD-Gammon, a self-teaching backgammon program, achieves master-level play," Neural Computation, vol. 6, no. 2, pp. 215-219, 1994.
90. G. Tesauro, "Temporal difference learning and TD-Gammon," Communications of the ACM, vol. 38, no. 3, pp. 58-67, 1995.
91. S. Thrun, "Learning to play the game of chess," In G. Tesauro, D. S. Touretzky, and T. K. Leen, editors, Advances in Neural Information Processing Systems, vol. 7, 1995.
92. S. Thrun and A. Schwartz, "Issues in using function approximation for reinforcement learning," Proc. of the Fourth Connectionist Models Summer School, Hillsdale, NJ, 1993.
93. J. N. Tsitsiklis, "Asynchronous stochastic approximation and Q-learning," Machine Learning, vol. 16, pp. 185-202, 1994.
94. J. N. Tsitsiklis and B. Van Roy, "An analysis of temporal-difference learning with function approximation," IEEE Trans. on Automatic Control, vol. 42, no. 5, pp. 674-690, 1997.
95. B. Van Roy, "Neuro-dynamic programming: Overview and recent trends," In E. Feinberg and A. Shwartz, editors, Handbook of Markov Decision Processes: Methods and Applications, Kluwer, Boston, MA, 2001.
96. C. J. C. H. Watkins, Learning from Delayed Rewards, PhD thesis, University of Cambridge, England, 1989.
97. C. J. C. H. Watkins and P. Dayan, "Q-leaming," Machine Learning, vol. 8, pp. 279-292, 1992.
98. P. J. Werbos, "Advanced forecasting methods for global crisis warning and models of intelligence'," General Systems Yearbook, vol. 22, pp. 25-38, 1977.
99. P. J. Werbos, "Approximate dynamic programming for real-time control and neural modeling," In D. A. White and D. A. Sofge, editors, Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches, Van Nostrand Reinhold, New York, pp. 493-525, 1992.
100. S. D. Whitehead, "Complexity and cooperation in Q-leaming," Proc. of the Eighth International Workshop on Machine Learning, Evanston, IL, 1991.
101. R. J. Williams and L. C. Baird III, "Analysis of some incremental variants of policy iteration: First steps toward understanding actor-critic learning systems," Technical Report NU-CCS-93-14, Northeastern University, College of Computer Science, Boston, MA, 1993.
102. J. A. Wilson and E. C. Martinez, "Neuro-fuzzy modeling and control of a batch process involving simultaneous reaction and distillation," Computers & Chemical Engineering, vol. 21S, pp. S1233-S1238, 1997.
103. M. Wonham, "Stochastic control problems," In B. Friedland, editor, Stochastic Problems in Control, ASME, New York, 1968.
104. W. Zhang, Reinforcement Learning for Job-Shop Scheduling, PhD thesis, Oregon State University, 1996. Also available as Technical Report CS-96-30-1.
105. W. Zhang, Reinforcement Learning for Job-Shop Scheduling, PhD thesis, Oregon State University, 1996. Also available as Technical Report CS-96-30-1.
106. W. Zhang and T. G. Dietterich, "A reinforcement learning approach to job-shop scheduling," Proc. of the Twelfth International Conference on Machine Learning, San Francisco, CA, pp. 1114-1120, 1995.
107. W. Zhang and T. G. Dietterich, "Highperformance job-shop scheduling with a time-delay TD(λ) network," In D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, editors, Advances in Neural Information Processing Systems, vol. 8, 1996.