Learning and optimization from a system theoretic perspective

Handbook of Learning and Approximate Dynamic Programming

Volumn , Issue , 2004, Pages 311-335

  Cao, Xi Ren ^a  

^a HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

Algorithm design and analysis; Estimation; Markov processes; Optimization; Sensitivity; Steady state; Stochastic systems

Indexed keywords

ESTIMATION; MARKOV PROCESSES; OPERATIONS RESEARCH; OPTIMIZATION; REINFORCEMENT LEARNING; STOCHASTIC CONTROL SYSTEMS; STOCHASTIC SYSTEMS;

ALGORITHM DESIGN AND ANALYSIS; MARKOV DECISION PROCESSES; PERFORMANCE POTENTIALS; PERFORMANCE SENSITIVITY; PERTURBATION ANALYSIS; PERTURBATION REALIZATION FACTORS; SENSITIVITY; STEADY STATE;

COMPUTER CONTROL SYSTEMS;

EID: 84986024692     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1109/9780470544785.ch12     Document Type: Chapter

References (49)

1. A. Arapostathis, V. S. Borkar, E. Femandez-Gaucherand, M. K. Ghosh, and S. I. Marcus, Discrete-time controlled Markov processes with average cost criterion: A survey, SIAM Journal Control and Optimization, vol. 31, pp. 282-344, 1993.
2. J. Baxter and P. L. Bartlett, Infinite-horizon policy-gradient estimation, Journal of Artificial Intelligence Research, vol. 15, pp. 319-350, 2001.
3. J. Baxter, P. L. Bartlett, and L. Weaver, Experiments with infinite-horizon policy-gradient estimation, Journal of Artificial Intelligence Research, vol. 15, pp. 351-381,2001.
4. D. P. Bertsekas, Dynamic Programming and Optimal Control, Volume I and II, Athena Scientific, Belmont, MA, 1995.
5. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming, Athena Scientific, Belmont, MA, 1996.
6. L. Breiman, Probability, Addison-Wesley, Reading, MA, 1968. Springer-Verlag, New York, 1994.
7. P. Bremaud, Markov Chains: Gibbs Fields, Monte Carlo Simulation, and Queues, Springer-Verlag, New York, 1998.
8. X. R. Cao, Convergence of parameter sensitivity estimates in a stochastic experiment, IEEE Trans. Automatic Control, vol. AC-30, pp. 834-843,1985.
9. X. R. Cao, Sensitivity estimates based on one realization of a stochastic system, Journal of Statistical Computation and Simulation, vol. 27, pp. 211-232,1987.
10. X. R. Cao, Realization Probabilities: The Dynamics of Queueing Systems, Springer-Verlag, New York, 1994.
11. X. R. Cao, X. M. Yuan, and L. Qiu, A single sample path-based performance sensitivity formula for Markov chains, IEEE Trans. Automatic Control, vol. 41, pp. 1814-1817,1996.
12. X. R. Cao, The relation among potentials, perturbation analysis, Markov decision processes, and other topics, Journal of Discrete Event Dynamic Systems, vol. 8, pp. 71-87,1998.
13. X. R. Cao, Single sample path based optimization of Markov chains, Journal of Optimization: Theory and Application, vol. 100, no. 3, pp. 527-548,1999.
14. X. R. Cao, A basic formula for on-line policy-gradient algorithms, submitted, 2004
15. X. R. Cao and H. F. Chen, Perturbation realization, potentials and sensitivity analysis of Markov processes, IEEE Trans. Automatic Control, vol. 42, pp. 1382-1393, 1997.
16. X. R. Cao, A unified approach to Markov decision problems and performance sensitivity analysis, Automatica, vol. 36, pp. 771-774, 2000.
17. X. R. Cao, Constructing performance sensitivities for Markov systems with potentials as building blocks, Proc. Of the 42nd IEEE Conference on Decision and Control, Maui, Hawaii, 2003.
18. X. R. Cao and H. T. Fang, Gradient-based policy iteration: An example, Proc. Of2002 IEEE Conference on Decision and Control, pp. 3367-3371, 2002.
19. X. R. Cao and X. P. Guo, A unified approach to Markov decision problems and sensitivity analysis with discounted and average criteria: The multichain case, submitted.
20. X. R. Cao and Y. W. Wan, Algorithms for sensitivity analysis of Markov systems through potentials and perturbation realization, IEEE Trans. Control System Tech, vol. 6, pp. 482-494,1998.
21. C. G. Cassandras and S. Lafortune, Introduction to Discrete Event Systems, Kluwer Academic Publishers, Boston, 1999.
22. E. K. P. Chong and P. J. Ramadge, Stochastic optimization of regenerative systems using infinitesimal perturbation analysis, IEEE Trans. Automatic Control, vol. 39, pp. 1400-1410,1994.
23. E. Cinlar, Introduction to Stochastic Processes, Prentice-Hall, Englewood Cliffs, NJ, 1975.
24. H.-T. Fang and X. R. Cao, Potential-based on-line policy iteration algorithms for Markov decision processes, IEEE Trans. Automatic Control, vol. 49, no. 4, pp. 493-505,2004.
25. P. W. Glynn, Likelihood ratio gradient estimation: An overview, in A. Thesen, H. Grant, and K. D. Kelton, (eds.), Proc. Of1987 Winter Simulation Conference, pp. 366-375, Society for Computer Simulation, San Diego, CA, 1988.
26. P. W. Glynn, Optimization of stochastic systems via simulation, in A. Thesen, H. Grant, and K. D. Kelton, (eds.), Proc. Of1987 Winter Simulation Conference, pp. 90-105, Society for Computer Simulation, San Diego, CA, 1988.
27. Y. C. Ho and X. R. Cao, Perturbation analysis and optimization of queueing networks, Journal of Optimization Theory and Applications, vol. 40, no. 4, pp. 559-582, 1983.
28. Y. C. Ho and X. R. Cao, Perturbation Analysis of Discrete-Event Dynamic Systems, Kluwer Academic Publisher, Boston, 1991.
29. L. H. Lee, E. T. K. Lau and Y. C. Ho, Explanation of goal softening in ordinal optimization, IEEE Trans. Automatic Control, vol. 44, pp. 94-99,1999.
30. T. Jaakkola, S. P. Singh, and M. I. Jordan, Reinforcement learning algorithm for partially observable Markov decision problems, Advances in Neural Information Processing Systems, vol. 7, pp. 345-352, Morgan Kaufman, San Francisco, CA, 1995.
31. L. Kleinrock, Queueing Systems, Vol. 1: Theory, Wiley, New York, 1975.
32. V. R. Konda and V. S. Borkar, Actor-critic-type learning algorithms for Markov decision processes, SIAM Journal of Control Optimization, vol. 38, pp. 94-123, 1999.
33. P. Marbach and J. N. Tsitsiklis, Simulation-based optimization of Markov reward processes, IEEE Trans. Automatic Control, vol. 46, pp. 191-209, 2001.
34. P. Marbach and J. N. Tsitsiklis, Approximate gradient methods in policy-space optimization of Markov reward processes, Journal of Discrete Event Dynamic Systems,1, vol. 13, no. 1, pp. 111-148, 2003.
35. S. P. Meyn and R. L. Tweedie, Markov Chains and Stochastic Stability, SpringerVerlag, London, 1993.
36. E. L. Plambeck, B. R. Fu, S. M. Robinson, and R. Suri, Sample-path optimization of convex stochastic performance functions, Math. Program. B, vol. 75, pp. 137-176, 1996.
37. M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, New York, 1994.
38. M. I. Reiman and A. Weiss, Sensitivity analysis via likelihood ratio, Operations Research, vol. 37, pp. 830-844,1989.
39. H. Robbins and S. Monro, A stochastic approximation method, Annals of Mathematical Statistics, vol. 22, pp. 400-407,1951.
40. R. V. Rubinstein, Monte Carlo Optimization, Simulation, and Sensitivity Analysis of Queueing Networks, Wiley, New York, 1986.
41. R. Sun and Y. T. Leung, Single run optimization of discrete event simulations-An empirical study using the M/M/l queue, HE Trans., vol. 21, pp. 35-49, 1989.
42. R. S. Sutton, Learning to predict by the methods of temporal differences, Machine Learning, vol. 3, pp. 835-846,1988.
43. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA, 1998.
44. J. N. Tsitsiklis and V. R. Konda, Actor-critic algorithms, Tech. Rep., Lab. Inform. Decision Systems, MIT, Cambridge, MA, 2001, Preprint.
45. J. N. Tsitsiklis and B. Van Roy, Feature-based methods for large-scale dynamic programming, Machine Learning, vol. 22, pp. 59-94,1994.
46. J. N. Tsitsiklis and B. Van Roy, An analysis of temporal-difference learning with function approximation, IEEE Trans. Automatic Control, vol. 42, pp. 674-690, 1997.
47. J. N. Tsitsiklis and B. Van Roy, Average cost temporal-difference learning, Automatica, vol. 35, pp. 1799-1808,1999.
48. C. Watkins, Learning from Delayed Rewards, Ph.D. Thesis, Cambridge University, Cambridge, UK, 1989.
49. C. Watkins and P. Dayan, Q-leaming, Machine Learning, vol. 8, pp. 279-292, 1992.