A new optimal stepsize for approximate dynamic programming

IEEE Transactions on Automatic Control

Volumn 60, Issue 3, 2015, Pages 743-758

  Ryzhov, Ilya O ^a   Frazier, Peter I ^b   Powell, Warren B ^c  

^a UNIVERSITY OF MARYLAND   (United States)

^b School of Operations Research and Industrial Engineering   (United States)

^c Princeton University   (United States)

Author keywords

Approximate dynamic programming (ADP); Kalman filter; simulation based optimization; stochastic approximation

Indexed keywords

APPROXIMATION ALGORITHMS; COMPUTER SYSTEMS PROGRAMMING; KALMAN FILTERS; OPERATIONS RESEARCH; OPTIMIZATION; STOCHASTIC SYSTEMS;

APPROXIMATE DYNAMIC PROGRAMMING; FASTER CONVERGENCE; LARGE-SCALE TRANSPORTATION; NUMERICAL EXPERIMENTS; RESEARCH APPLICATIONS; SIMULATION-BASED OPTIMIZATIONS; STOCHASTIC APPROXIMATIONS; VALUE FUNCTION APPROXIMATION;

DYNAMIC PROGRAMMING;

EID: 84923616759     PISSN: 00189286     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAC.2014.2357134     Document Type: Article

References (45)

1. D. Adelman and D. Klabjan, "Computing near-optimal policies in generalized joint replenishment," INFORMS J. Comp., vol. 24, no. 1, pp. 148-164, 2012.
2. M. S. Maxwell, M. Restrepo, S. G. Henderson, and H. Topaloglu, "Approximate dynamic programming for ambulance redeployment,"INFORMS J. Comp., vol. 22, no. 2, pp. 266-281, 2010.
3. M. He, L. Zhao, and W. B. Powell, "Approximate dynamic programming algorithms for optimal dosage decisions in controlled ovarian hyperstimulation,"Eur. J. Oper. Res., vol. 222, no. 2, pp. 328-340, 2012.
4. G. Lai, F. Margot, and N. Secomandi, "An approximate dynamic programming approach to benchmark practice-based heuristics for natural gas storage valuation," Oper. Res., vol. 58, no. 3, pp. 564-582, 2010.
5. N. Löhndorf and S. Minner, "Optimal day-ahead trading and storage of renewable energies - an approximate dynamic programming approach,"Energy Syst., vol. 1, no. 1, pp. 61-77, 2010.
6. N. Secomandi, "Optimal commodity trading with a capacitated storage asset," Manag. Sci., vol. 56, no. 3, pp. 449-467, 2010.
7. D. Zhang and D. Adelman, "An approximate dynamic programming approach to network revenue management with customer choice,"Transportation Sci., vol. 43, no. 3, pp. 381-394, 2009.
8. th IEEE Conf. Decision Control, 1997, vol. 2, pp. 1202-1207.
9. H. P. Simão, J. Day, A. P. George, T. Gifford, J. Nienow, and W. B. Powell, "An approximate dynamic programming algorithm for large-scale fleet management: A case application," Transportation Sci., vol. 43, no. 2, pp. 178-197, 2009.
10. W. B. Powell, A. George, A. Lamont, J. Stewart, and W. R. Scott, "SMART: A stochastic multiscale model for the analysis of energy resources, technology and policy," INFORMS J. Comp., vol. 24, no. 4, pp. 665-682, 2012.
11. R. Howard, Dynamic Probabilistic Systems, Volume II: Semimarkov and Decision Processes. New York, NY, USA: Wiley, 1971.
12. M. L. Puterman, Markov Decision Processes. New York, NY, USA: Wiley, 1994.
13. R. Bellman and S. Dreyfus, "Functional approximations and dynamic programming," Math. Tables Aids Comp., vol. 13, pp. 247-251, 1959.
14. D. Bertsekas and J. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA, USA: Athena Scientific, 1996.
15. R. Sutton and A. Barto, Reinforcement Learning. Cambridge, MA, USA: The MIT Press, 1998.
16. J. Si, A. G. Barto, W. B. Powell, and D. Wunsch, Eds., Handbook of Learning and Approximate Dynamic Programming. New York, NY, USA: IEEE Press, 2004.
17. nd ed.). New York, NY, USA: Wiley, 2011.
18. th IEEE Conf. Decision Control, 1997, vol. 4, pp. 4052-4057.
19. C. Watkins and P. Dayan, "Q-learning," Machine Learning, vol. 8, no. 3, pp. 279-292, 1992.
20. H. Topaloglu and W. B. Powell, "Dynamic programming approximations for stochastic, time-staged integer multicommodity flow problems,"INFORMS J. Comp., vol. 18, no. 1, pp. 31-42, 2006.
21. M. Wasan, Stochastic Approximation. Cambridge, U.K.: Cambridge Univ. Press, 1969.
22. H. J. Kushner and G. G. Yin, Stochastic Approximation Algorithms and Applications. New York, NY, USA: Springer-Verlag, 1997.
23. J. Tsitsiklis, "Asynchronous stochastic approximation and Q-learning,"Machine Learning, vol. 16, pp. 185-202, 1994.
24. T. Jaakkola, M. Jordan, and S. Singh, "Convergence of stochastic iterative dynamic programming algorithms," in Advances in Neural Information Processing Systems, vol. 6, J. Cowan, G. Tesauro, and J. Alspector, Eds. San Francisco, CA, USA: Morgan Kaufmann Publishers, 1994, pp. 703-710.
25. M. G. Azar, R. Munos, M. Ghavamzadeh, and H. J. Kappen, "Speedy Q-learning," Adv. Neural Inform. Processing Syst., vol. 24, pp. 2411-2419, 2011.
26. C. Szepesvári, "The asymptotic convergence-rate of Q-learning," in Advances in Neural Information Processing Systems, vol. 10, M. Jordan, M. Kearns, and S. Solla, Eds. Cambridge, MA, USA: MIT Press, 1997, pp. 1064-1070.
27. E. Even-Dar and Y. Mansour, "Learning rates for Q-learning," J. Machine Learning Res., vol. 5, pp. 1-25, 2003.
28. A. Gosavi, "On step sizes, stochastic shortest paths, survival probabilities in reinforcement learning," in Proc. Winter Simul. Conf., S. J. Mason, R. R. Hill, L. Mönch, O. Rose, T. Jefferson, and J. W. Fowler, Eds., 2008, pp. 525-531.
29. H. P. Simão and W. B. Powell, "Approximate dynamic programming for management of high-value spare parts," J. Manufact. Technol. Manag., vol. 20, no. 2, pp. 147-160, 2009.
30. A. Benveniste, M. Metivier, and P. Priouret, Adaptive Algorithms and Stochastic Approximations. New York, NY, USA: Springer-Verlag, 1990.
31. J. Duchi, E. Hazan, and Y. Singer, "Adaptive subgradient methods for online learning and stochastic optimization," J. Machine Learning Res., vol. 12, pp. 2121-2159, 2011.
32. th Int. Conf. Machine Learning, 2013, pp. 343-351.
33. th Nat. Conf. Artif. Intell., 1992, pp. 171-176.
34. A. R. Mahmood, R. S. Sutton, T. Degris, and P. M. Pilarski, "Tuning-free stepsize adaptation," in Proc. IEEE Int. Conf. Acous., Speech, Signal Processing, 2012, pp. 2121-2124.
35. R. Stengel, Optimal Control and Estimation. New York, NY, USA: Dover Publications, 1994.
36. D. P. Choi and B. Van Roy, "A generalized Kalman filter for fixed point approximation and efficient temporal-difference learning," Discrete Event Dyn. Syst., vol. 16, pp. 207-239, 2006.
37. A. George and W. B. Powell, "Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming," Machine Learning, vol. 65, no. 1, pp. 167-198, 2006.
38. M. Hutter and S. Legg, "Temporal difference updating without a learning rate," in Advances in Neural Information Processing Systems, vol. 20, J. C. Platt, D. Koller, Y. Singer, and S. Roweis, Eds. Cambridge, MA, USA: MIT Press, 2007, pp. 705-712.
39. R. Sutton, C. Szepesvári, and H. Maei, "A convergent O(n) algorithm for off-policy temporal-difference learning with linear function approximation,"Adv. Neural Inform. Processing Syst., vol. 21, pp. 1609-1616, 2008.
40. th Int. Conf. Machine Learning, 2013, pp. 924-932.
41. I. O. Ryzhov, P. I. Frazier, and W. B. Powell, "A new optimal stepsize for approximate dynamic programming," IEEE Trans. Autom. Control. [Online]. Available: http://arxiv.org/abs/1407.2676
42. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. New York, NY, USA: Springer, 2001, ser. Statistics.
43. M. Broadie, D. Cicek, and A. Zeevi, "General bounds and finite-time improvement for the Kiefer-Wolfowitz stochastic approximation algorithm,"Oper. Res., vol. 59, no. 5, pp. 1211-1224, 2011.
44. P. Bickel and K. Doksum, Mathematical Statistics - Basic Ideas and Selected Topics Volume 1. Upper Saddle River, NJ, USA: Prentice Hall, 2001.
45. A. George,W. B. Powell, and S. R. Kulkarni, "Value function approximation using multiple aggregation for multiattribute resource management,"J. Machine Learning Res., vol. 9, pp. 2079-2111, 2008.