An optimal approximate dynamic programming algorithm for concave, scalar storage problems with vector-valued controls

IEEE Transactions on Automatic Control

Volumn 58, Issue 12, 2013, Pages 2995-3010

  Nascimento, Juliana ^a   Powell, Warren B ^b  

^a KIMBERLY CLARK CORPORATION   (United States)

^b Princeton University   (United States)

Author keywords

Approximate dynamic programming; Resource allocation; Storage

Indexed keywords

APPROXIMATE DYNAMIC PROGRAMMING; DYNAMIC PROGRAMS; EXOGENOUS VARIABLES; EXPLORATION POLICY; OPTIMIZING ENERGY; STOCHASTIC CONTROL; TECHNICAL CONDITIONS; VALUE FUNCTION APPROXIMATION;

ENERGY STORAGE; OPTIMIZATION; PIECEWISE LINEAR TECHNIQUES; RESOURCE ALLOCATION; VIRTUAL STORAGE;

ALGORITHMS;

EID: 84889637960     PISSN: 00189286     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAC.2013.2272973     Document Type: Article

References (34)

1. R. K. Ahuja, T. L. Magnanti, and J. Orlin, "Network flows," in Theory, Algorithms, and Applications, R. K. Ahuja, J.B. Orlin, and D. Sharma, Eds. Englewood Cliffs, NJ: Prentice-Hall, 1993, vol. 91, pp. 71-97.
2. A. Antos, C. Szepesvári, and R. Munos, "Value-iteration based fitted policy iteration: Learning with a single trajectory," in Proc. 2007 IEEE Int. Symp. Approximate Dynamic Programming and Reinforcement Learning, 2007, pp. 330-337. (Pubitemid 47431404)
3. A. Antos, C. Szepesvári, and R. Munos, "Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path," Machine Learn., vol. 71, no. 1, pp. 89-129, 2008.
4. A. G. Barto, S. J. Bradtke, and S. P. Singh, "Learning to act using realtime dynamic programming," Artificial Intelligence, Special Volume on Computational Research on Interaction and Agency, vol. 72, pp. 81-138, 1995.
5. D. Bertsekas, A. Nedic, and E. Ozdaglar, Convex Analysis and Optimization. Belmont, MA: Athena Scientific, 2003.
6. D. P. Bertsekas, Dynamic Programming and Stochastic Control Vol. I. Belmont, MA: Athena Scientific, 2005.
7. D. P. Bertsekas, "Approximate policy iteration: A survey and some new methods," J. Control Theory Appl., vol. 9, no. 3, pp. 310-335, 2011.
8. D. P. Bertsekas, Approximate Dynamic Programming, 4th ed. Belmont, MA: Athena Scientific, 2012, vol. II, Dynamic Programming and Optimal Control, ch. 6.
9. D. P. Bertsekas, J. Abounadi, and V. Borkar, "Stochastic approximation for nonexpansive maps: Application to Q-learning algorithms," SIAM J. Control Optim., vol. 41, no. 1, pp. 1-22, 2003.
10. D. Bertsekas and J. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
11. V. S. Borkar and S. P. Meyn, "The O.D.E. method for convergence of stochastic approximation and reinforcement learning," SIAM J. Control Optim., vol. 38, no. 2, p. 447, 2000.
12. Z.-L. Chen and W. B. Powell, "A convergent cutting-plane and partial-sampling algorithm for multistage stochastic linear programs with recourse," J. Optimization Theory and Applications, vol. 102, no. 3, pp. 497-524, 1999.
13. J. Enders, W. B. Powell, and D. M. Egan, "Robust policies for the transformer acquisition and allocation problem," Energy Syst., vol. 1, no. 3, pp. 245-272, 2010.
14. A. George, W. B. Powell, and S. Kulkarni, "Value function approximation using multiple aggregation for multiattribute resource management," J. Mach. Learn. Res., vol. 9, pp. 2079-2111, 2008.
15. G. Godfrey and W. B. Powell, "An adaptive, distribution-free algorithm for the newsvendor problem with censored demands, with applications to inventory and distribution," Manage. Sci., vol. 47, no. 8, pp. 1101-1112, 2001. (Pubitemid 34192808)
16. M. He, L. Zhao, and W. B. Powell, "Optimal control of dosage decisions in controlled ovarian hyperstimulation," Ann. Oper. Res., vol. 178, pp. 223-245, 2010.
17. J. Higle and S. Sen, "Stochastic decomposition: An algorithm for two stage linear programs with recourse," Math. Oper. Res., vol. 16, no. 3, pp. 650-669, 1991.
18. T. Jaakkola, M. Jordan, and S. P. Singh, "On the convergence of stochastic iterative dynamic programming algorithms," Neural Comput., vol. 1201, no. 1988, pp. 1185-1201, 1994.
19. R. Munos and C. Szepesvári, "Finite-time bounds for fitted value iteration," J. Mach. Learn. Res., vol. 1, pp. 815-857, 2008.
20. J. M. Nascimento and W. B. Powell, "An optimal approximate dynamic programming algorithm for the lagged asset acquisition problem," Math. Oper. Res., vol. 34, no. 1, pp. 210-237, 2009.
21. W. B. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality, 2nd ed. Hoboken, NJ: Wiley, 2011.
22. W. B. Powell, A. George, A. Lamont, and J. Stewart, "SMART: A stochastic multiscale model for the analysis of energy resources, technology and policy," INFORMS J. Comput., vol. 24, no. 4, pp. 665-682, Fall 2011.
23. W. B. Powell, A. Ruszczyński, and H. Topaloglu, "Learning algorithms for separable approximations of stochastic optimization problems," Math. Oper. Res., vol. 29, no. 4, pp. 814-836, 2004. (Pubitemid 40055836)
24. A. Shapiro, "Monte Carlo sampling methods," in Handbooks in Operations Research and Management Science: Stochastic Programming, A. Ruszczyński and A. Shapiro, Eds. Amsterdam, The Netherlands: Elsevier, 2003, vol. 10, pp. 353-425.
25. A. Shapiro, D. Dentcheva, and A. Ruszczynski, Lectures on Stochastic Programming: Modeling and Theory. Philadelphia, PA: SIAM, 2009.
26. A. Shiryaev, "Probability theory," in Graduate Texts in Mathematics. New York: Springer-Verlag, 1996, vol. 95.
27. R. Sutton and A. Barto, Reinforcement Learning. Cambridge, MA: The MIT Press, 1998.
28. C. Szepesvári, "Algorithms for reinforcement learning," Synthesis Lectures on Artificial Intelligence and Machine Learning, vol. 4, no. 1, pp. 1-103, 2010.
29. H. Topaloglu and W. B. Powell, "Dynamic programming approximations for stochastic, time-staged integer multicommodity flow problems," INFORMS J. Comput., vol. 18, no. 1, pp. 31-42, 2006. (Pubitemid 43515944)
30. J. N. Tsitsiklis, "Asynchronous stochastic approximation and Q-learning," Machine Learn., vol. 16, pp. 185-202, 1994.
31. J. N. Tsitsiklis and B. Van Roy, "An analysis of temporal-difference learning with function approximation," IEEE Trans. Autom. Control, vol. 42, no. 7, pp. 674-690, May 1997. (Pubitemid 127760263)
32. R. Van Slyke and R. Wets, "L-shaped linear programs with applications to optimal control and stochastic programming," SIAM J. Appl. Math., vol. 17, no. 4, pp. 638-663, 1969.
33. R. Vanderbei, Linear Programming: Foundations and Extensions. New York: Kluwer's International Series, 1996.
34. P. J. Werbos, "Backpropagation and neurocontrol: A review and prospectus," Neural Networks, pp. 209-216, 1989.