F-Discrepancy for Efficient Sampling in Approximate Dynamic Programming

IEEE Transactions on Cybernetics

Volumn 46, Issue 7, 2016, Pages 1628-1639

  Cervellera, Cristiano ^a   MacCiò, Danilo ^a  

^a INSTITUTE OF INTELLIGENT SYSTEMS FOR AUTOMATION   (Italy)

Author keywords

Approximate dynamic programming (ADP); F discrepancy; Markovian decision problem (MDP); state sampling; value function approximation

Indexed keywords

ALGORITHMS; IMPORTANCE SAMPLING; PROBABILITY DISTRIBUTIONS;

APPROXIMATE DYNAMIC PROGRAMMING; APPROXIMATE SOLUTION; CONTINUOUS STATE; EFFICIENT SAMPLING; MARKOVIAN DECISION PROBLEMS; MARKOVIAN PROCESS; STATE TRAJECTORY; UNIFORM SAMPLING;

DYNAMIC PROGRAMMING;

EID: 84938514337     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2015.2453123     Document Type: Article

References (39)

1. D. Bertsekas, Dynamic Programming and Optimal Control, vol. 1, 2nd ed. Belmont, CA, USA: Athena Scientific, 2000.
2. W. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality, 2nd ed. Hoboken, NJ, USA: Wiley, 2011.
3. M. Puterman, Markov Decision Processes. New York, NY, USA: Wiley, 1994.
4. F. L. Lewis and K. G. Vamvoudakis, "Reinforcement learning for partially observable dynamic processes: Adaptive dynamic programming using measured output data," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 41, no. 1, pp. 14-25, Feb. 2011.
5. T. Mori and S. Ishii, "Incremental state aggregation for value function estimation in reinforcement learning," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 41, no. 5, pp. 1407-1416, Oct. 2011.
6. K. Senda, S. Hattori, T. Hishinuma, and T. Kohda, "Acceleration of reinforcement learning by policy evaluation using nonstationary iterative method," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2696-2705, Dec. 2014.
7. X. Xu, Z. Huang, D. Graves, and W. Pedrycz, "A clustering-based graph Laplacian framework for value function approximation in reinforcement learning," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2613-2625, Dec. 2014.
8. M. L. Koga, V. Freire, and A. H. R. Costa, "Stochastic abstract policies: Generalizing knowledge to improve reinforcement learning," IEEE Trans. Cybern., vol. 45, no. 1, pp. 77-88, Jan. 2015.
9. A. Heydari, "Revisiting approximate dynamic programming and its convergence," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2733-2743, Dec. 2014.
10. M. Palanisamy, H. Modares, F. L. Lewis, and M. Aurangzeb, "Continuous-time Q-learning for infinite-horizon discounted cost linear quadratic regulator problems," IEEE Trans. Cybern., vol. 45, no. 2, pp. 165-176, Feb. 2015.
11. Q. Wei, F.-Y. Wang, D. Liu, and X. Yang, "Finite-approximation-errorbased discrete-time iterative adaptive dynamic programming," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2820-2833, Dec. 2014.
12. B. Kiumarsi, F. L. Lewis, M.-B. Naghibi-Sistani, and A. Karimpour, "Optimal tracking control of unknown discrete-time linear systems using input-output measured data," IEEE Trans. Cybern., DOI: 10.1109/TCYB.2014.2384016.
13. A. Heydari and S. N. Balakrishnan, "Finite-horizon control-constrained nonlinear optimal control using single network adaptive critics," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 1, pp. 145-157, Jan. 2013.
14. K. G. Vamvoudakis, F. L. Lewis, and G. R. Hudas, "Multi-agent differential graphical games: Online adaptive learning solution for synchronization with optimality," Automatica, vol. 48, no. 8, pp. 1598-1611, 2012.
15. H. Zhang, C. Qin, and Y. Luo, "Neural-network-based constrained optimal control scheme for discrete-time switched nonlinear system using dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 3, pp. 839-849, Jul. 2014.
16. Q. Wei and D. Liu, "Adaptive dynamic programming for optimal tracking control of unknown nonlinear systems with application to coal gasification," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 4, pp. 1020-1036, Oct. 2014.
17. R. Bellman, R. Kalaba, and B. Kotkin, "Polynomial approximation-A new computational technique in dynamic programming allocation processes," Math. Comput., vol. 17, no. 82, pp. 155-161, 1963.
18. S. A. Johnson, J. Stedinger, C. A. Shoemaker, Y. Li, and J. A. Tejada-Guibert, "Numerical solution of continuous-state dynamic programs using linear and spline interpolation," Oper. Res., vol. 41, no. 3, pp. 484-500, 1993.
19. V. Chen, D. Ruppert, and C. A. Shoemaker, "Applying experimental design and regression splines to high-dimensional continuous-state stochastic dynamic programming," Oper. Res., vol. 47, no. 1, pp. 38-53, 1999.
20. C. Cervellera, M. Gaggero, and D. Macciò, "Low-discrepancy sampling for approximate dynamic programming with local approximators," Comput. Oper. Res., vol. 43, pp. 108-115, Mar. 2014.
21. H. Zhang, J. Zhang, G.-H. Yang, and Y. Luo, "Leader-based optimal coordination control for the consensus problem of multiagent differential games via fuzzy adaptive dynamic programming," IEEE Trans. Fuzzy Syst., vol. 23, no. 1, pp. 152-163, Feb. 2015.
22. D. Bertsekas and J. Tsitsiklis, Neuro-Dynamic Programming. Belmont, CA, USA: Athena Scientific, 1996.
23. M. Baglietto, C. Cervellera, M. Sanguineti, and R. Zoppoli, "Management of water resources systems in the presence of uncertainties by nonlinear approximators and deterministic sampling techniques," Comput. Optim. Appl., vol. 47, no. 2, pp. 349-376, 2010.
24. V. C. P. Chen, "Measuring the goodness of orthogonal array discretizations for stochastic programming and stochastic dynamic programming," SIAM J. Optim., vol. 12, no. 2, pp. 322-344, 2001.
25. C. Cervellera, V. Chen, and A. Wen, "Neural network and regression spline value function approximations for stochastic dynamic programming," Comput. Oper. Res., vol. 34, no. 1, pp. 70-90, 2006.
26. C. Cervellera and D. Macciò, "A comparison of global and semi-local approximation in T-stage stochastic optimization," Eur. J. Oper. Res., vol. 208, no. 2, pp. 109-118, 2011.
27. C. Cervellera and M. Muselli, "Efficient sampling in approximate dynamic programming algorithms," Comput. Optim. Appl., vol. 38, no. 3, pp. 417-443, 2007.
28. H. Fan, P. K. Tarun, and V. Chen, "Adaptive value function approximation for continuous-state stochastic dynamic programming," Comput. Oper. Res., vol. 40, no. 4, pp. 1076-1084, 2013.
29. J. Hartinger and R. Kainhofer, "Non-uniform low-discrepancy sequence generation and integration of singular integrands," in Monte Carlo and Quasi-Monte Carlo Methods 2004, H. Niederreiter and D. Talay, Eds. Berlin, Germany: Springer, 2006, pp. 163-179.
30. C. Cervellera, M. Gaggero, and D. Macciò, "An analysis based on F-discrepancy for sampling in regression tree learning," in Proc. Int. Joint Conf. Neural Netw. (IJCNN), Jul. 2014, Beijing, China, pp. 1115-1121.
31. K. T. Fang and Y. Wang, Number-Theoretic Methods in Statistics. London, U.K.: Chapman and Hall, 1994.
32. H. Niederreiter, Random Number Generation and Quasi-Monte Carlo Methods. Philadelphia, PA, USA: SIAM, 1992.
33. J. Dick, F. Y. Kuo, and I. H. Sloan, "High dimensional integration-The quasi-Monte Carlo way," Acta Numer., vol. 22, pp. 133-288, May 2013.
34. S. Heinrich, E. Novak, G. Wasilkowski, and H. Wózniakowski, "The inverse of the star-discrepancy depends linearly on the dimension," Acta Arith., vol. 96, no. 3, pp. 279-302, 2001.
35. M. Gnewuch, M. Wahlström, and C. Winzen, "A new randomized algorithm to approximate the star discrepancy based on threshold accepting," SIAM J. Numer. Anal., vol. 50, no. 2, pp. 781-807, 2012.
36. A. Barron, "Universal approximation bounds for superpositions of a sigmoidal function," IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 930-945, May 1993.
37. F. Girosi, M. Jones, and T. Poggio, "Regularization theory and neural networks architectures," Neural Comput., vol. 7, no. 2, pp. 219-269, 1995.
38. L. K. Jones, "A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training," Ann. Stat., vol. 20, no. 1, pp. 608-613, 1992.
39. M. Hagan and M. Menhaj, "Training feedforward networks with the Marquardt algorithm," IEEE Trans. Neural Netw., vol. 5, no. 6, pp. 989-993, Nov. 1994.