Error estimates for approximate optimization by the extended ritz method

SIAM Journal on Optimization

Volumn 15, Issue 2, 2005, Pages 461-487

  Kůrková, Věra ^a   Sanguineti, Marcello ^b  

^a INSTITUTE OF COMPUTER SCIENCE   (Czech Republic)

^b UNIVERSITY OF GENOA   (Italy)

Author keywords

(Extended) ritz method; Convex best approximation problems; Curse of dimensionality; Functional optimization; Learning from data by kernel methods; Rates of convergence of suboptimal solutions

Indexed keywords

CONVERGENCE OF NUMERICAL METHODS; ERROR ANALYSIS; FUNCTIONS; MATHEMATICAL TECHNIQUES; OPTIMIZATION; PROBLEM SOLVING;

CONVEX BEST APPROXIMATION PROBLEMS; CURSE OF DIMENSIONALITY; FUNCTIONAL OPTIMIZATION; LEARNING FROM DATA KERNEL METHODS; RATES OF CONVERGENCE OF SUBOPTIMAL SOLUTIONS; RITZ METHOD;

APPROXIMATION THEORY;

EID: 18744392546     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/S1052623403426507     Document Type: Article

References (82)

1. E. AARTS AND J. KORST, Simulated Annealing and Boltzmann Machines: A Stochastic Approach to Combinatorial Optimization and Neural Computing, John Wiley and Sons, New York, 1989.
2. R. A. ADAMS, Sobolev Spaces, Academic Press, New York, 1975.
3. A. ALESSANDRI, M. BAGLIETTO, T. PARISINI, AND R. ZOPPOLI, A neural state estimator with bounded errors for nonlinear systems, IEEE Trans. Automat. Control, 44 (1999), pp. 2028-2042.
4. A. ALESSANDRI, M. SANGUINETI, AND M. MAGGIORE, Optimization-based learning with bounded error for feedforward neural networks, IEEE Trans. Neural Networks, 13 (2002), pp. 261-273.
5. A. ALESSANDRI AND M. SANGUINETI, Optimization of approximating networks for optimal fault diagnosis, Optim. Methods Softw., 20 (2005), pp. 235-260.
6. W. ALT, On the approximation of infinite optimization problems with an application to optimal control problems, Appl. Math. Optim., 12 (1984), pp. 15-27.
7. N. ARONSZAJN, Theory of reproducing kernels, Trans. Amer. Math. Soc., 68 (1950), pp. 337-404.
8. M. BAGLIETTO, T. PARISINI, AND R. ZOPPOLI, Numerical solutions to the Witsenhausen counterexample by approximating networks, IEEE Trans. Automat. Control, 46 (2001), pp. 1471-1477.
9. M. BAGLIETTO, T. PARISINI, AND R. ZOPPOLI, Distributed-information neural control: The case of dynamic routing in traffic networks, IEEE Trans. Neural Networks, 12 (2001), pp. 485-502.
10. M. BAGLIETTO, M. SANGUINETI, AND R. ZOPPOLI, Facing the curse of dimensionality by the extended Ritz method in stochastic functional optimization: Dynamic routing in traffic networks, in High Performance Algorithms and Software for Nonlinear Optimization, G. Di Pillo and A. Murli, eds., Kluwer Academic Publishers, Norwell, MA, 2003, pp. 23-56.
11. A. R. BARRON, Neural net approximation, in Proceedings of the 7th Yale Workshop on Adaptive and Learning Systems, New Haven, CT, 1992, pp. 69-72.
12. A. R. BARRON, Universal approximation bounds for superpositions of a sigmoidal function, IEEE Trans. Inform. Theory, 39 (1993), pp. 930-945.
13. R. W. BEARD AND T. W. MCLAIN, Successive Galerkin approximation algorithms for nonlinear optimal and robust control, Internat. J. Control, 71 (1998), pp. 717-743.
14. R. BELLMAN, Dynamic Programming, Princeton University Press, Princeton, NJ, 1957.
15. M. BERTERO, Linear inverse and ill-posed problems, Adv. Electron. Electron. Phys., 75 (1989), pp. 1-120.
16. D.P. BERTSEKAS, A new class of incremental gradient methods for least squares problems, SIAM J. Optim., 7 (1997), pp. 913-926.
17. D. P. BERTSEKAS, Nonlinear Programming, Athena Scientific, Belmont, MA, 1999.
18. D. P. BERTSEKAS AND J. N. TSITSIKLIS, Neuro-Dynamic Programming, Athena Scientific, Belmont, MA, 1996.
19. W. E. BOSARGE, JR., O. G. JOHNSON, R. S. MCKNIGHT, AND W. P. TIMLAKE, The Ritz-Galerkin procedure for nonlinear control problems, SIAM J. Numer. Anal., 10 (1973), pp. 94-111.
20. D. BRAESS, Nonlinear Approximation Theory, Springer, Berlin, 1986.
21. L. BREIMAN, Hinging hyperplanes for regression, classification, and function approximation, IEEE Trans. Inform. Theory, 39 (1993), pp. 999-1013.
22. H. BREZIS, Analyse Fonctionnelle - Théorie et Applications, Masson, Paris, 1983.
23. F. C. CHEN AND H. KHALIL, Adaptive control of a class of nonlinear discrete-time systems using multilayer neural networks, IEEE Trans. Automat. Control, 40 (1995), pp. 791-801.
24. V. C. P. CHEN, D. RUPPERT, AND C. A. SHOEMAKER, Applying experimental design and regression splines to high-dimensional continuous-state stochastic dynamic programming, Oper. Res., 47 (1999), pp. 38-53.
25. F. CUCKER AND S. SMALE, On the mathematical foundations of learning, Bull. Amer. Math. Soc. (N.S.), 39 (2001), pp. 1-49.
26. J. W. DANIEL, The Approximate Minimization of Functionals, Prentice-Hall, Englewood Cliffs, NJ, 1971.
27. J. W. DANIEL, The Ritz-Galerkin method for abstract optimal control problems, SIAM J. Control, 11 (1973), pp. 53-63.
28. C. DARKEN, M. DONAHUE, L. GURVITS, AND E. SONTAG, Rate of approximation results motivated by robust neural network learning, in Proceedings of the Sixth Annual ACM Conference on Computational Learning Theory, The Association for Computing Machinery, New York, 1993, pp. 303-309.
29. F. DEUTSCH, Best Approximation in Inner Product Spaces, Springer, New York, 2001
30. R. A. DEVORE, R. HOWARD, AND C. MICCHELLI, Optimal nonlinear approximation, Manuscripta Math., 63 (1989), pp. 469-478.
31. R. A. DEVORE AND G. G. LORENTZ, Constructive Approximation, Grundlehren Math. Wiss. 303, Springer, Berlin, 1993.
32. R. A. DEVORE AND V. N. TEMLYAKOV, Nonlinear approximation by trigonometric sums, J. Fourier Anal. Appl., 2 (1995), pp. 29-48.
33. A. L. DONTCHEV, Perturbations, Approximations and Sensitivity Analysis of Optimal Control Systems, Lecture Notes in Control and Inform. Sci. 52, Springer, Berlin, Heidelberg, 1983.
34. A. L. DONTCHEV AND T. ZOLEZZI, Well-Posed Optimization Problems, Lecture Notes in Math. 1543, Springer, Berlin, Heidelberg, 1993.
35. I. EKELAND AND R. TEMAM, Convex Analysis and Variational Problems, North-Holland, Amsterdam, Oxford, American Elsevier, New York, 1976.
36. U. FELGENHAUER, On Ritz type discretizations for optimal control problems, in Proceedings of the 18th IFIP-ICZ Conference, Res. Notes in Math. 386, Chapman and Hall, London, 1999, pp. 91-99.
37. I. M. GELFAND AND S. V. FOMIN, Calculus of Variations, Prentice-Hall, Englewood Cliffs, NJ, 1963.
38. F. GIROSI, M. JONES, AND T. POGGIO, Regularization theory and neural networks architectures, Neural Computation, 7 (1995), pp. 219-269.
39. D. E. GOLDBERG, Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley, Reading, MA, 1989.
40. L. GRIPPO, Convergent on-line algorithms for supervised learning in neural networks, IEEE Trans. Neural Networks, 11 (2000), pp. 1284-1299.
41. W. W. HAGER, The Ritz-Trefftz method for state and control constrained optimal control problems, SIAM J. Numer. Anal., 12 (1975), pp. 854-867.
42. E. HEWIT AND K. STROMBERG, Abstract Analysis, Springer, Berlin, 1965.
43. R. B. HOLMES, Approximating best approximations, Nieuw Archief voor Wiskunde, XIV(3), (1966), pp. 106-113.
44. R. B. HOLMES, A Course on Optimization and Best Approximation, Lecture Notes in Math. 257, Springer, Berlin, 1972.
45. L. K. JONES, A simple lemma on greedy approximation in Hubert space and convergence rates for projection pursuit regression and neural network training, Ann. Statist., 20 (1992), pp. 608-613.
46. S. A. JOHNSON, J. R. STEDINGER, C. SHOEMAKER, Y. Li, AND J. TEJADA-GUIBERT, Numerical solution of continuous-state dynamic programs using linear and spline interpolation, Oper. Res., 41 (1993), pp. 484-500.
47. A. JUDITSKY, H. HJALMARSSON, A. BENVENISTE, B. DELYON, L. LJUNG, J. SJÖBERG, AND Q. ZHANG, Nonlinear black-box models in system identification: Mathematical foundations, Automatica, 31 (1995), pp. 1725-1750.
48. P. C. KAINEN, V. KŮRKOVÁ, AND M. SANGUINETI, Minimization of error functional over variable-basis functions, SIAM J. Optim., 14 (2003), pp. 732-742.
49. P. C. KAINEN, V. KŮRKOVÁ, AND A. VOGT, An integral formula for Heaviside neural networks, Neural Network World, 10 (2000), pp. 313-319.
50. P. C. KAINEN, V. KŮRKOVá, AND A. VOGT, Upper Bounds on Variation with Respect to Half-Spaces, Research Report ICS-2004-900, Institute of Computer Science, Prague, 2004.
51. V. KŮRKOVÁ, Dimension-independent rates of approximation by neural networks, in Computer-Intensive Methods in Control and Signal Processing. The Curse of Dimensionality, K. Warwick and M. Kárný, eds., Birkhäuser, Boston, 1997, pp. 261-270.
52. V. KŮRKOVá, Neural networks as universal approximators, in The Handbook of Brain Theory and Neural Networks, M. Arbib, ed., MIT Press, Cambridge, MA, 2002, pp. 1180-1183.
53. V. KŮRKOVÁ, High-dimensional approximation and optimization by neural networks, in Advances in Learning Theory: Methods, Models, and Applications, J. Suykens, G. Horvath, S. Basu, C. Micchelli, and J. Vandewalle, eds., IOS Press, Amsterdam, 2003, pp. 69-88.
54. V. KŮRKOVÁ, P. C. KAINEN, AND V. KREINOVICH, Estimates of the number of hidden units and variation with respect to half-spaces, Neural Networks, 10 (1997), pp. 1061-1068.
55. V. KŮRKOVÁ AND M. SANGUINETI, Bounds on rates of variable-basis and neural-network approximation, IEEE Trans. Inform. Theory, 47 (2001), pp. 2659-2665.
56. V. KŮRKOVÁ AND M. SANGUINETI, Comparison of worst case errors in linear and neural network approximation, IEEE Trans. Inform. Theory, 48 (2002), pp. 264-275.
57. V. KŮRKOVÁ AND M. SANGUINETI, Learning with generalization capability by kernel methods of bounded complexity, J. Complexity, to appear.
58. V. KŮRKOVÁ P. SAVICKÝ, AND K. HLAVÁČ KOVÁ, Representations and rates of approximation of real-valued Boolean functions by neural networks, Neural Networks, 11 (1998), pp. 651-659.
59. E. S. LEVITIN AND B. T. POLYAK, Convergence of minimizing sequences in conditional extremum problems, Dokl. Akad. Nauk SSSR, 168 (1966), pp. 764-767.
60. A. LINNEMANN, Convergent Ritz approximations of the set of stabilizing controllers, Systems Control Lett., 36 (1998), pp. 151-156.
61. K. S. NARENDRA AND S. MUKHOPADHYAY, Adaptive control using neural networks and approximate models, IEEE Trans. Neural Networks, 8 (1997), pp. 475-485.
62. K. S. NARENDRA AND K. PARTHASARATHI, Identification and control of dynamical systems using neural networks, IEEE Trans. Neural Networks, 4 (1990), pp. 4-26.
63. J. M. ORTEGA, Numerical Analysis: A Second Course, Classics Appl. Math. 3, SIAM, Philadelphia, 1990.
64. T. PARISINI, M. SANGUINETI, AND R. ZOPPOLI, Nonlinear stabilization by receding-horizon neural regulators, Internat. J. Control, 70 (1998), pp. 341-362.
65. T. PARISINI AND R. ZOPPOLI, Neural networks for feedback feedforward nonlinear control systems, IEEE Trans. Neural Networks, 5 (1994), pp. 436-449.
66. T. PARISINI AND R. ZOPPOLI, Neural approximations for multistage optimal control of nonlinear stochastic systems, IEEE Trans. Automat. Control, 41 (1996), pp. 889-895.
67. A. PINKUS, n- Widths in Approximation Theory, Springer, Berlin, Heidelberg, 1985.
68. G. PISIER, Remarques sur un résultat non publié de B. Maurey, Séminaire d'Analyse Fonctionnelle 1980-81, Exposé no. V, École Polytechnique, Centre de Mathémaiques, Palaiseau, France, pp. V.1-V.12.
69. T. POGGIO AND S. SMALE, The mathematics of learning: Dealing with data, Notices Amer. Math. Soc., 50 (2003), pp. 536-544.
70. B. T. POLAK, Existence theorems and convergence of minimizing sequences in extremum prob lems with restrictions, Dokl. Akad. Nauk SSSR, 166 (1966), pp. 72-75.
71. D. E. RUMELHART, G. E. HINTON, AND R. J. WILLIAMS, Learning internal representation by error propagation, in Parallel Distributed Processing: Explorations in the Microstructures of Cognition, I: Foundations, D. E. Rumelhart, J. L. McClelland, and the PDP Research Group, eds., MIT, Cambridge, MA, 1986, pp. 318-362.
72. I. SINGER, Best Approximation in Normed Linear Spaces by Elements of Subspaces, Springer, Berlin, 1970.
73. H. R. SIRISENA AND F. S. CHOU, Convergence of the control parametrization Ritz method for nonlinear optimal control problems, J. Optim. Theory Appl., 29 (1979), pp. 369-382.
74. J. SJÖBERG, Q. ZHANG, L. LJUNG, A. BENVENISTE, P.-Y. GLORENNEC, B. DELYON, H. HJAL-MARSSON, AND A. JUDITSKY, Nonlinear black-box modeling in system identification: A unified overview, Automatica, 31 (1995), pp. 1691-1724.
75. V. B. TJUHTIN, An error estimate for approximate solutions in one-sided variational problems, Vestnik Leningrad Univ. Math., 14 (1982), pp. 247-254.
76. P. TSENG, Incremental gradient(-projection) method with momentum term and adaptive step-size rule, SIAM J. Optim., 8 (1998), pp. 506-531.
77. J. F. TRAUB AND A. G. WERSCHULZ, Complexity and Information, Cambridge University Press, Cambridge, UK, 1998.
78. A. A. VLADIMIROV, YU. E. NESTEROV, AND YU. N. CHEKANOV, On uniformly convex functionals, Vestnik Moskov. Univ. Ser. 15 Vychisl. Mat. Kibernet., 3 (1978), pp. 12-23 (in Russian);
79. Moscow University Computational Mathematics and Cybernetics (1979), pp. 10-21 (in English).
80. G. YIN, Rates of convergence for a class of global stochastic optimization algorithms, SIAM J. Optim., 10 (1999), pp. 99-120.
81. R. ZOPPOLI AND T. PARISINI, Learning techniques and neural networks for the solution of N-stage nonlinear nonquadratic optimal control problems, in Systems, Models and Feedback: Theory and Applications, A. Isidori and T. J. Tarn, eds., Birkhäuser, Boston, 1992, pp. 193-210.
82. R. ZOPPOLI, M. SANGUINETI, AND T. PARISINI, Approximating networks and extended Ritz method for the solution of functional optimization problems, J. Optim. Theory Appl., 112 (2002), pp. 403-440.