The weight-decay technique in learning from data: An optimization point of view

Computational Management Science

Volumn 6, Issue 1, 2009, Pages 53-79

  Gnecco, Giorgio ^a   Sanguineti, Marcello ^a  

^a UNIVERSITY OF GENOA   (Italy)

Author keywords

Learning from data; Rates of convergence; Regularization; Suboptimal solutions; Weight decay

Indexed keywords

EID: 58149463469     PISSN: 1619697X     EISSN: 16196988     Source Type: Journal    
DOI: 10.1007/s10287-008-0072-5     Document Type: Article

References (46)

1. Aarts E, Korst J (1989) Simulated annealing and Boltzmann machines: A stochastic approach to combinatorial optimization and neural computing. Wiley,
2. Aronszajn N (1950) Theory of reproducing kernels. Trans AMS 68: 337-404
3. Bartlett PL (1998) The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network. IEEE Trans Inf Theory 44(2): 525-536
4. Berg C, Christensen JPR, Ressel P (1984) Harmonic analysis on semigroups. Springer, New York
5. Bertero M (1989) Linear inverse and ill-posed problems. Adv Electron Electron Phys 75: 1-120
6. Bertsekas DP (1999) Nonlinear programming. Athena Scientific, Belmont
7. Bishop C (1995) Neural networks for pattern recognition. Oxford University Press, London
8. Bishop C (2006) Pattern recognition and machine learning. Springer, Heidelberg
9. Burger M, Engl H (2000) Training neural networks with noisy data as an ill-posed problem. Adv Comput Math 13: 335-354
10. Burger M, Neubauer A (2002) Analysis of Tikhonov regularization for function approximation by neural networks. Neural Netw 16: 79-90
11. Cristianini N, Shawe-Taylor J (2000) An introduction to support vector machines and other kernel-based learning methods. Cambridge University Press, London
12. Cucker F, Smale S (2001) On the mathematical foundations of learning. Bull AMS 39: 1-49
13. Cucker F, Smale S (2002) Best choices for regularization parameters in learning theory: On the bias-variance problem. Found Comput Math 2: 413-428
14. Cuesta-Albertos JA, Wschebor M (2003) Some remarks on the condition number of a real random square matrix. J Compl 19: 548-554
15. Demmel J (1987) The geometry of ill-conditioning. J Compl 3: 201-229
16. Dontchev AL (1983) Perturbations, approximations and sensitivity analysis of optimal control systems. Lecture Notes in Control and Information Sciences, vol 52. Springer, Berlin
17. Friedman A (1970) Foundations of Modern Analysis. Holt, Rinehart, and Winston, New York
18. Girosi F, Jones M, Poggio T (1995) Regularization theory and neural networks architectures. Neural Comput 7: 219-269
19. Girosi F (1998) An equivalence between sparse approximation and support vector machines. Neural Comput 10: 1455-1480
20. Girosi F (1994) Regularization theory, radial basis functions and networks. In: Cherkassky JHFV, Wechsler H(eds) From Statistics to Neural Networks. Theory and pattern recognition applications, ser. NATO ASI Series F, Computer and Systems Sciences. Springer, Berlin, pp 166-187
21. Gnecco G, Sanguineti M (2007) Accuracy of suboptimal solutions to kernel principal component analysis. Comput Optim Appl. doi: 10.1007/ s10589-007-9108-y
22. Goldberg DE (1989) Genetic algorithms in search, optimization, and machine learning. Addison-Wesley, Reading
23. Golub GH, Loan CFV (1996) Matrix computations. John Hopkins University Press, London
24. Gupta A, Lam M (1998) The weight decay backpropagation for generalizations with missing values. Ann Oper Res 78: 165-187
25. Gupta A, Lam M (1998) Weight decay backpropagation for noisy data. Neural Netw 11: 1127-1138
26. Hofinger A (2006) Nonlinear function approximation: Computing smooth solutions with an adaptive greedy algorithm. J Approxim Theory 143: 159-175
27. Hofinger A, Pillichshammer F (2005) Learning a function from noisy samples at a finite sparse set of points. J. Kepler University, Linz, Technical Report, SFB F013
28. Kimeldorf GS, Wahba G (1970) A correspondence between Bayesian estimation on stochastic processes and smoothing by splines. Ann Math Stat 41: 495-502
29. Krogh A, Hertz JA (1992) A simple weight decay can improve generalization. In: Advances in neural information processing systems, vol. 4. Morgan Kaufmann Pub., pp 950-957
30. Kůrková V (1997) Dimension-independent rates of approximation by neural networks. In: Warwick K, Kárný M(eds) Computer-intensive methods in control and signal processing. The curse of dimensionality. Birkhäuser, Boston, pp 261-270
31. Kůrková V (2004) Learning from data as an inverse problem. In: Antoch J(eds) COMPSTAT 2004 - proceedings in computational statistics. Physica-Verlag/Springer, Heidelberg, pp 1377-1384
32. Kůrková V, Sanguineti M (2001) Bounds on rates of variable-basis and neural-network approximation. IEEE Trans Inf Theory 47: 2659-2665
33. Kůrková V, Sanguineti M (2005) Error estimates for approximate optimization by the extended Ritz method. SIAM J Optim 15: 461-487
34. Kůrková V, Sanguineti M (2005) Learning with generalization capability by kernel methods of bounded complexity. J Compl 21: 350-367
35. Kůrková V, Savická P, Hlaváčková K (1998) Representations and rates of approximation of real-valued Boolean functions by neural networks. Neural Netw 11: 651-659
36. Levitin ES, Polyak BT (1966) Convergence of minimizing sequences in conditional extremum problems. Dokl Akad Nauk SSSR 168(5): 764-767
37. Ortega JM (1990) Numerical analysis: A second course. SIAM, Philadelphia
38. Poggio T, Girosi F (1990) Networks for approximation and learning. Proc IEEE 78: 1481-1497
39. Poggio T, Smale S (2003) The mathematics of learning: Dealing with data. Notices AMS 50: 536-544
40. Poggio T, Mukherjee S, Rifkin R, Rakhlin A, Verri A (2002) "b". In: Winkler J, Niranjan M (eds) Uncertainty in Geometric Computations. Kluwer, Dordrecht, pp 131-141
41. Schílkopf B, Smola AJ (2002) Learning with kernels - support vector machines, regularization, optimization, and beyond. MIT Press, Cambridge
42. Schïlkopf B, Herbrich R, Smola AJ, Williamson RC (2001) A generalized representer theorem. In: Proceedings of COLT'01, Lecture Notes in Artificial Intelligence. Springer, Heidelberg, pp 416-424
43. Tikhonov AN, Arsenin VY (1977) Solutions of ill-posed problems. W.H. Winston, Washington
44. Treadgold NK, Gedeon TD (1998) Simulated annealing and weight decay in adaptive learning: The SARPROP algorithm. IEEE Trans Neural Netw 9(4): 662-668
45. Vapnik VN (1998) Statistical learning theory. Wiley, New York
46. Vladimirov AA, Nesterov YE, Chekanov YN (1978) On uniformly convex functionals. Vestnik Moskovskogo Universiteta. Seriya 15 - Vychislitel'naya Matematika i Kibernetika, vol 3, pp 12-23 (English translation: Moscow University Computational Mathematics and Cybernetics, pp 10-21, 1979)