Deterministic design for neural network learning: An approach based on discrepancy

IEEE Transactions on Neural Networks

Volumn 15, Issue 3, 2004, Pages 533-544

  Cervellera, Cristiano ^a   Muselli, Marco ^a  

^a CNR   (Italy)

Author keywords

Deterministic learning; Discrepancy; Empirical risk minimization (ERM); Learning rate; Variation

Indexed keywords

COMPUTATIONAL COMPLEXITY; COMPUTER SIMULATION; LEARNING ALGORITHMS; LEARNING SYSTEMS; OPTIMIZATION; PROBABILITY DENSITY FUNCTION;

DETERMINISTIC LEARNING; DISCREPANCY; EMPIRICAL RISK MINIMIZATION; LEARNING RATE; NEURAL NETWORK LEARNING; VARIATION;

NEURAL NETWORKS;

ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK;

ARTIFICIAL INTELLIGENCE; NEURAL NETWORKS (COMPUTER);

EID: 2542562008     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2004.824413     Document Type: Article

References (38)

1. G. Cybenko, "Approximation by superpositions of a sigmoidal function," Mathematics Control, Signals, Systems, vol. 2, pp. 303-314, 1989.
2. K. Hornik, M. Stinchcombe, and H. White, "Multilayer feedforward networks are universal approximators," Neural Netw., vol. 2, pp. 359-366, 1989.
3. A. Barron, "Universal approximation bounds for superpositions of a sigmoidal function," IEEE Trans. Inform. Theory, vol. 39, pp. 930-945, May 1993.
4. F. Girosi, M. Jones, and T. Poggio, "Regularization theory and neural networks architectures," Neural Comput., vol. 7, pp. 219-269, 1995.
5. V. N. Vapnik and A. Y. Chervonenkis, "On the uniform convergence of relative frequencies of events to their probabilities," Theory of Probability and Its Applications, vol. 16, pp. 264-280, 1971.
6. L. G. Valiant, "A theory of the learnable," Commun. ACM, vol. 27, pp. 1134-1142, 1984.
7. A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth, "Learnability and the Vapnik-Chervonenkis dimension," J. ACM, vol. 36, pp. 929-965, 1989.
8. L. Devroye, L. Györfi, and G. Lugosi, A Probabilistic Theory of Pattern Recognition. New York: Springer-Verlag, 1997.
9. D. Pollard, "Empirical processes: Theory and applications," in Proc. NSF-CBMS Regional Conf. Series Probability Statistics, vol. 2, 1990.
10. V. N. Vapnik, Statistical Learning Theory. New York: Wiley, 1998.
11. N. Alon, S. Ben-David, N. Cesa-Bianchi, and D. Haussler, "Scale-sensitive dimensions, uniform convergence, and learnability," J. ACM, vol. 44, pp. 615-631, 1997.
12. M. Vidyasagar, A Theory of Learning and Generalization. London, U.K.: Springer-Verlag, 1997.
13. W. Hoeffding, "Probability inequalities for sum of bounded random variables," J. Amer. Statistical Assn. Mathematical Soc. Trans., vol. 17, pp. 277-364, 1961.
14. K. Najarian, G. A. Dumont, M. S. Davies, and N. E. Heckman, "PAC learning in nonlinear FIR models," Int. J. Adaptive Control Signal Processing, vol. 15, pp. 37-52, 2001.
15. D. MacKay, "Information-based objective functions for active data selection," Neural Comput., vol. 4, pp. 305-318, 1992.
16. D. A. Cohn, "Neural network exploration using optimal experiment," in Advances in Neural Information Processing Systems 6, J. Cowan, G. Tesauro, and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann, 1994, pp. 679-686.
17. J. Kindermann, G. Paass, and F. Weber, "Query construction for neural networks using the bootstrap," in Proc. Int. Conf. Artificial Neural Networks, vol. 95, 1995, pp. 135-140.
18. K. Fukumizu, "Active learning in multilayer perceptrons," in Advances in Neural Information Processing Systems 8, D. Touretzky, M. Mozer, and M. Hasselmo, Eds. Cambridge, MA: MIT Press, 1996, pp. 295-301.
19. ____, "Statistical active learning in multilayer perceptrons," IEEE Trans. Neural Networks, vol. 11, pp. 17-26, Jan. 2000.
20. J. Pilz, Bayesian Estimation and Experimental Design in Linear Regression. New York: Wiley, 1991.
21. K.-T. Fang and Y. Wang, Number-Theoretic Methods in Statistics. London, U.K.: Chapman & Hall, 1994.
22. R. M. Dudley, Real Analysis and Probability. Pacific Grove, CA: Wadsworth & Brooks/Cole, 1989.
23. V. N. Vapnik and A. Y. Chervonenkis, "Uniform convergence of the frequencies of occurrence of events to their probabilities," Soviet Math. Doklady, vol. 9, pp. 915-918, 1968.
24. H. Niederreiter, Random Number Generation and Quasi-Monte Carlo Methods. Philadelphia, PA: SIAM, 1992.
25. N. Alon and J. H. Spencer, The Probabilistic Method. New York: Wiley, 2000.
26. L. Kuipers and H. Niederreiter, Uniform Distribution of Sequences. New York: Wiley, 1974.
27. E. Hlawka, "Funktionen von beschränkter variation in der theorie der gleichverteilung," Ann. Mat. Pura Appl., vol. 54, pp. 325-333, 1961.
28. M. Blumlinger and R. F. Tichy, "Bemerkungen zu einigen anwendungen gleichverteilter folgen," Sitzungsber. Österr. Akad. Wiss. Math.-Natur. Kl. II, vol. 195, pp. 253-265, 1986.
29. L. K. Hua and Y. Wang, Applications of Number Theory to Numerical Analysis. Berlin, Germany: Springer-Verlag, 1981.
30. R. F. Tichy, "Über eine zahlentheoretische methode zur numerischen integration und zur behandlung von integralgleichungen," Sitzungsber. Österr. Akad. Wiss. Math.-Natur. Kl. II, vol. 193, pp. 329-358, 1984.
31. C. M. Bishop, Neural Networks for Pattern Recognition. New York: Oxford Univ. Press, 1995.
32. I. M. Sobol, "The distribution of points in a cube and the approximate evaluation of integrals," Zh. Vychisl. Mat. i Mat. Fiz., vol. 7, pp. 784-802, 1967.
33. H. Niederreiter, "Point sets and sequences with small discrepancy," Monatsh. Math., vol. 104, pp. 273-337, 1987.
34. J. M. Hammersley and D. C. Handscomb, Monte Carlo Methods. London, U.K.: Methuen, 1964.
35. D. Angluin and L. Valiant, "Fast probabilistic algorithms for hamiltonian circuits and matchings," J. Comput. Syst. Sci., vol. 18, pp. 155-193, 1979.
36. V. Cherkassky and F. Mulier, Learning from Data: Concepts, Theory, and Methods. New York: Wiley, 1998.
37. P. Bratley, B. L. Fox, and H. Niederreiter, "Programs to generate Niederreiter's low-discrepancy sequences," ACM Trans. Math. Softw., vol. 20, no. 4, pp. 494-495, 1994.
38. M. Hagan and M. Menhaj, "Training feedforward networks with the Marquardt algorithm," IEEE Trans. Neural Networks, vol. 5, pp. 989-993, Nov. 1994.