A Statistical Approach to Learning and Generalization in Layered Neural Networks

Proceedings of the IEEE

Volumn 78, Issue 10, 1990, Pages 1568-1574

  Levin, Esther ^a   Tishby, Naftali ^a   Solla, Sara A ^a  

^a LUCENT TECHNOLOGIES   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

LEARNING SYSTEMS; MATHEMATICAL TECHNIQUES--ESTIMATION; STATISTICAL METHODS;

GIBBS DISTRIBUTION; LAYERED NEURAL NETWORKS; LEARNING CURVES;

NEURAL NETWORKS;

EID: 0025508916     PISSN: 00189219     EISSN: 15582256     Source Type: Journal    
DOI: 10.1109/5.58339     Document Type: Article

References (28)

1. R. P. Lippmann, "An introduction to computing with neural nets," ASSP Magazine, vol. 4, no. 2, pp. 4–22, 1987.
2. G. Cybenko, "Continuous valued neural networks with two hidden layers are sufficient," Tufts University preprint, 1988.
3. L. G. Valiant, "A theory of the learnable," Comm. ACM, vol. 27, no. 11, pp. 1134–1142, 1984.
4. L. D. Landau and E. M. Lifshitz, Course of theoretical physics, vol. 5, 3rd ed. Pergamon, 1980.
5. T. Cover, "Geometrical and statistical properties of systems of linear inequalities with applications to patterns recognition," IEEE Trans. Electron. Comput., vol. 14, pp. 326–334,1965.
6. E. B. Baum, "On the capabilities of multilayer perceptrons," J. Complexity, 1989.
7. H. White, "Learning in artificial neural networks: A statistical perspective," Neural Computation, vol. 1, pp. 425–464,1989.
8. J. Rissanen, "Stochastic complexity and modeling," Ann. Statist, vol. 14, pp. 1080–1100, 1986.
9. E. B. Baum and D. Haussler, "What size net gives valid generalization," Neural Computation, vol. 1, pp. 151–160,1989.
10. R. T. Cox, "Probability, frequency and reasonable expectation," Amer. J. Phys., vol. 14, pp. 1–26, 1946.
11. J. Aczél and Z. Daróczy, On Measures of Information and Their Characterizations. Academic Press, 1975, p. 16.
12. Y. Tikochinsky, N. Tishby, and R. D. Levine, "Alternative approach to maximum entropy inference," Phys. Rev. A, vol. 30, pp. 2638–2644, 1984.
13. S. Kirkpatrick, C. D. Celatt, and M. P. Vecchi, "Optimization by simulated annealing," Science, vol. 220, pp. 671–680,1983.
14. S. Kullback, Information Theory and Statistics. New York: Wiley, 1959.
15. J. Denker et al., "Large automatic learning, rule extraction, and generalization," Complex Systems, vol. 1, pp. 877–922, 1987.
16. P. Carnevali and S. Patranello, "Learning networks of neurons with Boolean logic," Europhys. Lett., vol. 4, pp. 503–508, 1199–1204, 1987.
17. E. Gardner, "The space of interactions of neural networks models," J. Phys. A, vol. 21, pp. 257–270,1988; E. Gardner and B. Derrida, J. Phys. A, vol. 21, pp. 271–284, 1988.
18. D. Hansel and H. Sompolinsky, "Learning from examples in a single-layer neural network," Europhys. Lett, 1990.
19. G. Györgyi and N. Tishby, "Statistical theory of learning a rule," in Neural Networks and Spin Classes. W. Theumann and R. Kobrele Eds. New Jersey: World Scientific, 1990.
20. H. Sompolinsky, S. Seung, and N. Tishby, "Learning from examples in large neural networks," to be published.
21. E. Levin, N. Tishby, and S. A. Solla, in Proc. 2nd Ann. Workshop on Computational Learning Theory (COLT’89), R. Rivest, D. Haussler, and M. K. Warmuth, Eds. San Mateo, CA: Morgan Kaufmann, 1989, pp. 245–260.
22. D. Haussler, N. Littlestone, and M. K. Warmuth, "Predicting {0, 1} Functions on Randomly Drawn Points," in Proc. COLT’88 San Mateo, CA: Morgan Kaufmann, 1988, pp. 280– 295.
23. N. Tishby, E. Levin, and S. A. Solla, "Consistent inference of probabilities in layered networks: Predictions and generalization," in Proc. Int. Joint Conf. on Neural Networks (IJCNN), vol. 2, pp. 403–409, IEEE Press, Washington DC, June 1989. New York: IEEE Press, 1989.
24. T. Maxwell, C. L. Giles,and Y. C. Lee, "Generalization in neural networks, the contiguity problem," in Proc. IEEE 1st Int. Conf. con Neural Networks, San Diego, 1987; T. Grossman, R. Meir, and E. Domany, "Learning by choice of internal representations," Complex Systems, vol. 2, pp. 555–563,1988.
25. M. Minsky, and S. Papert, Perceptrons. Cambridge, MA: M.I.T. Press, 1969.
26. M. Stone, "Cross-validatory choice and assessment of statistical predictions," J. R. Stat. Soc. Ser. B, vol. 36, pp. 111– 133, 1974.
27. D. B. Schwartz, V. K. Samalam, J. S. Denker, and S. A. Solla, "Exhaustive learning," Neural. Comp., 1990.
28. A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth, "Learnability and the Vapnik-Chervonenkis dimension," UCSC-CRL 87-20 and J. ACM, 1988