Discovering neural nets with low Kolmogorov complexity and high generalization capability

Neural Networks

Volumn 10, Issue 5, 1997, Pages 857-873

  Schmidhuber, Jürgen ^a  

^a DALLE MOLLE INSTITUTE FOR ARTIFICIAL INTELLIGENCE IDSIA   (Switzerland)

Author keywords

Generalization; Kolmogorov complexity; Levin complexity; Neural networks; Self sizing programs; Solomonoff Levin distribution; Universal search

Indexed keywords

COMPUTATIONAL COMPLEXITY; COMPUTER SIMULATION; LEARNING ALGORITHMS; LEARNING SYSTEMS; PROBABILITY;

KOLMOGOROV COMPLEXITY; SELF SIZING PROGRAMS; SOLOMONOFF LEVIN DISTRIBUTION;

NEURAL NETWORKS;

ARTIFICIAL NEURAL NETWORK; LEARNING; PRIORITY JOURNAL; REVIEW;

EID: 0031194381     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0893-6080(96)00127-X     Document Type: Article

References (33)

1. Adleman, L. (1979). Time, space and randomness (Technical Report MIT/LCS/79/TM-131). Boston, MA: Laboratory for Computer Science, MIT.
2. Allender, A. (1992). Application of time-bounded Kolmogorov complexity in complexity theory. In O. Watanabe (Ed.), Kolmogorov complexity and computational complexity (pp. 6-22). Berlin: EATCS Monographs on Theoretical Computer Science, Springer.
3. Amari, S., & Murata, N. (1993). Statistical theory of learning curves under entropic loss criterion. Neural Computation. 5 (1), 140-153.
4. Atick, J. J., Li, Z., & Redlich, A. N. (1992). Understanding retinal color coding from first principles. Neural Computation, 4, 559-572.
5. Barlow, H. B. (1989). Unsupervised learning. Neural Computation, 1 (3), 295-311.
6. Barron, A. R. (1988). Complexity regularization with application to artificial neural networks. In Nonparametric functional estimation and related topics (pp. 561-576). Dortrecht: Kluwer Academic Publishers.
7. Barto, A. G. (1989). Connectionist approaches for control (Technical Report COINS 89-89). Amherst, MA: University of Massachusetts.
8. Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13, 834-846.
9. Barzdin, Y. M. (1988). Algorithmic information theory. In D. Reidel (Ed.), Encyclopaedia of mathematics (Vol. 1, pp. 140-142). Dortrecht: Kluwer Academic Publishers.
10. Baum, E. B., & Haussler, D. (1989). What size net gives valid generalization?. Neural Computation, 1 (1), 151-160.
11. Becker, S. (1991). Unsupervised learning procedures for neural networks. International Journal of Neural Systems, 2 (1&2), 17-33.
12. Bennett, C. H. (1988). Logical depth and physical complexity. In The universal Turing machine: a lialf century survey (Vol. 1, pp. 227-258). Oxford: Oxford University Press, and Hamburg: Kammerer & Unverzagt.
13. Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M. K. (1987). Occam's razor. Information Processing Letters, 24, 377-380.
14. Chaitin, G. (1966). On the length of programs for computing finite binary sequences. Journal of the ACM, 13, 547-569.
15. Chaitin, G. (1969). On the length of programs for computing finite binary sequences: statistical considerations. Journal of the ACM, 16, 145-159.
16. Chaitin, G. (1975). A theory of program size formally identical to information theory. Journal of the ACM, 22, 329-340.
17. Chaitin, G. (1987). Algorithmic Information Theory. Cambridge: Cambridge University Press.
18. Cover, T. M., Gács, P., & Gray, R. M. (1989). Kolmogorov's contributions to information theory and algorithmic complexity. Annals of Probability Theory, 17, 840-865.
19. Dayan, P., & Sejnowski, T. (1994). TD(λ): Convergence with probability 1. Machine Learning, 14, 295-301.
20. Deco, G., Finnoff, W., & Zimmermann, H. G. (1993). Elimination of overtraining by a mutual information network. In Proceedings of the International Conference on Artificial Neural Networks, Amsterdam (pp. 744-749). Berlin: Springer.
21. Dietterich, T. G. (1989). Limitations of inductive learning. In Proceedings of the Sixth International Workshop on Machine Learning, Ithaca, NY (pp. 124-128). San Francisco, CA: Morgan Kaufmann.
22. Gàcs, P. (1974). On the symmetry of algorithmic information. Soviet Math. Dokl., 15, 1477-1480.
23. Gallant, S. I. (1990). A connectionist learning algorithm with provable generalization and scaling bounds. Neural Networks, 3, 191-201.
24. Gao, Q., & Li, M. (1989). The minimum description length principle and its application to online learning of handprinted characters. In Proceedings of the 11th IEEE International Joint Conference on Artificial Intelligence, Detroit, MI (pp. 843-848).
25. Guyon, I., Vapnik, V., Boser, B., Bottou, L., & Solla, S. A. (1992). Structural risk minimization for character recognition. In D. S. Lippman, J. E. Moody & D. Touretzky (Eds.), Advances in neural information processing systems 4 (pp. 471-479). San Mateo, CA: Morgan Kaufmann.
26. Hartmanis, J. (1983). Generalized Kolmogorov complexity and the structure of feasible computations. In Proceedings of the 24th IEEE Symposium on Foundations of Computer Science (pp. 439-445).
27. Hassibi, B., & Stork, D. G. (1993). Second order derivatives for network pruning: optimal brain surgeon. In D. S. Lippman, J. E. Moody & D. S. Touretzky (Eds.), Advances in neural information processing systems 5 (pp. 164-171). San Mateo, CA: Morgan Kaufmann.
28. Haussler, D. (1988). Quantifying inductive bias: AI learning algorithms and Valiant's learning framework. Artificial Intelligence, 36, 177-221.
29. Heil, S. (1995). Universelle Suche und inkrementelles Lernen. Unpublished diploma thesis. Fakultät für Informatik, Lehrstuhl Prof. Brauer, Technische Universität München.
30. Hinton, G. E., & van Camp, D. (1993). Keeping neural networks simple. In Proceedings of the International Conference on Artificial Neural Networks, Amsterdam (pp. 11-18). Berlin: Springer.
31. Hochreiter, S., & Schmidhuber, J. (1995). Simplifying neural nets by discovering flat minima. In G. Tesauro, D. S. Touretzky & T. K. Leen (Eds.), Advances in neural information processing systems 7 (pp. 529-536). Cambridge, MA: MIT Press.
32. Hochreiter, S., & Schmidhuber, J. (1996). Flat minima. Neural Computation. In press. Extended version available in WWW homepages of Hochreiter and Schmidhuber.
33. Kolmogorov, A. (1965). Three approaches to the quantitative definition of information. Problems of Information Transmission, 1, 1-11.