Stochastic complexity and generalization error of a restricted boltzmann machine in bayesian estimation

Journal of Machine Learning Research

Volumn 11, Issue , 2010, Pages 1243-1272

  Aoyagi, Miki ^a  

^a NIHON UNIVERSITY   (Japan)

Author keywords

Boltzmann machine; Non regular learning machine; Resolution of singularities; Zeta function

Indexed keywords

ASYMPTOTIC FORMS; BAYESIAN ESTIMATIONS; BOLTZMANN MACHINES; EIGENVALUE ANALYSIS; GENERALIZATION ERROR; HIERARCHICAL LEARNING; KULLBACK FUNCTION; NON-REGULAR LEARNING MACHINES; RESOLUTION OF SINGULARITIES; RESTRICTED BOLTZMANN MACHINE; STOCHASTIC COMPLEXITY; ZETA FUNCTION;

ASYMPTOTIC ANALYSIS; EIGENVALUES AND EIGENFUNCTIONS; FUNCTIONS; LEARNING SYSTEMS; POLES;

BAYESIAN NETWORKS;

EID: 77951945102     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (35)

1. H. Akaike. A new look at the statistical model identification. IEEE Trans, on Automatic Control, 19:716-723, 1974.
2. H. Akaike. Likelihood and bayes procedure. In J. M. Bernald, editor, Bayesian Statistics, pages 143-166. University Press, Valencia, Spain, 1980.
3. S. Amari and N. Murata. Statistical theory of learning curves under entropic loss. Neural Computation, 5:140-153, 1993.
4. S. Amari, N. Fujita, and S. Shinomoto. Four types of learning curves. Neural Computation, 4(4): 608-618, 1992.
5. M. Aoyagi. The zeta function of learning theory and generalization error of three layered neural perceptron. RIMS Kokyuroku, Recent Topics on Real and Complex Singularities, 1501:153-167, 2006.
6. M. Aoyagi and S. Watanabe. Stochastic complexities of reduced rank regression in bayesian estimation. Neural Networks, 18:924-933, 2005b.
7. M. Aoyagi and S. Watanabe. Resolution of singularities and the generalization error with bayesian estimation for layered neural network. IEICE Trans. J88-D-II, 10:2112-2124, 2005a.
8. K. Fukumizu. A regularity condition of the information matrix of a multilayer perceptron network. Neural Networks, 9(5):871-879, 1996.
9. W. Fulton. Introduction to toric varieties, Annals of Mathematics Studies. Princeton University Press, 1993.
10. K. Hagiwara, N. Toda, and S. Usui. On the problem of applying aie to determine the structure of a layered feed-forward neural network. In Proceedings of IJCNN Nagoya Japan, volume 3, pages 2263-2266, 1993.
11. E. J. Hannan and B. G. Quinn. The determination of the order of an autoregression. Journal of Royal Statistical Society Series B, 41:190-195, 1979.
12. J. A. Hartigan. A failure of likelihood asymptotics for normal mixtures. In Proceedings of the Berkeley Conference in Honor of J.Neyman and J.Kiefer, volume 2, pages 807-810, 1985.
13. G. E. Hinton. Reinforcement learning with factored states and actions. Brian Sallans, 5:1063-1088, 2004.
14. H. Hironaka. Resolution of singularities of an algebraic variety over a field of characteristic zero. Annals of Math, 79:109-326, 1964.
15. E. Levin, N. Tishby, and S. A. Solla. A statistical approaches to learning and generalization in layered neural networks. In Proceedings of IEEE, volume 78, pages 1568-1674, 1990.
16. D. J. Mackay. Bayesian interpolation. Neural Computation, 4(2):415-447, 1992.
17. N. J. Murata, S. G. Yoshizawa, and S. Amari. Network information criterion - determining the number of hidden units for an artificial neural network model. IEEE Trans, on Neural Networks, 5(6):865-872, 1994.
18. K. Nagata and S. Watanabe. A proposal and effectiveness of the optimal approximation for bayesian posterior distribution. In Workshop on Information-Based Induction Sciences, pages 99-104, 2005.
19. K. Nagata and S. Watanabe. Analysis of exchange ratio for exchange monte carlo method. In Proceedings of The First IEEE Symposium on Foundations of Computational Intelligence, pages 434-439, 2007.
20. Y. Nishiyama and S. Watanabe. Asymptotic behavior of free energy of general boltzmann machines in mean field approximation. Technical report of IEICE, NC2006, 38:1-6, 2006.
21. J. Rissanen. Universal coding, information, prediction, and estimation. IEEE Trans, on Information Theory, 30(4): 629-636, 1984.
22. J. Rissanen. Stochastic complexity and modeling. Annals of Statistics, 14:1080-1100, 1986.
23. D. Rusakov and D. Geiger. Asymptotic model selection for naive bayesian networks. Journal of Machine Learning Research, 6:1-35, 2005.
24. R. Salakhutdinov, A. Mnih, and G. E. Hinton. Restricted boltzmann machines for collaborative filtering. In Proceedings of the 24th International Conference on Machine Learning, pages 791-798. Omni Press, Zoubin Ghahramani, 2007.
25. G. Schwarz. Estimating the dimension of a model. Annals of Statistics, 6(2):461-464, 1978.
26. H. J. Sussmann. Uniqueness of the weights for minimal feed-forward nets with a given input-output map. Neural Networks, 5:589-593, 1992.
27. S. Takamatsu, S. Nakajima, and S. Watanabe. Generalization error of localized bayes estimation in reduced rank regression. In Workshop on Information-Based Induction Sciences, pages 81-86, 2005.
28. K. Takeuchi. Distribution of an information statistic and the criterion for the optimal model. Mathematical Science, 153:12-18, 1976.
29. S. Watanabe. Algebraic analysis for nonidentifiable learning machines. Neural Computation, 13 (4):899-933, 2001a. (Pubitemid 33594310)
30. S. Watanabe. Algebraic geometrical methods for hierarchical learning machines. Neural Networks, 14(8): 1049-1060, 2001b. (Pubitemid 32846604)
31. S. Watanabe. Algebraic geometry of learning machines with singularities and their prior distributions. Journal of Japanese Society of Artificial Intelligence, 16(2):308-315, 2001c.
32. S. Watanabe. Algebraic Geometry and Statistical Learning Theory, volume 25. Cambridge Monographs on Applied and Computational Mathematics, 2009.
33. S. Watanabe, K. Hagiwara, S. Akaho, Y. Motomura, K. Fukumizu, M. Okada, and M. Aoyagi. Theory and Application of Learning System. Morikita Press, 2005.
34. K. Yamanishi. A decision-theoretic extension of stochastic complexity and its applications to learning. IEEE Trans, on Information Theory, 44(4): 1424-1439, 1998.
35. K. Yamazaki and S. Watanabe. Singularities in complete bipartite graph-type boltzmann machines and upper bounds of stochastic complexities. IEEE Trans, on Neural Networks, 16(2):312-324, 2005.