Variational Bayes solution of linear neural networks and its generalization performance

Neural Computation

Volumn 19, Issue 4, 2007, Pages 1112-1153

  Nakajima, Shinichi ^a   Watanabe, Sumio ^b  

^a NIKON CORPORATION   (Japan)

^b TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 34247210751     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/neco.2007.19.4.1112     Document Type: Article

References (45)

1. Akaike, H. (1974). A new look at statistical model. IEEE Trans. on Automatic Control, 19(6), 716-723.
2. Akaike, H. (1980). Likelihood and Bayes procedure. In J. M. Bernald (Ed.), Bayesian statistics (pp. 143-166). Valencia, Spain: University Press.
3. Amari, S., Park, H., & Ozeki, T. (2006). Singularities affect dynamics of learning in neuromanifolds. Neural Computation, 18, 1007-1065.
4. Aoyagi, M., & Watanabe, S. (2004). Stochastic complerities of reduced rank regression in Bayesian estimation. In Neural Networks, 18, 924-933.
5. Attias, H. (1999). Inferring parameters and structure of latent variable models by variational Bayes. In Proc. of the Conference on Uncertainty in Artificial Intelligence. San Francisco: Morgan Kaufmann.
6. Baldi, P. F., & Hornik, K. (1995). Learning in linear neural networks: A survey. IEEE Trans, on Neural Networks, 6(4), 837-858.
7. Bickel, P., & Chernoff, H. (1993). Asymptotic distribution of the likelihood ratio statistic in a prototypical non regular problem. In J. K. Ghosh, S. K. Mitra, K. R. Parthasarathy, & B. L. S. Prakasa Rao (Eds.), Statistics and Probability: A Raghu Raj Balladur Festschrift (pp. 83-96). New York: Wiley.
8. Cramer, H. (1949). Mathematical methods of statistics. Princeton, NJ: Princeton University Press.
9. Dacunha-Castelle, D., & Gassiat, E. (1997). Testing in locally conic models, and application to mixture models. Probability and Statistics, 1, 285-317.
10. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood for incomplete data via the EM algorithm. J. R. Statistical Society, 39B, 1-38.
11. Efron, B., & Morris, C. (1973). Stein's estimation rule and its competitors - an empirical Bayes approach. J. Am. Stat. Assoc., 68, 117-130.
12. Fukumizu, K. (1999). Generalization error of linear neural networks in unidentifiable cases. In Proceedings of the Tenth International Conference, Algorithmic Learning Theory. (pp. 51-62). New York: Springer.
13. Fukumizu, K. (2003). Likelihood ratio of unidentifiable models and multilayer neural networks. Annals of Statistics, 31(3), 833-851.
14. Ghahramani, Z., & Beal, M. J. (2001). Graphical models and variational methods. In M. Opper & D. Saad (Eds.), Advanced mean field methods (pp. 161-177). Cambridge, MA: MIT Press.
15. Hagiwara, K. (2002). On the problem in model selection of neural network regression in overrealizable scenario. Neural Computation, 14, 1979-2002.
16. Hartigan, J. A. (1985). A failure of likelihood ratio asymptotics for normal mixtures. In Proc. of the Berkeley Conference in Honor of J. Neymun and J. Kiefer (pp. 807-810). Monterey, CA: Wadsworth.
17. Hinton, G. E., & van Camp, D. (1993). Keeping neural networks simple by minimizing the description length of the weights. In Proc. of the Conference on Computational Learning Theory, (pp. 5-13). New York: ACM.
18. Hosino, T., Watanabe, K., & Watanabe, S. (2005). Stochastic complexity of variational Bayesian hidden Markov models. In Proc. of the International Joint Conference on Neural Networks.
19. Jaakkola, T. S., & Jordan, M. I. (2000). Bayesian parameter estimation via variational methods. Statistics and Computing, 10, 25-37.
20. James, W., & Stein, C. (1961). Estimation with quadratic loss. In Proc. of the 4th Berkeley Symp. on Math. Stat. and Prob. (pp. 361-379). Berkeley: University of California Press.
21. Kuriki, S., & Takemura, A. (2001). Tail probabilities of the maxima of multilinear forms and their applications. Annals of Statistics, 29(2), 328-371.
22. Levin, E., Tishby, N., & Solla, S. A. (1990). A statistical approach to learning and generalization in layered neural networks. Proc. IEEE, 78, 568-1674.
23. MacKay, D. J. C. (1992). Bayesian interpolation. Neural Computation, 4(2), 415-447.
24. MacKay, D. J. C. (1995). Developments in probabilistic modeling with neural networks - ensemble learning. In Proc. of the 3rd Ann. Symp. on Neural Networks (pp. 191-198). New York: Springer-Verlag.
25. Nakajima, S., & Watanabe, S. (2005). Generalization error of linear neural networks in an empirical Bayes approach. In Proc. of IJCAI (pp. 804-810). Edinburgh, U.K.
26. Nakajima, S., & Watanabe, S. (2006). Generalization performance of subspace Bayes approach in linear neural networks. IEICE Trans., E89-D(3), 1128-1138.
27. Nakano, N., & Watanabe, S. (2005). Stochastic complexity of layered neural networks in mean field approximation. In Proc. of ICONIP. Taipei, Taiwan.
28. Reinsel, G. C., & Velu, R. P. (1998). Multivariate reduced-rank regression. New York: Springer.
29. Rissanen, J. (1986). Stochastic complexity and modeling. Annals of Statistics, 14(3), 1080-1100.
30. Rusakov, D., & Geiger, D. (2002). Asymptotic model selection for naive Bayesian networks. In Proc. of Conference on Uncertainty in Artificial Intelligence (pp. 438-445). San Francisco: Morgan Kaufmann.
31. Sato, M. (2001). Online model selection based on the variational Bayes. Neural Computation, 13, 1649-1681.
32. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461-464.
33. Stein, C. (1956). Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proc. of the 3rd Berkeley Symp. on Math. Stat. and Prob. (pp. 197-206).
34. Takemura, A., & Kuriki, S. (1997). Weights of chi-barsquare distribution for smooth or piecewise smooth cone alternatives. Annals of Statistics, 25(6), 2368-2387.
35. Wang, B., & Titterington, D. M. (2004). Convergence and asymptotic normality of variational Bayesian approximations for exponential family models with missing values. In Proc. of Conference on Uncertainly in Artificial Intelligence (pp. 577-584).
36. Watanabe, K., & Watanabe, S. (2006). Stochastic complexities of gaussian mixtures in variational Bayesian approximation. Journal of Machine Learning Research, 7, 625-644.
37. Watanabe, S. (1995). A generalized bayesian framework for neural networks with singular fisher information matrices. In Proc. of the International Symposium on Nonlinear Theory and Its Applications (Vol. 2, pp. 207-210).
38. Watanabe, S. (2001a). Algebraic analysis for nonidentifiable learning machines. Neural Computation, 13(4), 899-933.
39. Watanabe, S. (2001b). Algebraic information geometry for learning machines with singularities. In T. K. Leen, T. G. Dietterich, & V. Trep (Eds.), Advances in neural information processing systems, 13, (pp. 329-336). Cambridge, MA: MIT Press.
40. Watanabe, S., & Amari, S. (2003). Learning coefficients of layered models when the true distribution mismatches the singularities. Neural Computation, 15, 1013-1033.
41. Watcher, K. W. (1978). The strong limits of random matrix spectra for sample matrices of independent elements. Ann. Prob., 6, 1-18.
42. Yamazaki, K., & Watanabe, S. (2002). Resolution of singularities in mixture models and its stochastic complexity. In Proc. of the 9th International Conference on Neural Information Processing (pp. 1355-1359). Singapore.
43. Yamazaki, K., & Watanabe, S. (2003a). Stochastic complexities of hidden Markov models. In Proc. of Neural Networks for Signal Processing XIII (NNSP) (pp. 179-188). Toulouse, France.
44. Yamazaki, K., & Watanabe, S. (2003b). Stochastic complexity of Bayesian networks. In Proceedings of the 19th Conference on Uncertainty in Artificial Intelligence (pp. 592-599). San Francisco: Morgan Kaufmann.
45. Yamazaki, K., Nagata, K., & Watanabe, S. (2005). A new method of model selection based on learning coefficient. In Proceedings of the Inter national Symposium on Nonlinear Theory and Its Applications (pp. 389-392). Bruges, Belgium.