Generalization performance of subspace bayes approach in linear neural networks

IEICE Transactions on Information and Systems

Volumn E89-D, Issue 3, 2006, Pages 1128-1138

  Nakajima, Shinichi ^a,b   Watanabe, Sumio ^a  

^a TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

^b NIKON CORPORATION   (Japan)

Author keywords

Empirical Bayes; James Stein; Neural networks; Reduced rank regression; Unidentifiable; Variational Bayes

Indexed keywords

APPROXIMATION THEORY; ERROR ANALYSIS; MATHEMATICAL MODELS; MAXIMUM LIKELIHOOD ESTIMATION; MONTE CARLO METHODS; REGRESSION ANALYSIS;

EMPIRICAL BAYES; JAMES-STEIN; REDUCED-RANK REGRESSION; VARIATIONAL BAYES;

NEURAL NETWORKS;

EID: 33645781604     PISSN: 09168532     EISSN: 17451361     Source Type: Journal    
DOI: 10.1093/ietisy/e89-d.3.1128     Document Type: Conference Paper

References (34)

1. S. Nakajima and S. Watanabe, "Generalization error of linear neural networks in an empirical Bayes approach," Proc. IJCAI, pp.804-810, Edinburgh, U.K., 2005.
2. J.A. Hartigan, "A Failure of likelihood ratio asymptotics for normal mixtures," Proc. Berkeley Conference in Honor of J. Neyman and J. Kiefer, pp.807-810, 1985.
3. S. Watanabe, "A generalized Bayesian framework for neural networks with singular fisher information matrices," Proc. NOLTA, pp.207-210, 1995.
4. S. Amari, H. Park, and T. Ozeki, "Geometrical singularities in the neuromanifold of multilayer perceptrons," Advances in NIPS, vol.14, pp.343-350, 2002.
5. K. Hagiwara, "On the problem in model selection of neural network regression in overrealizable scenario," Neural Comput., vol. 14, pp.1979-2002, 2002.
6. H. Akaike, "A new look at statistical model," IEEE Trans. Autom. Control, vol.19, no.6, pp.716-723, 1974.
7. G. Schwarz, "Estimating the dimension of a model," Annals of Statistics, vol.6, no.2, pp.461-464, 1978.
8. J. Rissanen, "Stochastic complexity and modeling," Annals of Statistics, vol.14, no.3, pp. 1080-1100, 1986.
9. P. Bickel and H. Chernoff, Asymptotic Distribution of the Likelihood Ratio Statistic in a Prototypical Non Regular Problem, pp.83-96, Wiley Eastern Limited, 1993.
10. A. Takemura and S. Kuriki, "Weights of chi-bar-square distribution for smooth or piecewise smooth cone alternatives," Annals of Statistics, vol.25, no.6, pp.2368-2387, 1997.
11. S. Kuriki and A. Takemura, "Tail probabilities of the maxima of multilinear forms and their applications," Annals of Statistics, vol.29, no.2, pp.328-371, 2001.
12. K. Fukumizu, "Likelihood ratio of unidentifiable models and multilayer neural networks," Annals of Statistics, vol.31, no.3, pp.833-851, 2003.
13. D. Dacunha-Castelle and E. Gassiat, "Testing in locally conic models, and application to mixture models," Probability and Statistics, vol.1, pp.285-317, 1997.
14. K. Fukumizu, "Generalization error of linear neural networks in unidentifiable cases," Proc. ALT, pp.51-62, Springer, 1999.
15. F.F. Baldi and K. Hornik, "Learning in linear neural networks: A survey," IEEE Trans. Neural Netw., vol.6, no.4, pp.837-858, 1995.
16. S. Watanabe, "Algebraic analysis for nonidentifiable learning machines," Neural Comput., vol.13, no.4, pp.899-933, 2001.
17. K. Yamazaki and S. Watanabe, "Resolution of singularities in mixture models and its stochastic complexity," Proc. ICONIP, pp.1355-1359, Singapore, 2002.
18. D. Rusakov and D. Geiger, "Asymptotic model selection for naive Bayesian networks," Proc. UAI, pp.438-445, Alberta, Canada, 2002.
19. K. Yamazaki and S. Watanabe, "Stochastic complexities of hidden Markov models," Proc. Neural Networks for Signal Processing XIII (NNSP), pp. 179-188, Toulouse, France, 2003.
20. K. Yamazaki and S. Watanabe, "Stochastic complexity of Bayesian networks," Proc. UAI, pp.592-599, Acapulco, Mexico, 2003.
21. M. Aoyagi and S. Watanabe, "The generalization error of reduced rank regression in Bayesian estimation," Proc. ISITA, pp. 1068-1073, Parma, Italy, 2004.
22. S. Watanabe, "Algebraic information geometry for learning machines with singularities," Advances in NIPS, vol.13, pp.329-336, 2001.
23. G.E. Hinton and D. van Camp, "Keeping neural networks simple by minimizing the description length of the weights," Proc. COLT, pp.5-13, 1993.
24. D.J.C. MacKay, "Developments in probabilistic modeling with neural networks - Ensemble learning," Proc. 3rd Ann. Symp. on Neural Networks, pp. 191-198, 1995.
25. H. Attias, "Inferring parameters and structure of latent variable models by variational Bayes," Proc. UAI, 1999.
26. Z. Ghahramani and M.J. Beal, "Graphical models and variational methods," in Advanced Mean Field Methods, pp. 161-177, MIT Press, 2001.
27. S. Nakajima and S. Watanabe, "Generalization error and free energy of variational Bayes approach of linear neural networks," Proc. ICONIP, pp.55-60, Taipei, Taiwan, 2005.
28. W. James and C. Stein, "Estimation with quadratic loss," Proc. 4th Berkeley Symp. on Math. Stat. and Prob., pp.361-379, 1961.
29. S. Watanabe and S. Amari, "Learning coefficients of layered models when the true distribution mismatches the singularities," Neural Comput., vol.15, pp. 1013-1033, 2003.
30. G.C. Reinsel and R.P. Velu, Multivariate Reduced-Rank Regression, Springer, 1998.
31. B. Efron and C. Morris, "Stein's estimation rule and its competitors - An empirical Bayes approach," J. Am. Stat. Assoc., vol.68, pp.117-130, 1973.
32. H. Akaike, "Likelihood and Bayes Procedure," in Bayesian Statistics, ed. J.M. Bemald, pp. 143-166, University Press, 1980.
33. R.E. Kass and D. Steffey, "Approximate Bayesian inference in conditionally independent hierarchical models (Parametric empirical Bayes models)," J. Am. Stat. Assoc., vol.84, pp.717-726, 1989.
34. K. W. Watcher, "The strong limits of random matrix spectra for sample matrices of independent elements," Ann. Prob., vol.6, pp. 1-18, 1978.