BYY learning, regularized implementation, and model selection on modular networks with one hidden layer of binary units

Neurocomputing

Volumn 51, Issue , 2003, Pages 277-301

  Xu, Lei ^a  

^a CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Binary hidden layer; BYY modular system; Conflict and redundancy; Harmony learning; Hidden unit number; Mixture of experts; Model selection; Regularization; Rule extraction

Indexed keywords

DATA REDUCTION; LEARNING ALGORITHMS; NEURAL NETWORKS;

MODULAR NETWORKS;

NEUROLOGY;

ALGORITHM; ARTICLE; AUTOMATION; BAYES THEOREM; CLUSTER ANALYSIS; DATA ANALYSIS; EXPERIMENTAL MODEL; FACTORIAL ANALYSIS; LEARNING; MATHEMATICAL COMPUTING; NORMAL DISTRIBUTION; PATTERN RECOGNITION; PRINCIPAL COMPONENT ANALYSIS; PRIORITY JOURNAL; REGRESSION ANALYSIS;

EID: 0037380850     PISSN: 09252312     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0925-2312(02)00622-7     Document Type: Article

References (41)

1. H. Akaike, A new look at the statistical model identification, IEEE Trans. Automat. Control 19 (1974) 714-723.
2. T.W. Anderson, H. Rubin, Statistical inference in factor analysis, Proceedings of the Berkeley Symposium on Mathematical and Statistical Probability 3rd 5, UC Berkeley, 1956, pp. 111-150.
3. H.B. Barlow, Unsupervised learning, Neural Comput. 1 (1989) 295-311.
4. C.M. Bishop, Training with noise is equivalent to Tikhonov regularization, Neural Comput. 7 (1995) 108-116.
5. H. Bourlard, Y. Kamp, Auto-association by multilayer perceptrons and singular value decomposition, Biol. Cybernet. 59 (1988) 291-294.
6. H. Bozdogan, Model selection and akaike's information criterion: the general theory and its analytical extension Psychometrika 52 (1987) 345-370.
7. K.C. Chiu, L. Xu, A comparative study of Gaussian TFA learning and statistical tests for determination of factor number in APT, Proceedings of IJCNN '02, Honolulu, Hawaii, USA, May 12-17, 2002, pp. 2243-2248.
8. P. Dayan, R.S. Zemel, Competition and multiple cause models, Neural Comput. 7 (1995) 565-579.
9. A.P. Dempster, et al., Maximum-likelihood from incomplete data via the EM algorithm, J. Roy. Statist. Soc. B39 (1977) 1-38.
10. L. Devroye, et al., A Probability Theory of Pattern Recognition, Springer, Berlin, 1996.
11. G.E. Hinton, R. S. Zemel, Autoencoders, minimum description length and Helmholtz free energy, Adv. NIPS 6 (1994) 3-10.
12. R.A. Jacobs, et al., Adaptive mixtures of local experts, Neural Comput. 3 (1991) 79-87.
13. M.I. Jordan, R.A. Jacobs, Hierarchical mixtures of experts and the EM algorithm, Neural Comput. 6 (1994) 181-214.
14. M.I. Jordan, L. Xu, Convergence results for the EM approach to mixtures of experts, Neural Networks 8 (1995) 1409-1431.
15. J. Karhunen, J. Joutsensalo, Representation and separation of signals using nonlinear PCA type learning, Neural Networks 7 (1994) 113-127.
16. S. Kirkpatrick, et al., Optimization by simulated annealing, Science 220 (1983) 671-680.
17. Z.B. Lai, P. Guo, T.J. Wang, L. Xu, Comparison on Bayesian Ying-Yang theory based clustering number selection criterion with information theoretical criteria, Proceedings of IJCNN98, Vol. I, Anchorage, Alaska, May 5-9, 1998, pp. 725-729.
18. W.K. Lam, L. Xu, An experimental comparison of the Bayesian YING-YANG criteria and cross validation for selection on number of hidden units in feedforward networks, Proceedings of ICASSP98, Vol. 2, Seattle, WA, May 12-15, 1998, pp. 1189-1192.
19. W.K. Lam, L. Xu, An experimental comparison of the Bayesian Ying-Yang criteria and cross validation on experts number selection in original and alternative model for mixture of experts. Proceeding of ICONIP'98, Vol. 1, Kitakyushu, Japan, October 21-23, 1998, pp. 71-44.
20. Z.Y. Liu, K.C. Chiu, L. Xu, Local subspace analysis for astronomical object detection and star/galaxy classification, Proceedings of IJCNN'02, Honolulu, Hawaii, USA, May 12-17, 2002, pp. 962-967.
21. D.J.C. Mackey, A practical Bayesian framework for backpropagation, Neural Comput. 4 (1992) 415-447.
22. O. Ning, W.K. Lam, K. Yamauchi, L. Xu, Using an improved back propagation learning method to diagnose the sites of cardiac, hypertrophy, MD Comput. 16 (1) (1999) 79-81.
23. V. Ramamurti, J. Ghosh, Regularization and error bars for the mixture of experts network, Proceedings of IEEE ICNN97, 1997, pp. 221-225.
24. R.A. Redner, H.F. Walker, Mixture densities, maximum likelihood, and the EM algorithm, SIAM Rev. 26 (1984) 195-239.
25. J. Rissanen, Stochastic complexity, J. Roy. Statist. Soc. 49 (3) (1987) 223-239.
26. J. Rissanen, Stochastic Complexity in Statistical Inquiry, World Scientific, Singapore, 1989.
27. I. Rivals, L. Personnaz, On cross validation for model selection, Neural Comput. 11 (1999) 863-870.
28. D.E. Rumelhart, et al., Learning internal representations by error propagation, Parallel Distributed Processing, Vol. 1, MIT Press, Cambridge, MA, 1986.
29. E. Saund, A multiple cause mixture model for unsupervised learning, Neural Comput. 7 (1995) 51-71.
30. A.N. Tikhonov, V.Y. Arsenin, Solutions of Ill-posed Problems, V.H. Winston and Sons, Washington, DC, 1977.
31. V.N. Vapnik, The Nature of Statistical Learning Theory, Springer, Berlin, 1995.
32. C.S. Wallace, D.M. Boulton, An information measure for classification, Comput. J. 11 (1968) 185-194.
33. C.S. Wallace, D.R. Dowe, Minimum message length and Kolmogorov complexity, Comput. J. 42 (4) (1999) 270-280.
34. L. Xu, Least mean square error reconstruction for self-organizing neural-nets, Neural Networks 6 (1993) 627-648. Its early version on Proceedings of 1991 International Joint Conference on Neural Networks (IJCNN91' Singapore), 1991, pp. 2363-2373.
35. L. Xu, Bayesian Ying-Yang machine, clustering and number of clusters, Pattern Recognition Lett. 18 (11-13) (1997) 1167-1178.
36. L. Xu, RBF nets, mixture experts, and Bayesian Ying-Yang learning, Neurocomput. 19 (1-3) (1998) 223-257.
37. L. Xu, Temporal BYY learning for state space approach, hidden Markov model and blind source separation, IEEE Trans. Signal Process. 48 (7) (2000) 2132-2144.
38. L. Xu, BYY harmony learning, independent state space and generalized APT financial analyses, IEEE Trans. Neural Networks 12 (4) (2001) 822-847.
39. L. Xu, Bayesian Ying Yang harmony learning, in: M.A. Arbib (Ed.), The Handbook of Brain Theory and Neural Networks, 2nd Edition, The MIT Press, Cambridge, MA, 2002, in press.
40. L. Xu, C.C. Cheung, S.I. Amari, Learned parametric mixture based ICA algorithm, Neurocomput. 22(1-3) (1998) 69-80. A part of its preliminary version on Proceedings of ESANN97, Bruges, April 16-18, 1997, pp. 291-296.
41. L. Xu, M.I. Jordan, G.E. Hinton, An alternative model for mixtures of experts, Adv. Neural Inform. Process. Systems, 7 (1995) 633-640. Its preliminary version on Proc. WCNN'94 2 (1994) 405-410.