Neural networks for predicting conditional probability densities: Improved training scheme combining EM and RVFL

Neural Networks

Volumn 11, Issue 1, 1998, Pages 89-116

  Husmeier, Dirk ^a   Taylor, John G ^a  

^a NONE

Author keywords

Automatic relevance determination; Conditional probability density; Expectation maximization algorithm; Gaussian mixture model; Generalization performance; Maximum likelihood; Model complexity; Network committees; Random vector functional link net approach; Time series prediction

Indexed keywords

COMPUTATIONAL COMPLEXITY; COMPUTER ARCHITECTURE; COMPUTER SIMULATION; LEARNING ALGORITHMS; LEARNING SYSTEMS; MATHEMATICAL MODELS; MAXIMUM LIKELIHOOD ESTIMATION; PROBABILITY DENSITY FUNCTION; RANDOM PROCESSES; TIME SERIES ANALYSIS; VECTORS;

CONDITIONAL PROBABILITY DENSITY; EXPECTATION MAXIMIZATION (EM) ALGORITHMS; GAUSSIAN MIXTURE MODEL; RANDOM VECTOR FUNCTIONAL LINK NET APPROACH;

NEURAL NETWORKS;

ALGORITHM; ARTICLE; INFORMATION PROCESSING; NERVE CELL NETWORK; PREDICTION; PRIORITY JOURNAL; PROBABILITY;

EID: 0031913824     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0893-6080(97)00089-0     Document Type: Article

References (33)

1. Allen, D. & Taylor, J.G. (1994). Learning time series by neural networks. In M. Marinaro & P. Morasso (Eds), Proceedings of ICANN 94 (pp. 529-532). Berlin: Springer.
2. Bishop, C.M. (1994). Mixture density networks. Technical Report, Aston University, Birmimgham, UK.
3. Bishop, C.M. (1995). Neural networks for pattern recognition. New York: Oxford University Press.
4. Casdagli M., Eubank S., Farmer J.D., Gibson J. State space reconstruction in the presence of noise. Physica. D51:1991;52-98.
5. Dempster A.P., Laird N.M., Rubin D.B. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. B39(1):1997;1-38.
6. Harrison D., Rubinfeld D.L. Hedonic prices and the demand for clean air. Journal of Environmental Economics and Management. 5:1978;81-102.
7. Husmeier D., Taylor J.G. Predicting conditional probability densities of stationary stochastic time series. Neural Networks. 10(3):1997;479-497.
8. Igelnik B., Pao Y.H. Stochastic choice of basis functions in adaptive functional approximation and the functional-link net. IEEE Transactions on Neural Networks. 6:1995;1320-1329.
9. Jordan M.I., Jacobs R.A. Hierarchical mixtures of experts and the EM algorithm. Neural Computation. 6:1994;181-214.
10. Jordan M.I., Xu L. Convergence results for the EM approach to mixtures of experts architectures. Neural Networks. 8:1995;1409-1431.
11. Krogh A., Sollich P. Statistical mechanics of ensemble learning. Physical Review. E 55:1997;811-825.
12. LeBlanc, M. & Tibshirani, R. (1993). Combining estimates in regression and classification. Technical Report, Department of Preventive Medicine and Biostatistics, University of Toronto.
13. MacKay, D.J.C. (1992a). Bayesian interpolation. Neural Computation, 4, 415-447.
14. MacKay, D.J.C. (1992b). A practical Bayesian framework for backpropagation networks. Neural Computation, 4, 448-472.
15. MacKay, D.J.C. (1993). Bayesian non-linear modeling for the prediction competition. In G.R. Heidbreder (Ed.), Maximum entropy and Bayesian methods (pp. 221-234) Santa Barbara, CA: Kluwer Academic.
16. Moody, J. & Utans, J. (1992). Principled architecture selection for neural networks: application to corporate bond rating prediction. In J.E. Moody, S.J. Hanson & R.P. Lippman (Eds), Advances in neural information processing systems 4 (pp. 683-690), San Mateo, CA: Morgan Kaufmann.
17. Neal, R.M. (1995). Bayesian learning for neural networks. Ph.D. thesis, University of Toronto, Canada.
18. Neuneier, R., Hergert, F., Finnoff, W. & Ormoneit, D. (1994). Estimation of conditional densities: a comparison of neural network approaches. In M. Marinaro & P. Morasso (Eds), Proceedings of ICANN 94 (pp. 689-692). Berlin: Springer.
19. Nowlan, S.J. (1991) Soft competitive adaptation: neural network learning algorithms based on fitting statistical mixtures. Ph.D. thesis, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA.
20. Ormoneit, D. (1993). Estimation of probability densities using neural networks. Diplomarbeit, Fachbereich Informatik, Technische Universität München (D).
21. Ormoneit, D. & Tresp, V. (1996). Improved gaussian mixture density estimates using Bayesian penalty terms and network averaging. In D.S. Touretzky, M.C. Mozer & M.E. Hasselmo (Eds), Advances in Neural Information Processing Systems 8 (pp. 542-548). San Mateo, CA: Morgan Kaufmann.
22. Pao Y.H., Park G.H., Sobajic D.J. Learning and generalization characteristics of the random vector functional-link net. Neurocomputing. 6:1994;163-180.
23. Parzen E. On estimation of a probability density function and mode. Annals of Mathematical Statistics. 33:1962;1065-1076.
24. Press, W.H., Teukolsky, S.A., Vetterling, W.T. & Flannery, B.P. (1992). Numerical recipes in C. New York: Cambridge University Press.
25. Rögnvaldsson T. On langevin updating in multilayer perceptrons. Neural Computation. 6:1994;916-926.
26. Sharkey, A.J.C., Sharkey, N.E. & Gopinath, O.C. (1995). Diversity, neural nets and safety critical applications. In L. Niklasson & M. Boden (Eds), Current trends in connectionism (pp. 165-178). Hillsdale, NJ: Lawrence Erlbaum Associates.
27. Sharkey, A.J.C., Sharkey, N.E. & Chandroth, G.O. (1996). Neural nets and diversity. In Proceedings of the 14th International Conference on Computer Safety, Reliability and Security (pp. 375-389).
28. Sollich, P. & Krogh, A. (1996). Learning with ensembles: how over-fitting can be useful. In D.S. Tourezky, M.C. Mozer & M.E. Hasselmo (Eds), Advances in neural information processing systems 8 (pp. 190-196). San Mateo, CA: Morgan Kaufmann.
29. Specht D.F. Probabilistic neural networks. Neural Networks. 3(1):1990;109-118.
30. Srivastava, A.N. & Weigend, A.S. (1994). Computing the probability density in connectionist regression. In M. Marinaro & P. Morasso (Eds), Proceedings of ICANN 94 (pp. 685-688). Berlin: Springer.
31. Takens, F. (1981). Detecting strange attractors in fluid turbulence. In D. Rand & L.S. Young (Eds), Dynamical systems and turbulence. Berlin: Springer.
32. Weigend A.S., Srivastava A.N. Predicting conditional probability distributions: a connectionist approach. International Journal of Neural Systems. 6:1995;preprint.
33. Wolpert D.H. Stacked generalization. Neural Networks. 5(2):1992;241-259.