Fast training of recurrent networks based on the EM algorithm

IEEE Transactions on Neural Networks

Volumn 9, Issue 1, 1998, Pages 11-26

  Ma, Sheng ^b   Ji, Chuanyi ^a,b  

^a IEEE

^b RENSSELAER POLYTECHNIC INSTITUTE   (United States)

Author keywords

EM algorithm; Fast; Mean field approximation; Moving targets; Probability model; Recurrent networks

Indexed keywords

APPROXIMATION THEORY; ERROR ANALYSIS; LEARNING ALGORITHMS; LEARNING SYSTEMS; MATHEMATICAL MODELS; PROBABILITY DENSITY FUNCTION; REGRESSION ANALYSIS; TRANSFER FUNCTIONS; VECTORS;

EXPECTATION MAXIMIZATION (EM) ALGORITHMS; RECURRENT NETWORKS;

FEEDFORWARD NEURAL NETWORKS;

EID: 0031675929     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.655025     Document Type: Article

References (35)

1. S. Amari, K. Kurata, and H. Nagaoka, "Information geometry of Boltzmann machines," IEEE Trans. Neural Networks, vol. 3, pp. 260-271, 1992.
2. R. Andrews, J. Diederich, and A. B. Tickle, "A survey and crique of techniques for extracting rules from trained artificial neural networks," to appear in Knowledge-Based Systems, available FTP:ftp.qut.edu.au//pub/NRC/ps/QUTNRC-95-01-02.ps.Z
3. L. E. Baum, T. Petrie, G. Soules, and N. Weiss, "A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains," Annu. Math. Statist., vol. 41, pp. 164-171, 1970.
4. Y. Bengio, "Credit assignment through time: Alternative to backpropagation," Advances in Neural Inform. Processing Syst., vol. 6, pp. 75-82, 1994.
5. Y. Bengio, P. Frasconi, and P. Simard, "The problem of learning long-term dependencies in recurrent networks," IEEE Trans. Neural Networks, vol. 5, pp. 157-166, 1993.
6. L. E. Breiman and J. H. Friedman, "Function approximation using ramps," Neural Networks for Computing. Snowbird, UT: Snowbird, 1993.
7. W. Byne, "Alternating minimization and Boltzmann machine learning," IEEE Trans. Neural Networks, vol. 3, 1992.
8. T. M. Cover and J. Thomas, Elements of Information Theory. New York: Wiley, 1991.
9. S. Das and M. C. Mozor, "A unified gradient descent clustering architecture for finite-state machine induction," Advances in Neural Inform. Processing Syst., vol. 6, 1994.
10. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via EM algorithm," J. Roy. Statist. Soc., vol. 39, pp. 1-33, 1977.
11. S. E. Fahlman, "The recurrent cascade-correlation architecture," Advanices in Neural Inform. Processing Syst., vol. 3, pp. 190-196, 1991.
12. T. Grossman, "Learning, capacity and implementation of neural-network models," Ph.D. dissertation, Weizmann Institute of Science, 1992.
13. A. Herz, A. Krough, and P. G. Palmer, Introduction to the Theory of Neural Computation. Reading, MA: Addison-Wesley.
14. B. G. Home and C. L. Giles, "An experimental comparison of recurrent neural networks," Advances in Neural Inform. Processing Syst., vol. 7, 1995.
15. M. Jordan and R. A. Jacobs, "Hierarchical mixture of experts and the EM algorithm," Neural Computa., vol. 6, pp. 181-214, 1994.
16. A. Krough, G. I. Thorberaso, and J. A. Hertz, "A cost function for internal representations," Advances in Neural Inform. Processing Syst., vol. 2, pp. 733-745, 1990.
17. K. J. Lang, A. H. Waibel, and G. E. Hinton, "A time-delay neural-network architecture for isolated word recognition," Neural Networks, vol. 3, pp. 23-44, 1990.
18. E. Levin, N. Tishby, and S. A. Solla, "A statistical approach to learning and generalization in layered neural network," Proc. IEEE, vol. 78, no. 10, pp. 1568-1574, 1990.
19. S. Ma, C. Ji, and J. Farmer, "An efficient EM-based training algorithm for feedforward neural networks," Neural Networks, to be pulished.
20. D. J. Mackey, "A practical Bayesian framework for backpropagation networks," Neural Computa., vol. 4, pp. 448-472, 1992.
21. K. S. Narendra, "Adaptive control of dynamical systems using neural networks," in Handbook of Intelligent Control, D. A. White and D. A. Sofge, Eds. NY: Van Nostrand Reinhold, 1992.
22. C. Omlin and C. L. Giles, "Extraction of rules from discrete-time recurrent neural networks," Neural Networks, vol. 9, no. 1, p. 41, 1996.
23. J. B. Pollack, "The induction of dynamical recognizers," Machine Learning, vol. 7, no. 23, 1991.
24. R. Rohwer, "The moving target training algorithm," Advances in Neural Inform. Processing Syst., vol. 2, pp. 558-565, 1990.
25. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, vol. 1, Parallel Distributed Processing. Cambridge, MA: MIT Press, 1986, ch. 8.
26. D. Saad and E. Marom, "Learning by choice of internal representations: An energy minimization approach," Complex Syst., vol. 4, pp. 107-118, 1990.
27. L. K. Saul, T. Jaakkola, and M. I. Jordan, "Mean field theory for sigmoid belief networks," J. Artificial Intell. Res., vol. 4, pp. 61-76, 1996.
28. J. W. Shavlik, "Combining symbolic and neural learning," Machine Learning, vol. 14, no. 3, 1994.
29. R. S. Shadafan and M. Niranjan, "A dynamic neural network architecture by sequential partitioning of the input space," Neural Computa., vol. 6, pp. 1203-1222, 1994.
30. E. Sontag, "Systems combining linearity and saturations and relations to neural networks," Rutgers Center for Syst. and Contr., Tech. Rep. SYCON92-01, 1992.
31. P. Tino and J. Sajda, "Learning and extracting initial mealy machines with a modular neural network model," Neural Computa., vol. 7, no. 4, 1995.
32. P. Werbos, "Beyond regression," Ph.D dissertation, Harvard Univ., Cambridge, MA, 1974.
33. R. J. Williams and J. Peng, "An efficient gradient-based algorithm of online training of recurrent network trajectories," Neural Computa., vol. 2, pp. 491-501, 1990.
34. R. J. Williams and D. Zipser, "A learning algorithm for continually running fully recurrent neural networks," Neural Computa., vol. 1, pp. 270-280, 1989.
35. J. Zhang, "The mean-field theory in EM procedures for Markov random fields," IEEE Trans. Signal Processing, vol. 40, pp. 2570-2583, 1992.