Hierarchical Recurrent Neural Networks for Long-Term Dependencies

NIPS 1995: Proceedings of the 8th International Conference on Neural Information Processing Systems

Volumn , Issue , 1995, Pages 493-499

  El Hihi, Salah ^a   Bengio, Yoshua ^a,b  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

^b LUCENT TECHNOLOGIES   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DYNAMICAL SYSTEMS; RECURRENT NEURAL NETWORKS;

DETERMINISTIC DYNAMICAL SYSTEMS; HIDDEN-MARKOV MODELS; INPUT/OUTPUT HIDDEN MARKOV MODELS; LONG-TERM DEPENDENCIES; MODEL OUTPUTS; NETWORK MODELS; PRIORI KNOWLEDGE; PROBABILISTIC MODELS; RECURRENT NETWORKS; SEQUENTIAL DATA;

HIDDEN MARKOV MODELS;

EID: 85057230110     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (22)

1. Bengio, Y. and Frasconi, P. (1994). Credit assignment through time: Alternatives to back propagation. In Cowan, J., Tesauro, G., and Alspector, J., editors, Advances in Neural Information Processing Systems 6. Morgan Kaufmann.
2. Bengio, Y. and Frasconi, P. (1995a). Diffusion of context and credit information in markovian models. Journal of Artificial Intelligence Research, 3:223-244.
3. Bengio, Y. and Frasconi, P. (1995b). An input/output HMM architecture. In Tesauro, G., Touretzky, D., and Leen, T., editors, Advances in Neural Information Processing Systems 7, pages 427-434. MIT Press, Cambridge, MA.
4. Bengio, Y., LeCun, Y., and Henderson, D, (1994). Globally trained handwritten word recognizer using spatial representation, space displacement neural networks and hidden Markov models. In Cowan, J., Tesauro, G, and Alspector, J., editors, Advances in Neural Information Processing Systems 6, pages 937-944.
5. Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5 (2): 157-166.
6. Brugnara, F., DeMori, R, Giuliani, D., and Omologo, M. (1992). A family of parallel nidden markov models. In International Conference on Acoustics, Speech and Signal Processing, pages 377-370, New York, NY, USA. IEEE.
7. Daubechies, I. (1990). The wavelet transform, time-frequency localization and signal analysis. IEEE Transaction on Information Theory, 36 (5):961-1005.
8. Dayan, P. and Hinton, G. (1993). Feudal reinforcement learning. In Hanson, S. J., Cowan, J. D., and Giles, C. L., editors, Advances in Neural Information Processing Systems 5, San Mateo, CA. Morgan Kaufmann.
9. Frasconi, P., Gori, M., Maggini, M., and Soda, G. (1993). Unified integration of explicit rules and learning by example in recurrent networks. IEEE Transactions on Knowledge and Data Engineering. (in press).
10. Giles, C. L. and Omlin, C. W. (1992). Inserting rules into recurrent neural networks. In Kung, Fallside, Sorenson, and Kamm, editors, Neural Networks for Signal Processing II, Proceedings of the 1992 IEEE workshop, pages 13-22. IEEE Press.
11. Lang, K. J., Waibel, A. H., and Hinton, G. E. (1990). A time-delay neural network architecture for isolated word recognition. Neural Networks, 3:23-43.
12. LeCun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., and Jackei, L. (1989). Back propagation applied to handwritten zip code recognition. Neural Computation, 1:541-551.
13. Levinson, S., Rabiner, L., and Sondhi, M. (1983). An introduction to the application of the theory of probabilistic functions of a Markov process to automatic speech recognition. Bell System Technical Journal, 64 (4):1035-1074.
14. Lin, T., Home, B., Tino, P., and Giles, C. (1995). Learning long-term dependencies is not as difficult with NARX recurrent neural networks. Technical Report UMICAS-TR-95-78, Institute for Advanced Computer Studies, University of Mariland.
15. Rabiner, L. and Juang, B. (1986). An introduction to hidden Markov models. IEEE ASSP Magazine, pages 257-285.
16. Rumelhart, D., Hinton, G., and Williams, R. (1986). Learning internal representations by error propagation. In Rumelhart, D. and McClelland, J., editors, Parallel Distributed Processing, volume 1, chapter 8, pages 318-362. MIT Press, Cambridge.
17. Saul, L. and Jordan, M. (1995). Boltzmann chains and hidden markov models. In Tesauro, G., Touretzky, D., and Leen, T., editors, Advances in Neural Information Processing Systems 7, pages 435-442, MIT Press, Cambridge, MA.
18. Schmidhuber, J. (1992). Learning complex, extended sequences using the principle of history compression. Neural Computation, 4 (2):234-242.
19. Simard, P. and LeCun, Y. (1992). Reverse TDNN: An architecture for trajectory generation. In Moody, J., Hanson, S., and Lipmann, R., editors, Advances in Neural Information Processing Systems 4, pages 579-588, Denver, CO. Morgan Kaufmann, San Mateo.
20. Singh, S. (1992). Reinforcement learning with a hierarchy of abstract models. In Proceedings of the 10th National Conference on Artificial Intelligence, pages 202-207. MIT/AAAI Press.
21. Suaudeau, N. (1994). Un modèle probabiliste pour intégrer la dimension temporelle dans un système de reconnaissance automatique de la parole. PhD thesis, Université de Rennes I, France.
22. Sutton, R, (1995). TD models: modeling the world at a mixture of time scales. In Proceedings of the 12th International Conference on Machine Learning. Morgan Kaufmann.