Input-output HMM's for sequence processing

IEEE Transactions on Neural Networks

Volumn 7, Issue 5, 1996, Pages 1231-1249

  Bengio, Yoshua ^b,c   Frasconi, Paolo ^a,d  

^a IEEE

^b UNIVERSITÉ DE MONTRÉAL   (Canada)

^c LUCENT TECHNOLOGIES   (United States)

^d UNIVERSITY OF FLORENCE   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTIVE SYSTEMS; ALGORITHMS; COMPUTATIONAL GRAMMARS; DATA PROCESSING; ESTIMATION; LEARNING SYSTEMS; MARKOV PROCESSES; MATHEMATICAL MODELS; PROBLEM SOLVING; STATE SPACE METHODS; STATISTICAL METHODS;

BENCHMARK PROBLEM; DISCRETE STATE MODEL; ESTIMATION MAXIMATION; GRAMMATICAL INFERENCE PROBLEMS; HIDDEN MARKOV MODEL; SEQUENCE PROCESSING; TEMPORAL CREDIT ASSIGNMENT; TOMITA GRAMMARS;

NEURAL NETWORKS;

EID: 0030242097     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.536317     Document Type: Article

References (61)

1. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. Lang, "Phoneme recognition using time-delay neural networks," IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 328-339, 1989.
2. D. Seidl and D. Lorenz, "A structure by which a recurrent neural network can approximate a nonlinear dynamic system," in Proc. Int. Joint Conf. Neural Networks, vol. 2, July 1991 pp. 709-714.
3. E. Sontag, "Systems combining linearity and saturations and relations to neural networks," Rutgers Center Syst. Contr., Tech. Rep. SYCON-92-01, 1992.
4. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing, D. E. Rumelhart and J. L. McClelland, Eds., vol. 1, ch. 8. Cambridge, MA: MIT Press, 1986, pp. 318-362.
5. B. A. Pearlmutter, "Learning state space trajectories in recurrent neural networks," Neural Computa., vol. 1, pp. 263-269, 1989.
6. G. Kuhn, "A first look at phonetic discrimination using connectionist models with recurrent links," CCRP - IDA SCIMP working paper 4/87, Inst. Defense Anal., Princeton, NJ, 1987.
7. A. J. Robinson and F. Fallside, "Static and dynamic error propagation networks with application to speech coding," in Neural Information Processing Systems, D. Z. Anderson, Ed. New York: Amer. Inst. Physics, 1988, pp. 632-641.
8. R. J. Williams and D. Zipser, "A learning algorithm for continually running fully recurrent neural networks," Neural Computa., vol. 1, pp. 270-280, 1989.
9. M. Gori, Y. Bengio, and R. De Mori, "BPS: A learning algorithm for capturing the dynamical nature of speech," in Proc. Int. Joint Conf. Neural Networks, Washington, D.C., 1989, pp. 643-644.
10. M. C. Mozer, "A focused backpropagation algorithm for temporal pattern recognition." Complex Syst., vol. 3, pp. 349-381, 1989.
11. P. Frasconi, M. Gori, and G. Soda, "Local feedback multilayered networks," Neural Computa., vol. 4, no. 1, pp. 120-130, 1992.
12. A. Tsoi and A. Back, "Locally recurrent globally feedforward networks, a critical review of architectures," IEEE Trans. Neural Networks, vol. 5, pp. 229-239, 1994.
13. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult," IEEE Trans. Neural Networks, vol. 5, pp. 157-166, 1994.
14. M. C. Mozer, "The induction of multiscale temporal structure," in Advances in Neural Information Processing Systems 4, J. Moody, S. Hanson, and R. Lipmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 275-282.
15. R. Rohwer, "The time dimension of neural-network models," ACM Sigart Bulleting, vol. 5, pp. 36-44, July 1994.
16. Y. Bengio, P. Frasconi, and P. Simard, "The problem of learning long-term dependencies in recurrent networks," in Proc. IEEE Int. Conf. Neural Networks, San Francisco, CA, 1993, pp. 1183-1195 (invited paper).
17. R. Rohwer, "The 'moving targets' training algorithm," in Advances in Neural Information Processing Systems, vol. 2, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, pp. 558-565.
18. Y. Bengio and P. Frasconi, "Credit assignment through time: Alternatives to backpropagation," in Advances in Neural Information Processing Systems 6, J. D. Cowan, G. Tesauro, and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann, 1994.
19. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum-likelihood from incomplete data via the EM algorithm," J. Roy. Statist. Soc. B, vol. 39, pp. 1-38, 1977.
20. L. E. Baum, T. Petrie, G. Soules, and M. Weiss, "A maximization technique occuring in the statistical analysis of probabilistic functions of Markov chains," Ann. Math. Statist., vol. 41, pp. 164-171, 1970.
21. S. E. Levinson, L. R. Rabiner, and M. M. Sondhi, "An introduction to the application of the theory of probabilistic functions of a Markov process to automatic speech recognition," Bell Syst. Tech. J., vol. 64, no. 4, pp. 1035-1074, 1983.
22. A. Poritz, "Hidden Markov models: A guided tour," in Proc. Int. Conf. Acoust., Speech, Signal Processing, 1988, pp. 7-13.
23. H. Leprieur and P. Haffner, "Discriminant learning with minimum memory loss for improved nonvocabulary rejection," in Proc. EUROSPEECH'95, Madrid, Spain, 1995.
24. Y. Bengio, Y. LeCun, and D. Henderson, "Globally trained handwritten word recognizer using spatial representation, space displacement neural networks and hidden Markov models," in Advances in Neural Information Processing Systems 6, J. D. Cowan, G. Tesauro, and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann, 1994, pp. 937-944.
25. G. J. McLachlan and K. E. Basford, Mixture Models: Inference and Applications to Clustering. New York: Marcel Dekker, 1988.
26. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, "Adaptive mixture of local experts," Neural Computa., vol. 3, pp. 79-87, 1991.
27. T. W. Cacciatore and S. J. Nowlan, "Mixtures of controllers for jump linear and nonlinear plants," in Advances in Neural Information Processing Systems 6, J. D. Cowan, G. Tesauro, and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann, 1994.
28. A. Kehagias, "Stochastic recurrent networks: Prediction and classification of time series," Brown Univ. Division Appl. Math., Providence, RI, Tech. Rep., 1991.
29. H. Bourlard and C. Wellekens, "Links between hidden Markov models and multilayer perceptions," IEEE Trans. Pattern Anal. Machine Intell., vol. 12, pp. 1167-1178, 1990.
30. H. Bourlard and N. Morgan, "Connectionist speech recognition: A hybrid approach," in The Kluwer International Series in Engineering Computer Science, vol. 247. Boston, MA: Kluwer, 1993.
31. Y. Bengio, R. De Mori, G. Flammia, and R. Kompe, "Global optimization of a neural network-hidden Markov model hybrid," IEEE Trans. Neural Networks, vol. 3, pp. 252-259, 1992.
32. E. Levin, "Word recognition using hidden control neural architecture," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Processing, Albuquerque, NM, pp. 433-436.
33. J. Bridle, "Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition," in Neuro-Computing: Algorithms, Architectures, and Applications, F. Fogelman-Soulie and J. Hérault, Eds. New York: Springer-Verlag, 1989.
34. J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Mateo, CA: Morgan Kaufmann, 1988.
35. J. S. Bridle, "Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters," in Advances in Neural Information Processing Systems, vol. 2, D. S. Touretzky, Ed. San Mateo. CA: Morgan Kaufmann, 1989, pp. 211-217.
36. P. Baldi and Y. Chauvin, "Smooth on-line learning algorithms for hidden Markov models," Neural Computa., vol. 6, no. 4, pp. 307-318, 1994.
37. L. R. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition," Proc. IEEE, vol. 77, no. 2, pp. 257-286, 1989.
38. P. Brown, "The acoustic-modeling problem in automatic speech recognition," Ph.D. dissertation, Dep. Comput. Sci., Carnegie-Mellon Univ., Pittsburgh, PA, 1987.
39. K. Hornik, M. Stinchcombe, and H. White, "Multilayer feedforward networks are universal approximators," Neural Networks, vol. 2, pp. 359-366, 1989.
40. C. L. Giles and C. W. Omlin, "Inserting rules into recurrent neural networks," in Neural Networks for Signal Processing II, S. Y. Kung, Fallside, F. Sorenson, and J. A. Kamm, Eds., 1992, pp. 13-22.
41. G. Z. Sun, H. H. Chen, Y. C. Lee, and C. L. Giles, "Recurrent neural networks, hidden Markov models and stochastic grammars," in Proc. Int. Joint Conf. Neural Networks, vol. I, San Diego, CA, 1990, pp. 729-734.
42. M. I. Jordan and R. A. Jacobs, "Hierarchies of adaptive experts," in Advances in Neural Information Processing Systems 4, J. Moody, S. Hanson, and R. Lipmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 985-992.
43. J. L. Elman, "Finding structure in time," Cognitive Sci., vol. 14, pp. 179-211, 1990.
44. Y. Bengio and P. Frasconi, "Diffusion of context and credit information in Markovian models," J. Artificial Intell. Res., vol. 3, pp. 249-270, 1995. [Online] Available gopher://p.gp.cs.cmu.edu/00/volume3/bengio95a.ps.Z.
45. E. Seneta, Nonnegative Matrices and Markov Chains. New York: Springer-Verlag, 1981.
46. Y. S. Abu-Mostafa, "Learning from hints in neural networks," J. Complexity, vol. 6, pp. 192-198, 1990.
47. G. G. Towell, J. W. Shavlik, and M. O. Noordewier, "Refinement of approximate domain theories by knowledge-based neural networks," in Proc. 8th Nat. Conf. Artificial Intell., Boston, MA, 1990, pp. 861-866.
48. V. Tresp, J. Hollatz, and S. Ahmad, "Network structuring and training using rule-based knowledge," in Advances in Neural Information Processing Systems 5, S. J. Hanson, J. D. Cowan, and C. L. Giles, Eds. San Mateo, CA: Morgan Kaufman, 1993.
49. C. W. Omlin and C. L. Giles, "Training second-order recurrent neural networks using hints," in Machine Learning: Proc. 9th Int. Conf. Artificial Intell, D. Sleeman and P. Edwards, Eds. San Mateo, CA: Morgan Kaufmann, 1992.
50. R. Maclin and J. W. Shavlik, "Refining domain theories expressed as finite-state automata," in Machine Learning: Proc. 8th Int. Wkshp., L. Birnbaum and G. Collins, Eds. San Mateo, CA: Morgan Kaufmann, 1991.
51. P. Frasconi, M. Gori, M. Maggini, and G. Soda, "Unified integration of explicit rules and learning by example in recurrent networks," IEEE Trans. Knowledge Data Eng., vol. 7, pp. 340-346, 1995.
52. R. Bakis, "Continuous speech recognition via centisecond acoustic states," in Proc. 19th Mtg. Acoust. Soc. Amer., Apr. 1976.
53. S. Becker and Y. L. Cun, "Improving the convergence of backpropagation learning with second-order methods," in Proc. 1988 Connectionist Models Summer School, D. Touretzky, G. Hinton, and T. Sejnowski, Eds. San Mateo, CA: Morgan Kaufmann, 1989, pp. 29-37.
54. D. Angluin and C. Smith, "Inductive inference: Theory and methods," Computing Surveys, vol. 15, no. 3. pp. 237-269, 1983.
55. C. L. Giles, C. B. Miller, D. Chen, H. H. Chen, G. Z. Sun, and Y. C. Lee, "Learning and extracted finite state automata with second-order recurrent neural networks," Neural Computa., vol. 4, no. 3, pp. 393-405, 1992.
56. J. B. Pollack, "The induction of dynamical recognizers," Machine Learning, vol. 7, no. 2, pp. 196-227, 1991.
57. R. L. Watrous and G. M. Kuhn, "Induction of finite-state languages using second-order recurrent networks," Neural Computa., vol. 4, no. 3, pp. 406-414, 1992.
58. P. Frasconi, M. Gori, M. Maggini, and G. Soda, "Representation of finite state automata into recurrent radial basis function networks," Machine Learning, vol. 23, pp. 5-32, 1996.
59. S. Das and M. C. Mozer, "A unified gradient-descent/clustering architecture for finite state machine induction," in Advances in Neural Information Processing Systems 6, J. D. Cowan, G. Tesauro, and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann, 1994.
60. M. Tomita, "Dynamic construction of finite-state automata from examples using hill-climbing," in Proc. 4th Annu. Cognitive Sci. Conf., Ann Arbor, MI, 1982, pp. 105-108.
61. J. E. Hopcroft and J. D. Ullman, Introduction to Automata Theory, Languages, and Computation. Reading, MA: Addison-Wesley, 1979.