Links between probabilistic automata and hidden Markov models: Probability distributions, learning models and induction algorithms

Pattern Recognition

Volumn 38, Issue 9, 2005, Pages 1349-1371

  Dupont, P ^a   Denis, F ^b   Esposito, Y ^b  

^a UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

^b AIX MARSEILLE UNIV   (France)

Author keywords

Bayesian learning; Grammar induction; Hidden Markov models; HMM topology learning; Induction algorithms; PAC learning; Probabilistic automata

Indexed keywords

ALGORITHMS; COMPUTATIONAL GRAMMARS; ERROR CORRECTION; LEARNING SYSTEMS; MARKOV PROCESSES; MATHEMATICAL MODELS; PROBABILITY DISTRIBUTIONS; TOPOLOGY;

BAYESIAN LEARNING; GRAMMAR INDUCTION; HIDDEN MARKOV MODELS (HMM); HMM TOPOLOGY LEARNING; INDUCTION ALGORITHMS; PAC LEARNING; PROBABILISTIC AUTOMATA;

AUTOMATA THEORY;

EID: 19944415906     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2004.03.020     Document Type: Article

References (73)

1. L. Bahl, P. Brown, P. de Souza, and R. Mercer Estimating hidden Markov models parameters so as to maximize speech recognition accuracy IEEE Trans. Speech Audio Process. 1 1 1993 77 83
2. K.F. Lee, Large vocabulary speaker independent continuous speech recognition: the SPHINX system, Ph.D. Dissertation, Computer Science Department, Carnegie Mellon University, 1988.
3. L. Rabiner, and B.-H. Juang Fundamentals of Speech Recognition 1993 Prentice-Hall Englewood Cliffs, NJ
4. F. Jelinek Statistical Methods for Speech Recognition 1998 MIT Press Cambridge, MA
5. R. Durbin, S. Eddy, A. Krogh, and G. Mitchison Biological Sequence Analysis 1998 Cambridge University Press Cambridge
6. P. Baldi, and S. Brunak Bioinformatics A Machine Learning Approach second ed. 2001 MIT Press
7. K. Seymore, A. McCallum, and R. Rosenfeld Learning hidden markov model structure for information extraction AAAI'99 Workshop on Machine Learning for Information Extraction Orlando, Florida 1999 37 42
8. E. Levin, and R. Pieraccini Planar hidden markov modeling: from speech to optical character recognition C.L. Giles S.J. Hanton J.D. Cowan Advances in Neural Information Processing Systems vol. 5 1993 Morgan Kauffman Los Altos, CA 731 738
9. K.S. Fu, and T.L. Booth Grammatical inference: introduction and survey, Part 1 IEEE T. Syst. Man Cyb. 5 1975 85 111
10. L. Miclet, Grammatical inference, in: H. Bunke, A. Sanfeliu (Eds.), Syntactic and Structural Pattern Recognition: Theory and Applications, Series in Computer Science, vol. 7, World Scientific, Singapore, 1990, pp. 237-290.
11. E. Vidal, P. Casacuberta, P. García, Grammatical inference and applications to automatic speech recognition, in: A.J. Rubio, J.M. López (Eds.), NATO ASI, Speech Recognition and Coding, New Advances and Trends, 1995, pp. 174-191.
12. Y. Sakakibara Recent advances of grammatical inference Theoret. Comput. Sci. 185 1 1997 15 45
13. J.J. Horning, A study of grammatical inference, Ph.D. Dissertation, Computer Science Department, Stanford University, Stanford, California, 1969.
14. K.S. Fu, and T.L. Booth Grammatical inference: introduction and survey, Part 2 IEEE T. Syst. Man Cyb. 5 1975 409 423
15. F.J. Maryanski, Inference of probabilistic grammars, Ph.D. Dissertation, University of Connecticut, 1974.
16. A. Van der Mude, and A. Walker On the inference of stochastic regular grammars Inf. Control 38 1978 310 329
17. H. Rulot, and E. Vidal An efficient algorithm for the inference of circuit-free automata G. Ferratè T. Pavlidis A. Sanfeliu H. Bunke Advances in Structural and Syntactic Pattern Recognition 1988 NATO ASI Springer, Berlin 173 184
18. R. Carrasco, J. Oncina, Learning stochastic regular grammars by means of a state merging method, in: Grammatical Inference and Applications, ICGI'94, Lecture Notes in Artificial Intelligence, vol. 862, Alicante, Spain, 1994. Springer, Berlin, 1994, pp. 139-150.
19. D. Ron, Y. Singer, and N. Tishby Learning probabilistic automata with variable memory length Proceedings of the Seventh Annual Conference on Computational Learning Theory 1994 New Brunswick, NJ ACM Press
20. D. Ron, Y. Singer, and N. Tishby On the learnability and usage of acyclic probabilistic automata Proceedings of the Eighth Annual Conference on Computational Learning Theory 1995 Santa Cruz, CA ACM Press 31 40
21. R. Carrasco, and J. Oncina Learning deterministic regular grammars from stochastic samples in polynomial time Theoret. Informatics Appl. 33 1 1999 1 19
22. F. Thollard, P. Dupont, and C. de la Higuera Probabilistic DFA inference using Kullback-Leibler divergence and minimality Seventeenth International Conference on Machine Learning 2000 Morgan Kauffman Los Altos, CA 975 982
23. H. Huang, and K.S. Fu On stochastic context-free languages Inf. Sci. 3 1971 201 224
24. C. Wetherell Probabilistic languages: a review and some open questions Comput. Surv. 12 4 1980 361 379
25. K.S. Fu, Syntactic Methods in Pattern Recognition, Mathematics in Science and Engineering, vol. 112, Academic Press, New York, 1974.
26. R.C. Gonzalez, and M.G. Thomason Syntacic Pattern Recognition, An Introduction 1978 Addision-Wesley Reading, MA
27. A. Paz Introduction to Probabilistic Automata 1971 Academic Press New York
28. F. Casacuberta Some relations among stochastic finite state networks used in automatic speech recognition IEEE Trans. Pattern Anal. Mach. Intell. 12 7 1990 691 695
29. V. Balasubramanian, Equivalence and reduction of hidden Markov models, Ph.D. Thesis, MIT, Cambridge, MA, 1993.
30. N. Abe, and M. Warmuth On the computational complexity of approximating distributions by probabilistic automata Mach. Learn. 9 1992 205 260
31. M.J. Kearns, Y. Mansour, D. Ron, R. Rubinfeld, R.E. Schapire, and L. Sellie On the learnability of discrete distributions Proceedings of the 25th Annual ACM Symposium on Theory of Computing 1994 273 282
32. L. Bahl, F. Jelinek, and R. Mercer A maximum likelihood approach to continuous speech recognition IEEE Trans. Pattern Anal. Mach. Intell. 5 2 1983 179 190
33. A. Poritz Hidden markov models: a guided tour International Conference on Acoustic, Speech and Signal Processing 1988 7 13
34. L. Rabiner A tutorial on hidden markov models and selected applications in speech recognition Proc. IEEE 77 2 1989 257 286
35. L.G. Valiant A theory of the learnable Commun. Assoc. Comput. Mach. 27 11 1984 1134 1142
36. T. Cover, and J. Thomas Elements of Information Theory 1991 Wiley New York
37. A.R. Barron, T.M. Cover, Minimum complexity density estimation, IEEE Trans. Inf. Theory (1991) 1034-1054.
38. E.M. Gold Language identification in the limit Inf. Control 10 5 1967 447 474
39. D. Angluin, Identifying languages from stochastic examples, Technical Report YALEU/DCS/RR-614, Yale University, March 1988.
40. C. de la Higuera, F. Thollard, Identification in the limit with probability one of stochastic deterministic finite automata, in: A. Oliveira (Ed.), Grammatical Inference: Algorithms and Applications, Lecture Notes in Artificial Intelligence, number 1891, Springer, Berlin, 2000, pp. 15-24.
41. T. Mitchell Machine Learning 1997 McGraw-Hill New York
42. D. Ron, and R. Rubinfeld Learning fallible deterministic finite automata Mach. Learn. 18 1995 149 185
43. L. Baum An inequality and associated maximization technique in statistical estimation for probabilistic functions of Markov processes Inequalities 3 1972 1 8
44. A. Dempster, N. Laird, and D. Rubin Maximum likelihood from incomplete data via the EM algorithm J. R. Stat. Soc. Ser. B (Methodological) 39 1977 1 38
45. Y. Esposito, A. Lemay, F. Denis, P. Dupont, Learning probabilistic residual finite automata, in: P. Adriaans, H. Fernau, M. van Zaanen (Eds.), Proceedings of the Sixth International Colloquium on Grammatical Inference: Algorithms and Applications, Lecture Notes in Artificial Intelligence, number 2484, Amsterdam, The Netherlands, September 2002, Springer, Berlin, pp. 77-91.
46. W.-G. Tzeng The equivalence and learning of probabilistic automata 30th Annual Symposium on Fundations of Computer Science 1989 268 273
47. N. Abe, J. Takeuchi, and M.K. Warmuth Polynomial learnability of probabilistic concepts with respect to Kullback-Leibler divergence Proceedings of the Fourth Annual Workshop on Computational Learning Theory 1991 277 289
48. K. Yamanishi A learning criterion for stochastic rules Mach. Learn. 9 1992 165 203
49. M. Kearns, R. Schapire, Efficient distribution-free learning of probabilistic concepts, in: S.J. Hanson, G.A. Drastal, R.L. Rivest (Eds.), Computational Learning Theory and Natural Learning Systems, Volume I: Constraints and Prospect, vol. 1, MIT Press, Cambridge, MA, 1994.
50. L. Baum, T. Petrie, G. Soules, and N. Weiss A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains Ann. Math. Stat. 41 1 1970 164 171
51. A.J. Viterbi Error bounds for convutional codes and asymptotically optimal decoding algorithm IEEE Trans. Inf. Theory 13 1967 260 269
52. G.D. Forney The Viterbi algorithm IEEE Proc. 3 1973 268 278
53. Y. Kamp, An introduction to the Baum and EM algorithms for maximum likelihood estimation, Technical Report 830, Institute for Perception Research, 1991.
54. R. Chaudhuri, and A.N.V. Rao Approximating grammar probabilities: solution of a conjecture J. Assoc. Comput. Mach. 33 4 1986 702 705
55. J.K. Baker Trainable grammars for speech recognition D. Klatt J. Wolf Speech Communication Papers for the 97th Meeting of the Acoustical Society of America 1979 547 550
56. K. Lari, and S.J. Young The estimation of stochastic context-free grammars using the inside-outside algorithm Comput. Speech Lang. 4 1990 35 56
57. K. Lari, and S.J. Young Applications of stochastic context-free grammars using the inside-outside algorithm Comput. Speech Lang. 5 1991 237 257
58. T. Pao, and J. Carr A solution to the syntactic induction-inference problem for regular languages Comput. Lang. 3 1978 53 64
59. P. Dupont, L. Miclet, E. Vidal, What is the search space of the regular inference?, in: Grammatical Inference and Applications, ICGI'94, Lecture Notes in Artificial Intelligence, number 862, Alicante, Spain, Springer, Berlin, 1994, pp. 25-37.
60. W. Hoeffding Probability inequalities for sums of bounded random variables J. Am. Statist. Assoc. 58 301 1963 13 30
61. M. Young-Lai, and F.W. Tompa Stochastic grammatical inference of text database structure Mach. Learn. 40 2 2000 111 137
62. L. Hirschman Multi-site data collection for a spoken language corpus Proceedings of DARPA Speech and Natural Language Workshop 1992 Arden House NY 7 14
63. A. Stolcke, and S.M. Omohundro Hidden markov model induction by bayesian model merging C.L. Giles S.J. Hanton J.D. Cowan Advances in Neural Information Processing Systems 1993 Morgan Kauffman Los Altos, CA
64. A. Stolcke, Bayesian learning of probabilistic language models, Ph.D. Dissertation, University of California, 1994.
65. J. Takami, S. Sagayama, A successive state-splitting algorithm for efficient allophone modeling, in: Proceedings of the International Conference on Acoustics, Speech and Signal Processing, vol. 1, 1992, pp. 573-576.
66. M. Ostendorf, and H. Singer Hmm topology design using maximum likelihood successive state splitting Comput. Speech Lang. 11 1997 17 41
67. B. Stenger, V. Ramesh, N. Paragios, F. Coetzee, J. Buhmann, Topology free hidden markov models: application to background modeling, in: Proceedings of the Eight International Conference on Computer Vision, vol. 1, 2001, pp. 294-301.
68. M.E. Brand Structure learning in conditional probability models via an entropic prior and parameter extinction Neural Comput. J. 11 5 1999 1155 1182
69. R. Vasko, A. El-Jaroudi, and J. Boston An algorithm to determine hidden markov model topology International Conference on Acoustic, Speech and Signal Processing 1996
70. M. Thomason, and E. Granum Dynamic programming inference of markov networks from finite sets of sample strings IEEE Trans. Pattern Anal. Mach. Intell. 8 4 1986 491 501
71. P. Dupont, L. Chase, Using symbol clustering to improve probabilistic automation inference, in: Grammatical Inference, ICGI'98, Lecture Notes in Artificial Intelligence, number 1433, Ames, Iowa, Springer, Berlin, 1998, pp. 232-243.
72. P. Dupont, J.C. Amengual, Smoothing probabilistic automata: an error-correcting approach, in: A. Oliveira (Ed.), Grammatical Inference: Algorithms and Applications, Lecture Notes in Artificial Intelligence, number 1891, Springer, Berlin, 2000, pp. 51-64.
73. D. Llorens, J.-M. Vilar, and F. Casacuberta Finite state language models smoothed using n-grams Int. J. Pattern Recogn. Artif. Intell. 16 3 2002 275 289