Stable encoding of finite-state machines in discrete-time recurrent neural nets with Sigmoid units

Neural Computation

Volumn 12, Issue 9, 2000, Pages 2129-2174

  Carrasco, Rafael C ^a   Forcada, Mikel L ^a   Valdés Muñoz, M Ángeles ^a   Ñeco, Ramón P ^b  

^a UNIVERSITY OF ALICANTE   (Spain)

^b MIGUEL HERNÁNDEZ UNIVERSITY   (Spain)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; BIOLOGICAL MODEL; NERVE CELL NETWORK; STATISTICAL MODEL;

MODELS, NEUROLOGICAL; MODELS, STATISTICAL; NERVE NET;

EID: 0034268994     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/089976600300015097     Document Type: Article

References (56)

1. Alon, N., Dewdney, A. K., & Ott, T. J. (1991). Efficient simulation of finite automata by neural nets. Journal of the Association of Computing Machinery, 38, 495-514.
2. Alquézar, R., & Sanfeliu, A. (1995). An algebraic framework to represent finite state automata in single-layer recurrent neural networks. Neural Computation, 7, 931-949.
3. Arai, K.-I., & Nakano, R. (1996). Annealing RNN learning of finite state automata. In Proceedings of ICANN'96 (pp. 519-524).
4. Arai, K.-I., & Nakano, R. (1997). Adaptive β-scheduling learning method of finite state automata by recurrent neural networks. In Progress in connectionist-based information systems: Proceedings of the 1997 International Conference on Neural Information Processing and Intelligent Information Systems (Vol. 1, pp. 351-354). Singapore: Springer-Verlag.
5. Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157-166.
6. Blair, A., & Pollack, J. B. (1997). Analysis of dynamical recognizers. Neural Computation, 9(5), 1127-1142.
7. Carrasco, R. C., Forcada, M. L., & Santamaría, L. (1996). Inferring stochastic regular grammars with recurrent neural networks. In L. Miclet & C. de la Higuera (Eds.), Grammatical inference: Learning syntax from sentences (pp. 274-281). Berlin: Springer-Verlag.
8. Carrasco, R. C., Oncina, J., & Forcada, M. L. (1999). Efficient encodings of finite automata in discrete-time recurrent neural networks. In Proceedings of ICANN'99, International Conference on Artificial Neural Networks (Vol. 2, pp. 673-677).
9. Casey, M. (1996). The dynamics of discrete-time computation, with application to recurrent neural networks and finite state machine extraction. Neural Computation, 8(6), 1135-1178.
10. Casey, M. (1998). Correction to proof that recurrent neural networks can robustly recognize only regular languages. Neural Computation, 10(5), 1067-1069.
11. Cleeremans, A., Servan-Schreiber, D., & McClelland, J. L. (1989). Finite state automata and simple recurrent networks. Neural Computation, 1(3), 372-381.
12. Das, S., & Das, R. (1991). Induction of discrete state-machine by stabilizing a continuous recurrent network using clustering. Computer Science and Informatics, 21(2), 35-40.
13. Das, S., & Mozer, M. (1998). Dynamic on-line clustering and state extraction: An approach to symbolic learning. Neural Networks, 11(1), 53-64.
14. Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14, 179-211.
15. Floyd, T. L. (1996). Digital fundamentals (6th ed.). Englewood Cliffs, NJ: Prentice-Hall.
16. Forcada, M. L., & Carrasco, R. C. (1995). Learning the initial state of a second-order recurrent neural network during regular-language inference. Neural Computation, 7(5), 923-930.
17. Frasconi, P., Gori, M., Maggini, M., & Soda, G. (1996). Representation of finite-state automata in recurrent radial basis function networks. Machine Learning, 23, 5-32.
18. Giles, C. L., Miller, C. B., Chen, D., Chen, H. H., Sun, G. Z., & Lee, Y. C. (1992). Learning and extracted finite state automata with second-order recurrent neural networks. Neural Computation, 4(3), 393-405.
19. Gori, M., Maggini, M., Martinelli, E., & Soda, G. (1998). Inductive inference from noisy examples using the hybrid finite state filter. IEEE Transactions on Neural Networks, 9(3), 571-575.
20. Goudreau, M. W., Giles, C. L., Chakradhar, S. T., & Chen, D. (1994). First-order vs. second-order single layer recurrent neural networks. IEEE Transactions on Neural Networks, 5(3), 511-513.
21. Haykin, S. (1998). Neural networks - A comprehensive foundation (2nd ed.). Upper Saddle River, NJ: Prentice-Hall.
22. Hertz, J., Krogh, A., & Palmer, R. G. (1991). 1991. Introduction to the theory of neural computation. Redwood City, CA: Addison-Wesley.
23. Hopcroft, J. E., & Ullman, J. D. (1979). Introduction to automata theory, languages, and computation. Reading, MA: Addison-Wesley.
24. Home, B. G., & Giles, C. L. (1995). An experimental comparison of recurrent neural networks. In G. Tesauro, D. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems, 7 (pp. 697-704). Cambridge, MA: MIT Press.
25. Home, B. G., & Hush, D. R. (1996). Bounds on the complexity of recurrent neural network implementations of finite state machines. Neural Networks, 9(2), 243-252.
26. Hush, D., & Home, B. (1993). Progress in supervised neural networks. IEEE Signal Processing Magazine, 10(1), 8-39.
27. Indyk, P. (1995). Optimal simulation of automata by neural nets. In Proceedings of the 12th Annual Symposium on Theoretical Aspects of Computer Science (pp. 337-348). Berlin: Springer-Verlag.
28. Kleene, S. C. (1956). Representation of events innerve nets and finite automata. In C. E. Shannon & J. McCarthy (Eds.), Automata studies Princeton, NJ: Princeton University Press.
29. Kolen, J. F. (1994). Fool's gold: Extracting finite state machines from recurrent network dynamics. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems, 6 (pp. 501-508). San Mateo, CA: Morgan Kaufmann.
30. Kolen, J. F., & Pollack, J. B. (1995). The observer's paradox: Apparent computational complexity in physical systems. Journal of Experimental and Theoretical Artificial Intelligence, 7, 253-277.
31. Kremer, S. C. (1995). On the computational power of Elman-style recurrent networks. IEEE Transactions on Neural Networks, 6(4), 1000-1004.
32. Kremer, S. (1996). A theory of grammatical induction in the donnectionist paradigm. Unpublished Ph.D. dissertation, University of Alberta, Edmonton, Alberta.
33. Kremer, S., Ñeco, R. P., & Forcada, M. L. (1998). Constrained second-order recurrent networks for finite-state automata induction. In L. Niklasson, M. Bodén, & T. Ziemke (Eds.), Proceedings of the 8th International Conference on Artificial Neural Networks ICANN'98 (Vol. 2, pp. 529-534). London: Springer.
34. Li, L.K. (1992). Fixed point analysis for discrete-time recurrent neural networks. In Proceedings of IJCNN (Vol. 4, pp. 134-139).
35. Maass, W., & Orponen, P. (1998). On the effect of analog noise in discrete-time analog computations. Neural Computation, 10(5), 1071-1095.
36. Maass, W., & Sontag, E. D. (1999). Analog neural nets with gaussian or other common noise distribution cannot recognize arbitrary regular languages. Neural Computation, 21(3), 771-782.
37. Manolios, P., & Fanelli, R. (1994). First order recurrent neural networks and deterministic finite state automata. Neural Computation, 6(6), 1154-1172.
38. Maskara, A., & Noetzel, A. (1992). Forcing simple recurrent neural networks to encode context. In Proceedings of the 1992 Long Island Conference on Artificial Intelligence and Computer Graphics.
39. McCulloch, W. S., & Pitts, W. H. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5, 115-133.
40. Miller, C. B., & Giles, C. L. (1993). Experimental comparison of the effect of order in recurrent neural networks. International Journal of Pattern Recognition and Artificial Intelligence, 7(4), 849-872.
41. Minsky, M. L. (1967). Computation: Finite and infinite machines. Englewood Cliffs, NJ: Prentice-Hall.
42. Neco, R. P., & Forcada, M. L. (1996). Beyond Mealy machines: Learning translators with recurrent neural networks. In Proceedings of the World Conference on Neural Networks '96 (pp. 408-411). San Diego.
43. Neco, R. P., Forcada, M. L, Carrasco, R. C., & Valdés-Muñoz, A. (1999). Encoding of sequential translators in discrete-time recurrent neural nets. In Proceedings of the European Symposium on Artificial Neural Networks ESANN'99 (pp. 375-380).
44. Omlin, C. W., & Giles, C. L. (1996a). Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 43(6), 937-972.
45. Omlin, C. W., & Giles, C. L. (1996b). Stable encoding of large finite-state automata in recurrent neural networks with sigmoid discriminants. Neural Computation, 8, 675-696.
46. Pollack, J. B. (1991). The induction of dynamical recognizers. Machine Learning, 7, 227-252.
47. Robinson, T., & Fallside, F. (1991). A recurrent error propagation network speech recognition system. Computer Speech and Language, 5, 259-274.
48. Sanfeliu, A., & Alquézar, R. (1994). Active grammatical inference: A new learning methodology. In D. Dori & A. Bruckstein (Eds.), Shape and structure in pattern recognition. Singapore: World Scientific.
49. Šíma, J. (1997). Analog stable simulation of discrete neural networks. Neural Network World, 7, 679-686.
50. Šíma, J., & Wiedermann, J. (1998). Theory of neuromata. Journal of the ACM, 45(1), 155-178.
51. Tiňo, P., Horne, B. G., Giles, C. L., & Colingwood, P. C. (1998). Finite state machines and recurrent neural networks - automata and dynamical systems approaches. In J. E. Dayhoff & O. Omidvar (Eds.), Neural networks and pattern recognition (pp. 171-220). San Diego, CA: Academic.
52. Tiňo, P., & Sajda, J. (1995). Learning and extracting initial Mealy automata with a modular neural network model. Neural Computation, 7(4).
53. Tsoi, A. C., & Back, A. (1997). Discrete time recurrent neural network architectures: A unifying review. Neurocomputing, 15, 183-223.
54. Watrous, R. L., & Kuhn, G. M. (1992). Induction of finite-state languages using second-order recurrent networks. Neural Computation, 4(3), 406-414.
55. Zeng, Z., Goodman, R. M., & Smyth, P. (1994). Discrete recurrent neural networks for grammatical inference. IEEE Transactions on Neural Networks, 5(2), 320-330.
56. Zeng, Z., Goodman, R. M., & Smyth, P. (1993). Learning finite state machines with self-clustering recurrent networks. Neural Computation, 5(6), 976-990.