Markovian architectural bias of recurrent neural networks

IEEE Transactions on Neural Networks

Volumn 15, Issue 1, 2004, Pages 6-15

  Tiňo, Peter ^a   Čerňanský, Michal ^b   Beňušková, Ľubica ^c  

^a UNIVERSITY OF BIRMINGHAM   (United Kingdom)

^b SLOVAK UNIVERSITY OF TECHNOLOGY   (Slovakia)

^c Physics and Informatics   (Slovakia)

Author keywords

Complex symbolic sequences; Information latching problem; Iterative function systems; Markov models; Recurrent neural networks (RNNs)

Indexed keywords

CHAOS THEORY; CONTEXT FREE LANGUAGES; ITERATIVE METHODS; MARKOV PROCESSES; MATHEMATICAL MODELS; RECURSIVE FUNCTIONS; STATE SPACE METHODS;

COMPLEX SYMBOLIC SEQUENCES; INFORMATION LATCHING PROBLEM; MARKOV MODELS;

RECURRENT NEURAL NETWORKS;

ARTICLE; ARTIFICIAL NEURAL NETWORK; PROBABILITY;

MARKOV CHAINS; NEURAL NETWORKS (COMPUTER);

EID: 6044234526     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2003.820839     Document Type: Article

References (49)

1. J. F. Kolen, "Recurrent networks: state machines or iterated function systems?," in Proc. 1993 Connectionist Models Summer School, M. C. Mozer, P. Smolensky, D. S. Touretzky, J. L. Elman, and A. S. Weigend, Eds. Hillsdale, NJ: Erlbaum Associates, 1994, pp. 203-210.
2. ____, "The origin of clusters in recurrent neural network state space," in Proc. 16th Annu. Conf. the Cognitive Science Society. Hillsdale, NJ: Erlbaum Associates, 1994, pp. 508-513.
3. S. C. Kremer, "Spatio-temporal connectionist networks: a taxonomy and review," Neural Comput., vol. 13, no. 2, pp. 249-306, 2001.
4. A. D. Blair and J. B. Pollack, "Analysis of dynamical recognizers," Neural Computation, vol. 9, pp. 1127-1142, 1997.
5. Y. Bengio, P. Frasconi, and P. Simard, "The problem of learning long-term dependencies in recurrent networks," in Proc. 1993 IEEE Int. Conf. Neural Networks, vol. 3, 1993, pp. 1183-1188.
6. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult," IEEE Trans. Neural Networks, vol. 5, no. 2, pp. 157-166, 1994.
7. J. Hochreiter and J. Schmidhuber, "Long short term memory," Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997.
8. A. Cleeremans, D. Servan-Schreiber, and J. L. McClelland, "Finite state automata and simple recurrent networks," Neural Comput., vol. 1, no. 3, pp. 372-381, 1989.
9. J. L. Elman, "Finding structure in time," Cognitive Sci., vol. 14, pp. 179-211, 1990.
10. C. L. Giles, C. B. Miller, D. Chen, H. H. Chen, G. Z. Sun, and Y. C. Lee, "Learning and extracting finite state automata with second-order recurrent neural networks," Neural Comput., vol. 4, no. 3, pp. 393-405, 1992.
11. P. Manolios and R. Fanelli, "First order recurrent neural networks and deterministic finite state automata," Neural Comput., vol. 6, no. 6, pp 1155-1173, 1994.
12. R. L. Watrous and G. M. Kuhn, "Induction of finite-state languages using second-order recurrent networks," Neural Comput., vol. 4, no. 3, pp. 406-414, 1992.
13. P. Tiňo and J. Sajda, "Learning and extracting initial mealy machines with a modular neural network model," Neural Comput., vol. 7, no. 4, pp. 822-844, 1995.
14. M. P. Casey, "The dynamics of discrete-time computation, with application to recurrent neural networks and finite state machine extraction," Neural Comput., vol. 8, no. 6, pp. 1135-1178, 1996.
15. P. Frasconi, M. Gori, M. Maggini, and G. Soda, "Insertion of finite state automata in recurrent radial basis function networks," Machine Learning, vol. 23, pp. 5-32, 1996.
16. C. W. Omlin and C. L. Giles, "Extraction of rules from discrete-time recurrent neural networks," Neural Networks, vol. 9, no. 1, pp. 41-51, 1996.
17. P. Tiňo, B. G. Horne, C. L. Giles, and P. C. Collingwood, "Finite state machines and recurrent neural networks - automata and dynamical systems approaches," in Neural Networks and Pattern Recognition, J. E. Dayhoff and O. Omidvar, Eds. New York: Academic, 1998, pp. 171-220.
18. P. Tiňo and M. Koteles, "Extracting finite state representations from recurrent neural networks trained on chaotic symbolic sequences," IEEE Trans. Neural Networks, vol. 10, no. 2, pp. 284-302, 1999.
19. M. H. Christiansen and N. Chater, "Toward a connectionist model of recursion in human linguistic performance," Cognitive Sci., vol. 23, pp. 417-437, 1999.
20. D. Ron, Y. Singer, and N. Tishby, "The power of amnesia," in Advances in Neural Information Processing Systems 6. San Mateo, CA: Morgan Kaufmann, 1994, pp. 176-183.
21. I. Guyon and F. Pereira, "Design of a linguistic postprocessor using variable memory length Markov models," in Proc. Int. Conf. Document Analysis and Recognition, Canada, Montreal, QC, Canada, 1995, pp. 454-457.
22. M. J. Weinberger, J. J. Rissanen, and M. Feder, "A universal finite memory source," IEEE Trans. Inform. Theory, vol. 41, pp. 643-652, May 1995.
23. P. Buhlmann and A. J. Wyner, "Variable length Markov chains," Ann. Statist., vol. 27, pp. 480-513, 1999.
24. J. Rissanen, "A universal data compression system," IEEE Trans. Inform. Theory, vol. IT-29, pp. 656-664, Sept. 1983.
25. P. Rodriguez, J. Wiles, and J. L. Elman, "A recurrent neural network that learns to count," Connection Sci., vol. 11, pp. 5-40, 1999.
26. R. J. Williams and D. Zipser, "A learning algorithm for continually running fully recurrent neural networks," Neural Comput., vol. 1, no. 2, pp. 270-280, 1989.
27. R. J. Williams, "Training recurrent networks using the extended Kalman filter," in Proc. 1992 Int. Joint Conf. Neural Networks, vol. 4, 1992, pp. 241-246.
28. G. Patel, S. Becker, and R. Racine, "2D image modeling as a time-series prediction problem," in Kalman Filtering Applied to Neural Networks, S. Haykin, Ed. New York: Wiley, 2001.
29. M. R. Anderberg, Cluster Analysis for Applications. New York: Academic, 1973.
30. M. F. Barnsley, Fractals Everywhere. New York: Academic, 1988.
31. P. Tiňo and G. Dorffner, "Predicting the future of discrete sequences from fractal representations of the past," Machine Learning, vol. 45, no. 2, pp. 187-218, 2001.
32. P. Tiňo, B. G. Horne, and C. L. Giles, "Attractive periodic sets in discrete time recurrent networks (with emphasis on fixed point stability and bifurcations in two-neuron networks)," Neural Comput., vol. 13, no. 6, pp. 1379-1414, 2001.
33. S. Lawrence, C. L. Giles, and S. Fong, "Natural language grammatical inference with recurrent neural networks," IEEE Trans. Knowledge Data Eng., vol. 12, no. 1, pp. 126-140, 2000.
34. A. Katok and B. Hasselblatt, Introduction to the Modern Theory of Dynamical Systems. Cambridge, U.K.: Cambridge Univ. Press, 1995.
35. C. L. Giles and C. W. Omlin, "Insertion and refinement of production rules in recurrent neural networks," Connection Sci., vol. 5, no. 3, pp. 307-318, 1993.
36. K. Doya, "Bifurcations in the learning of recurrent neural networks," in Proc. 1992 IEEE Int. Symp. Circuits and Systems, 1992, pp. 2777-2780.
37. D. Servan-Schreiber, A. Cleeremans, and J. McClelland, "Encoding sequential structure in simple recurrent networks," in Advances in Neural Information Processing Systems 1, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1989.
38. D. L. T. Rohde and D. C. Plaut, "Language acquisition in the absence of explicit negative evidence: How important is starting small?," Cognition, vol. 72, pp. 67-109, 1999.
39. P. Tiňo, "Spatial representation of symbolic sequences through iterative function system," IEEE Trans. Systems, Man Cybern. Pt. A, vol. 29, no. 4, pp. 386-392, 1999.
40. ____, "Multifractal properties of Hao's geometric representations of DNA sequences," Physica A: Statist. Mech. its Applic., vol. 304, no. 3-4, pp. 480-494, 2002.
41. Y. Pesin, Dimension Theory in Dynamical Systems: Rigorous Results and Applications. Chicago, IL: Univ. Chicago Press, 1997.
42. Y. Pesin and H. Weiss, "On the dimension of deterministic and Cantor-like sets, symbolic dynamics, and the Eckmann-Ruelle conjecture," Commun. Math. Phys, vol. 182, no. 1, pp. 105-153, 1996.
43. W. Tabor, "Dynamical Automata," Cornell Univ., Computer Science Depth., Ithaca, NY, Tech. Rep. TR98-1694, 1998.
44. W. Tabor and M. K. Tanenhaus, "Dynamical models of sentence processing," Cognitive Sci., vol. 23, no. 4, pp. 491-515, 2000.
45. W. Tabor, "Fractal encoding of context free grammars in connectionist networks," Expert Syst.: Int. J. Knowledge Eng. Neural Networks, vol. 17, no. 1, pp. 41-56, 2000.
46. L. Feldkamp, D. Prokhorov, C. Eagen, and F. Yuan, "Enhanced multi-stream Kalman filter training for recurrent networks," in Nonlinear Modeling: Advanced Black-Box Techniques, J. Suykens and J. Vandewalle, Eds. Norwell, MA, 1998, pp. 29-53.
47. E. A. Wan and A. T. Nelson, "Dual EKF methods," in Kalman Filtering and Neural Networks, S. Haykin, Ed. New York: Wiley, 2000.
48. G. Welch and G. Bishop, "An Introduction to the Kalman Filter," Univ. North Carolina, Raleigh, Tech. Rep. TR NC27599-3175, 1997.
49. P. J. Werbos, "Backpropagation through time; what it does and how to do it," Proc. IEEE, vol. 78, pp. 1550-1560, 1990.