The Dynamics of Discrete-Time Computation, with Application to Recurrent Neural Networks and Finite State Machine Extraction

Neural Computation

Volumn 8, Issue 6, 1996, Pages 1135-1178

  Casey, Mike ^a,b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b BRANDEIS UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; ARTIFICIAL NEURAL NETWORK; TIME;

NEURAL NETWORKS (COMPUTER); TIME FACTORS;

EID: 0030586641     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/neco.1996.8.6.1135     Document Type: Article

References (45)

1. Arbib, M. A., and Zeiger, H. P. 1969. On the relevance of abstract algebra to control theory. Automatica 5, 589-606.
2. Beer, R. D. 1996. On the dynamics of small continuous-time recurrent neural networks. Adaptive Behavior 3(4), 471-511.
3. Bowen, R. 1975. Equilibrium States and the Ergodic Theory of Anosov Diffeomorphisms, Lecture Notes in Mathematics, Vol. 470. Springer-Verlag, Berlin.
4. Casey, M. 1993. Computation dynamics in discrete-time recurrent neural networks. In UCSD's Institute for Neural Computation's Proceedings of the Annual Research Symposium, June, 78-95.
5. Casey, M. 1995a. Relaxing the symmetric weight condition for convergent dynamics in discrete-time recurrent networks. Institute for Neural Computation Tech. Rep. Series No. INC-9503, April.
6. Casey, M. 1995b. Computation dynamics in discrete-time dynamical systems. Ph.D. Thesis, Department of Mathematics, UC, San Diego, March.
7. Cleermans, A., McClelland, J. L., and Servan-Schreiber, D. 1989. Finite state automata and simple recurrent networks. Neural Comp. 1, 372-381.
8. Crutchfield, J. P. 1994a. Observing complexity and the complexity of observation. In Inside versus Outside, H. Atmanspacher, ed., pp. 235-272. Springer-Verlag, Berlin.
9. Crutchfield, J. P. 1994b. The calculi of emergence: Computation, dynamics, and induction. Physica D, 75, 11-54.
10. Crutchfield, J. P., and Young, K. 1989a. Computation at the onset of chaos. In Entropy, Complexity, and the Physics of Information, W. Zurek, ed. Addison-Wesley, Reading, MA.
11. Crutchfield, J. P., and Young, K. 1989b. Inferring statistical complexity. Phys. Rev. Let. 63, 105-108.
12. Cummins, F. 1993. Representation of temporal patterns in recurrent neural networks. Preprint, submitted to The 15th Annual Conference of the Cognitive Science Society.
13. Devaney, R. L. 1987. An Introduction to Chaotic Dynamical Systems. Addison-Wesley, New York.
14. Elman, J. L. 1989. Representation and structure in connectionist models. Tech. Rep. 8903, Center for Research in Language, University of California, San Diego, La Jolla, CA, August.
15. Giles, C. L., Sun, G. Z., Chen, H. H., Lee, Y. C., and Chen, D. 1990. Higher order recurrent networks and grammatical inference. In Advances in Neural Information Processing Systems 2, D. S. Touretzky, ed., p. 38. Morgan Kaufmann, San Mateo, CA.
16. Giles, C. L., Miller, C. B., Chen, D., Chen, H. H., Sun, G. Z., and Lee, Y. C. 1992. Learning and extracting finite state automata with second-order recurrent neural networks. Neural Comp. 4, 393-405.
17. Grebogi, C., Ott, E., and Yorke, J. A. 1987. Chaos, strange attractors, and fractal basin boundaries in nonlinear dynamics. Science 238 632-638.
18. Hertz, J., Krough, A., and Plamer, R. G. 1991. Introduction to the Theory of Neural Computation. Addison-Wesley, Menlo Park, CA.
19. Hopfield, J. J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554-2558.
20. Kaiman, R. E., Farb, P. L., and Arbib, M. A. 1969. Topics in Mathematical Systems Theory. McGraw-Hill, New York.
21. Kolen, J. F. 1994. Fool's Gold: Extracting finite state machines from recurrent network dynamics. In Advances in Neural Information Processing Systems 6, J. D. Cowan, G. Tesauro, and J. Alspector, eds., pp. 501-508. Morgan Kaufmann, San Mateo, CA.
22. Kristan, W. B., Jr. 1980. Generation of rhythmic motor patterns. In Information Processing in the Nervous System, H. M. Pinsker and W. D. Willis, Jr., eds., pp. 241-261, Raven Press, New York.
23. Kuan, C. M., Hornik, K., and White, H. 1994. A convergence result for learning in recurrent neural networks. Neural Comp. 6, 420.
24. Lewis, R. L., and Papadimitriou, C. H. 1981. Elements of the Theory of Computation. Prentice-Hall, Englewood Cliffs, NJ.
25. Milnor, J. 1985. On the concept of attractor. Commun. Math. Phys. 99, 177-195.
26. Munkres, J. R. 1975. Topology: A first Course. Prentice-Hall, Englewood Cliffs, NJ.
27. Narendra, K. S., and Parthasarathy, K. 1990. Identification and control of dynamical systems using neural networks. IEEE Transact. Neural Networks 1, 4-27.
28. Nerode, A. 1958. Linear automata transformations. Proc. Am. Math. Soc. 9, 541-544.
29. Omlin, C., and Giles, C. 1994. Constructing deterministic finite-state automata in sparse recurrent neural networks. IEEE Int. Conf. Neural Networks (ICNN'94) 1732-1737.
30. Pollack J. B. 1991. The induction of dynamic recognizers. Machine Learn. 7 227-252.
31. Selverston, A. I., and Moulins, M. 1985. Oscillatory neural networks. Annu. Rev. Physiol. 47, 29-48.
32. Shub, M. 1987. Global Stability of Dynamical Systems. Springer-Verlag, Berlin.
33. Siegelmann, H. T., and Sontag, E. D. 1992. "On the computational power of neural networks. Proc. Fifth ACM Workshop Comp. Learning Theory, Pittsburgh, PA.
34. Siegelmann, H. A., Sontag, E. D., and Giles, C. L. 1992. The complexity of language recognition by neural networks. In Algorithms, Software, Architecture - Information Processing 92, J. van Leeuwen, ed., Vol. 1, pp. 329-335. Elsevier Science, Amsterdam.
35. Tani, J. 1995. Essentail dynamical structure in learnable autonomous robots. In Advances in Artificial Life, F. Moran, A. Moreno, J. J. Merelo, and P. Chacon, eds., Lecture Notes in Artificial Intelligence 929, pp. 721-732. Springer-Verlag, New York.
36. Tino, P., Horne, B. G., and Giles, C. L. 1995a. Fixed points in two-neuron discrete time recurrent networks: Stability and bifurcation considerations. Tech. Rep. UMIACS-TR-95-51 and CS-TR-3461, Institute for Advanced Computer Studies, University of Maryland, College Park, MD.
37. Tino, P., Horne, B. G., Giles, C. L., and Collingwood, P. C. 1995b. Finite state machines and recurrent neural networks - Automata and dynamical systems approaches. Tech. Rep. UMIACS-TR-95-1 and CS-TR-3396, Institute for Advanced Computer Studies, University of Maryland, College Park, MD.
38. Tsung, F-S., and Cottrell, G. W. 1993. Learning temporal signals in phase space. Int. Symp. Nonlinear Theory Its Appl., Hawaii.
39. Tsung, F-S., Cottrell, G. W., and Selverston, A. I. 1990. Some experiments on learning stable network oscillations. International Joint Conference on Neural Networks, San Diego. IEEE, New York.
40. Turing, A. M. 1936. On computable numbers, with an application to the Entscheidungs problem. Proc. London Math. Soc. Ser. 2-42, 230-265.
41. Wang, X. 1991. Period doublings to chaos in a simple neural network: An analytical proof. Complex Syst. 5(4), 425-441.
42. Watrous, R. L., and Kuhn, G. M. Induction of finite-state languages using second-order recurrent networks. Neural Comp. 4, 406-414.
43. Wiggins, S. 1990. Introduction to Applied Nonlinear Dynamical Systems and Chaos. Springer-Verlag, New York.
44. Williams, R. J., and Zipser, D. 1989. A learning algorithm for continually running fully recurrent networks. Neural Comp. 1, 270-280.
45. Yamauchi, B., and Beer, R. D. 1994. Sequential behavior and learning in evolved dynamical neural networks. Adapt. Behav. 2(3), 219-246.