A recurrent neural network that learns to count

Connection Science

Volumn 11, Issue 1, 1999, Pages 5-40

  Rodriguez, Paul ^a   Wiles, Janet ^b   Elman, Jeffrey L ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b NONE

Author keywords

Context free languages; Dynamical systems; Recurrent neural network

Indexed keywords

EID: 0033098329     PISSN: 09540091     EISSN: None     Source Type: Journal    
DOI: 10.1080/095400999116340     Document Type: Article

References (43)

1. Barton, G.E., Berwick, R.C. & Ristad, E.S. (1987) Computational Complexity and Natural Language. Cambridge, MA: MIT Press.
2. Batali, J. (1994) Innate biases and critical periods: combining evolution and learning the acquisition of syntax. Artificial Life IV, pp. 160-171. Cambridge, MA: MIT Press.
3. Blair, A.D. & Pollack, J.B. (1997) Analysis of dynamical recognizers. Neural Computation, 9, 1127-1142.
4. Blank, D.S., Meeden, L.A. & Marshall, J.B. (1992) Exploring the symbolic/subsymbolic continuum: a case study of RAAM. In J. Dinsmore (Ed.), Closing the Gap: Symbolism vs. Connectionism, pp. 113-147. Hillsdale, NJ: Lawrence Erlbaum Associates.
5. Casey, M. (1996) The dynamics of discrete-time computation, with application to recurrent neural networks and finite state machine extraction. Neural Computation, 8, 1135-1178.
6. Christiansen, M.H. & Chater, N. (1994) Natural Language Recursion and Recurrent Neural Networks. TR 94-13 in Archive of Philosophy/Neuroscience/Psychology, Washington University.
7. Cottrell, G.W. & Tsung, F.-S. (1993) Learning simple arithmetic procedures. Connection Science, 5, 37-58.
8. Das, S., Giles, C.L. & Sun, G.Z. (1992) Learning context-free grammars: capabilities and limitations of a recurrent neural network with an external memory stack. Proceedings of the 14th Annual Conference of the Cognitive Science Society, pp. 791-796. Hillsdale, NJ: Lawrence Erlbaum Associates.
9. Elman, J.L. (1991) Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7, 195-225.
10. Giles, C.L. & Omlin, C.W. (1993) Extraction, insertion and refinement of symbolic rules in dynamically driven recurrent neural networks. Connection Science, 5, 307-337.
11. Giles, C.L., Sun, G.Z., Chen, H.H., Lee, Y.C. & Chen, D. (1992) Extracting and learning an unknown grammar with recurrent neural networks. In J.E. Moody, S.J. Hanson & R.P. Lippmann (Eds), Advances in Information Processing 4, pp. 317-323. San Mateo, CA: Morgan Kaufman.
12. Hopcroft, J.E. & Ullman, J.D. (1979) Introduction to Automata Theory, Languages, and Computation. Addison-Wesley.
13. Kolen, J.F. (1994) Exploring the Computational Capabilities of Recurrent Neural Networks. PhD dissertation, The Ohio State University.
14. Kutas, M. & Hillyard, S.A. (1984) Brain potentials during reading reflect word expectancy and semantic association. Nature, 307, 161-163.
15. Kwasny, S.C. & Kalman, B.L. (1995) Tail-recursive distributed representations and simple recurrent networks. Connection Science, 7, 61-80.
16. Martelli, M. (1992) Discrete Dynamical Systems and Chaos. Longman Scientific and Technical.
17. McClelland, J.L. (1987) The case for interactionism in language processing. In M. Coltheart (Ed.), Attention and Performance XII: The Psychology of Reading. London: Erlbaum.
18. McClelland, J.L. & Kawamoto, A.H. (1986) Mechanisms of sentence processing: assigning roles to constituents. IE. D.E. Rumelhart & J.L. McClelland (Eds), Parallel Distributed Processing, Vol. 2. Cambridge, MA: MIT Press.
19. Miikulainen, R. (1992) Script recognition with hierarchical feature maps. In N. Sharkey (Ed.), Connectionist Natural language Processing, pp. 196-214. Dordrecht: Kluwer Academic.
20. Moore, C. (1996) Dynamical recognizers: real-time language recognition by analog computers. Santa Fe Institute Working Paper 96-05-023.
21. Pollack, J.B. (1991) The induction of dynamical recognizers. Machine Learning, 7, 227-252.
22. Pollack, J.B. (1990) Recursive autoassociative memories. Artificial Intelligence, 46, 77-105.
23. Pollack, J.B. (1988) Recursive autoassociative memory: devising compositional distributed representations. Proceedings of the 10th Annual Conference of the Cognitive Science Society, pp. 33-38. Hillsdale, NJ: Lawrence Erlbaurn Associates.
24. Pollack, J.B. (1987a) Cascaded back propagation on dynamic connectionist networks. Proceedings of the 9th Annual Conference of the Cognitive Science Society, pp. 391-404. Hillsdale, NJ: Lawrence Erlbaum Associates.
25. Pollack, J.B. (1987b) On Connectionist Models of Language, PhD dissertation, Computer Science Department, University of Illinois at Urbana-Champaign.
26. Robinson, C. (1995) Dynamical Systems: Stability, Symbolic Dynamics, and Chaos. Boca Raton: CRC Press.
27. Rodriguez, P. & Wiles, J. (1998) A recurrent neural network can learn to implement symbol-sensitive counting. In M. Jordan, M. Kearns & S. Solla (Eds), Advances in Neural Information Processing Systems 10, pp. 87-93. Cambridge, MA: MIT Press.
28. Rumelhart, D.E., Hinton, G.E. & Williams, R.J. (1986) Learning internal representations by error propagation. In D.E. Rumelhart & J.L. McClelland (Eds), Parallel Distributed Processing, Vol. 2. Cambridge, MA: MIT Press.
29. Servan-Schreiber, D., Cleermans, A. & McClelland, J.L. (1988) Encoding Sequential Structure in Simple Recurrent Networks. CMU TR CS-88-183, Carnegie Mellon University.
30. Siegelmann, H.T. (1993) Foundations of Recurrent Neural Networks. PhD dissertation, New Brunswick Rutgers, The State University of New Jersey.
31. Smith, A.W. & Zipser, D. (1989) Learning sequential structure with the real-time recurrent learning algorithm. Interactional Journal of Neural Systems, 1, 125-131.
32. St. John, M. (1992) The story Gestalt: a model of knowledge-intensive processes in text comprehension. Cognitive Science, 16, 271-306.
33. Stolcke, A. (1990) Learning Feature-based Semantics with Simple Recurrent Networks. TR-90-015, International Computer Science Institute, University of California at Berkeley.
34. Sun, G.Z., Chen, H.H., Giles, C.L., Lee, Y.C. & Chen, D. (1990) Connectionist pushdown automata that learn context-free grammars. Proceedings of the International Joint Conference on Neural Networks, pp. I-577-580. Washington, DC.
35. Tino, P., Horne, B.G. & Giles, C.L. (1995) Finite State Machines and Recurrent Neural Networks Automata and Dynamical Systems Approaches. TR-UMCP-CSD:CS-TR-3396, University of Maryland, College Park.
36. Tonkes, B. (1998) Recurrent networks are unstable in learning a simple context free language. 4th Biannual Australian Conference of Cognitive Science (in press).
37. Tsung, F.-S. (1994) Modeling Dynamical Systems with Recurrent Neural Networks. PhD dissertation, Department of Computer Science, University of California, San Diego.
38. Tsung, F.-S. & Cottrell, G.W. (1994) Phase-space learning. In G. Tesauro, D. Touretzky & T. Leen (Eds), Advances in Neural Information Processing Systems 7. Cambridge, MA: MIT Press.
39. Watrous, R.L. & Kuhn, G.M. (1992) Induction of finite-state automata using second order recurrent networks. In J.E. Moody, S.J. Hanson & R.P. Lippmann (Eds), Advances in Information Processing 4, pp. 309-316. San Mateo, CA: Morgan Kaufman.
40. Weckerley, J. & Elman, J.L. (1992) A PDP approach to processing center-embedded sentences. Proceedings of the 14th Annual Conference of the Cognitive Science Society, pp. 414-419. Hillsdale, NJ: Lawrence Erlbaum Associates.
41. Wiggins, S. (1990) Introduction to Applied Nonlinear Dynamical Systems and Chaos. New York: Springer.
42. Wiles, J. & Elman, J.L. (1995) Learning to count without a counter: a case study of dynamics and activation landscapes in recurrent neural networks. Proceedings of the 17th Annual Conference of the Cognitive Science Society, pp. 482-487. Cambridge, MA: MIT Press.
43. Williams, R.J. & Zipser, D. (1989) Experimental analysis of the real-time recurrent learning algorithm. Connection Science, 1, 87-111.