On the implicit acquisition of a context-free grammar by a simple recurrent neural network

Neurocomputing

Volumn 71, Issue 7-9, 2008, Pages 1527-1537

  Cartling, Bo ^a  

^a ALBANOVA UNIVERSITY CENTER   (Sweden)

Author keywords

Context free grammar; Generalization capacity; Internal representation; Language acquisition; Simple recurrent network

Indexed keywords

CONTEXT FREE GRAMMARS; DATA ACQUISITION; PRINCIPAL COMPONENT ANALYSIS;

GENERALIZATION CAPACITY; INTERNAL REPRESENTATION; LANGUAGE ACQUISITION;

RECURRENT NEURAL NETWORKS;

ARTICLE; ARTIFICIAL NEURAL NETWORK; CONTROLLED STUDY; GRAMMAR; HUMAN; LANGUAGE PROCESSING; LEARNING ALGORITHM; NERVE CELL NETWORK; NEUROBIOLOGY; PREDICTION; PRINCIPAL COMPONENT ANALYSIS; PRIORITY JOURNAL; SIMPLE RECURRENT NEURAL NETWORK; SYMBOLISM; SYNAPSE; TASK PERFORMANCE;

EID: 40649110282     PISSN: 09252312     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neucom.2007.05.006     Document Type: Article

References (40)

1. Abeles M. Corticonics. Neural Circuits of the Cerebral Cortex (1991), Cambridge University Press, Cambridge
2. Bodén M., and Wiles J. Context-free and context-sensitive dynamics in recurrent neural networks. Connection Sci. 12 (2000) 197-210
3. Casey M. The dynamics of discrete-time computation, with application to recurrent neural networks and finite state machine extraction. Neural Comput. 8 (1996) 1135-1178
4. Chomsky N. Syntactic Structures (1957), Mouton, The Hague
5. Chomsky N. On certain formal properties of grammars. Inf. Control 2 (1959) 137-167
6. N. Chomsky, Context-free grammars and pushdown storage, MIT Research Laboratory Electronics Quarterly Progress Report 65, 1962, pp.187-194.
7. Christiansen M.H., and Chater N. Generalization and connectionist language learning. Mind Lang. 9 (1994) 273-287
8. Christiansen M.H., and Chater N. Toward a connectionist model of recursion in human linguistic performance. Cognit. Sci. 23 (1999) 157-205
9. Cleeremans A., Servan-Schreiber D., and McClelland J.L. Finite state automata and simple recurrent networks. Neural Comput. 1 (1989) 372-381
10. Elman J.L. Finding structure in time. Cognit. Sci. 14 (1990) 179-211
11. Elman J.L. Distributed representations, simple recurrent networks, and grammatical structure. Mach. Learn. 7 (1991) 195-225
12. Elman J.L. Learning and development in neural networks: the importance of starting small. Cognition 48 (1993) 71-99
13. Gers F.A., and Schmidhuber J. LSTM recurrent networks learn simple context-free and context-sensitive languages. IEEE Trans. Neural Networks 12 (2001) 1333-1340
14. Giles C.L., Miller C.B., Chen D., Chen H.H., Sun G.Z., and Lee Y.C. Learning and extracting finite state automata with second-order recurrent neural networks. Neural Comput. 4 (1992) 393-405
15. Grüning A. Stack-like and queue-like dynamics in recurrent neural networks. Connection Sci. 18 (2006) 23-42
16. Hadley R.F. Systematicity in connectionist language learning. Mind Lang. 9 (1994) 247-272
17. Hadley R.F. Systematicity revisited: reply to Christiansen and Chater and Niklasson and van Gelder. Mind Lang. 9 (1994) 431-444
18. Hertz J., Krogh A., and Palmer R.G. Introduction to the Theory of Neural Computation (1991), Addison-Wesley, Redwood City, CA
19. Hochreiter S., and Schmidhuber J. Long short-term memory. Neural Comput. 9 (1997) 1735-1780
20. Hopcroft J.E., Motwani R., and Ullman J.D. Introduction to Automata Theory, Languages, and Computation (2003), Addison-Wesley, New York
21. Jacobsson H. Rule extraction from recurrent neural networks: a taxonomy and review. Neural Comput. 17 (2005) 1223-1263
22. M.I. Jordan, Serial order: A parallel distributed processing approach, Tech. Rep. No. 8604 (Institute for Cognitive Science, UCSD, 1986).
23. Lawrence S., Giles C.L., and Fong S. Natural language grammatical inference with recurrent neural networks. IEEE Trans. Knowl. Data Eng. 12 (2000) 126-140
24. Nelder J.A., and Mead R. A simplex method for function minimization. Computer J. 7 (1965) 308-313
25. Niklasson L.F., and van Gelder T. On being systematically connectionist. Mind Lang. 9 (1994) 288-302
26. A.G. Oettinger, Automatic syntactic analysis and the pushdown store, in: Proceedings of the Symposia in Applied Mathematics, vol. 12, American Mathematical Society, Providence, RI, 1961, pp. 104-129.
27. Papadimitriou C.H. Computational Complexity (1995), Addison-Wesley, New York
28. Pollack J.B. The induction of dynamical recognizers. Mach. Learn. 7 (1991) 227-252
29. Press W.H., Teukolsky S.A., Vetterling W.T., and Flannery B.P. Numerical Recipes. The Art of Scientific Computing (1992), Cambridge University Press, Cambridge
30. Pulvermüller F. A brain perspective on language mechanisms: from discrete neuronal ensembles to serial order. Prog. Neurobiol. 67 (2002) 85-111
31. Pulvermüller F. The Neuroscience of Language. On Brain Circuits of Words and Serial Order (2002), Cambridge University Press, Cambridge
32. Rodriguez P., Wiles J., and Elman J.L. A recurrent neural network that learns to count. Connection Sci. 11 (1999) 5-40
33. Rodriguez P. Simple recurrent networks learn context-free and context-sensitive languages by counting. Neural Comput. 13 (2001) 2093-2118
34. Rohde D.L.T., and Plaut D.C. Language acquisition in the absence of explicit negative evidence: how important is starting small?. Cognition 72 (1999) 67-109
35. Rumelhart D.E., Hinton G.E., and Williams R.J. Learning internal representations by error propagation. In: Rumelhart D.E., and McClelland J.L. (Eds). Parallel Distributed Processing: Explorations in the Microstructure of Cognition vol. 1 (1986), MIT Press, Cambridge, MA 318-362
36. Savitch W.J. Infinity is in the eyes of the beholder. In: Georgopoulos C., and Ishihara R. (Eds). Interdisciplinary Approaches to Language: Essays in Honor of S.-Y. Kuroda (1991), Kluwer Academic Publishers, Dordrecht 487-500
37. W.J. Savitch, Why it might pay to assume that languages are infinite, Newsletter 6(4), Center for Research in Language, UCSD, 1992.
38. Siegelmann H.T., and Sontag E.D. Analog computation via neural networks. Theor. Comput. Sci. 131 (1994) 331-360
39. Siegelmann H.T., and Sontag E.D. On the computational power of neural nets. J. Comput. Syst. Sci. 50 (1995) 132-150
40. Wiles J., Blair A.D., and Bodén M. Representation beyond finite states: alternatives to push-down automata. In: Kolen J.F., and Kremer S.C. (Eds). A Field Guide to Dynamical Recurrent Networks (2001), IEEE Press, New York 129-142