Inferring algorithmic patterns with stack-augmented recurrent nets

Advances in Neural Information Processing Systems

Volumn 2015-January, Issue , 2015, Pages 190-198

  Joulin, Armand ^a   Mikolov, Tomas ^a  

^a FACEBOOK AI RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; INFORMATION SCIENCE; LEARNING SYSTEMS;

DEEP LEARNING; RECURRENT NETWORKS; SEQUENCE PREDICTION; SEQUENTIAL DATA;

COMPLEX NETWORKS;

EID: 84965117324     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (37)

1. Y. Bengio and Y. Le Cun. Scaling learning algorithms towards ai. Large-scale kernel machines, 2007.
2. C. M. Bishop. Pattern recognition and machine learning. springer New York, 2006.
3. M. Bodén and J. Wiles. Context-free and context-sensitive dynamics in recurrent neural networks. Connection Science, 2000.
4. L. Bottou. Large-scale machine learning with stochastic gradient descent. In COMPSTAT. Springer, 2010.
5. L. Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
6. J. S. Bridle. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing, pages 227-236. Springer, 1990.
7. M. H. Christiansen and N. Chater. Toward a connectionist model of recursion in human linguistic performance. Cognitive Science, 23(2):157-205, 1999.
8. J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Gated feedback recurrent neural networks. arXiv, 2015.
9. D. C. Ciresan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmidhuber. High-performance neural networks for visual object classification. arXiv preprint, 2011.
10. M. W. Crocker. Mechanisms for sentence processing. University of Edinburgh, 1996.
11. G. E. Dahl, D. Yu, L. Deng, and A. Acero. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. Audio, Speech, and Language Processing, 20(1):30-42, 2012.
12. S. Das, C. Giles, and G. Sun. Learning context-free grammars: Capabilities and limitations of a recurrent neural network with an external stack memory. In ACCSS, 1992.
13. S. Das, C. Giles, and G. Sun. Using prior knowledge in a nnpda to learn context-free languages. NIPS, 1993.
14. J. L. Elman. Finding structure in time. Cognitive science, 14(2):179-211, 1990.
15. M. Fanty. Context-free parsing in connectionist networks. Parallel natural language processing, 1994.
16. F. A. Gers and J. Schmidhuber. Lstm recurrent networks learn simple context-free and context-sensitive languages. Transactions on Neural Networks, 12(6):1333-1340, 2001.
17. A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv preprint, 2014.
18. P. Grünwald. A recurrent network that performs a context-sensitive prediction task. In ACCSS, 1996.
19. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.
20. S. Holldobler, Y. Kalinke, and H. Lehmann. Designing a counter: Another case study of dynamics and activation landscapes in recurrent networks. In Advances in Artificial Intelligence, 1997.
21. A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
22. Y. Le Cun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. 1998.
23. T. Mikolov. Statistical language models based on neural networks. PhD thesis, Brno University of Technology, 2012.
24. T. Mikolov, A. Joulin, S. Chopra, M. Mathieu, and M. A. Ranzato. Learning longer memory in recurrent neural networks. arXiv preprint, 2014.
25. M. Minsky and S. Papert. Perceptrons. MIT press, 1969.
26. M. C. Mozer and S. Das. A connectionist symbol manipulator that discovers the structure of context-free languages. NIPS, 1993.
27. J. B. Pollack. The induction of dynamical recognizers. Machine Learning, 7(2-3):227-252, 1991.
28. B. Recht, C. Re, S. Wright, and F. Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In NIPS, 2011.
29. P. Rodriguez, J. Wiles, and J. L. Elman. A recurrent neural network that learns to count. Connection Science, 1999.
30. D. E. Rumelhart, G. Hinton, and R. J. Williams. Learning internal representations by error propagation. Technical report, DTIC Document, 1985.
31. W. Tabor. Fractal encoding of context-free grammars in connectionist networks. Expert Systems, 2000.
32. P. Werbos. Generalization of backpropagation with application to a recurrent gas market model. Neural Networks, 1(4):339-356, 1988.
33. J. Weston, S. Chopra, and A. Bordes. Memory networks. In ICLR, 2015.
34. J. Wiles and J. Elman. Learning to count without a counter: A case study of dynamics and activation landscapes in recurrent networks. In ACCSS, 1995.
35. R. J. Williams and D. Zipser. Gradient-based learning algorithms for recurrent networks and their computational complexity. Back-propagation: Theory, architectures and applications, pages 433-486, 1995.
36. W. Zaremba and I. Sutskever. Learning to execute. arXiv preprint, 2014.
37. Z. Zeng, R. M. Goodman, and P. Smyth. Discrete recurrent neural networks for grammatical inference. Transactions on Neural Networks, 5(2):320-330, 1994.