Neural GPUs learn algorithms

4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings

Volumn , Issue , 2016, Pages

  Kaiser, Łukasz ^a   Sutskever, Ilya ^a  

^a GOOGLE INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER PROGRAMMING; NETWORK ARCHITECTURE; PROGRAM PROCESSORS; RECURRENT NEURAL NETWORKS; TURING MACHINES;

GRADIENT NOISE; HIGHLY PARALLELS; PARAMETER SHARING; RECURRENT NETWORKS;

BACKPROPAGATION ALGORITHMS;

EID: 85083953090     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (31)

1. Angluin, Dana. Learning regaular sets from queries and counterexamples. Information and Computation, 75: 87–106, 1987.
2. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473, 2014. URL http://arxiv.org/abs/1409.0473.
3. Blumensath, Achim and Grädel, Erich. Automatic Structures. In Proceedings of LICS 2000, pp. 51–62, 2000. URL http://www.logic.rwth-aachen.de/pub/graedel/BlGr-lics00.ps.
4. Chan, William, Jaitly, Navdeep, Le, Quoc V., and Vinyals, Oriol. Listen, attend and spell. In International Conference on Acoustics, Speech and Signal Processing, ICASSP’16, 2016.
5. Cho, Kyunghyun, van Merrienboer, Bart, Gulcehre, Caglar, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoder-decoder for statistical machine translation. CoRR, abs/1406.1078, 2014. URL http://arxiv.org/abs/1406.1078.
6. Chung, Junyoung, Gülçehre, Çaglar, Cho, Kyunghyun, and Bengio, Yoshua. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555, 2014. URL http://arxiv.org/abs/1412.3555.
7. Dahl, George E., Yu, Dong, Deng, Li, and Acero, Alex. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on Audio, Speech & Language Processing, 20 (1):30–42, 2012.
8. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural turing machines. CoRR, abs/1410.5401, 2014. URL http://arxiv.org/abs/1410.5401.
9. Grefenstette, Edward, Hermann, Karl Moritz, Suleyman, Mustafa, and Blunsom, Phil. Learning to transduce with unbounded memory. CoRR, abs/1506.02516, 2015. URL http://arxiv.org/abs/1506.02516.
10. Greff, Klaus, Srivastava, Rupesh Kumar, Koutník, Jan, Steunebrink, Bas R., and Schmidhuber, Jürgen. LSTM: A search space odyssey. CoRR, abs/1503.04069, 2015. URL http://arxiv.org/abs/1503.04069.
11. Gulwani, Sumit. Dimensions in program synthesis. In Proceedings of PPDP 2010, PPDP’10, pp. 13–24, 2010.
12. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
13. Joulin, Armand and Mikolov, Tomas. Inferring algorithmic patterns with stack-augmented recurrent nets. CoRR, abs/1503.01007, 2015. URL http://arxiv.org/abs/1503.01007.
14. Kaiser, Łukasz. Learning games from videos guided by descriptive complexity. In Proceedings of the AAAI-12, pp. 963–970. AAAI Press, 2012. URL http://goo.gl/mRbfV5.
15. Kalchbrenner, Nal and Blunsom, Phil. Recurrent continuous translation models. In Proceedings EMNLP 2013, pp. 1700–1709, 2013. URL http://nal.co/papers/KalchbrennerBlunsom_EMNLP13.
16. Kalchbrenner, Nal, Danihelka, Ivo, and Graves, Alex. Grid long short-term memory. In International Conference on Learning Representations, 2016. URL http://arxiv.org/abs/1507.01526.
17. Kingma, Diederik P. and Ba, Jimmy. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. URL http://arxiv.org/abs/1412.6980.
18. Kitzelmann, Emanuel. Inductive programming: A survey of program synthesis techniques. In Approaches and Applications of Inductive Programming, AAIP 2009, volume 5812 of LNCS, pp. 50–73, 2010.
19. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey. Imagenet classification with deep convolutional neural network. In Advances in Neural Information Processing Systems, 2012.
20. Lavin, Andrew and Gray, Scott. Fast algorithms for convolutional neural networks. CoRR, abs/1509.09308, 2015. URL http://arxiv.org/abs/1509.09308.
21. Pham, Vu, Bluche, Théodore, Kermorvant, Christopher, and Louradour, Jérôme. Dropout improves recurrent neural networks for handwriting recognition. In International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 285–290. IEEE, 2014. URL http://arxiv.org/pdf/1312.4569.pdf.
22. Shi, Xingjian, Chen, Zhourong, Wang, Hao, Yeung, Dit-Yan, kin Wong, Wai, and chun Woo, Wang. Convo-lutional LSTM network: A machine learning approach for precipitation nowcasting. In Advances in Neural Information Processing Systems, 2015. URL http://arxiv.org/abs/1506.04214.
23. Srivastava, Rupesh Kumar, Greff, Klaus, and Schmidhuber, Jürgen. Highway networks. CoRR, abs/1505.00387, 2015. URL http://arxiv.org/abs/1505.00387.
24. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104–3112, 2014. URL http://arxiv.org/abs/1409.3215.
25. Toderici, George, O’Malley, Sean M., Hwang, Sung Jin, Vincent, Damien, Minnen, David, Baluja, Shumeet, Covell, Michele, and Sukthankar, Rahul. Variable rate image compression with recurrent neural networks. In International Conference on Learning Representations, 2016. URL http://arxiv.org/abs/1511.06085.
26. Vinyals & Kaiser, Koo, Petrov, Sutskever, and Hinton. Grammar as a foreign language. In Advances in Neural Information Processing Systems, 2015. URL http://arxiv.org/abs/1412.7449.
27. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. CoRR, abs/1411.4555, 2014. URL http://arxiv.org/abs/1411.4555.
28. Vivien, Helene. An Introduction to cellular automata. 2003. http://www.liafa.univ-paris-diderot.fr/yunes/ca/archives/bookvivien.pdf. URL
29. Welling, Max and Teh, Yee Whye. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of ICML 2011, pp. 681–688, 2011.
30. Zaremba, Wojciech and Sutskever, Ilya. Learning to execute. CoRR, abs/1410.4615, 2015a. URL http://arxiv.org/abs/1410.4615.
31. Zaremba, Wojciech and Sutskever, Ilya. Reinforcement learning neural turing machines. CoRR, abs/1505.00521, 2015b. URL http://arxiv.org/abs/1505.00521.