Neural programmer: Inducing latent programs with gradient descent

4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings

Volumn , Issue , 2016, Pages

  Neelakantan, Arvind ^a   Le, Quoc V ^b   Sutskever, Ilya ^b  

^a UNIVERSITY OF MASSACHUSETTS   (United States)

^b GOOGLE INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPLEX NETWORKS; COMPUTER CIRCUITS; DEEP NEURAL NETWORKS; GRADIENT METHODS; IMAGE RECOGNITION; SPEECH RECOGNITION;

COMPLEX ARITHMETIC; GRADIENT DESCENT; LOGIC OPERATIONS; LOGIC REASONING; QUESTION ANSWERING; RECURRENT NETWORKS; SEQUENCE LEARNING; SUPERVISED CLASSIFICATION;

RECURRENT NEURAL NETWORKS;

EID: 85083953004     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (56)

1. Andreas, Jacob, Rohrbach, Marcus, Darrell, Trevor, and Klein, Dan. Learning to compose neural networks for question answering. ArXiv, 2016.
2. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. ICLR, 2014.
3. Bahdanau, Dzmitry, Chorowski, Jan, Serdyuk, Dmitriy, Brakel, Philemon, and Bengio, Yoshua. End-to-end attention-based large vocabulary speech recognition. arXiv preprint arxiv:1508.04395, 2015.
4. Bordes, Antoine, Chopra, Sumit, and Weston, Jason. Question answering with subgraph embeddings. In EMNLP, 2014.
5. Cantlon, Jessica F., Brannon, Elizabeth M., Carter, Elizabeth J., and Pelphrey, Kevin A. Functional imaging of numerical processing in adults and 4-y-old children. PLoS Biology, 2006.
6. Chan, William, Jaitly, Navdeep, Le, Quoc V., and Vinyals, Oriol. Listen, attend and spell. arXiv preprint arxiv:1508.01211, 2015.
7. Clarke, James, Goldwasser, Dan, Chang, Ming-Wei, and Roth, Dan. Driving semantic parsing from the world’s response. In CoNLL, 2010.
8. Das, Sreerupa, Giles, C. Lee, and zheng Sun, Guo. Learning context-free grammars: Capabilities and limitations of a recurrent neural network with an external stack memory. In CogSci, 1992a.
9. Das, Sreerupa, Giles, C. Lee, and zheng Sun, Guo. Using prior knowledge in an NNPDA to learn context-free languages. In NIPS, 1992b.
10. Dastjerdi, Mohammad, Ozker, Muge, Foster, Brett L, Rangarajan, Vinitha, and Parvizi, Josef. Numerical processing in the human parietal cortex during experimental and natural conditions. Nature communications, 4, 2013.
11. Eisenstein, Jacob, Clarke, James, Goldwasser, Dan, and Roth, Dan. Reading to learn: Constructing features from semantic abstracts. In EMNLP, 2009.
12. Fias, Wim, Lammertyn, Jan, Caessens, Bernie, and Orban, Guy A. Processing of abstract ordinal knowledge in the horizontal segment of the intraparietal sulcus. The Journal of Neuroscience, 2007.
13. Gers, Felix A. and Schmidhuber, Jürgen. LSTM recurrent networks learn simple context free and context sensitive languages. IEEE Transactions on Neural Networks, 2001.
14. Graves, Alex. Generating sequences with recurrent neural networks. arXiv preprint arxiv:1308.0850, 2013.
15. Graves, Alex and Jaitly, Navdeep. Towards end-to-end speech recognition with recurrent neural networks. In ICML, 2014.
16. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural Turing Machines. arXiv preprint arxiv:1410.5401, 2014.
17. Hannun, Awni Y., Case, Carl, Casper, Jared, Catanzaro, Bryan C., Diamos, Greg, Elsen, Erich, Prenger, Ryan, Satheesh, Sanjeev, Sengupta, Shubho, Coates, Adam, and Ng, Andrew Y. Deep Speech: Scaling up end-to-end speech recognition. arXiv preprint arxiv:1412.5567, 2014.
18. Hermann, Karl Moritz, Kociský, Tomás, Grefenstette, Edward, Espeholt, Lasse, Kay, Will, Suley-man, Mustafa, and Blunsom, Phil. Teaching machines to read and comprehend. NIPS, 2015.
19. Hinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George, rahman Mohamed, Abdel, Jaitly, Navdeep, Senior, Andrew, Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara, and Kingsbury, Brian. Deep neural networks for acoustic modeling in speech recognition. Signal Processing Magazine, 2012.
20. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural Computation, 1997.
21. Huber, Peter. Robust estimation of a location parameter. In The Annals of Mathematical Statistics, 1964.
22. Iyyer, Mohit, Boyd-Graber, Jordan L., Claudino, Leonardo Max Batista, Socher, Richard, and III, Hal Daumé. A neural network for factoid question answering over paragraphs. In EMNLP, 2014.
23. Joulin, Armand and Mikolov, Tomas. Inferring algorithmic patterns with stack-augmented recurrent nets. NIPS, 2015.
24. Kingma, Diederik P. and Ba, Jimmy. Adam: A method for stochastic optimization. ICLR, 2014.
25. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep con-volutional neural networks. In NIPS, 2012.
26. Kucian, Karin, Loenneker, Thomas, Dietrich, Thomas, Dosch, Mengia, Martin, Ernst, and Von Aster, Michael. Impaired neural networks for approximate calculation in dyscalculic children: a functional mri study. Behavioral and Brain Functions, 2006.
27. Kumar, Ankit, Irsoy, Ozan, Su, Jonathan, Bradbury, James, English, Robert, Pierce, Brian, On-druska, Peter, Gulrajani, Ishaan, and Socher, Richard. Ask me anything: Dynamic memory networks for natural language processing. ArXiv, 2015.
28. Liang, Percy, Jordan, Michael I., and Klein, Dan. Learning programs: A hierarchical Bayesian approach. In ICML, 2010.
29. Liang, Percy, Jordan, Michael I., and Klein, Dan. Learning dependency-based compositional semantics. In ACL, 2011.
30. Lin, Yankai, Liu, Zhiyuan, Luan, Huan-Bo, Sun, Maosong, Rao, Siwei, and Liu, Song. Modeling relation paths for representation learning of knowledge bases. In EMNLP, 2015.
31. Luong, Thang, Sutskever, Ilya, Le, Quoc V., Vinyals, Oriol, and Zaremba, Wojciech. Addressing the rare word problem in neural machine translation. ACL, 2014.
32. Neelakantan, Arvind, Roth, Benjamin, and McCallum, Andrew. Compositional vector space models for knowledge base completion. In ACL, 2015.
33. Neelakantan, Arvind, Vilnis, Luke, Le, Quoc V., Sutskever, Ilya, Kaiser, Lukasz, Kurach, Karol, and Martens, James. Adding gradient noise improves learning for very deep networks. ICLR Workshop, 2016.
34. Pasupat, Panupong and Liang, Percy. Compositional semantic parsing on semi-structured tables. In ACL, 2015.
35. Peng, Baolin, Lu, Zhengdong, Li, Hang, and Wong, Kam-Fai. Towards neural network-based reasoning. arXiv preprint arxiv:1508.05508, 2015.
36. Piantadosi, Steven T., Goodman, N.D., Ellis, B.A., and Tenenbaum, J.B. A Bayesian model of the acquisition of compositional semantics. In CogSci, 2008.
37. Piazza, Manuela, Izard, Veronique, Pinel, Philippe, Le Bihan, Denis, and Dehaene, Stanislas. Tuning curves for approximate numerosity in the human intraparietal sulcus. Neuron, 2004.
38. Poon, Hoifung. Grounded unsupervised semantic parsing. In ACL, 2013.
39. Reed, Scott and Freitas, Nando De. Neural programmer-interpreters. ICLR, 2016.
40. Schmidhuber, J. A self-referentialweight matrix. In ICANN, 1993.
41. Shang, Lifeng, Lu, Zhengdogn, and Li, Hang. Neural responding machine for short-text conversation. arXiv preprint arXiv:1503.02364, 2015.
42. Steijvers, Mark. A recurrent network that performs a context-sensitive prediction task. In CogSci, 1996.
43. Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, and Fergus, Rob. End-to-end memory networks. arXiv preprint arXiv:1503.08895, 2015.
44. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. Sequence to sequence learning with neural networks. In NIPS, 2014.
45. Vinyals, Oriol and Le, Quoc V. A neural conversational model. ICML DL Workshop, 2015.
46. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. In CVPR, 2015.
47. Von Neumann, John. First draft of a report on the EDVAC. Technical report, 1945.
48. Wang, Yushi, Berant, Jonathan, and Liang, Percy. Building a semantic parser overnight. In ACL, 2015.
49. Welling, Max and Teh, Yee Whye. Bayesian learning via stochastic gradient Langevin dynamics. In ICML, 2011.
50. Werbos, P. Backpropagation through time: what does it do and how to do it. In Proceedings of IEEE, 1990.
51. Weston, Jason, Chopra, Sumit, and Bordes, Antoine. Memory Networks. 2015.
52. Xu, Kelvin, Ba, Jimmy, Kiros, Ryan, Cho, Kyunghyun, Courville, Aaron C., Salakhutdinov, Ruslan, Zemel, Richard S., and Bengio, Yoshua. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.
53. Yin, Pengcheng, Lu, Zhengdong, Li, Hang, and Kao, Ben. Neural enquirer: Learning to query tables with natural language. ArXiv, 2015.
54. Zelle, John M. and Mooney, Raymond J. Learning to parse database queries using inductive logic programming. In AAAI/IAAI, 1996.
55. Zeng, Z., Goodman, R., and Smyth, P. Discrete recurrent neural networks for grammatical inference. IEEE Transactions on Neural Networks, 1994.
56. Zettlemoyer, Luke S. and Collins, Michael. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In UAI, 2005.