Listen, attend, andwalk: Neural mapping of navigational instructions to action sequences

30th AAAI Conference on Artificial Intelligence, AAAI 2016

Volumn , Issue , 2016, Pages 2772-2778

  Mei, Hongyuan ^a   Bansal, Mohit ^a   Walter, R Matthew ^a  

^a TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; AUTONOMOUS AGENTS; SYNTACTICS;

ACTION SEQUENCES; ENCODER-DECODER; LINGUISTIC RESOURCES; LONG SHORT TERM MEMORY; MODEL USE; MULTILEVELS; NATURAL LANGUAGES; SEQUENCE MODELING;

RECURRENT NEURAL NETWORKS;

EID: 85007207124     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (36)

1. Andreas, J., and Klein, D. 2015. Alignment-based compositional semantics for instruction following. In EMNLP.
2. Artzi, Y., and Zettlemoyer, L. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. TACL 1:49- 62.
3. Artzi, Y.; Das, D.; and Petrov, S. 2014. Learning compact lexicons for ccg semantic parsing. In EMNLP.
4. Ba, J.; Mnih, V.; and Kavukcuoglu, K. 2014. Multiple object recognition with visual attention. arXiv:1412.7755.
5. Bahdanau, D.; Cho, K.; and Bengio, Y. 2014. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473.
6. Chen, D. L., and Mooney, R. J. 2011. Learning to interpret natural language navigation instructions from observations. In AAAI.
7. Chen, X., and Zitnick, C. L. 2015. Mind's eye: A recurrent visual representation for image caption generation. In CVPR.
8. Chen, D. L. 2012. Fast online lexicon learning for grounded language acquisition. In ACL.
9. Cho, K.; van Merrienboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP.
10. Donahue, J.; Hendricks, L. A.; Guadarrama, S.; Rohrbach, M.; Venugopalan, S.; Saenko, K.; and Darrell, T. 2014. Long-Term recurrent convolutional networks for visual recognition and description. arXiv:1411.4389.
11. Graves, A.; Abdel-rahman, M.; and Hinton, G. 2013. Speech recognition with deep recurrent neural networks. In ICASSP.
12. Harnad, S. 1990. The symbol grounding problem. Physica D 42:335-346.
13. Hemachandra, S.; Duvallet, F.; Howard, T. M.; Roy, N.; Stentz, A.; and Walter, M. R. 2015. Learning models for following natural language directions in unknown environments. In ICRA.
14. Hochreiter, S., and Schmidhuber, J. 1997. Long short-Term memory. Neural Computation 9(8).
15. Karpathy, A., and Fei-Fei, L. 2015. Deep visual-semantic alignments for generating image descriptions. In CVPR.
16. Kim, J., and Mooney, R. J. 2012. Unsupervised PCFG induction for grounded language learning with highly ambiguous supervision. In EMNLP, 433-444.
17. Kim, J., and Mooney, R. J. 2013. Adapting discriminative reranking to grounded language learning. In ACL.
18. Kingma, D., and Ba, J. 2015. Adam: A method for stochastic optimization. In ICLR.
19. Kiros, R.; Salakhutdinov, R.; and Zemel, R. S. 2014. Unifying visual-semantic embeddings with multimodal neural language models. arXiv:1411.2539.
20. Kollar, T.; Tellex, S.; Roy, D.; and Roy, N. 2010. Toward understanding natural language directions. In HRI.
21. Kong, C.; Lin, D.; Bansal, M.; Urtasun, R.; and Fidler, S. 2014. What are you talking about? text-To-image coreference. In Proceedings of CVPR.
22. MacMahon, M.; Stankiewicz, B.; and Kuipers, B. 2006. Walk the talk: Connecting language, knowledge, and action in route instructions. In AAAI.
23. Mao, J.; Xu, W.; Yang, Y.; Wang, J.; and Yuille, A. 2014. Deep captioning with multimodal recurrent neural networks (m-RNN). arXiv:1412.6632.
24. Matuszek, C.; Fox, D.; and Koscher, K. 2010. Following directions using statistical machine translation. In HRI.
25. Mnih, V.; Hees, N.; Graves, A.; and Kavukcuoglu, K. 2014. Recurrent models of visual attention. In NIPS.
26. Mooney, R. J. 2008. Learning to connect language and perception. In AAAI.
27. Pascanu, R.; Gulcehre, C.; Cho, K.; and Bengio, Y. 2014. How to construct deep recurrent neural networks. arXiv:1312.6026.
28. Rush, A. M.; Chopra, S.; and Weston, J. 2015. A neural attention model for abstractive sentence summarization. In EMNLP.
29. Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014. Dropout: A simple way to prevent neural networks from overfitting. J. Machine Learning Research 15(1):1929-1958.
30. Sutskever, I.; Vinyals, O.; and Lee, Q. V. 2014. Sequence to sequence learning with neural networks. In NIPS.
31. Tellex, S.; Kollar, T.; Dickerson, S.;Walter, M. R.; Banerjee, A. G.; Teller, S.; and Roy, N. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI.
32. Vinyals, O.; Toshev, A.; Bengio, S.; and Erhan, D. 2015. Show and tell: A neural image caption generator. In CVPR.
33. Wen, T.-H.; Gǎsíc, M.; Mrksíc, N.; Su, P.-H.; Vandyke, D.; and Young, S. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In EMNLP.
34. Winograd, T. 1970. Proceedures as a Representation for Data in a Computer Program for Understanding Natural Language. Ph.D. Dissertation, MIT.
35. Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhutdinov, R.; Zemel, R.; and Bengio, Y. 2015. Show, attend and tell: Neural image caption generation with visual attention. In ICML.
36. Zaremba, W.; Sutskever, I.; and Vinyals, O. 2014. Recurrent neural network regularization. arXiv:1409.2329.