Inferring and Executing Programs for Visual Reasoning

Proceedings of the IEEE International Conference on Computer Vision

Volumn 2017-October, Issue , 2017, Pages 3008-3017

  Johnson, Justin ^a   Hariharan, Bharath ^b   Maaten, Laurens Van Der ^b   Hoffman, Judy ^a   Fei Fei, Li ^a   Zitnick, C Lawrence ^b   Girshick, Ross ^b  

^a STANFORD UNIVERSITY   (United States)

^b FACEBOOK AI RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ENGINES;

BLACK BOXES; BLACK-BOX MODEL; EXECUTION ENGINE; EXPLICIT REPRESENTATION; MODULE NETWORKS; REASONING PROCESS; VISUAL REASONING;

COMPUTER VISION;

EID: 85041924656     PISSN: 15505499     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICCV.2017.325     Document Type: Conference Paper

References (50)

1. J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Learning to compose neural networks for question answering. In NAACL, 2016.
2. J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks. In CVPR, 2016.
3. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Zitnick, and D. Parikh. VQA: Visual question answering. In ICCV, 2015.
4. M. Balog, A. Gaunt, M. Brockschmidt, S. Nowozin, and D. Tarlow. Deepcoder: Learning to write programs. In ICLR, 2017.
5. J. Cai, R. Shin, and D. Song. Making neural programming architectures generalize via recursion. In ICLR, 2017.
6. A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. Moura, D. Parikh, and D. Batra. Visual dialog. In CVPR, 2017.
7. J. Devlin, S. Gupta, R. Girshick, M. Mitchell, and C. L. Zitnick. Exploring nearest neighbor approaches for image captioning. arXiv preprint arXiv:1505. 04467, 2015.
8. A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In arXiv:1606. 01847, 2016.
9. Y. Gao, O. Beijbom, N. Zhang, and T. Darrell. Compact bilinear pooling. In CVPR, 2016.
10. R. Girshick. Fast R-CNN. In ICCV, 2015.
11. Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
12. A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv preprint arXiv:1410. 5401, 2014.
13. A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Barwinska, S. Colmenarejo, E. Grefenstette, T. Ramalho, J. Agapiou, A. Badia, K. Hermann, Y. Zwols, G. Ostrovski, A. Cain, H. King, C. Summerfield, P. Blunsom, K. Kavukcuoglu, and D. Hassabis. Hybrid computing using a neural network with dynamic external memory. Nature, 2016.
14. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
15. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9 (8):1735-1780, 1997.
16. R. Hu, J. Andreas, M. Rohrbach, T. Darrell, and K. Saenko. Learning to reason: End-to-end module networks for visual question answering. In ICCV, 2017.
17. R. Hu, M. Rohrbach, J. Andreas, T. Darrell, and K. Saenko. Modeling relationships in referential expressions with compositional modular networks. In CVPR, 2017.
18. A. Jabri, A. Joulin, and L. van der Maaten. Revisiting visual question answering baselines. In ECCV, 2016.
19. J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
20. A. Joulin and T. Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In NIPS, 2015.
21. K. Kafle and C. Kanan. Visual question answering: Datasets, algorithms, and future challenges. In arXiv preprint arXiv:1610. 01465, 2016.
22. L. Kaiser and I. Sutskever. Neural GPUs learn algorithms. In ICLR, 2016.
23. D. Klein and C. D. Manning. Accurate unlexicalized parsing. In ACL, pages 423-430, 2003.
24. R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
25. A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS. 2012.
26. K. Kurach, M. Andrychowicz, and I. Sutskever. Neural random-access machines. In ICLR, 2016.
27. B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 2016.
28. C. Liang, J. Berant, Q. Le, K. D. Forbus, and N. Lao. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. ACL, 2017.
29. P. Liang, M. I. Jordan, and D. Klein. Learning dependencybased compositional semantics. In ACL, 2011.
30. J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical question-image co-attention for visual question answering. In NIPS, 2016.
31. M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, 2014.
32. M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions about images. In ICCV, 2015.
33. A. Mallya and S. Lazebnik. Learning models for actions and person-object interactions with transfer to question answering. In ECCV, 2016.
34. A. Neelakantan, Q. V. Le, and I. Sutskever. Neural programmer: Inducing latent programs with gradient descent. In ICLR, 2016.
35. S. Reed and N. De Freitas. Neural programmer-interpreters. In ICLR, 2016.
36. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. ImageNet large scale visual recognition challenge. IJCV, 2015.
37. S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus. End-toend memory networks. In NIPS, 2015.
38. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In NIPS, 2014.
39. M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler. Movieqa: Understanding stories in movies through question-answering. In CVPR, 2016.
40. J. Weston, S. Chopra, and A. Bordes. Memory networks. In ICLR, 2015.
41. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8 (23), 1992.
42. T. Winograd. Understanding Natural Language. Academic Press, 1972.
43. Q. Wu, D. Teney, P. Wang, C. Shen, A. Dick, and A. van den Hengel. Visual question answering: A survey of methods and datasets. In arXiv preprint arXiv:1607. 05910, 2016.
44. C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering. ICML, 2016.
45. Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering. In CVPR, 2016.
46. W. Zaremba, T. Mikolov, A. Joulin, and R. Fergus. Learning simple algorithms from examples. In ICML, 2016.
47. W. Zaremba and I. Sutskever. Learning to execute. arXiv preprint arXiv:1410. 4615, 2014.
48. W. Zaremba and I. Sutskever. Reinforcement learning neural turing machines. arXiv preprint arXiv:1505. 00521, 2015.
49. P. Zhang, Y. Goyal, D. Summers-Stay, D. Batra, and D. Parikh. Yin and yang: Balancing and answering binary visual questions. In CVPR, 2016.
50. Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. Visual7W: Grounded question answering in images. In CVPR, 2016.