Human attention in visual question answering: Do humans and deep networks look at the same regions?

EMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Proceedings

Volumn , Issue , 2016, Pages 932-937

  Das, Abhishek ^d   Agrawal, Harsh ^d   Lawrence Zitnick, C ^b   Parikh, Devi ^a,c   Batra, Dhruv ^a,c  

^a VIRGINIA POLYTECHNIC INSTITUTE AND STATE UNIVERSITY   (United States)

^b FACEBOOK AI RESEARCH   (United States)

^c GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

^d NONE

Author keywords

[No Author keywords available]

Indexed keywords

ATTENTION MODEL; BLURRED IMAGE; DESIGN AND TESTS; HUMAN ATTENTION; LARGE-SCALE STUDIES; QUESTION ANSWERING; RANK ORDER; STATE OF THE ART;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 85072846928     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.18653/v1/d16-1092     Document Type: Conference Paper

References (25)

1. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural networks for question answering. In NAACL HLT. 2
2. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In ICCV. 2
3. Jimmy Lei Ba, Volodymyr Mnih, and Koray Kavukcuoglu. 2015. Multiple Object Recognition With Visual Attention. In ICLR. 1
4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR. 1
5. KyungHyun Cho, Aaron C. Courville, and Yoshua Bengio. 2015. Describing Multimedia Content using Attention-based Encoder-Decoder Networks. volume abs/1507.01053. 1
6. Jia Deng, Jonathan Krause, Michael Stark, and Li Fei-Fei. 2015. Leveraging the Wisdom of the Crowd for Fine-Grained Recognition. PAMI. 3
7. Jacob Devlin, Saurabh Gupta, Ross B. Girshick, Margaret Mitchell, and C. Lawrence Zitnick. 2015. Exploring nearest neighbor approaches for image captioning. volume abs/1505.04467. 1
8. Li Fei-Fei, Asha Iyer, Christof Koch, and Pietro Perona. 2007. What do we perceive in a glance of a real-world scene? Journal of Vision, 7(1):10. 2
9. Orhan Firat, KyungHyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. volume abs/1601.01073. 1
10. Jia Deng and Jonathan Krause and Li Fei-Fei. 2013. Fine-Grained Crowdsourcing for Fine-Grained Recognition. In CVPR. 3
11. Ming Jiang, Juan Xu, and Qi Zhao. 2014. Saliency in Crowd. In ECCV. 2
12. Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. 2015. Salicon: Saliency in context. In CVPR. 2, 3
13. Tilke Judd, Krista Ehinger, Frédo Du-rand, and Antonio Torralba. 2009. Learning to predict where humans look. In ICCV. 2, 4
14. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollr, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In ECCV. 2
15. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical Question-Image Co-Attention for Visual Question Answering. In NIPS. 1, 2, 4
16. Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. 2014. Recurrent Models of Visual Attention. In NIPS. 1
17. Ronald A. Rensink. 2000. The dynamic representation of scenes. Visual Cognition, 7(1-3):17-42. 1
18. Pierre Sermanet, Andrea Frome, and Esteban Real. 2014. Attention for Fine-Grained Categorization. volume abs/1412.7054. 1
19. Benjamin W. Tatler. 2007. The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributions. Journal of Vision, 7(14):4. 2, 4
20. Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In CHI. 3
21. Caiming Xiong, Stephen Merity, and Richard Socher. 2016. Dynamic memory networks for visual and textual question answering. In ICML. 1
22. Huijuan Xu and Kate Saenko. 2015. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. volume abs/1511.05234. 1
23. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In ICML. 1
24. Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander J. Smola. 2016. Stacked Attention Networks for Image Question Answering. In CVPR. 1, 2, 4
25. A. L. Yarbus. 1967. Eye Movements and Vision. Plenum. New York. 2