Learning to answer questions from image using convolutional neural network

30th AAAI Conference on Artificial Intelligence, AAAI 2016

Volumn , Issue , 2016, Pages 3567-3573

  Ma, Lin ^a   Lu, Zhengdong ^a   Li, Hang ^a  

^a HUAWEI NOAH S ARK LAB   (China)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; NEURAL NETWORKS;

BENCHMARK DATASETS; CONVOLUTIONAL NEURAL NETWORK; END TO END; IMAGE CONTENT; MODAL INTERACTIONS; MULTI-MODAL; QUESTION ANSWERING; STATE OF THE ART;

CONVOLUTION;

EID: 85007153677     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (35)

1. Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Zitnick, C. L.; and Parikh, D. 2015. VQA: visual question answering. arXiv 1505.00468.
2. Chen, X., and Zitnick, C. L. 2014. Learning a recurrent visual representation for image caption generation. arXiv 1411.5654.
3. Dahl, G. E.; Sainath, T. N.; and Hinton, G. E. 2013. Improving deep neural networks for LVCSR using rectified linear units and dropout. In ICASSP.
4. Donahue, J.; Hendricks, L. A.; Guadarrama, S.; Rohrbach, M.; Venugopalan, S.; Saenko, K.; and Darrell, T. 2014. Longterm recurrent convolutional networks for visual recognition and description. arXiv 1411.4389.
5. Fang, H.; Gupta, S.; Iandola, F. N.; Srivastava, R.; Deng, L.; Dolĺar, P.; Gao, J.; He, X.; Mitchell, M.; Platt, J. C.; Zitnick, C. L.; and Zweig, G. 2014. From captions to visual concepts and back. arXiv 1411.4952.
6. Frome, A.; Corrado, G.; Shlens, J.; Bengio, S.; Dean, J.; Ranzato, M.; and Mikolov, T. 2013. Devise: A deep visualsemantic embedding model. In NIPS.
7. Gao, H.; Mao, J.; Zhou, J.; Huang, Z.; Wang, L.; and Xu, W. 2015. Are you talking to a machine? dataset and methods for multilingual image question answering. arXiv 1505.05612.
8. Geman, D.; Geman, S.; Hallonquist, N.; and Younes, L. 2015. Visual turing test for computer vision systems. In PNAS.
9. Hu, B.; Lu, Z.; Li, H.; and Chen, Q. 2014. Convolutional neural network architectures for matching natural language sentences. In NIPS.
10. Kalchbrenner, N.; Grefenstette, E.; and Blunsom, P. 2014. A convolutional neural network for modelling sentences. In ACL.
11. Karpathy, A., and Li, F.-F. 2014. Deep visual-semantic alignments for generating image descriptions. arXiv 1412.2306.
12. Karpathy, A.; Joulin, A.; and Li, F.-F. 2014. Deep fragment embeddings for bidirectional image sentence mapping. In NIPS.
13. Kim, Y. 2014. Convolutional neural networks for sentence classification. In EMNLP.
14. Kiros, R.; Salakhutdinov, R.; and Zemel, R. 2014a. Multimodal neural language models. In ICML.
15. Kiros, R.; Salakhutdinov, R.; and Zemel, R. S. 2014b. Unifying visual-semantic embeddings with multimodal neural language models. arXiv 1411.2539.
16. Klein, B.; Lev, G.; Sadeh, G.; and Wolf, L. 2015. Fisher vectors derived from hybrid gaussian-laplacian mixture models for image annotation. In CVPR.
17. Lecun, Y., and Bengio, Y. 1995. Convolutional networks for images, speech and time series. The Handbook of Brain Theory and Neural Networks.
18. Ma, L.; Lu, Z.; Shang, L.; and Li, H. 2015. Multimodal convolutional neural networks for matching image and sentence. In ICCV.
19. Malinowski, M., and Fritz, M. 2014a. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS.
20. Malinowski, M., and Fritz, M. 2014b. Towards a visual turing challenge. arXiv 1410.8027.
21. Malinowski, M., and Fritz, M. 2015. Hard to cheat: A turing test based on answering questions about images. arXiv 1501.03302.
22. Malinowski, M.; Rohrbach, M.; and Fritz, M. 2015. Ask your neurons: A neural-based approach to answering questions about images. arXiv 1505.01121.
23. Mao, J.; Xu, W.; Yang, Y.; Wang, J.; and Yuille, A. L. 2014a. Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv 1412.6632.
24. Mao, J.; Xu, W.; Yang, Y.; Wang, J.; and Yuille, A. L. 2014b. Explain images with multimodal recurrent neural networks. arXiv 1410.1090.
25. Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J. 2013. Efficient estimation of word representations in vector space. arXiv 1301.3781.
26. Nakamura, T.; Nagai, T.; Funakoshi, K.; Nagasaka, S.; Taniguchi, T.; and Iwahashi, N. 2013. Mutual learning of an object concept and language model based on mlda and npylm. In IROS.
27. Ordonez, V.; Kulkarni, G.; and Berg, T. L. 2011. Im2text: Describing images using 1 million captioned photographs. In NIPS.
28. Ren, M.; Kiros, R.; and Zemel, R. S. 2015. Exploring models and data for image question answering. arXiv 1505.02074.
29. Simonyan, K., and Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition. arXiv 1409.1556.
30. Socher, R.; Karpathy, A.; Le, Q. V.; Manning, C. D.; and Ng, A. Y. 2014. Grounded compositional semantics for finding and describing images with sentences. In TACL.
31. Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; and Rabinovich, A. 2015. Going deeper with convolutions. In CVPR.
32. Vinyals, O.; Toshev, A.; Bengio, S.; and Erhan, D. 2014. Show and tell: A neural image caption generator. arXiv 1411.4555.
33. Wu, Z., and Palmer, M. S. 1994. Verb semantics and lexical selection. In ACL.
34. Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhutdinov, R.; Zemel, R.; and Bengio, Y. 2015a. Show, attend and tell: Neural image caption generation with visual attention. arXiv 1502.03044.
35. Xu, R.; Xiong, C.; Chen, W.; and Corso, J. 2015b. Jointly modeling deep video and compositional text to bridge vision and language in a unified framework. In AAAI.