Visual question answering: Datasets, algorithms, and future challenges

Computer Vision and Image Understanding

Volumn 163, Issue , 2017, Pages 3-20

  Kafle, Kushal ^a   Kanan, Christopher ^a  

^a ROCHESTER INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Image understanding; Natural language processing; Vision and language

Indexed keywords

COMPUTER VISION; DEEP LEARNING; IMAGE UNDERSTANDING; VISUAL LANGUAGES;

EVALUATION METRICS; FUTURE CHALLENGES; LARGE AMOUNTS; POSSIBLE FUTURES; PROBLEM FORMULATION; QUESTION ANSWERING;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 85020874517     PISSN: 10773142     EISSN: 1090235X     Source Type: Journal    
DOI: 10.1016/j.cviu.2017.06.005     Document Type: Article

References (75)

1. Agrawal, A., Batra, D., Parikh, D., Analyzing the behavior of visual question answering models. Conference on Empirical Methods on Natural Language Processing (EMNLP), 2016.
2. Andreas, J., Rohrbach, M., Darrell, T., Klein, D., Deep compositional question answering with neural module networks. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
3. Andreas, J., Rohrbach, M., Darrell, T., Klein, D., Learning to compose neural networks for question answering. Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL), 2016.
4. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D., VQA: Visual question answering. The IEEE International Conference on Computer Vision (ICCV), 2015.
5. Antol, S., Zitnick, C.L., Parikh, D., Zero-shot learning via visual abstraction. European Conference on Computer Vision (ECCV), 2014.
6. Ba, J., Mnih, V., Kavukcuoglu, K., Multiple object recognition with visual attention. International Conference on Learning Representations (ICLR), 2015.
7. Bahdanau, D., Cho, K., Bengio, Y., Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations (ICLR), 2015.
8. Banerjee, S., Lavie, A., Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, Vol. 29, 2005, 65–72.
9. Bernardi, R., Cakici, R., Elliott, D., Erdem, A., Erdem, E., Ikizler-Cinbis, N., Keller, F., Muscat, A., Plank, B., Automatic description generation from images: A survey of models, datasets, and evaluation measures. Journal of Artificial Intelligence Research 55 (2016), 409–442.
10. Callison-Burch, C., Osborne, M., Koehn, P., Re-evaluation the role of BLEU in machine translation research. 2006.
11. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y., Learning phrase representations using RNN encoder-decoder for statistical machine translation. Conference on Empirical Methods on Natural Language Processing (EMNLP), 2014.
12. Das, A., Agrawal, H., Zitnick, C.L., Parikh, D., Batra, D., Human attention in visual question answering: Do humans and deep networks look at the same regions?. Conference on Empirical Methods on Natural Language Processing (EMNLP), 2016.
13. Devlin, J., Gupta, S., Girshick, R., Mitchell, M., Zitnick, C.L., Exploring nearest neighbor approaches for image captioning., 2015 arXiv preprint arXiv: 1505.04467.
14. Donahue, J., Hendricks, L.A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., Darrell, T., Long-term recurrent convolutional networks for visual recognition and description. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
15. Fang, H., Gupta, S., Iandola, F., Srivastava, R., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J., et al. From captions to visual concepts and back. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
16. Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., Rohrbach, M., Multimodal compact bilinear pooling for visual question answering and visual grounding. Conference on Empirical Methods on Natural Language Processing (EMNLP), 2016.
17. Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., Xu, W., Are you talking to a machine? Dataset and methods for multilingual image question answering. Advances in Neural Information Processing Systems (NIPS), 2015.
18. Geman, D., Geman, S., Hallonquist, N., Younes, L., Visual turing test for computer vision systems. Proceedings of the National Academy of Sciences, 112(12), 2015.
19. He, K., Zhang, X., Ren, S., Sun, J., Deep residual learning for image recognition. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
20. Hochreiter, S., Schmidhuber, J., Long short-term memory. Neural computation 9:8 (1997), 1735–1780.
21. Ilievski, I., Yan, S., Feng, J., A focused dynamic attention model for visual question answering., 2016 arXiv preprint arXiv: 1604.01485.
22. Jabri, A., Joulin, A., van der Maaten, L., Revisiting visual question answering baselines. European Conference on Computer Vision (ECCV), 2016.
23. Johnson, J., Karpathy, A., Fei-Fei, L., Densecap: Fully convolutional localization networks for dense captioning. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
24. Kafle, K., Kanan, C., Answer-type prediction for visual question answering. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
25. Karpathy, A., Fei-Fei, L., Deep visual-semantic alignments for generating image descriptions. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
26. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L., Large-scale video classification with convolutional neural networks. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
27. Kim, J.-H., Lee, S.-W., Kwak, D.-H., Heo, M.-O., Kim, J., Ha, J.-W., Zhang, B.-T., Multimodal residual learning for visual QA. Advances in Neural Information Processing Systems (NIPS), 2016.
28. Kim, J.-H., On, K.-W., Kim, J., Ha, J.-W., Zhang, B.-T., Hadamard product for low-rank bilinear pooling., 2016 arXiv preprint arXiv: 1610.04325.
29. Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R.S., Torralba, A., Urtasun, R., Fidler, S., Skip-thought vectors. Advances in Neural Information Processing Systems (NIPS), 2015.
30. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D.A., et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123:1 (2017), 32–73.
31. Kumar, A., Irsoy, O., Su, J., Bradbury, J., English, R., Pierce, B., Ondruska, P., Gulrajani, I., Socher, R., Ask me anything: Dynamic memory networks for natural language processing. International Conference on Machine Learning (ICML), 2016.
32. Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P.N., Hellmann, S., Morsey, M., van Kleef, P., Auer, S., et al. DBpedia–a large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web 6:2 (2015), 167–195.
33. Lin, C.-Y., Rouge: A package for automatic evaluation of summaries. Text summarization branches out: Proceedings of the ACL-04 workshop, Vol. 8, 2004, Barcelona, Spain.
34. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L., Microsoft COCO: Common objects in context. European Conference on Computer Vision (ECCV), 2014.
35. Lin, T.-Y., RoyChowdhury, A., Maji, S., Bilinear cnn models for fine-grained visual recognition. The IEEE International Conference on Computer Vision (ICCV), 2015.
36. Long, J., Shelhamer, E., Darrell, T., Fully convolutional networks for semantic segmentation. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
37. Lu, J., Yang, J., Batra, D., Parikh, D., Hierarchical question-image co-attention for visual question answering. Advances in Neural Information Processing Systems (NIPS), 2016.
38. Luong, M.-T., Pham, H., Manning, C.D., 2015. Effective approaches to attention-based neural machine translation.
39. Malinowski, M., Fritz, M., A multi-world approach to question answering about real-world scenes based on uncertain input. Advances in Neural Information Processing Systems (NIPS), 2014.
40. Malinowski, M., Fritz, M., Towards a visual turing challenge., 2014 arXiv preprint arXiv: 1410.8027.
41. Malinowski, M., Rohrbach, M., Fritz, M., Ask your neurons: A neural-based approach to answering questions about images. The IEEE International Conference on Computer Vision (ICCV), 2015.
42. Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., Yuille, A., Deep captioning with multimodal recurrent neural networks (m-rnn). International Conference on Learning Representations (ICLR), 2015.
43. Mikolov, T., Dean, J., Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems (NIPS), 2013.
44. Nam, H., Ha, J.-W., Kim, J., Dual attention networks for multimodal reasoning and matching., 2016 arXiv preprint arXiv: 1611.00471.
45. Noh, H., Han, B., Training recurrent answering units with joint loss minimization for VQA., 2016 arXiv preprint arXiv: 1606.03647.
46. Noh, H., Hong, S., Han, B., Learning deconvolution network for semantic segmentation. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
47. Noh, H., Seo, P.H., Han, B., Image question answering using convolutional neural network with dynamic parameter prediction. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
48. Papineni, K., Roukos, S., Ward, T., Zhu, W.-J., BLEU: a method for automatic evaluation of machine translation. Annual Meeting of the Association for Computational Linguistics (ACL), 2002.
49. Redmon, J., Divvala, S., Girshick, R., Farhadi, A., You only look once: Unified, real-time object detection. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
50. Ren, M., Kiros, R., Zemel, R., Exploring models and data for image question answering. Advances in Neural Information Processing Systems (NIPS), 2015.
51. Ren, S., He, K., Girshick, R., Sun, J., Faster r-CNN: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems (NIPS), 2015.
52. Saito, K., Shin, A., Ushiku, Y., Harada, T., Dualnet: Domain-invariant network for visual question answering., 2016 arXiv preprint arXiv: 1606.06108.
53. Shih, K.J., Singh, S., Hoiem, D., Where to look: Focus regions for visual question answering. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
54. Silberman, N., Hoiem, D., Kohli, P., Fergus, R., Indoor segmentation and support inference from RGBD images. European Conference on Computer Vision (ECCV), 2012.
55. Silberman, N., Sontag, D., Fergus, R., Instance segmentation of indoor scenes using a coverage loss. European Conference on Computer Vision (ECCV), 2014.
56. Simonyan, K., Zisserman, A., Two-stream convolutional networks for action recognition in videos. Advances in Neural Information Processing Systems (NIPS), 2014.
57. Simonyan, K., Zisserman, A., Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015.
58. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., Going deeper with convolutions. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
59. Szegedy, C., Toshev, A., Erhan, D., Deep neural networks for object detection. Advances in Neural Information Processing Systems (NIPS), 2013.
60. Thomee, B., Shamma, D.A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., Li, L.-J., Yfcc100m: The new data in multimedia research. Communications of the ACM 59:2 (2016), 64–73.
61. Torralba, A., Efros, A., Unbiased look at dataset bias. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
62. Vedantam, R., Lawrence Zitnick, C., Parikh, D., Cider: Consensus-based image description evaluation. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
63. Vinyals, O., Toshev, A., Bengio, S., Erhan, D., Show and tell: A neural image caption generator. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
64. Wu, Q., Wang, P., Shen, C., van den Hengel, A., Dick, A.R., Ask me anything: Free-form visual question answering based on knowledge from external sources. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
65. Wu, Z., Palmer, M., Verbs semantics and lexical selection. Annual Meeting of the Association for Computational Linguistics (ACL), 1994.
66. Xiong, C., Merity, S., Socher, R., Dynamic memory networks for visual and textual question answering. International Conference on Machine Learning (ICML), 2016.
67. Xu, H., Saenko, K., Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. European Conference on Computer Vision (ECCV), 2016.
68. Xu, K., Ba, J., Kiros, R., Courville, A., Salakhutdinov, R., Zemel, R., Bengio, Y., Show, attend and tell: Neural image caption generation with visual attention. International Conference on Machine Learning (ICML), 2015.
69. Yang, Z., He, X., Gao, J., Deng, L., Smola, A.J., Stacked attention networks for image question answering. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
70. Zhang, P., Goyal, Y., Summers-Stay, D., Batra, D., Parikh, D., Yin and yang: balancing and answering binary visual questions. CVPR, 2016.
71. Zhang, Z., Fidler, S., Urtasun, R., Instance-level segmentation with deep densely connected MRFs. CVPR, 2016.
72. Zhang, Z., Schwing, A.G., Fidler, S., Urtasun, R., Monocular object instance segmentation and depth ordering with CNNs. CVPR, 2015, 2614–2622.
73. Zhou, B., Tian, Y., Sukhbaatar, S., Szlam, A., Fergus, R., Simple baseline for visual question answering., 2015 arXiv preprint arXiv: 1512.02167.
74. Zhu, Y., Groth, O., Bernstein, M., Fei-Fei, L., Visual7w: Grounded question answering in images. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
75. Zitnick, C.L., Dollár, P., Edge boxes: Locating object proposals from edges. European Conference on Computer Vision, 2014, Springer, 391–405.