Multimodal convolutional neural networks for matching image and sentence

Proceedings of the IEEE International Conference on Computer Vision

Volumn 2015 International Conference on Computer Vision, ICCV 2015, Issue , 2015, Pages 2623-2631

  Ma, Lin ^a   Lu, Zhengdong ^a   Shang, Lifeng ^a   Li, Hang ^a  

^a HUAWEI NOAH S ARK LAB   (China)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; CONVOLUTION; NEURAL NETWORKS; SEMANTICS;

CNN MODELS; CONVOLUTIONAL NEURAL NETWORK; END TO END; IMAGE CONTENT; IMAGE REPRESENTATIONS; MULTI-MODAL; STATE-OF-THE-ART APPROACH;

IMAGE MATCHING;

EID: 84973864182     PISSN: 15505499     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICCV.2015.301     Document Type: Conference Paper

References (37)

1. O. Abdel Hamid, A. R. Mohamed, H. Jiang, and G. Penn. Applying convolutional neural networks concepts to hybrid nn-hmm model for speech recognition. ICASSP, 2012. 3
2. R. Caruana, S. Lawrence, and C. L. Giles. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. NIPS, 2000. 6
3. X. Chen and C. L. Zitnick. Learning a recurrent visual representation for image caption generation. arXiv: 1411. 5654, 2014. 2, 6, 7
4. G. E. Dahl, T. N. Sainath, and G. E. Hinton. Improving deep neural networks for lvcsr using rectified linear units and dropout. ICASSP, 2013. 2
5. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darreell. Long-term recurrent convolutional networks for visual recognition and description. arXiv: 1411. 4389, 2014. 2, 7
6. A. Frame, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. A. Ranzato, and T. Mikolov. Devise: A deep visual-semantic embedding model. NIPS, 2013. 1, 2, 6, 7, 8
7. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierachies for accurate object detection and semantic segmentation. CVPR, 2014. 8
8. Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. lazebnik. Improving image-sentence embeddings using large weakly annotated photo collections. ECCV, 2014. 1
9. D. Grangier and S. Bengio. A neural network to retrieve images from text queries. ICANN, 2006. 2
10. K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. ECCV, 2014. 1
11. K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: surpassing human-level performance on imagenet classification. arXiv: 1502. 01852, 2015. 1
12. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Improving neural networks by proventing co-adaptation of feature detecters. arXiv: 1207. 0580, 2012. 6
13. M. Hodosh, P. Young, and J. Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47: 853-899, 2013. 1, 2, 6
14. B. Hu, Z. Lu, H. Li, and Q. Chen. Convolutional neural network architectures for matching natural language sentences. NIPS, 2014. 1, 3
15. N. Kalchbrenner, E. Grefenstette, and P. Blunsom. A convolutional neural network for modelling sentences. ACL, 2014. 1
16. A. Karpathy, A. Joulin, and F.-F. Li. Deep fragment embeddings for bidirectional image sentence mapping. NIPS, 2014. 1, 2, 6, 7, 8
17. A. Karpathy and F.-F. Li. Deep visual-semantic alignments for generating image descriptions. arXiv: 1412. 2306, 2014. 2, 6, 7
18. Y. Kim. Convolutional neural network for sentence classification. EMNLP, 2014. 1
19. R. Kiros, R. Salakhutdinov, and R. Zemel. Multimodal neural language model. ICML, 2014. 2
20. R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv: 1411. 2539, 2014. 2, 6, 7
21. R. Kiros and C. Szepesvári. Deep representations and codes for image auto-annotation. NIPS, 2012. 1
22. Y. LeCun and Y. Bengio. Convolutional networks for images, speech and time series. The Handbook of Brain Theory and Neural Networks, 3361, 1995. 3
23. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Deep captioning with multimodal recurrent neural networks (mrnn). arXiv: 1412. 6632, 2014. 2, 7
24. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Explain images with multimodal recurrent neural networks. arXiv: 1410. 1090, 2014. 2, 6, 7
25. T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv: 1301. 3781, 2013. 6
26. V. Ordonez, G. Kulkarni, and T. L. Berg. Im2txt: Describing images using 1 million captioned photogrphs. NIPS, 2011. 1
27. M. A. Sadeghi and A. Farhadi. Recognition using visual phrases. CVPR, 2011. 1, 2
28. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Intergrated recognition, localization and detection using convolutional networks. arXiv: 1312. 6229, 2014. 2, 5, 6, 7
29. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv: 1409. 1556, 2014. 1, 2, 5, 6, 7
30. R. Socher, Q. V. L. A. Karpathy, C. D. Manning, and A. Y. Ng. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2: 207-218, 2014. 1, 2, 6, 7, 8
31. N. Srivastava and R. Salakhutdinov. Learning representations for multimodal data with deep belief nets. ICML Representation Learning Workshop, 2012. 1, 2
32. N. Srivastava and R. Salakhutdinov. Multimodal learning with deep boltzmann machines. NIPS, 2012. 1, 2
33. C. Szegedy, W. Liu, Y. Jia, P. Sermannet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv: 1409. 4842, 2014. 1, 7
34. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. arXiv: 1411. 4556, 2014. 2, 6, 7
35. J. Weston, S. Bengio, and N. Usunier. Wsabie: Scaling up to large vocabulary image annotation. IJCAI, 2011. 2
36. P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014. 7
37. C. L. Zitnick, D. Parikh, and L. Vanderwende. Learning the visual interpretation of sentences. ICCV, 2013. 1, 2