Learning image embeddings using convolutional neural networks for improved multi-modal semantics

EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

Volumn , Issue , 2014, Pages 36-45

  Kiela, Douwe ^a   Bottou, Léon ^b  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CONVOLUTION; DEEP NEURAL NETWORKS; LARGE DATASET; MULTILAYER NEURAL NETWORKS; NATURAL LANGUAGE PROCESSING SYSTEMS; OBJECT RECOGNITION; SEMANTIC WEB; SEMANTICS;

BAG-OF-VISUAL-WORDS; CONVOLUTIONAL NEURAL NETWORK; LINGUISTIC REPRESENTATIONS; PERFORMANCE GAIN; SEMANTIC RELATEDNESS; TRANSFER LEARNING; VISUAL CONCEPT; VISUAL FEATURE;

IMAGE ENHANCEMENT;

EID: 84952650015     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.3115/v1/d14-1005     Document Type: Conference Paper

References (40)

1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and wordnet-based approaches. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL '09, pages 19-27, Boulder, Colorado.
2. Lawrence W. Barsalou. 2008. Grounded cognition. Annual Review of Psychology, 59:617-845.
3. Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. 2008. SURF: Speeded Up Robust Features. In Computer Vision and Image Understanding (CVIU), volume 110, pages 346-359.
4. Steven Bird, Edward Loper, and Ewan Klein. 2009. Natural Language Processing with Python. O'Reilly Media Inc.
5. Anna Bosch, Andrew Zisserman, and Xavier Munoz. 2007. Image classification using random forests and ferns. In Proceedings of ICCV.
6. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam- Khanh Tran. 2012. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 136-145. Association for Computational Linguistics.
7. Elia Bruni, Nam Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1-47.
8. Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients for human detection. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1 - Volume 01, CVPR '05, pages 886-893.
9. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE.
10. Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2014. DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. In International Conference on Machine Learning (ICML 2014).
11. Xavier Driancourt and Léon Bottou. 1990. TDNNextracted features. In Proceedings of Neuro Nimes 90, Nimes, France. EC2.
12. Christiane Fellbaum. 1999. WordNet. Wiley Online Library.
13. Yansong Feng and Mirella Lapata. 2010. Visual information in semantic representation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 91-99. Association for Computational Linguistics.
14. Fangxiang Feng, Ruifan Li, and Xiaojie Wang. 2013. Constructing hierarchical image-tags bimodal representations for word tags alternative choice. CoRR.
15. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th international conference on World Wide Web, pages 406- 414. ACM.
16. Andrea Frome, Greg Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, MarcÁurelio Ranzato, and Tomas Mikolov. 2013. DeViSE: A Deep Visual- Semantic Embedding Model. In NIPS.
17. R. Girshick, J. Donahue, T. Darrell, and J. Malik. 2013. Rich feature hierarchies for accurate object detection and semantic segmentation. arXiv preprint:1311.2524, November.
18. Douwe Kiela and Stephen Clark. 2014. A Systematic Study of Semantic Vector Space Model Parameters. In Proceedings of EACL 2014, Workshop on Continuous Vector Space Models and their Compositionality (CVSC).
19. Douwe Kiela, Felix Hill, Anna Korhonen, and Stephen Clark. 2014. Improving Multi-Modal Representations Using Image Dispersion: Why Less is Sometimes More. In Proceedings of ACL 2014.
20. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1106- 1114.
21. Angeliki Lazaridou, Elia Bruni, and Marco Baroni. 2014. Is this a wampimuk? cross-modal mapping between distributional semantics and the visual world. In Proceedings of ACL 2014.
22. Geoffrey Leech, Roger Garside, and Michael Bryant. 1994. Claws4: The tagging of the British National Corpus. In Proceedings of the 15th conference on Computational linguistics-Volume 1, pages 622- 628. Association for Computational Linguistics.
23. Ben Leong and Rada Mihalcea. 2011. Going Beyond Text: A Hybrid Image-Text Approach for Measuring Word Relatedness. In Proceedings of Joint International Conference on Natural Language Processing (IJCNLP), Chiang Mai, Thailand.
24. Omer Levy and Yoav Goldberg. 2014. Dependencybased word embeddings. In Proceedings of ACL 2014.
25. M. M. Louwerse. 2011. Symbol interdependency in symbolic and embodied cognition. TopiCS in Cognitive Science, 3:273-302.
26. David G. Lowe. 1999. Object recognition from local scale-invariant features. In Proceedings of the International Conference on Computer Vision-Volume 2 - Volume 2, ICCV '99.
27. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of International Conference of Learning Representations, Scottsdale, Arizona, USA.
28. M. Oquab, L. Bottou, I. Laptev, and J. Sivic. 2014. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
29. A.S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. 2014. CNN features off-the-shelf: An astounding baseline for recognition. arXiv preprint:1403.6382.
30. Stephen Roller and Sabine Schulte im Walde. 2013. A multimodal LDA model integrating textual, cognitive and visual modalities. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1146-1157, Seattle, Washington, USA, October. Association for Computational Linguistics.
31. D Sculley. 2010. Web-scale k-means clustering. In Proceedings of the 19th international conference on World wide web, pages 1177-1178. ACM.
32. Carina Silberer and Mirella Lapata. 2012. Grounded models of semantic representation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1423-1433. Association for Computational Linguistics.
33. Carina Silberer and Mirella Lapata. 2014. Learning Grounded Meaning Representations with Autoencoders. In Proceedings of ACL 2014, Baltimore, MD.
34. J. Sivic and A. Zisserman. 2003. Video Google: A text retrieval approach to object matching in videos. In Proceedings of the Ninth IEEE International Conference on Computer Vision, volume 2, pages 1470- 1477, Oct.
35. Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng. 2014. Grounded Compositional Semantics for Finding and Describing Images with Sentences. Transactions of the Association for Computational Linguistics (TACL 2014).
36. Nitish Srivastava and Ruslan Salakhutdinov. 2012. Multimodal learning with deep boltzmann machines. In F. Pereira, C.J.C. Burges, L. Bottou, and K.Q.Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 2222-2230.
37. A. Vedaldi and B. Fulkerson. 2008. VLFeat: An open and portable library of computer vision algorithms. http://www.vlfeat.org/.
38. Luis Von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 319-326. ACM.
39. PengchengWu, Steven C.H. Hoi, Hao Xia, Peilin Zhao, Dayong Wang, and Chunyan Miao. 2013. Online multimodal deep similarity learning with application to image retrieval. In Proceedings of the 21st ACM International Conference on Multimedia, MM '13, pages 153-162.
40. Matthew D. Zeiler and Rob Fergus. 2013. Visualizing and understanding convolutional networks. CoRR.