Distributional semantics with eyes: Using image analysis to improve computational representations of word meaning

MM 2012 - Proceedings of the 20th ACM International Conference on Multimedia

Volumn , Issue , 2012, Pages 1219-1228

  Bruni, Elia ^a   Uijlings, Jasper ^a   Baroni, Marco ^a   Sebe, Nicu ^a  

^a UNIVERSITY OF TRENTO   (Italy)

Author keywords

language; object recognition; semantics; visual words

Indexed keywords

IMAGE ANALYSIS TECHNIQUES; IMAGE ANNOTATION; LANGUAGE; PROCESSING TECHNIQUE; SEMANTIC RELATEDNESS; SEMANTIC REPRESENTATION; VISUAL FEATURE; VISUAL WORD; WORD MEANING;

OBJECT RECOGNITION; SEMANTICS; TEXT PROCESSING; VISUAL LANGUAGES;

IMAGE ANALYSIS;

EID: 84871385304     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2393347.2396422     Document Type: Conference Paper

References (61)

1. M. Andrews, G. Vigliocco, and D. Vinson. Integrating experiential and distributional data to learn semantic representations. Psych. Rev., 116(3):463-498, 2009.
2. K. Barnard, P. Duygulu, D. Forsyth, N. de Freitas, D. Blei, and M. Jordan. Matching words and pictures. J. Machine Learning Research, 3:1107-1135, 2003.
3. K. Barnard and M. Johnson. Word sense disambiguation with pictures. Artificial Intelligence, 167:13-30, 2005.
4. M. Baroni, E. Barbu, B. Murphy, and M. Poesio. Strudel: A distributional semantic model based on properties and types. Cognitive Science, 34(2):222-254, 2010.
5. M. Baroni and A. Lenci. Concepts and properties in word spaces. Italian Journal of Linguistics, 20(1):55-88, 2008.
6. M. Baroni and A. Lenci. How we BLESSed distributional semantic evaluation. In Proceedings of the EMNLP GEMS Workshop, pages 1-10, 2011.
7. T. Berg, A. Berg, and J. Shih. Automatic attribute discovery and characterization from noisy Web data. In Proceedings of ECCV, pages 663-676, 2010.
8. S. Bergsma and R. Goebel. Using visual information to predict lexical preference. In Proceedings of Recent Advances in Natural Language Processing, pages 399-405, 2011.
9. D. Blei, A. Ng, and M. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993-1022, 2003.
10. A. Bosch, A. Zisserman, and X. Munoz. Scene classification using a hybrid generative/discriminative approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(4):712-727, 2008.
11. E. Bruni, G. Boleda, M. Baroni, and N. K. Tran. Distributional semantics in technicolor. In Proceedings of the ACL 2012, South Korea, 2012.
12. E. Bruni, G. B. Tran, and M. Baroni. Distributional semantics from text and images. In Proceedings of the EMNLP GEMS Workshop, pages 22-32, Edinburgh, 2011.
13. J. Caicedo, J. Ben-Abdallah, F. González, and O. Nasraoui. Multimodal representation, indexing, automated annotation and retrieval of image collections via non-negative matrix factorization. Neurocomputing, 76(1):50-60, 2012.
14. G. Csurka, C. Dance, L. Fan, J. Willamowski, and C. Bray. Visual categorization with bags of keypoints. In ECCV Workshop on Statistical Learning in Computer Vision, pages 1-22, 2004.
15. N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In Proceedings of CVPR, 2005.
16. R. Datta, D. Joshi, J. Li, and J. Wang. Image retrieval: Ideas, inuences, and trends of the new age. ACM Computing Surveys, 40(2), 2008.
17. H. Escalante, C. Hérnadez, L. Sucar, and M. Montes. Late fusion of heterogeneous methods for multimedia image retrieval. In ACM Conference on Multimedia Information Retrieval, 2008.
18. M. Everingham, L. van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes (VOC) challenge. International Jounal of Computer Vision, 2010.
19. S. Evert. The Statistics of Word Cooccurrences. Dissertation, Stuttgart University, 2005.
20. A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth. Describing objects by their attributes. In Proceedings of CVPR, pages 1778-1785, 2009.
21. A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences from images. In Proceedings of ECCV, 2010.
22. P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32:1627-1645, 2010.
23. Y. Feng and M. Lapata. Visual information in semantic representation. In Proceedings of HLT-NAACL, pages 91-99, 2010.
24. K. Grauman and T. Darrell. The pyramid match kernel: Discriminative classification with sets of image features. In Proceedings of ICCV, pages 1458-1465, 2005.
25. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. Baby talk: Understanding and generating simple image descriptions. In Proceedings of CVPR, 2011.
26. C. H. Lampert, M. B. Blaschko, and T. Hofmann. Efficient Subwindow Search: A Branch and Bound Framework for Object Localization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31:2129-2142, 2009.
27. T. Landauer and S. Dumais. A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psych. Rev., 104(2):211-240, 1997.
28. I. Laptev. Improving Object Detection with Boosted Histograms. Image and Vision Computing, 27:535-544, 2009.
29. V. Lavrenko, R. Manmatha, and J. Jeon. A model for learning the semantics of pictures. In NIPS, 2003.
30. S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proceedings of CVPR, pages 2169-2178, 2006.
31. C. W. Leong and R. Mihalcea. Going beyond text: A hybrid image-text approach for measuring word relatedness. In Proceedings of IJCNLP, pages 1403-1407, 2011.
32. D. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), Nov. 2004.
33. W. Lowe. Towards a theory of semantic space. In Proceedings of CogSci, pages 576-581, 2001.
34. K. Lund and C. Burgess. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, 28:203-208, 1996.
35. C. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval. Cambridge University Press, Cambridge, 2008.
36. G. Miller and W. Charles. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1-28, 1991.
37. D. Moore and G. McCabe. Introduction to the Practice of Statistics. Freeman, New York, 5 edition, 2005.
38. F. Moosmann, B. Triggs, and F. Jurie. Fast discriminative visual codebooks using randomized clustering forests. In Neural Information Processing Systems, pages 985-992, 2006.
39. B. Murphy. A Study of Notions of Participation and Discourse in Argument Structure Realisation. Dissertation, Trinity College Dublin, 2007.
40. D. Nister and H. Stewenius. Scalable recognition with a vocabulary tree. In Proceedings of CVPR, pages 2161-2168, 2006.
41. S. Padó and M. Lapata. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161-199, 2007.
42. D. Parikh and K. Grauman. Relative attributes. In Proceedings of ICCV, 2011.
43. T.-T. Pham, N. Maillot, J.-H. Lim, and J.-P. Chevallet. Latent semantic fusion model for image retrieval and annotation. In Proceedings of CIKM, pages 439-443, 2007.
44. R. Rapp. Word sense discovery based on sense descriptor dissimilarity. In Proceedings of MT Summit, pages 315-322, 2003.
45. N. Rasiwasia, J. Costa Pereira, E. Coviello, G. Doyle, G. Lanckriet, R. Levy, and N. Vasconcelos. A new approach to cross-modal multimedia retrieval. In ACM MM, 2010.
46. B. Riordan and M. Jones. Redundancy in perceptual and linguistic experience: Comparing feature-based and distributional models of semantic representation. Topics in Cognitive Science, 3(2):1-43, 2011.
47. M. Rohrbach, M. Stark, G. Szarvas, I. Gurevych, and B. Schiele. What helps where-and why? semantic relatedness for knowledge transfer. In Proceedings of CVPR, 2010.
48. M. Sahlgren. An introduction to random indexing. http://www.sics.se/ ~mange/papers/RI-intro.pdf, 2005.
49. M. Sahlgren. The Word-Space Model. Dissertation, Stockholm University, 2006.
50. H. Schütze. Ambiguity Resolution in Natural Language Learning. CSLI, Stanford, CA, 1997.
51. J. Sivic and A. Zisserman. Video Google: A Text Retrieval Approach to Object Matching in Videos. In Proceedings of ICCV, 2003.
52. C. G. M. Snoek. The Authoring Metaphor to Machine Understanding of Multimedia. PhD thesis, University of Amsterdam, 2005.
53. P. Turney and P. Pantel. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141-188, 2010.
54. J. R. R. Uijlings, A. W. M. Smeulders, and R. J. H. Scha. Real-time Visual Concept Classification. IEEE Transactions on Multimedia, 12, 2010.
55. K. E. A. van de Sande, T. Gevers, and C. G. M. Snoek. Evaluating Color Descriptors for Object and Scene Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010.
56. K. E. A. van de Sande, J. Uijlings, T. Gevers, and A. Smeulders. Segmentation as Selective Search for Object Recognition. In Proceedings of ICCV, 2011.
57. P. Viola and M. J. Jones. Robust real-time face detection. International Journal of Computer Vision, 57:137-154, 2004.
58. D. Vreeswijk, B. Huurnink, and A. Smeulders. Text and image subject classifiers: dense works better. In ACM MM, 2011.
59. G. Wang and D. Forsyth. Joint learning of visual attributes, object classes and visual saliency. In Proceedings of ICCV, 2009.
60. J. Yang, Y.-G. Jiang, A. G. Hauptmann, and C.-W. Ngo. Evaluating bag-of-visual-words representations in scene classification. In Multimedia Information Retrieval, pages 197-206, 2007.
61. Z.-J. Zha, L. Yang, T. Mei, M. Wang, and Z. Wang. Visual query suggestion. In Proceedings of ACM Multimedia, 2009.