A multimodal LDA model integrating textual, cognitive and visual modalities

EMNLP 2013 - 2013 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

Volumn , Issue , 2013, Pages 1146-1157

  Roller, Stephen ^a   Walde, Sabine Schulte Im ^b  

^a University of Texas at Austin   (United States)

^b UNIVERSITY OF STUTTGART   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; STATISTICS;

BIMODAL MODELS; DIMENSIONAL MODEL; HOLISTIC VIEW; LATENT DIRICHLET ALLOCATION; LATENT STRUCTURES; UNSUPERVISED CLUSTER; VISUAL FEATURE; VISUAL MODALITIES;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 84906929694     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (49)

1. Mark Andrews, Gabriella Vigliocco, and David Vinson. 2009. Integrating experiential and distributional data to learn semantic representations. Psychological Review, 116(3): 463.
2. Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. In Transactions of the Association for Computational Linguistics, Volume 1, pages 49-62.
3. Lisa Aziz-Zadeh, Stephen M. Wilson, Giacomo Rizzolatti, and Marco Iacoboni. 2006. Congruent embodied representations for visually presented actions and linguistic phrases describing actions. Current Biology, 16(18): 1818-1823.
4. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The wacky wide web: a collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3): 209-226.
5. Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. 2008. Surf: Speeded up robust features. Computer Vision and Image Understanding, 110(3): 346-359, June.
6. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. The Journal of Machine Learning Research, 3: 993-1022.
7. Elia Bruni, Giang Binh Tran, and Marco Baroni. 2011. Distributional semantics from text and images. Proceedings of the EMNLP 2011 Geometrical Models for Natural Language Semantics, pages 22-32.
8. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012a. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, pages 136-145.
9. Elia Bruni, Jasper Uijlings, Marco Baroni, and Nicu Sebe. 2012b. Distributional semantics with eyes: Using image analysis to improve computational representations of word meaning. In Proceedings of the 20th ACM International Conference on Multimedia, pages 1219-1228.
10. David L. Chen and Raymond J. Mooney. 2011. Learning to interpret natural language navigation instructions from observations. Proceedings of the 25th AAAI Conference on Artificial Intelligence, pages 859-865, August.
11. David L. Chen, Joohyun Kim, and Raymond J. Mooney. 2010. Training a multilingual sportscaster: Using perceptual context to learn language. Journal of Artificial Intelligence Research, 37(1): 397-436.
12. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6): 391-407.
13. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE.
14. Thomas Deselaers and Vittorio Ferrari. 2011. Visual and semantic similarity in imagenet. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1777-1784.
15. Matthijs Douze, Hervé Jégou, Harsimrat Sandhawalia, Laurent Amsaleg, and Cordelia Schmid. 2009. Evaluation of gist descriptors for web-scale image search. In Proceedings of the ACM International Conference on Image and Video Retrieval, pages 19:1-19:8.
16. Jacob Eisenstein, Brendan O'Connor, Noah A. Smith, and Eric P. Xing. 2010. A latent variable model for geographic lexical variation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1277-1287.
17. Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. 2009. Describing objects by their attributes. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1778-1785.
18. Yansong Feng and Mirella Lapata. 2010a. How many words is a picture worth? Automatic caption generation for news images. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1239-1249.
19. Yansong Feng and Mirella Lapata. 2010b. Visual information in semantic representation. In Human Language Technologies: the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 91-99.
20. Thomas L. Griffiths, Mark Steyvers, and Joshua B. Tenenbaum. 2007. Topics in semantic representation. Psychological Review, 114(2): 211.
21. Matthew Hoffman, David M. Blei, and Francis Bach. 2010. Online learning for latent dirichlet allocation. Advances in Neural Information Processing Systems, 23: 856-864.
22. Matthew Hoffman, David M. Blei, Chong Wang, and John Paisley. 2012. Stochastic variational inference. ArXiv e-prints, June.
23. Brendan T. Johns and Michael N. Jones. 2012. Perceptual inference through global lexical similarity. Topics in Cognitive Science, 4(1): 103-120.
24. Dhiraj Joshi, James Z. Wang, and Jia Li 2006. The story picturing engine - a system for automatic text illustration. ACM Transactions on Multimedia Computing, Communications, and Applications, 2(1): 68-89.
25. Rohit J. Kate and Raymond J. Mooney. 2007. Learning language semantics from ambiguous supervision. In Proceedings of the 22nd Conference on Artificial Intelligence, Volume 7, pages 895-900.
26. David G. Lowe. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2): 91-110.
27. Stefan Mathe, Afsaneh Fazly, Sven Dickinson, and Suzanne Stevenson. 2008. Learning the abstract motion semantics of verbs from captioned videos. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages 1-8.
28. Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. 2012. Learning to parse natural language commands to a robot control system. In Proceedings of the 13th International Symposium on Experimental Robotics.
29. Ken McRae, George S. Cree, Mark S. Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods, 37(4): 547-559.
30. Tanvi S. Motwani and Raymond J. Mooney. 2012. Improving video activity recognition using object recognition and text mining. In ECAI, pages 600-605.
31. Douglas L. Nelson, Cathy L. McEvoy, and Thomas A. Schreiber. 2004. The University of South Florida free association, rhyme, and word fragment norms. Behavior Research Methods, Instruments, & Computers, 36(3): 402-407.
32. Aude Oliva and Antonio Torralba. 2001. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision, 42(3): 145-175.
33. Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. 2011. Im2text: Describing images using 1 million captioned photographs. In Advances in Neural Information Processing Systems, pages 1143-1151.
34. Devi Parikh and Kristen Grauman. 2011. Relative attributes. In International Conference on Computer Vision, pages 503-510. IEEE.
35. Friedemann Pulvermüller, Olaf Hauk, Vadim V. Nikulin, and Risto J Ilmoniemi. 2005. Functional links between motor and language systems. European Journal of Neuroscience, 21(3): 793-797.
36. Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. 2013. Grounding action descriptions in videos. In Transactions of the Association for Computational Linguistics, Volume 1, pages 25-36.
37. Marcus Rohrbach, Michael Stark, György Szarvas, Iryna Gurevych, and Bernt Schiele. 2010. What helps where-and why? Semantic relatedness for knowledge transfer. In IEEE Conference on Computer Vision and Pattern Recognition, pages 910-917.
38. Stephen Roller, Michael Speriosu, Sarat Rallapalli, Benjamin Wing, and Jason Baldridge. 2012. Supervised text-based geolocation using language models on an adaptive grid. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1500-1510.
39. Sabine Schulte im Walde, Susanne Borgwaldt, and Ronny Jauch. 2012. Association norms of german noun compounds. In Proceedings of the 8th International Conference on Language Resources and Evaluation, pages 632-639, Istanbul, Turkey.
40. D. Sculley. 2010. Web-scale k-means clustering. In Proceedings of the 19th International Conference on World Wide Web, pages 1177-1178.
41. Carina Silberer and Mirella Lapata. 2012. Grounded models of semantic representation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1423-1433, Jeju Island, Korea, July.
42. Carina Silberer, Vittorio Ferrari, and Mirella Lapata. 2013. Models of semantic representation with visual attributes. In Proceedings of the 51th Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, August.
43. Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, and Andrew Y. Ng. 2013. Zero-shot learning through cross-modal transfer. International Conference on Learning Representations.
44. Mark Steyvers. 2010. Combining feature norms and text data with topic models. Acta Psychologica, 133(3): 234-243.
45. Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth J. Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. Proceedings of the 25th AAAI Conference on Artificial Intelligence.
46. Marco Tettamanti, Giovanni Buccino, Maria Cristina Saccuman, Vittorio Gallese, Massimo Danna, Paola Scifo, Ferruccio Fazio, Giacomo Rizzolatti, Stefano F. Cappa, and Daniela Perani. 2005. Listening to action-related sentences activates fronto-parietal motor circuits. Journal of Cognitive Neuroscience, 17(2): 273-281.
47. Luis Von Ahn. 2006. Games with a purpose. Computer, 39(6): 92-94.
48. Claudia von der Heide and Susanne Borgwaldt. 2009. Assoziationen zu Unter-, Basis- und Oberbegriffen. Eine explorative Studie. In Proceedings of the 9th Norddeutsches Linguistisches Kolloquium, pages 51-74.
49. Benjamin Wing and Jason Baldridge. 2011. Simple supervised document geolocation with geodesic grids. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Volume 11, pages 955-964.