Framing image description as a ranking task: Data, models and evaluation metrics

Journal of Artificial Intelligence Research

Volumn 47, Issue , 2013, Pages 853-899

  Hodosh, Micah ^a   Young, Peter ^a   Hockenmaier, Julia ^a  

^a University of Illinois at Urbana Champaign   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

SEMANTICS;

AUTOMATIC EVALUATION; EVALUATION METRICS; FULLY AUTOMATED; HUMAN JUDGMENTS; IMAGE DESCRIPTIONS; NATURAL LANGUAGES; RELEVANCE JUDGMENT; SEMANTIC FEATURES;

IMAGE ANALYSIS;

EID: 84883394520     PISSN: None     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.3994     Document Type: Article

References (63)

1. Artstein, R., & Poesio, M. (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4), 555-596.
2. Bach, F. R., & Jordan, M. I. (2002). Kernel independent component analysis. Journal of Machine Learning Research, 3, 1-48.
3. Barnard, K., Duygulu, P., Forsyth, D., Freitas, N. D., and Blei, D. M., & Jordan, M. I. (2003). Matching words and pictures. Journal of Machine Learning Research, 3, 1107-1135.
4. Blei, D. M., & Jordan, M. I. (2003). Modeling annotated data. In SIGIR 2003: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 127-134, Toronto, Ontario, Canada.
5. Bloehdorn, S., Basili, R., Cammisa, M., & Moschitti, A. (2006). Semantic kernels for text classification based on topological measures of feature similarity. In Proceedings of the 6th IEEE International Conference on Data Mining (ICDM 2006), pp. 808-812, Hong Kong, China.
6. BNC Consortium (2007). The British National Corpus, version 3 (BNC XML edition). http://www.natcorp.ox.ac.uk.
7. Brown, P. F., Pietra, V. J. D., Pietra, S. A. D., & Mercer, R. L. (1993). The mathematics of statistical machine translation: parameter estimation. Computational Linguistics, 19(2), 263-311.
8. Callison-Burch, C., Osborne, M., & Koehn, P. (2006). Re-evaluation the role of bleu in machine translation research. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pp. 249-256, Trento, Italy.
9. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
10. Croce, D., Moschitti, A., & Basili, R. (2011). Structured lexical similarity via convolution kernels on dependency trees. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1034-1046, Edinburgh, UK.
11. Dale, R., & White, M. (Eds.). (2007). Workshop on Shared Tasks and Comparative Evaluation in Natural Language Generation: Position Papers, Arlington, VA, USA.
12. Datta, R., Joshi, D., Li, J., & Wang, J. Z. (2008). Image retrieval: Ideas, influences, and trends of the new age. ACM Computing Surveys, 40(2), 5:1-5:60.
13. Deschacht, K., & Moens, M.-F. (2007). Text analysis for automatic image annotation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL), pp. 1000-1007, Prague, Czech Republic.
14. Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7), 1895-1923. (Pubitemid 128463689)
15. Everingham, M., and Gool, L. V., Williams, C., Winn, J., & Zisserman, A. (2008). The PASCAL Visual Object Classes Challenge 2008 (VOC2008) Results. http://www.pascal-network.org/challenges/VOC/voc2008/workshop/.
16. Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., & Forsyth, D. (2010). Every picture tells a story: Generating sentences from images. In Proceedings of the European Conference on Computer Vision (ECCV), Part IV, pp. 15-29, Heraklion, Greece.
17. Fellbaum, C. (1998). WordNet: An Electronic Lexical Database. Bradford Books.
18. Felzenszwalb, P., McAllester, D., & Ramanan, D. (2008). A discriminatively trained, mul-tiscale, deformable part model. In Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-8, Anchorage, AK, USA.
19. Feng, Y., & Lapata, M. (2008). Automatic image annotation using auxiliary text information. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-08: HLT), pp. 272-280, Columbus, OH, USA.
20. Feng, Y., & Lapata, M. (2010). How many words is a picture worth? automatic caption generation for news images. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1239-1249, Uppsala, Sweden.
21. Fisher, R. A. (1935). The Design of Experiments. Olyver and Boyd, Edinburgh, UK.
22. Grangier, D., & Bengio, S. (2008). A discriminative kernel-based approach to rank images from text queries. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30, 1371-1384.
23. Grice, H. P. (1975). Logic and conversation. In Davidson, D., & Harman, G. H. (Eds.), The Logic of Grammar, pp. 64-75. Dickenson Publishing Co., Encino, CA, USA.
24. Grubinger, M., Clough, P., Müller, H., & Deselaers, T. (2006). The IAPR benchmark: A new evaluation resource for visual information systems. In OntoImage 2006, Workshop on Language Resources for Content-based Image Retrieval during LREC 2006, pp. 13-23, Genoa, Italy.
25. Gupta, A., Verma, Y., & Jawahar, C. (2012). Choosing linguistics over vision to describe images. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, Toronto, Ontario, Canada.
26. Hardoon, D. R., Saunders, C., Szedmak, S., & Shawe-Taylor, J. (2006). A correlation approach for automatic image annotation. In Li, X., Zaïane, O. R., & Li, Z.-H. (Eds.), Advanced Data Mining and Applications, Vol. 4093 of Lecture Notes in Computer Science, pp. 681-692. Springer Berlin Heidelberg. (Pubitemid 44503211)
27. Hardoon, D. R., and Szedmak, S. R., & Shawe-Taylor, J. R. (2004). Canonical correlation analysis: An overview with application to learning methods. Neural Computation, 16, 2639-2664. (Pubitemid 39604012)
28. Hotelling, H. (1936). Relations between two sets of variates. Biometrika, 28(3/4), 321-377.
29. Hwang, S., & Grauman, K. (2012). Learning the relative importance of objects from tagged images for retrieval and cross-modal search. International Journal of Computer Vision, 100(2), 134-153.
30. Jaimes, A., Jaimes, R., & Chang, S.-F. (2000). A conceptual framework for indexing visual information at multiple levels. In Internet Imaging 2000, Vol. 3964 of Proceedings of SPIE, pp. 2-15, San Jose, CA, USA. (Pubitemid 30535439)
31. Jurafsky, D., & Martin, J. H. (2008). Speech and Language Processing (2nd edition). Prentice Hall.
32. Krippendorff, K. (2004). Content analysis: An introduction to its methodology. Sage.
33. Kulkarni, G., Premraj, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2011). Baby talk: Understanding and generating simple image descriptions. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1601-1608.
34. Kuznetsova, P., Ordonez, V., Berg, A., Berg, T., & Choi, Y. (2012). Collective generation of natural image descriptions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 359-368, Jeju Island, Korea.
35. Lavrenko, V., Manmatha, R., & Jeon, J. (2004). A model for learning the semantics of pictures. In Thrun, S., Saul, L., & Schölkopf, B. (Eds.), Advances in Neural Information Processing Systems 16, Cambridge, MA, USA.
36. Lazebnik, S., Schmid, C., & Ponce, J. (2009). Spatial pyramid matching. In S. Dickinson, A. Leonardis, B. S., & Tarr, M. (Eds.), Object Categorization: Computer and Human Vision Perspectives, chap. 21, pp. 401-415. Cambridge University Press.
37. Li, S., Kulkarni, G., Berg, T. L., and Berg, A. C., & Choi, Y. (2011). Composing simple image descriptions using web-scale n-grams. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning (CoNLL), pp. 220-228, Portland, OR, USA.
38. Lin, C.-Y. (2004). Rouge: A package for automatic evaluation of summaries. In Marie-Francine Moens, S. S. (Ed.), Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pp. 74-81, Barcelona, Spain.
39. Lin, C.-Y., & Hovy, E. H. (2003). Automatic evaluation of summaries using n-gram cooccurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL), pp. 71-78, Edmonton, AB, Canada.
40. Lin, D. (1998). An information-theoretic definition of similarity. In Proceedings of the Fifteenth International Conference on Machine Learning (ICML), pp. 296-304, Madison, WI, USA.
41. Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. Internationa Journal of Computer Vision, 60(2), 91-110.
42. Makadia, A., Pavlovic, V., & Kumar, S. (2010). Baselines for image annotation. International Journal of Computer Vision, 90(1), 88-105.
43. Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.
44. Mitchell, M., Dodge, J., Goyal, A., Yamaguchi, K., Stratos, K., Han, X., Mensch, A., Berg, A., Berg, T., & Daume III, H. (2012). Midge: Generating image descriptions from computer vision detections. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pp. 747-756, Avignon, France.
45. Moschitti, A. (2009). Syntactic and semantic kernels for short text pair categorization. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pp. 576-584, Athens, Greece.
46. Moschitti, A., Pighin, D., & Basili, R. (2008). Tree kernels for semantic role labeling. Computational Linguistics, 34(2), 193-224.
47. Och, F. J., & Ney, H. (2003). A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1), 19-51. (Pubitemid 37049767)
48. Ordonez, V., Kulkarni, G., & Berg, T. L. (2011). Im2text: Describing images using 1 million captioned photographs. In Advances in Neural Information Processing Systems 24, pp. 1143-1151.
49. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 311-318, Philadelphia, PA, USA.
50. Popescu, A., Tsikrika, T., & Kludas, J. (2010). Overview of the Wikipedia retrieval task at ImageCLEF 2010. In CLEF (Notebook Papers/LABs/Workshops), Padua, Italy.
51. Porter, M. F. (1980). An algorithm for suffix stripping. Program, 14(3), 130-137.
52. Rashtchian, C., Young, P., Hodosh, M., & Hockenmaier, J. (2010). Collecting image annotations using Amazon's Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk, pp. 139-147, Los Angeles, CA, USA.
53. Rasiwasia, N., and Pereira, J. C., Coviello, E., Doyle, G., Lanckriet, G. R., Levy, R., & Vasconcelos, N. (2010). A new approach to cross-modal multimedia retrieval. In Proceedings of the International Conference on Multimedia (MM), pp. 251-260, New York, NY, USA.
54. Reiter, E., & Belz, A. (2009). An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4), 529-558.
55. Shatford, S. (1986). Analyzing the subject of a picture: A theoretical approach. Cataloging & Classification Quarterly, 6, 39-62.
56. Shawe-Taylor, J., & Cristianini, N. (2004). Kernel Methods for Pattern Analysis. Cambridge University Press.
57. Smucker, M. D., Allan, J., & Carterette, B. (2007). A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management (CIKM), pp. 623-632, Lisbon, Portugal.
58. Socher, R., & Li, F.-F. (2010). Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora. In Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 966-973, San Francisco, CA, USA.
59. van Erp, M., & Schomaker, L. (2000). Variants of the Borda count method for combining ranked classifier hypotheses. In Proceedings of the Seventh International Workshop on Frontiers in Handwriting Recognition (IWFHR), pp. 443-452, Nijmegen, Netherlands.
60. Varma, M., & Zisserman, A. (2005). A statistical approach to texture classification from single images. International Journal of Computer Vision, 62, 61-81.
61. Vedaldi, A., & Fulkerson, B. (2008). VLFeat: An open and portable library of computer vision algorithms. http://www.vlfeat.org/.
62. Weston, J., Bengio, S., & Usunier, N. (2010). Large scale image annotation: learning to rank with joint word-image embeddings. Machine Learning, 81(1), 21-35.
63. Yang, Y., Teo, C., Daume III, H., & Aloimonos, Y. (2011). Corpus-guided sentence generation of natural images. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 444-454, Edinburgh, UK.