Multilingual summarization evaluation without human models

Coling 2010 - 23rd International Conference on Computational Linguistics, Proceedings of the Conference

Volumn 2, Issue , 2010, Pages 1059-1067

  Saggion, Horacio ^a   Torres Moreno, Juan Manuel ^b   Da Cunha, Iria ^a   Sanjuan, Eric ^c   Velázquez Morales, Patricia ^d  

^a UNIVERSITAT POMPEU FABRA   (Spain)

^b ÉCOLE POLYTECHNIQUE DE MONTRÉAL   (Canada)

^c Universite d'Avignon (LIA)   (France)

^d VM Labs   (France)

Author keywords

[No Author keywords available]

Indexed keywords

CONTENT-BASED; EVALUATION FRAMEWORK; EVALUATION MEASURES; EVALUATION METHOD; HUMAN MODEL; MULTI-DOCUMENT SUMMARIZATION; MULTILINGUAL SUMMARIZATION; TEXT SUMMARIZATION;

COMPUTATIONAL LINGUISTICS; TEXT PROCESSING;

PROBABILITY DISTRIBUTIONS;

EID: 80053423337     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (23)

1. da Cunha, Iria, Leo Wanner, and M. Teresa Cabré. 2007. Summarization of specialized discourse: The case of medical articles in spanish. Terminology, 13(2):249-286.
2. Donaway, Robert L., Kevin W. Drummey, and Laura A. Mather. 2000. A comparison of rankings produced by summarization evaluation measures. In NAACL-ANLP 2000 Workshop on Automatic Summarization, pages 69-78, Morristown, NJ, USA. ACL.
3. Fernandez, Silvia, Eric SanJuan, and Juan-Manuel Torres-Moreno. 2007. Textual Energy of Associative Memories: performants applications of Enertex algorithm in text summarization and topic segmentation. In MICAI'07, pages 861-871.
4. Kullback, S. and R.A. Leibler. 1951. On information and sufficiency. Annals of Mathematical Statistics, 22(1):79-86.
5. Lin, C.-Y. and E. Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of HLT-NAACL 2003, pages 71-78, Morristown, NJ, USA. ACL.
6. Lin, Chin-Yew, Guihong Cao, Jianfeng Gao, and Jian-Yun Nie. 2006. An information-theoretic approach to automatic evaluation of summaries. In Conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pages 463-470, Morristown, NJ, USA. ACL.
7. Lin, J. 1991a. Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory, 37(145-151).
8. Lin, Jianhua. 1991b. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37:145-151.
9. Lin, Chin-Yew. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Marie-Francine Moens, Stan Szpakowicz, editor, Text Summarization Branches Out: ACL-04 Workshop, pages 74-81, Barcelona, Spain, July.
10. Louis, Annie and Ani Nenkova. 2009. Automatically Evaluating Content Selection in Summarization without Human Models. In Conference on Empirical Methods in Natural Language Processing, pages 306-314, Singapore, August. ACL.
11. Mani, I., G. Klein, D. House, L. Hirschman, T. Firmin, and B. Sundheim. 2002. Summac: a text summarization evaluation. Natural Language Engineering, 8(1):43-68.
12. Nenkova, Ani and Rebecca Passonneau. 2004. Evaluating Content Selection in Summarization: The Pyramid Method. In Proceedings of NAACL-HLT 2004.
13. Over, Paul, Hoa Dang, and Donna Harman. 2007. Duc in context. Information Processing & Management, 43(6):1506-1520. (Pubitemid 47268109)
14. Owkzarzak, Karolina and Hoa Trang Dang. 2009. Evaluation of automatic summaries: Metrics under varying data conditions. In Proceedings of the 2009 Workshop on Language Generation and Summarisation (UCNLG+Sum 2009), pages 23-30, Suntec, Singapore, August. ACL.
15. Papineni, K., S. Roukos, T. Ward, and W. J. Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL'02: 40th Annual meeting of the Association for Computational Linguistics, pages 311-318.
16. Pastra, K. and H. Saggion. 2003. Colouring summaries Bleu. In Proceedings of Evaluation Initiatives in Natural Language Processing, Budapest, Hungary, 14 April. EACL.
17. Radev, Dragomir R., Simone Teufel, Horacio Saggion, Wai Lam, John Blitzer, Hong Qi, Arda Cȩlebi, Danyu Liu, and Elliott Drábek. 2003. Evaluation challenges in large-scale document summarization. In ACL, pages 375-382.
18. Saggion, H., D. Radev, S. Teufel, and W. Lam. 2002. Meta-evaluation of Summaries in a Cross-lingual Environment using Content-based Metrics. In Proceedings of COLING 2002, pages 849-855, Taipei, Taiwan, August 24-September 1.
19. Spärck-Jones, Karen and Julia Rose Galliers, editors. 1996. Evaluating Natural Language Processing Systems, An Analysis and Review, volume 1083 of Lecture Notes in Computer Science. Springer.
20. Spiegel, S. and N.J. Castellan, Jr. 1998. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill International.
21. Torres-Moreno, Juan-Manuel, Patricia Velźquez-Morales, and Jean-Guy Meunier. 2002. Condensś de textes par des méthodes numŕiques. In JADT'02, volume 2, pages 723-734, St Malo, France.
22. Vivaldi, Jorge, Iria da Cunha, Juan-Manuel Torres-Moreno, and Patricia Velázquez-Morales. 2010. Automatic summarization using terminological and semantic resources. In LREC'10, volume 2, page 10, Malta.
23. Yatsko, V.A. and T.N. Vishnyakov. 2007. A method for evaluating modern systems of automatic text summarization. Automatic Documentation and Mathematical Linguistics, 41(3):93-103.