Evaluation of automatic summaries: Metrics under varying data conditions

ACL-IJCNLP 2009 - UCNLG+Sum 2009: 2009 Workshop on Language Generation and Summarisation, Proceedings of the Workshop

Volumn , Issue , 2009, Pages 23-30

  Owczarzak, Karolina ^a   Dang, Hoa Trang ^a  

^a NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; NATURAL LANGUAGE PROCESSING SYSTEMS;

AUTOMATIC METRICS; CONDITION; DATA SET; HUMAN JUDGMENTS;

STATISTICAL TESTS;

EID: 80053407364     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (10)

1. Donna Harman and Paul Over. 2004. The effects of human variation in DUC summarization evaluation. In Proceedings of the ACL-04 Workshop: Text Summarization Branches Out, pages 10-17, Barcelona, Spain.
2. Eduard Hovy, Chin-Yew Lin, and Liang Zhou. 2005. Evaluating DUC 2005 using Basic Elements. In Proceedings of the 5th Document Understanding Conference (DUC).
3. Chin-Yew Lin. 2004a. Looking for a few good metrics: Automatic summarization evaluation-how many samples are enough? In Proceedings of NT-CIR Workshop 4, Tokyo, Japan.
4. Chin-Yew Lin. 2004b. ROUGE: A package for automatic evaluation of summaries. In Proceedings of the ACL 2004 Workshop: Text Summarization Branches Out, pages 74-81.
5. Ani Nenkova and Rebecca J. Passonneau. 2004. Evaluating content selection in summarization: The Pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 145-152, Boston, MA.
6. Rebecca J. Passonneau, Ani Nenkova, Kathleen McKeown, and Sergey Sigelman. 2005. Applying the Pyramid method in DUC 2005. In Proceedings of the 5th Document Understanding Conference (DUC), Vancouver, Canada.
7. Karen Spärck Jones. 2007. Automatic summarising: The state of the art. Information Processing and Management, 43(6):1449-1481.
8. Hans van Halteren and Simone Teufel. 2003. Examining the consensus between human summaries: Initial experiments with factoid analysis. In Proceedings of the HLT-NAACL DUC Workshop 2003, pages 57-64, Edmonton, Canada.
9. Ellen M. Voorhees and Chris Buckley. 2002. Effect of topic set size on retrieval experiment error. In Proceedings of the 25th Annual International ACM SI-GIR conference on Research and Development in Information Retrieval, pages 317-323, Tampere, Finland.
10. Ying Zhang and Stephan Vogel. 2004. Measuring confidence intervals for the Machine Translation evaluation metrics. In Proceedings of the 10th Conference on Theoretical and Methodological Issues in Machine Translation, pages 85-94, Baltimore, MD.