A methodology for extrinsic evaluation of text summarization: Does ROUGE correlate?

Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Proceedings of the Workshop ACL 2005

Volumn , Issue , 2005, Pages 1-8

  Dorr, Bonnie J ^a   Monz, Christof ^a   President, Stacy ^a   Schwartz, Richard ^b   Zajic, David ^a  

^a University of Maryland ^*  (United States)

^b BBN TECHNOLOGIES   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

FULL-TEXT DOCUMENTS; GOLD STANDARDS; IMPLICATIONS FOR FUTURES; NEGATIVE CORRELATION; POSITIVE CORRELATIONS; RELEVANCE JUDGEMENT; RELEVANCE PREDICTION; TEXT SUMMARISATION; WEAK CORRELATION;

EID: 57849148109     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. James Allan, Hubert Jin, Martin Rajman, Charles Wayne, Daniel Gildea, Victor Lavrenko, Rose Hoberman, and David Caputo. 1999. Topic-based Novelty Detection. Technical Report 1999 Summer Workshop at CLSP Final Report, Johns Hopkins, Maryland.
2. Jean Carletta. 1996. Assessing Agreement on Classification Tasks: The Kappa Statistic. Computational Lingusitics, 22(2):249–254, June.
3. Bonnie J. Dorr, Christof Monz, Douglas Oard, Stacy President, and David Zajic. 2004. Extrinsic Evaluation of Automatic Metrics for Summarization. Technical report, University of Maryland, College Park, MD. LAMP-TR-115, CAR-TR-999, CS-TR-4610, UMIACS-TR-2004-48.
4. Barbara Di Eugenio and Michael Glass. 2004. Squibs and Discussions - The Kappa Statistic: A Second Look. Computational Linguistics, pages 95–101.
5. Donna Harman and Paul Over. 2004. Proceedings of the DUC 2004. Boston, MA.
6. Hongyan Jing, Regina Barzilay, Kathleen McKeown, and Michael Elhadad. 1998. Summarization evaluation methods: Experiments and analysis. In Proceedings of the AAAI Symposium on Intelligent Summarization, Stanford University, CA, March 23-25.
7. Chin-Yew Lin and Eduard Hovy. 2003. Automatic Evaluation of Summaries Using N-gram Co-Occurrence Statistics. In Proceedings of HLT-NAACL 2003 Workshop, pages 71–78, Edmonton Canada, May-June.
8. Chin-Yew Lin and Franz Joseph Och. 2004. ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation. In Proceedings of the 20th International Conference on Computational Linguistics (COLING 2004), Geneva, Switzerland, August 23–27.
9. Chin-Yew Lin. 2004. ROUGE: a Package for Automatic Evaluation of Summaries. In Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), Barcelona, Spain, July 25–26.
10. I. Mani and E. Bloedorn. 1998. Summarizing Similarities and Differences Among Related Documents. Information Retrieval, 1(1):35–67.
11. Inderjeet Mani, Gary Klein, David House, and Lynette Hirschman. 2002. SUMMAC: a text summarization evaluation. Natural Language Engineering, 8(1):43–68.
12. Inderjeet Mani. 2001. Summarization Evaluation: An Overview. In Proceedings of the NAACL 2001 Workshop on Automatic Summarization.
13. Ani Nenkova and Rebecca J. Passonneau. 2004. Evaluating Content Selection in Summarization: The Pyramid Method. In Proceedings of the NAACL 2004, Boston, MA.
14. Mamiko Oka and Yoshihiro Ueda. 2000. Evaluation of Phrase-Representation Summarization Based on an Information Retrieval Task. In Proceedings of the ANLP/NAACL 2000 Workshop on Automatic Summarization, pages 59–68, New Brunswick, NJ.
15. Dragomir Radev and Kathleen McKeown. 1998. Generating Natural Language Summaries from Multiple On-Line Sources. Computational Linguistics, pages 469–500.
16. Tetsuya Sakai and Karen Sparck-Jones. 2001. Generic Summaries for Indexing in Information Retrieval - Detailed Test Results. Technical Report TR513, Computer Laboratory, University of Cambridge.
17. Sidney Siegel and N. John Castellan, Jr. 1988. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill, New York, second edition.
18. Karen Sparck-Jones and J.R. Gallier. 1996. Evaluating Natural Language Processing Systems: An Analysis and Review. Springer, Berlin.
19. Anastasios Tombros and Mark Sanderson. 1998. Advantages of query biased summaries in information retrieval. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2–10.
20. Hans van Halteren and Simone Teufel. 2003. Examining the Consensus Between Human Summaries: Initial Experiments with Factoid Analysis. In Proceedings of the HLT-NAACL 03 Text Summarization Workshop.
21. David Zajic, Bonnie J. Dorr, Richard Schwartz, and Stacy President. 2004. Headline Evaluation Experiment Results. Technical report, University of Maryland, College Park, MD. UMIACS-TR-2004-18.