A decade of automatic content evaluation of news summaries: Reassessing the state of the art

ACL 2013 - 51st Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference

Volumn 2, Issue , 2013, Pages 131-136

  Rankel, Peter A ^a   Conroy, John M ^b   Dang, Hoa Trang ^c   Nenkova, Ani ^d  

^a UNIVERSITY OF MARYLAND   (United States)

^b IDA   (United States)

^c NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY   (United States)

^d UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATIC CONTENT; AUTOMATIC EVALUATION; AUTOMATIC PREDICTION; PRECISION AND RECALL; STATE OF THE ART; STATISTICALLY SIGNIFICANT DIFFERENCE; SYNTACTIC DEPENDENCIES; TEXT ANALYSIS;

COMPUTATIONAL LINGUISTICS;

EID: 84907356594     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (20)

1. Enrique Amigó, Julio Gonzalo, and Felisa Verdejo. 2012. The heterogeneity principle in evaluation measures for automatic summarization. In Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization, pages 36-43, Montréal, Canada, June. Association for Computational Linguistics.
2. John M. Conroy and Hoa Trang Dang. 2008. Mind the gap: Dangers of divorcing evaluations of summary content from linguistic quality. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 145-152, Manchester, UK, August. Coling 2008 Organizing Committee.
3. John M. Conroy, Judith D. Schlesinger, and Dianne P. O'Leary. 2011. Nouveau-ROUGE: A Novelty Metric for Update Summarization. Computational Linguistics, 37(1):1-8.
4. George Giannakopoulos, Vangelis Karkaletsis, George A. Vouros, and Panagiotis Stamatopoulos. 2008. Summarization system evaluation revisited: N-gram graphs. TSLP, 5(3).
5. George Giannakopoulos, George A. Vouros, and Vangelis Karkaletsis. 2010. Mudosng: Multidocument summaries using n-gram graphs (tech report). CoRR, abs/1012.2042.
6. Eduard Hovy, Chin-Yew Lin, Liang Zhou, and Junichi Fukumoto. 2006. Automated summarization evaluation with basic elements. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06), pages 899-902.
7. Alistair Kennedy, Anna Kazantseva Saif Mohammad, Terry Copeck, Diana Inkpen, and Stan Szpakowicz. 2011. Getting emotional about news. In Fourth Text Analysis Conference (TAC 2011).
8. Niraj Kumar, Kannan Srinathan, and Vasudeva Varma. 2011. Using unsupervised system with least linguistic features for tac-aesop task. In Fourth Text Analysis Conference (TAC 2011).
9. N. Kumar, K. Srinathan, and V. Varma. 2012. Using graph based mapping of co-occurring words and closeness centrality score for summarization evaluation. Computational Linguistics and Intelligent Text Processing, pages 353-365.
10. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Stan Szpakowicz Marie-Francine Moens, editor, Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74-81, Barcelona, Spain, July. Association for Computational Linguistics.
11. Annie Louis and Ani Nenkova. 2013. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39:267-300.
12. Ani Nenkova and Annie Louis. 2008. Can you summarize this? identifying correlates of input difficulty for multi-document summarization. In ACL, pages 825-833.
13. Ani Nenkova, Rebecca J. Passonneau, and Kathleen McKeown. 2007. The pyramid method: Incorporating human content selection variation in summarization evaluation. TSLP, 4(2).
14. Ani Nenkova. 2005. Discourse factors in multidocument summarization. In AAAI, pages 1654-1655.
15. Karolina Owczarzak, John M. Conroy, Hoa Trang Dang, and Ani Nenkova. 2012. An assessment of the accuracy of automatic evaluation in summarization. In Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization, pages 1-9, Montréal, Canada, June. Association for Computational Linguistics.
16. Peter Rankel, John Conroy, Eric Slud, and Dianne O'Leary. 2011. Ranking human and machine summarization systems. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 467-473, Edinburgh, Scotland, UK., July. Association for Computational Linguistics.
17. Peter A. Rankel, John M. Conroy, and Judith D. Schlesinger. 2012. Better metrics to automatically predict the quality of a text summary. Algorithms, 5(4):398-420.
18. Stephen Tratz and Eduard Hovy. 2008. Summarisation evaluation using transformed basic elements. In Proceedings TAC 2008. NIST.
19. John von Neumann and Oskar Morgenstern. 1953. Theory of games and economic behavior. Princeton Univ. Press, Princeton, NJ, 3. ed. edition.
20. Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In Proceedings of the 18th conference on Computational linguistics - Volume 2, COLING '00, pages 947-953, Stroudsburg, PA, USA. Association for Computational Linguistics.