Automatically evaluating content selection in summarization without human models

EMNLP 2009 - Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: A Meeting of SIGDAT, a Special Interest Group of ACL, Held in Conjunction with ACL-IJCNLP 2009

Volumn , Issue , 2009, Pages 306-314

  Louis, Annie ^a   Nenkova, Ani ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATIC METHOD; HUMAN JUDGMENTS; HUMAN MODEL; JENSEN-SHANNON DIVERGENCE; MODEL-BASED OPC; PARTICIPATING SYSTEMS; TEXT ANALYSIS;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 78650467116     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.3115/1699510.1699550     Document Type: Conference Paper

References (16)

1. J. Conroy and H. Dang. 2008. Mind the gap: Dangers of divorcing evaluations of summary content from linguistic quality. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 145-152.
2. J. Conroy, J. Schlesinger, and D. O'Leary. 2006. Topic-focusedmulti- document summarization using an approximate oracle score. In Proceedings of ACL, short paper.
3. R. Donaway, K. Drummey, and L. Mather. 2000. A comparison of rankings produced by summarization evaluation measures. In NAACL-ANLP Workshop on Automatic Summarization.
4. H. Jing, R. Barzilay, K. Mckeown, and M. Elhadad. 1998. Summarization evaluation methods: Experiments and analysis. In In AAAI Symposium on Intelligent Summarization, pages 60-68.
5. M. Lapata and R. Barzilay. 2005. Automatic evaluation of text coherence: Models and representations. In IJCAI'05.
6. C. Lin and E. Hovy. 2000. The automated acquisition of topic signatures for text summarization. In Proceedings of the 18th conference on Computational linguistics, pages 495-501.
7. C. Lin and E. Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurance statistics. In Proceedings of HLT-NAACL 2003.
8. C. Lin, G. Cao, J. Gao, and J. Nie. 2006. An information-theoretic approach to automatic evaluation of summaries. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 463-470.
9. C. Lin. 2004. ROUGE: a package for automatic evaluation of summaries. In ACL Text Summarization Workshop.
10. K. McKeown, R. Barzilay, D. Evans, V. Hatzivassiloglou, B. Schiffman, and S. Teufel. 2001. Columbia multi-document summarization: Approach and evaluation. In DUC'01.
11. A. Nenkova and R. Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In HLT/NAACL.
12. A. Nenkova, R. Passonneau, and K. McKeown. 2007. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Trans. Speech Lang. Process., 4(2):4.
13. P. Over, H. Dang, and D. Harman. 2007. Duc in context. Inf. Process. Manage., 43(6):1506-1520.
14. D. Radev, S. Teufel, H. Saggion, W. Lam, J. Blitzer, H. Qi, A. Ç elebi, D. Liu, and E. Drabek. 2003. Evaluation challenges in large-scalemulti-document summarization: the mead project. In Proceedings of ACL 2003, Sapporo, Japan.
15. G. J. Rath, A. Resnick, and R. Savage. 1961. The formation of abstracts by the selection of sentences: Part 1: sentence selection by man and machines. American Documentation, 2(12):139-208.
16. I. Soboroff and D. Harman. 2005. Novelty detection: the trec experience. In HLT '05: Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 105-112