Correlation between rouge and human evaluation of extractive meeting summaries

ACL-08: HLT - 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference

Volumn , Issue , 2008, Pages 201-204

  Liu, Feifan ^a   Liu, Yang ^a  

^a UNIVERSITY OF TEXAS AT DALLAS   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; TEXT PROCESSING;

AUTOMATIC SUMMARIZATION; DISFLUENCIES; EVALUATING SYSTEMS; HUMAN EVALUATION; POTENTIAL PROBLEMS; SUMMARIZATION SYSTEMS; TEXT SUMMARIZATION;

PETROLEUM RESERVOIR EVALUATION;

EID: 84859896976     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.3115/1557690.1557747     Document Type: Conference Paper

References (20)

1. J. Carbonell and J. Goldstein. 1998. The use of mmr, diversitybased reranking for reordering documents and producing summaries. In SIGIR, pages 335-336.
2. M. Galley. 2006. A skip-chain conditional random field for ranking meeting utterances by importance. In EMNLP, pages 364-372.
3. C. Hori, T. Hori, and S. Furui. 2003. Evaluation methods for automatic speech summarization. In EUROSPEECH, pages 2825-2828.
4. E. Hovy, C. Lin, L. Zhou, and J. Fukumoto. 2006. Automated summarization evaluation with basic elements. In LREC.
5. A. Janin, D. Baron, J. Edwards, D. Ellis, G. Gelbart, N. Norgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, and C. Wooters. 2003. The icsi meeting corpus. In ICASSP.
6. K. S. Jones and J. Galliers. 1996. Evaluating natural language processing systems: An analysis and review. Lecture Notes in Artificial Intelligence.
7. D. Jones, F. Wlof, E. Gilbson, E. Williams, E. Fedorenko, D. Reynolds, and M. Zissman. 2003. Measuring the readability of automatic speech-to-text transcripts. In EUROSPEECH, pages 1585-1588.
8. C. Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Workshop on Text Summarization Branches Out at ACL, pages 74-81.
9. Y. Liu, F. Liu, B. Li, and S. Xie. 2007. Do disfluencies affect meeting summarization? a pilot study on the impact of disfluencies. In MLMI Workshop, Poster Session.
10. I. Mani, T. Firmin, D. House, M. Chrzanowski, G. Klein, L. Hirschman, B. Sundheim, and L. Obrst. 1998. The tipster summac text summarization evaluation: Final report. Technical report, The MITRE Corporation.
11. G. Murray, S. Renals, J. Carletta, and J. Moore. 2005. Evaluating automatic summaries of meeting recordings. In ACL 2005 MTSE Workshop, pages 33-40.
12. A. Nenkova and R. Passonneau. 2004. Evaluating content selection in summarization: the pyramid method. In HLT/NAACL.
13. NIST. 2007. Document understanding conference (DUC). http://duc.nist.gov/ .
14. D. Radev, T. Allison, S. Blair-Goldensohn, J. Blitzer, A. Ç elebi, E. Drabek, W. Lam, D. Liu, H. Qi, H. Saggion, S. Teufel, M. Topper, and A. Winkel. 2003. The MEAD Multidocument Summarizer. http://www.summarization.com/ mead/.
15. D. R. Radev, H. Jing, M. Stys, and T. Daniel. 2004. Centroidbased summarization of multiple documents. Information Processing and Management, 40:919-938.
16. E. Shriberg, R. Dhillon, S. Bhagat, J. Ang, and H. Carvey. 2004. The icsi meeting recorder dialog act (mrda) corpus. In SIGDAL Workshop, pages 97-100.
17. S. Teufel and H. Halteren. 2004. Evaluating information content by factoid analysis: Human annotation and stability. In EMNLP.
18. S. Xie and Y. Liu. 2008. Using corpus and knowledge-based similarity measure in maximum marginal relevance for meeting summarization. In ICASSP.
19. X. Zhu and G. Penn. 2005. Evaluation of sentence selection for speech summarization. In ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization.
20. X. Zhu and G. Penn. 2006. Comparing the roles of textual, acoustic and spoken-language features on spontaneousconversation summarization. In HLT/NAACL.