Exploring Correlation Between ROUGE and Human Evaluation on Meeting Summaries

IEEE Transactions on Audio, Speech and Language Processing

Volumn 18, Issue 1, 2010, Pages 187-196

  Liu, Feifan ^a   Liu, Yang ^a  

^a The University of Texas at Dallas   (United States)

Author keywords

Correlation; disfluencies; evaluation; meeting summarization; ROUGE

Indexed keywords

EID: 85008539481     PISSN: 15587916     EISSN: 15587924     Source Type: Journal    
DOI: 10.1109/TASL.2009.2025096     Document Type: Article

References (27)

1. F. Liu and Y. Liu, “Correlation between rouge and human evaluation of extractive meeting summaries,” in Proc. ACL, 2008, pp. 201–204.
2. S. Furui, T. Kikuchi, Y. Shinnaka, and C. Hori, “Speech-to-text and speech-to-speech summarization of spontaneous speech,” IEEE Trans. Speech Audio Process., vol. 12, no. 4, pp. 401–408, Jul. 2004.
3. S. Maskey and J. Hirschberg, “Comparing lexical, acoustic/prosodic, structural and discourse features for speech summarization,” in Proc. Interspeech, 2005, pp. 621–624.
4. X. Zhu and G. Penn, “Evaluation of sentence selection for speech summarization,” in Proc. RANLP, Workshop Crossing Barriers in Text Summarization Research, 2005, pp. 39–45.
5. L. Lee and B. Chen, “Spoken document understanding and organization,” IEEE Signal Process. Mag., vol. 22, no. 5, pp. 42–60, Sep. 2005.
6. K. McKeown, R. J. Passonneau, D. K. Elson, A. Nenkova, and J. Hirschberg, “Do summaries help?,” in Proc. SIGIR, 2005, pp. 210–217.
7. M. Galley, “A skip-chain conditional random held for ranking meeting utterances by importance,” in Proc. EMNLP, 2006, pp. 364–372.
8. G. Murray, S. Renals, J. Carletta, and J. Moore, “Evaluating automatic summaries of meeting recordings,” in Proc. ACL MTSE Workshop, 2005, pp. 33–40.
9. S. Xie and Y. Liu, “Using corpus and knowledge-based similarity measure in maximum marginal relevance for meeting summarization,” in Proc. ICASSP, 2008, pp. 4985–4988.
10. C. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Proc. Workshop Text Summarization Branches Out ACL, 2004, pp. 74–81.
11. K. S. Jones and J. Galliers, “Evaluating natural language processing systems: An analysis and review,” Lecture Notes in Artificial Intelligence, vol. 1083, 1996.
12. D. R. Radev, H. Jing, M. Stys, and T. Daniel, “Centroid-based summarization of multiple documents,” Inf. Process. Manage., vol. 40, pp. 919–938, 2004.
13. S. Teufel and H. Halteren, “Evaluating information content by factoid analysis: Human annotation and stability,” in Proc. EMNLP, 2004, pp. 419–426.
14. A. Nenkova and R. Passonneau, “Evaluating content selection in summarization: The pyramid method,” in Proc. HLT/NAACL, 2004, pp. 145–152.
15. E. Hovy, C. Lin, L. Zhou, and J. Fukumoto, “Automated summarization evaluation with basic elements,” in Proc. LREC, 2006, pp. 899–902.
16. I. Mani, T. Firmin, D. House, M. Chrzanowski, G. Klein, L. Hirschman, B. Sundheim, and L. Obrst, “The Tipster Summac Text Summarization Evaluation: Final Report,” 1998, The MITRE Corp., Tech. Rep.
17. S. P. Hobson, B. J. Dorr, C. Monz, and R. Schwartz, “Task-based evaluation of text summarization using relevance prediction,” Inf. Process. Manage. Special Iss. Summarization, vol. 43, pp. 1482–1499, 2007.
18. NIST, “DUC2007: Task, Documents, and Measures,” in Proc. Document Understanding Conf. (DUG), 2007 [Online]. Available: http:// www-nlpir.nist.gov/projects/duc/duc2007/tasks.html, NIST
19. C. Hori, T. Hori, and S. Furui, “Evaluation methods for automatic speech summarization,” Eurospeech, pp. 2825–2828, 2003.
20. X. Zhu, C. Barras, S. Meignier, and J.-L. Gauvain, “Combining speaker identification and BIC for speaker diarization,” in Proc. Interspeech '05, 2005, pp. 2441–2444.
21. Y. Liu, F. Liu, B. Li, and S. Xie, “Do disfluencies affect meeting summarization? a pilot study on the impact of disfluencies,” in Proc. MLMI Workshop, Poster Session, 2007.
22. X. Zhu and G. Penn, “Comparing the roles of textual, acoustic and spoken-language features on spontaneous-conversation summarization,” in Proc. HLT/NAACL, 2006, pp. 197–200.
23. D. Jones, F. Wlof, E. Gilbson, E. Williams, E. Fedorenko, D. Reynolds, and M. Zissman, “Measuring the readability of automatic speech-to-text transcripts,” in Proc. Eurospeech, 2003, pp. 1585–1588.
24. A. Janin, D. Baron, J. Edwards, D. Ellis, G. Gelbart, N. Norgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, and C. Wooters, “The ICSI meeting corpus,” in Proc. ICASSP, 2003, pp. 364–367.
25. E. Shriberg, R. Dhillon, S. Bhagat, J. Ang, and H. Carvey, “The ICSI meeting recorder dialog act (MRDA) corpus,” in Proc. SIGDIAL Workshop, 2004, pp. 97–100.
26. J. Carbonell and J. Goldstein, “The use of MMR, diversity-based reranking for reordering documents and producing summaries,” Proc. SIGIR, pp. 335–336, 1998.
27. K. S. Jones “A statistical interpretation of term specificity and its application in retrieval,” J. Documentation, vol. 28, no. 1, pp. 132–142, 1972.