Automatically assessing machine summary content without a gold standard

Computational Linguistics

Volumn 39, Issue 2, 2013, Pages 267-300

  Louis, Annie ^a   Nenkova, Ani ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATIC EVALUATION; GOLD STANDARDS; HUMAN ASSESSMENT; HUMAN JUDGMENTS; HUMAN MODEL; PSEUDO-MODELS; SINGLE MODELS; SOURCE TEXT;

QUALITY CONTROL;

EID: 84877033915     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/COLI_a_00123     Document Type: Article

References (48)

1. Albrecht, Joshua and Rebecca Hwa. 2007. Regression for sentence-level MT evaluation with pseudo references. In Proceedings of ACL, pages 296-303, Prague.
2. Albrecht, Joshua and Rebecca Hwa. 2008. The role of pseudo references in MT evaluation. In Proceedings of the Third Workshop on Statistical Machine Translation, ACL, pages 187-190, Columbus, OH.
3. Best, D. J. and D. E. Roberts. 1975. Algorithm as 89: The upper tail probabilities of Spearman's rho. Journal of the Royal Statistical Society. Series C (Applied Statistics), 24(3):377-379.
4. Callison-Burch, Chris, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 17-53, Uppsala.
5. Callison-Burch, Chris, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 22-64, Edinburgh.
6. Carbonell, Jaime and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of SIGIR, pages 335-336, Melbourne.
7. Conroy, John M., Jade Goldstein, Judith D. Schlesinger, and Dianne P. O'Leary. 2004. Left-brain/right-brain multi-document summarization. In Proceedings of the 4th Document Understanding Conference (DUC'04), Boston, MA. Available at: http://duc.nist.gov/pubs/2004papers/ida.conroy.ps.
8. Conroy, John M. and Dianne P. O'Leary. 2001. Text summarization via hidden Markov models. In Proceedings of SIGIR, pages 406-407, New Orleans, LA.
9. Conroy, John M., Judith D. Schlesinger, Jeff Kubina, Peter A. Rankel, and Dianne P. O'Leary. 2011. Classy 2011 at TAC: Guided and multi-lingual summaries and evaluation metrics. In Proceedings of TAC, Gaithersburg, MD. Available at: http://www.nist.gov/tac/publications/2011/participant.papers/CLASSY.proceedings.pdf.
10. Conroy, John M., Judith D. Schlesinger, and Dianne P. O'Leary. 2006. Topic-focused multi-document summarization using an approximate oracle score. In Proceedings of the COLING-ACL, pages 152-159, Sydney.
11. Deerwester, Scott, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the Americal Society for Information Science, 41(6):391-407.
12. Donaway, Robert L., Kevin W. Drummey, and Laura A. Mather. 2000. A comparison of rankings produced by summarization evaluation measures. In Proceedings of the NAACL-ANLP Workshop on Automatic Summarization, pages 69-78, Seattle, WA.
13. Erkan, Günȩ and Dragomir R. Radev. 2004. Lexpagerank: Prestige in multi-document text summarization. In Proceedings of EMNLP, pages 365-371, Barcelona.
14. Gillick, Dan and Benoit Favre. 2009. A scalable global model for summarization. In Proceedings of the Workshop on Integer Linear Programming for Natural Language Processing, pages 10-18, Boulder, CO.
15. Gillick, Dan and Yang Liu. 2010. Non-expert evaluation of summarization systems is risky. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 148-151, Los Angeles, CA.
16. Gong, Yihong and Xin Liu. 2001. Generic text summarization using relevance measure and latent semantic analysis. In Proceedings of SIGIR, pages 19-25, New Orleans, LA.
17. Haghighi, Aria and Lucy Vanderwende. 2009. Exploring content models for multi-document summarization. In Proceedings of HLT-NAACL, pages 362-370, Boulder, CO.
18. Harman, Donna and Paul Over. 2004. The effects of human variation in DUC summarization evaluation. In Proceedings of the ACL-04 Workshop: Text Summarization Branches Out, pages 10-17, Barcelona.
19. Jing, Hongyan, Regina Barzilay, Kathleen Mckeown, and Michael Elhadad. 1998. Summarization evaluation methods: Experiments and analysis. In AAAI Symposium on Intelligent Summarization, pages 60-68, Palo Alto, CA.
20. Kumar, Shankar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of HLT-NAACL, pages 169-176, Boston, MA.
21. Lin, Chin-Yew. 2004a. Looking for a few good metrics: Automatic summarization evaluation-how many samples are enough. In Proceedings of the NTCIR Workshop, volume 4, pages 1-10, Tokyo.
22. Lin, Chin-Yew. 2004b. ROUGE: A package for automatic evaluation of summaries. In Proceedings of the ACL Text Summarization Workshop, pages 74-81, Barcelona.
23. Lin, Chin-Yew, Guihong Cao, Jianfeng Gao, and Jian-Yun Nie. 2006. An information-theoretic approach to automatic evaluation of summaries. In Proceedings of HLT-NAACL, pages 463-470, New York, NY.
24. Lin, Chin-Yew and Eduard Hovy. 2000. The automated acquisition of topic signatures for text summarization. In Proceedings of COLING, pages 495-501, Saarbrücken.
25. Lin, Chin-Yew and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of HLT-NAACL, pages 71-78, Edmonton.
26. Louis, Annie and Ani Nenkova. 2008. Automatic summary evaluation without human models. In Proceedings of TAC, Gaithersburg, MD. Available at: http://www.nist.gov/tac/publications/2008/additional.papers/Penn.proceedings.pdf.
27. Louis, Annie and Ani Nenkova. 2009a. Automatically evaluating content selection in summarization without human models. In Proceedings of EMNLP, pages 306-314, Singapore.
28. Louis, Annie and Ani Nenkova. 2009b. Performance confidence estimation for automatic summarization. In Proceedings of EACL, pages 541-548, Athens.
29. Louis, Annie and Ani Nenkova. 2009c. Predicting summary quality using limited human input. In Proceedings of TAC, Gaithersburg, MD. Available at: http://www.nist.gov/tac/publications/2009/participant.papers/UPenn.proceedings.pdf.
30. Madnani, Nitin, Necip Fazil Ayan, Philip Resnik, and Bonnie J. Dorr. 2007. Using paraphrases for parameter tuning in statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 120-127, Prague.
31. McDonald, Ryan. 2007. A study of global inference algorithms in multi-document summarization. In Proceedings of ECIR, pages 557-564, Rome.
32. McKeown, Kathy, Regina Barzilay, David Evans, Vasileios Hatzivassiloglou, Barry Schiffman, and Simone Teufel. 2001. Columbia multi-document summarization: Approach and evaluation. In Proceedings of DUC, New Orleans, LA. Available at: http://www-nlpir.nist.gov/projects/duc/pubs/2001papers/columbiaredo.pdf.
33. Mihalcea, Rada and Paul Tarau. 2005. Multi-document summarization with iterative graph-based algorithms. In Proceedings of the First International Conference on Intelligent Analysis Methods and Tools (IA 2005), McLean, VA.
34. Nenkova, Ani and Annie Louis. 2008. Can you summarize this? Identifying correlates of input difficulty for multi-document summarization. In Proceedings of ACL-HLT, pages 825-833, Columbus, OH.
35. Nenkova, Ani and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of HLT-NAACL, pages 145-152, Boston, MA.
36. Nenkova, Ani, Rebecca Passonneau, and Kathleen McKeown. 2007. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Transactions on Speech and Language Processing, 4(2):4.
37. Nenkova, Ani, Lucy Vanderwende, and Kathleen McKeown. 2006. A compositional context sensitive multi-document summarizer: Exploring the factors that influence summarization. In Proceedings of SIGIR, pages 573-580, Seattle, WA.
38. Owkzarzak, Karolina and Hoa Trang Dang. 2009. Evaluation of automatic summaries: Metrics under varying data conditions. In Proceedings of the Workshop on Language Generation and Summarisation, pages 23-30, Singapore.
39. R Development Core Team. 2011. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna.
40. Radev, Dragomir, Timothy Allison, Sasha Blair-Goldensohn, John Blitzer, Arda Çelebi, Stanko Dimitrov, Elliott Drabek, Ali Hakim, Wai Lam, Danyu Liu, Jahna Otterbacher, Hong Qi, Horacio Saggion, Simone Teufel, Michael Topper, Adam Winkel, and Zhu Zhang. 2004a. MEAD-A platform for multidocument multilingual text summarization. In Proceedings of LREC 2004, pages 1-4, Lisbon.
41. Radev, Dragomir, Hongyan Jing, Malgorzata Sty, and Daniel Tam. 2004b. Centroid-based summarization of multiple documents. Information Processing and Management, 40:919-938.
42. Radev, Dragomir and Daniel Tam. 2003. Single-document and multi-document summary evaluation via relative utility. In Proceedings of CIKM, pages 508-511, New Orleans, LA.
43. Rath, G. J., A. Resnick, and R. Savage. 1961. The formation of abstracts by the selection of sentences: Part 1: Sentence selection by man and machines. American Documentation, 2(12):139-208.
44. Saggion, Horacio, Juan-Manuel, Torres Moreno, Iria da Cunha, Eric SanJuan, and Patricia Velazquez-Morales. 2010. Multilingual summarization evaluation without human models. In Proceedings of COLING, pages 1059-1067, Beijing.
45. Soboroff, Ian, Charles Nicholas, and Patrick Cahan. 2001. Ranking retrieval systems without relevance judgments. In Proceedings of SIGIR, pages 66-73, New Orleans, LA.
46. Tromble, Roy W., Shankar Kumar, Franz Och, and Wolfgang Macherey. 2008. Lattice minimum Bayes-risk decoding for statistical machine translation. In Proceedings of EMNLP, pages 620-629, Honolulu, HI.
47. van Halteren Hans and Simone Teufel. 2003. Examining the consensus between human summaries: Initial experiments with factoid analysis. In Proceedings of the HLT-NAACL DUC on Text Summarization Workshop, pages 57-64, Edmonton.
48. Zhang, Ying and Stephan Vogel. 2010. Significance tests of automatic machine translation evaluation metrics. Machine Translation, 24(1):51-65.