Task-based evaluation of text summarization using Relevance Prediction

Information Processing and Management

Volumn 43, Issue 6, 2007, Pages 1482-1499

  Hobson, Stacy President ^a   Dorr, Bonnie J ^a   Monz, Christof ^b   Schwartz, Richard ^c  

^a University of Maryland ^*  (United States)

^b QUEEN MARY UNIVERSITY OF LONDON   (United Kingdom)

^c BBN TECHNOLOGIES   (United States)

Author keywords

Relevance prediction; Summarization evaluation; Summary usefulness

Indexed keywords

DATA REDUCTION; STATISTICAL METHODS; THEOREM PROVING; WEB BROWSERS;

RELEVANCE PREDICTION; SUMMARIZATION EVALUATION; SUMMARY USEFULNESS;

DATA PROCESSING;

EID: 34547945128     PISSN: 03064573     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ipm.2007.01.002     Document Type: Article

References (36)

1. Ahmad, K., Vrusias, B., & de Oliveira, P. C. F. (2003). Summary evaluation and text categorization. In Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval. Toronto, Canada (July).
2. Allan, J., Jin, H., Rajman, M., Wayne, C., Gildea, D., & Lavrenko, V. (1999). Topic-based novelty detection. Tech. Rep. 1999 summer workshop at CLSP final report. Maryland: Johns Hopkins.
3. Brandow R., Mitze K., and Rau L.F. Automatic condensation of electronic publications by sentence selection. Information Processing and Management 31 5 (1995) 675-685
4. Carletta J. Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics 22 2 (1996) 249-254 June
5. Dang, H. T. (2005). Overview of DUC 2005. In Proceedings of the document understanding conferences (DUC). Vancouver: Canada (October).
6. In: Davison A., and Hinkley D. (Eds). Bootstrap methods and their application (1997), Cambridge University Press, Cambridge, United Kingdom
7. Dorr, B. J., Zajic, D., & Schwartz, R. (2003). Hedge trimmer: a parse-and-trim approach to headline generation. In Proceedings of the human language technology - North American chapter of the association for computational linguistics (HLT-NAACL) text summarization workshop. Alta., Canada (May).
8. Edmundson H.P. New methods in automatic abstracting. Journal of the Association for Computing Machinery 16 2 (1969) 264-285
9. Eugenio B.D., and Glass M. Squibs and discussions - the kappa statistic: a second look. Computational Linguistics (2004) 95-101
10. Hand, T. F. (1997). A proposal for task-based evaluation of text summarization systems. In Proceedings of the ACL/EACL-97 summarization workshop. Madrid: Spain (July).
11. Harman, D., & Over, P. (2004). In Proceedings of the document understanding conference (DUC) 2004. Boston, MA.
12. Hinton P.R. Statistics explained: A guide for social science students (1995), Routledge, New York, NY
13. Hovy, E., & Lin, C.-Y. (1997). Automated text summarization in SUMMARIST. In Proceedings of the association for computational linguistics (ACL) workshop on intelligent scalable text summarization. Madrid, Spain (August).
14. Hovy, E., Lin, C.-Y., & Zhou, L. (2005). Evaluating DUC 2005 using basic elements. In Proceedings of the document understanding conferences (DUC). Vancouver, Canada (October).
15. Jing, H., Barzilay, R., McKeown, K., & Elhadad, M. (1998). Summarization evaluation methods: experiments and analysis. In Proceedings of the AAAI symposium on intelligent summarization. Stanford University, CA (March 23-25).
16. LDC (2006). Data annotation. Linguistic Data Consortium, University of Pennsylvania: Philadelphia, PA. .
17. Lin, C.-Y. (2004). ROUGE: a package for automatic evaluation of summaries. In Proceedings of the workshop on text summarization branches out (WAS 2004). Barcelona, Spain (July 25-26).
18. Lin, C.-Y., & Hovy, E. (2003). Automatic evaluation of summaries using N-gram co-occurrence statistics. In Proceedings of the joint annual meeting of human language technology (HLT) and the North American chapter of the association for computational linguistics (HLT-NAACL) (pp. 71-78). Edmonton, Canada (May-June).
19. Lin, C.-Y., & Och, F.J. (2004). ORANGE: a method for evaluating automatic evaluation metrics for machine translation. In Proceedings of the 20th international conference on computational linguistics (COLING 2004). Geneva, Switzerland (August 23-27).
20. Lin, J., & Demner-Fushman, D. (2005). Automatically evaluating answers to definition questions. In Proceedings of the 2005 human language technology conference and conference on empirical methods in natural language processing (HLT/EMNLP 2005) (pp. 931-938). Vancouver, Canada.
21. Mani, I. (2001). Summarization evaluation: an overview. In Proceedings of the North American chapter of the association for computational linguistics (NAACL) workshop on automatic summarization.
22. Mani I., Klein G., House D., and Hirschman L. SUMMAC: a text summarization evaluation. Natural Language Engineering 8 1 (2002) 43-68
23. Monz, C. (2004). Minimal span weighting retrieval for question answering. In Proceedings of the special interest group on information retrieval (SIGIR) workshop on information retrieval for question answering. Pittsburgh, PA (May).
24. Monz, C., & de Rijke, M. (2001). The university of Amsterdam at CLEF 2001. In Proceedings of the cross language evaluation forum workshop (CLEF 2001) (pp. 165-169). Darmstadt, Germany (September).
25. Nenkova, A., & Passonneau, R. J. (2004). Evaluating content selection in summarization: the pyramid method. In Proceedings of the joint annual meeting of human language technology and the North American chapter of the association for computational linguistics (HLT-NAACL). Boston, MA (May).
26. Over, P., & Yen, J. (2003). An introduction to DUC-2003: intrinsic evaluation of generic news text summarization systems. In Proceedings of the DUC 2003 workshop on text summarization. .
27. Paice C.D. Constructing literature abstracts by computer: techniques and prospects. Information Processing and Management 26 1 (1990) 171-186
28. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the association for computational linguistics (ACL 2004). Philadelphia, PA (July).
29. President, S., & Dorr, B. (2006). Text summarization evaluation: correlating human performance on an extrinsic task with automatic intrinsic metrics. Tech. rep., University of Maryland, College Park, MD, LAMP-TR-133, CS-TR-4808, UMIACS-TR-2006-28.
30. Schwartz, R., Sista, S., & Leek, T. 2001. Unsupervised topic discovery. In: Proceedings of the advanced research and development activity in information technology (ARDA) workshop on language modeling and information retrieval. Pittsburgh, PA (May).
31. Siegel S., and Castellan Jr. N.J. Nonparametric statistics for the behavioral sciences. 2nd ed. (1988), McGraw-Hill, New York
32. Tombros, A., & Sanderson, M. (1998). Advantages of query biased summaries in information retrieval. In Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval (pp. 2-10).
33. Zajic, D., Dorr, B. J., & Schwartz, R. (2004). BBN/UMD at DUC 2004: topiary. In Proceedings of the document understanding conference (DUC). Boston, MA (May).
34. Zajic, D., Dorr, B. J., Schwartz, R., & President, S. (2004). Headline evaluation experiment results. Tech. rep., University of Maryland, College Park, MD, UMIACS-TR-2004-18.
35. Zhou, L., & Hovy, E. (2003). Web-trained extraction summarization system. In: Proceedings of the joint annual meeting of human language technology (HLT) and the North American chapter of the association for computational linguistics (HLT-NAACL). Alta., Canada (May).
36. Zhou, L., Lin, C.-Y., Munteanu, D. S., & Hovy, E. (2006). ParaEval: using paraphrases to evaluate summaries automatically. In Proceedings of HLT/NAACL-2006 (pp. 447-454).