Evaluating information content by factoid analysis: Human annotation and stability

Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, EMNLP 2004 - A meeting of SIGDAT, a Special Interest Group of the ACL held in conjunction with ACL 2004

Volumn , Issue , 2004, Pages 419-426

  Teufel, Simone ^a   van Halteren, Hans ^b  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

GOLD STANDARDS; HUMAN ANNOTATIONS; HUMAN STABILITIES; INFORMATION CONTENTS; NEW APPROACHES; OVERLAP MEASURES; STABLE SYSTEMS; STRING SIMILARITY; SYSTEM RANKINGS;

EID: 85009196770     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (12)

1. DUC. 2002. Document Understanding Conference (DUC). Electronic proceedings, http://www-nlpir.nist.gov/projects/duc/pubs.html.
2. Jing, H., R. Barzilay, K. R. McKeown, and M. Elhadad. 1998. Summarization Evaluation Methods: Experiments and Analysis. In Working Notes of the AAAI Spring Symposium on Intelligent Text Summarization, 60-68.
3. Lin, C., and E. Hovy. 2002. Manual and automatic evaluation of summaries. In DUC 2002.
4. Mani, I. 2001. Automatic Summarization. John Benjamins.
5. Mani, I., T. Firmin, D. House, G. Klein, B. Sundheim, and L. Hirschman. 1999. The TIPSTER Summac Text Summarization Evaluation. In Proceedings of EACL-99, 77-85.
6. Nenkova, A., and R. Passonneau. 2004. Evaluating Content Selection in Summarization: the Pyramid Method. In Proceedings of NAACL/HLT-2003.
7. Papineni, K, S. Roukos, T Ward, and W-J. Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL-02, 311-318.
8. Radev, D., and D. Tam. 2003. Summarization evaluation using relative utility. In Proceedings of the Twelfth International Conference on Information and Knowledge Management, 508-511.
9. Rath, G.J, A. Resnick, and T. R. Savage. 1961. The Formation of Abstracts by the Selection of Sentences. American Documentation 12(2): 139-143.
10. Spärck Jones, K. 1999. Automatic Summarising: Factors and Directions. In I. Mani and M. May-bury, eds., Advances in Automatic Text Summarization, 1-12. Cambridge, MA: MIT Press.
11. van Halteren, H., and S. Teufel. 2003. Examining the consensus between human summaries: initial experiments with factoid analysis. In Proceedings of the HLT workshop on Automatic Summarization.
12. Voorhees, E. 2000. Variations in relevance judgements and the measurement of retrieval effectiveness. Information Processing and Management 36: 697-716.