Comparing automatic and human evaluation of NLG systems

EACL 2006 - 11th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference

Volumn , Issue , 2006, Pages 313-320

  Belz, Anja ^a   Reiter, Ehud ^b  

^a UNIVERSITY OF BRIGHTON   (United Kingdom)

^b UNIVERSITY OF ABERDEEN   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATIC EVALUATION; AUTOMATIC METRICS; EVALUATION PROBLEMS; EVALUATION RESULTS; HUMAN EVALUATION; HUMAN JUDGMENTS; NATURAL LANGUAGE GENERATION; STATISTICAL SYSTEMS;

KNOWLEDGE BASED SYSTEMS;

COMPUTATIONAL LINGUISTICS;

EID: 84893365704     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (23)

1. S. Bangalore, O. Rambow, and S. Whittaker. 2000. Evaluation metrics for generation. In Proc. 1st International Conference on Natural Language Generation, pages 1-8.
2. A. Belz. 2005. Statistical generation: Three methods compared and evaluated. In Proc. 10th European Workshop on Natural Language Generation (ENLG'05), pages 15-23.
3. A. Belz. 2006. pCRU: Probabilistic generation using representational underspecification. Technical Report ITRI-06-01, ITRI, University of Brighton.
4. J. Coch. 1996. Evaluating and comparing three text production techniques. In Proc. 16th International Conference on Computational Linguistics (COLING-1996).
5. G. Doddington. 2002. Automatic evaluation of machine translation quality using n-gram cooccurrence statistics. In Proc. ARPA Workshop on Human Language Technology.
6. M. E. Foster and J. Oberlander. 2006. Data-driven generation of emphatic facial displays. In Proceedings of EACL-2006.
7. N. Habash. 2004. The use of a structural n-gram language model in generation-heavy hybrid machine translation. In Proc. 3rd International Conference on Natural Language Generation (INLG'04), Volume 3123 of LNAI, pages 61-69. Springer.
8. I. Langkilde. 1998. An empirical verification of coverage and correctness for a general-purpose sentence generator. In Proc. 2nd International Natural Language Generation Conference (INLG '02).
9. J. Lester and B. Porter. 1997. Developing and empirically evaluating robust explanation generators: The KNIGHT experiments. Computational Linguistics, 23(1): 65-101.
10. C.-Y. Lin and E. Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proc. HLT-NAACL 2003, pages 71-78.
11. T. Marciniak and M. Strube. 2004. Classification-based generation using TAG. In Natural Language Generation: Proceedings of INLG-2994, pages 100-109. Springer.
12. C. Mellish and R. Dale. 1998. Evaluation in the context of natural language generation. Computer Speech and Language, 12: 349-373.
13. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proc. ACL-2002, pages 311-318.
14. E. Reiter and S. Sripada. 2002. Should corpora texts be gold standards for NLG? In Proc. 2nd International Conference on Natural Language Generation, pages 97-104.
15. E. Reiter, S. Sripada, and R. Robertson. 2003. Acquiring correct knowledge for natural language generation. Journal of Artificial Intelligence Research, 18: 491-516.
16. E. Reiter, S. Sripada, J. Hunter, and J. Yu. 2005. Choosing words in computer-generated weather forecasts. Artificial Intelligence, 167: 137-169.
17. S. Riezler and J. T. Maxwell III. 2005. On some pitfalls in automatic evaluation and significance testing for MT. In Proc. ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, pages 57-64.
18. K. Sparck Jones and J. R. Galliers. 1996. Evaluating Natural Language Processing Systems: An Analysis and Review. Springer Verlag.
19. S. Sripada, E. Reiter, J. Hunter, and J. Yu. 2003. Exploiting a parallel TEXT-DATA corpus. In Proc. Corpus Linguistics 2003, pages 734-743.
20. S. Sripada, E. Reiter, and L. Hawizy. 2005. Evaluation of an NLG system used post-edit data: Lessons learned. In Proc. ENLG-2005, pages 133-139.
21. A. Stolcke. 2002. SRILM: An extensible language modeling toolkit. In Proc. 7th International Conference on Spoken Language Processing (ICSLP'02), pages 901-904.
22. S. Williams and E. Reiter. 2005. Generating readable texts for readers with low basic skills. In Proc. ENLG-2005, pages 140-147.
23. M. Young. 1999. Using Grice's maxim of quantity to select the content of plan descriptions. Artificial Intelligence, 115: 215-256.