Correlating human and automatic evaluation of a German surface realiser

ACL-IJCNLP 2009 - Joint Conf. of the 47th Annual Meeting of the Association for Computational Linguistics and 4th Int. Joint Conf. on Natural Language Processing of the AFNLP, Proceedings of the Conf.

Volumn , Issue , 2009, Pages 97-100

  Cahill, Aoife ^a  

^a UNIVERSITY OF STUTTGART   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS;

AUTOMATIC EVALUATION; AUTOMATIC METRICS; AUTOMATICALLY GENERATED; RELATIVE RANKINGS; STRONG CORRELATION; EVALUATION METRICS; GENERAL CORRELATIONS; HUMAN EVALUATION; HUMAN JUDGMENTS;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 84859906372     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.3115/1667583.1667615     Document Type: Conference Paper

References (15)

1. Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In Proceedings of EACL 2006, pages 313-320, Trento, Italy.
2. Aoife Cahill and Martin Forst. 2009. Human Evaluation of a German Surface Realisation Ranker. In Proceedings of EACL 2009, pages 112-120, Athens, Greece, March.
3. Aoife Cahill, Martin Forst, and Christian Rohrer. 2007. Stochastic Realisation Ranking for a Free Word Order Language. In Proceedings of ENLG-07, pages 17-24, Saarbrücken, Germany, June.
4. Richard Crouch, Ron Kaplan, Tracy Holloway King, and Stefan Riezler. 2002. A comparison of evaluation metrics for a broad coverage parser. In Proceedings of the LREC Workshop: Beyond PARSEVAL, pages 67-74, Las Palmas, Spain.
5. Johan Hall and Joakim Nivre. 2008. A dependencydriven parser for German dependency and constituency representations. In Proceedings of the Workshop on Parsing German, pages 47-54, Columbus, Ohio, June.
6. Eduard Hovy, Chin yew Lin, and Liang Zhou. 2005. Evaluating duc 2005 using basic elements. In Proceedings of DUC-2005.
7. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Stan Szpakowicz Marie-Francine Moens, editor, Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74-81, Barcelona, Spain, July.
8. I. Dan Melamed, Ryan Green, and Joseph P. Turian. 2003. Precision and recall of machine translation. In Proceedings of NAACL-03, pages 61-63, NJ, USA.
9. Karolina Owczarzak, Josef van Genabith, and Andy Way. 2008. Evaluating machine translation with LFG dependencies. Machine Translation, 21:95-119.
10. Karolina Owczarzak. 2009. DEPEVAL(summ): Dependency-based Evaluation for Automatic Summaries. In Proceedings of ACL-IJCNLP 2009, Singapore.
11. Kishore Papineni, Salim Roukos, ToddWard, andWei- Jing Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL-02, pages 311-318, NJ, USA.
12. Ehud Reiter and Anja Belz. 2009. An Investigation into the Validity of Some Metrics for Automatically Evaluating Natural Language Generation Systems. Computational Linguistics, 35.
13. Christian Rohrer and Martin Forst. 2006. Improving Coverage and Parsing Quality of a Large-Scale LFG for German. In Proceedings of LREC 2006, Genoa, Italy.
14. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and Ralph Weischedel. 2006. A study of translation error rate with targeted human annotation. In Proceedings of AMTA 2006, pages 223-231.
15. Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presense of variation. In Proceedings of CICLING, pages 341-351.