Human evaluation of a German surface realisation ranker

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 5790 LNAI, Issue , 2010, Pages 201-221

  Cahill, Aoife ^a   Forst, Martin ^b  

^a UNIVERSITY OF STUTTGART   (Germany)

^b MICROSOFT   (United States)

Author keywords

automatic metrics; correlation; generation evaluation; German; human evaluation; human judgements; surface realisation

Indexed keywords

AUTOMATIC METRICS; CORRELATION; GENERATION EVALUATION; GERMAN; HUMAN EVALUATION; HUMAN JUDGEMENTS;

COMPUTATIONAL LINGUISTICS;

REGRESSION ANALYSIS;

EID: 77956337876     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-642-15573-4_11     Document Type: Conference Paper

References (26)

1. Bangalore, S., Rambow, O., Whittaker, S.: Evaluation metrics for generation. In: Proceedings of the First International Natural Language Generation Conference (INLG 2000), Mitzpe Ramon, Israel, pp. 1-8 (2000)
2. Belz, A., Kow, E.: Assessing the trade-off between system building cost and output quality in data-to-text generation. In: Krahmer, E., Theune, M. (eds.) Empirical Methods in NLG. LNCS (LNAI), vol.5790, pp. 180-200. Springer, Heidelberg (2010)
3. Belz, A., Reiter, E.: Comparing automatic and human evaluation of NLG systems. In: Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy, pp. 313-320 (2006)
4. Brants, S., Dipper, S., Hansen, S., Lezius, W., Smith, G.: The TIGER treebank. In: Proceedings of the Workshop on Treebanks and Linguistic Theories, Sozopol, Bulgaria, pp. 24-41 (2002)
5. Bresnan, J.: Lexical-Functional Syntax. Blackwell, Oxford (2001)
6. Cahill, A., Forst, M., Rohrer, C.: Stochastic realisation ranking for a free word order language. In: Proceedings of the Eleventh European Workshop on Natural Language Generation, DFKI GmbH, Saarbrücken, Germany, June 2007, pp. 17-24 (2007) (document D-07-01)
7. Cahill, A., Riester, A.: Incorporating information status into generation ranking. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, August 2009, pp. 817-825. Association for Computational Linguistics, Suntec (August 2009)
8. Callaway, C.: Evaluating coverage for large symbolic NLG grammars. In: Proceedings of the 18th International Joint Conference on Artificial Intelligence (IJCAI 2003), Acapulco, Mexico, pp. 811-817 (2003)
9. Crouch, R., Kaplan, R., King, T.H., Riezler, S.: A comparison of evaluation metrics for a broad coverage parser. In: Proceedings of the LREC Workshop: Beyond PARSEVAL - Towards Improved Evaluation Measures for Parsing Systems, Las Palmas, Canary Islands, Spain, pp. 67-74 (2002)
10. Filippova, K., Strube, M.: Generating constituent order in German clauses. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 320-327. Association for Computational Linguistics, Prague (June 2007)
11. Gatt, A., Belz, A.: Introducing shared task evaluation to NLG: The TUNA shared task evaluation challenges. In: Krahmer, E., Theune, M. (eds.) Empirical Methods in NLG. LNCS (LNAI), vol.5790, pp. 264-293. Springer, Heidelberg (2010)
12. Hall, J., Nivre, J.: A dependency-driven parser for German dependency and constituency representations. In: Proceedings of the Workshop on Parsing German, pp. 47-54. Association for Computational Linguistics, Columbus (June 2008)
13. Hovy, E., Yew Lin, C., Zhou, L.: Evaluating duc 2005 using basic elements. In: Proceedings of DUC 2005 (2005)
14. Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Marie- Francine Moens, S.S. (ed.) Text Summarization Branches Out: Proceedings of the ACL 2004 Workshop, pp. 74-81. Association for Computational Linguistics, Barcelona (July 2004)
15. Melamed, I.D., Green, R., Turian, J.P.: Precision and recall of machine translation. In: NAACL 2003: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, pp. 61-63. Association for Computational Linguistics, Morristown (2003)
16. Nakanishi, H., Miyao, Y., Tsujii, J.: Probabilistic models for disambiguation of an HPSG-based chart generator. In: Proceedings of the Ninth International Workshop on Parsing Technology, pp. 93-102. Association for Computational Linguistics, Vancouver (October 2005)
17. Nenkova, A., Chae, J., Louis, A., Pitler, E.: Structural features for predicting the linguistic quality of text: Applications to machine translation, automatic summarization and human-authored text. In: Krahmer, E., Theune, M. (eds.) Empirical Methods in NLG. LNCS (LNAI), vol.5790, pp. 222-241. Springer, Heidelberg (2010)
18. Owczarzak, K.: DEPEVAL(summ): Dependency-based evaluation for automatic summaries. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 190-198. Association for Computational Linguistics, Suntec (August 2009)
19. Owczarzak, K., van Genabith, J., Way, A.: Evaluating machine translation with LFG dependencies. Machine Translation 21, 95-119 (2008)
20. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318. Association for Computational Linguistics, Philadelphia (July 2002)
21. Reiter, E., Sripada, S.: Should corpora texts be gold standards for NLG? In: Proceedings of INLG 2002, Harriman, NY, pp. 97-104 (2002)
22. Rohrer, C., Forst, M.: Improving coverage and parsing quality of a large-scale LFG for German. In: Proceedings of the Language Resources and Evaluation Conference (LREC 2006), Genoa, Italy, pp. 2206-2211 (2006)
23. Snover, M., Dorr, B., Schwartz, R., Micciulla, L.,Weischedel, R.: A study of translation error rate with targeted human annotation. In: Proceedings of the Association for Machine Translation in the Americas Conference 2006, pp. 223-231 (2006)
24. Stent, A., Marge, M., Singhai, M.: Evaluating evaluation methods for generation in the presence of variation. In: Proceedings of CICLING, pp. 341-351 (2005)
25. Velldal, E.: Empirical Realization Ranking. Ph.D. thesis, University of Oslo (2008)
26. Velldal, E., Oepen, S.: Statistical ranking in tactical generation. In: Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pp. 517-525. Association for Computational Linguistics, Sydney (July 2006)