Further meta-evaluation of broad-coverage surface realization

EMNLP 2010 - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

Volumn , Issue , 2010, Pages 564-574

  Espinosa, Dominic ^a   Rajkumar, Rajakrishnan ^a   White, Michael ^a   Berleant, Shoshana ^a  

^a OHIO STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATIC EVALUATION; AUTOMATIC METRICS; RESEARCH COMMUNITIES; SYSTEM LEVELS; TREEBANKS; WORDNET;

COMPUTATIONAL LINGUISTICS;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 80053285631     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. Jason Baldridge. 2002. Lexically Specified Derivational Control in Combinatory Categorial Grammar. Ph.D. thesis, University of Edinburgh.
2. S. Banerjee and A. Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72.
3. R. Barzilay and L. Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In proceedings of HLT-NAACL, volume 2003, pages 16-23.
4. Aoife Cahill. 2009. Correlating human and automatic evaluation of a german surface realiser. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 97-100, Suntec, Singapore, August. Association for Computational Linguistics.
5. C. Callison-Burch, M. Osborne, and P. Koehn. 2006. Re-evaluating the role of BLEU in machine translation research. In Proceedings of EACL, volume 2006, pages 249-256.
6. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In StatMT '07: Proceedings of the Second Workshop on Statistical Machine Translation, pages 136-158, Morristown, NJ, USA. Association for Computational Linguistics.
7. C. Callison-Burch, T. Cohn, and M. Lapata. 2008. Parametric: An automatic evaluation metric for paraphrasing. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, pages 97-104. Association for Computational Linguistics.
8. J. Carletta. 1996. Assessing agreement on classification tasks: the kappa statistic. Computational linguistics, 22(2):249-254.
9. Dick Crouch, Mary Dalrymple, Ron Kaplan, Tracy King, John Maxwell, and Paula Newman. 2008. Xle documentation. Technical report, Palo Alto Research Center.
10. Philip Koenh. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing.
11. J.R. Landis and G.G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159-174. (Pubitemid 8054849)
12. Chin-Yew Lin and Franz Josef Och. 2004. Orange: a method for evaluating automatic evaluation metrics for machine translation. In COLING '04: Proceedings of the 20th international conference on Computational Linguistics, page 501, Morristown, NJ, USA. Association for Computational Linguistics.
13. K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. Technical report, IBM Research.
14. E. Reiter and A. Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529-558.
15. Stefan Riezler, Tracy H. King, Ronald M. Kaplan, Richard Crouch, John T. III Maxwell, and Mark Johnson. 2002. Parsing the wall street journal using a lexical-functional grammar and discriminative estimation techniques. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 271-278, Philadelphia, Pennsylvania, USA, July. Association for Computational Linguistics.
16. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pages 223-231.
17. M. Snover, N. Madnani, B.J. Dorr, and R. Schwartz. 2009. Fluency, adequacy, or HTER?: exploring different human judgments with a tunable MT metric. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 259-268. Association for Computational Linguistics.
18. Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In Proceedings of CICLing.
19. J.P. Turian, L. Shen, and I.D. Melamed. 2003. Evaluation of machine translation and its evaluation. recall (C-R), 100:2.
20. Michael White and Rajakrishnan Rajkumar. 2009. Per-ceptron reranking for CCG realization. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 410-419, Singapore, August. Association for Computational Linguistics.
21. Michael White. 2006. Efficient Realization of Coordinate Structures in Combinatory Categorial Grammar. Research on Language and Computation, 4(1):39-75. (Pubitemid 43830209)