Match without a Referee: Evaluating MT Adequacy without Reference Translations

WMT 2012 - 7th Workshop on Statistical Machine Translation, Proceedings of the Workshop

Volumn , Issue , 2012, Pages 171-180

  Mehdad, Yashar ^a   Negri, Matteo ^a   Federico, Marcello ^a  

^a FONDAZIONE BRUNO KESSLER   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

BENCHMARKING; COMPUTATIONAL LINGUISTICS; MACHINE TRANSLATION;

AUTOMATIC MACHINES; CROSS-LINGUAL; ECONOMIC PERSPECTIVE; HUMAN JUDGMENTS; MACHINE TRANSLATION EVALUATIONS; MT EVALUATIONS; SYSTEM EVALUATION; TEXTUAL ENTAILMENT; WITHOUT REFERENCE TRANSLATIONS;

SEMANTICS;

EID: 84962896686     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. E. Avramidis, M. Popovic, V. Vilar Torres, and A. Burchardt. 2011. Evaluate with Confidence Estimation: Machine Ranking of Translation Outputs using Grammatical Features. In Proceedings of the Sixth Workshop on Statistical Machine Translation (WMT '11).
2. N. Bach, F. Huang, and Y. Al-Onaizan. 2011. Goodness: A Method for Measuring Machine Translation Confidence. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL 2011).
3. S. Banerjee and A. Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization.
4. J. Blatz, E. Fitzgerald, G. Foster, S. Gandrabur, C. Goutte, A. Kulesza, A. Sanchis, and N. Ueffing. 2004. Confidence Estimation for Machine Translation. In Proceedings of the 20th international conference on Computational Linguistics (COLING '04). Association for Computational Linguistics.
5. C. Callison-Burch, C. Fordyce, P. Koehn, C. Monz, and J. Schroeder. 2007. (Meta-) Evaluation of Machine Translation. In Proceedings of the Second Workshop on Statistical Machine Translation (WMT '07).
6. X. Carreras, I. Chao, L. Padŕo, and M. Padŕo. 2004. FreeLing: An Open-Source Suite of Language Analyzers. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC'04).
7. C.C. Chang and C.J. Lin. 2011. LIBSVM: A Library for Support Vector Machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3).
8. I. Dagan and O. Glickman. 2004. Probabilistic Textual Entailment: Generic Applied Modeling of Language Variability. In Proceedings of the PASCAL Workshop of Learning Methods for Text Understanding and Mining.
9. G. Doddington. 2002. Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurrence Statistics. In Proceedings of the second international conference on Human Language Technology Research, HLT '02.
10. J. Giḿenez and L. M`arquez. 2007. Linguistic Features for Automatic Evaluation of Heterogenous MT Systems. In Proceedings of the Second Workshop on Statistical Machine Translation (StatMT '07).
11. P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions (ACL 2007).
12. Y. Mehdad, M. Negri, and M. Federico. 2010. Towards Cross-Lingual Textual Entailment. In Proceedings of the 11th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL HLT 2010).
13. Y. Mehdad, M. Negri, and M. Federico. 2011. Using Bilingual Parallel Corpora for Cross-Lingual Textual Entailment. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL HLT 2011).
14. Y. Mehdad, M. Negri, and M. Federico. 2012. Detecting Semantic Equivalence and Information Disparity in Cross-lingual Documents. In Proceedings of the ACL'12.
15. F.J. Och and H. Ney. 2000. Improved Statistical Alignment Models. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics (ACL 2000).
16. P.G. Otero and I.G. Lopez. 2011. A Grammatical Formalism Based on Patterns of Part-of-Speech Tags. International journal of corpus linguistics, 16(1).
17. S. Pado, M. Galley, D. Jurafsky, and C. D. Manning. 2009. Textual Entailment Features for Machine Translation Evaluation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (StatMT '09).
18. S. Pado, 2006. User's guide to sigf: Significance testing by approximate randomisation.
19. K. Papineni, S. Roukos, T. Ward, and W.J. Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation (ACL 2002. In Proceedings of the 40th annual meeting on association for computational linguistics.
20. C.B. Quirk. 2004. Training a Sentence-Level Machine Translation Confidence Measure. In Proceedings of LREC 2004.
21. M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of Association for Machine Translation in the Americas (AMTA 2006).
22. L. Specia and A. Farzindar. 2010. Estimating Machine Translation Post-Editing Effort with HTER. In Proceedings of the AMTA-2010 Workshop, Bringing MT to the User: MT Research and the Translation Industry.
23. L. Specia, N. Cancedda, and M. Dymetman. 2010a. A Dataset for Assessing Machine Translation Evaluation Metrics. In Proceedings of the 7th international conference on Language Resources and Evaluation (LREC10).
24. L. Specia, D. Raj, and M. Turchi. 2010b. Machine Translation Evaluation Versus Quality Estimation. Machine translation, 24(1).
25. L. Specia, N. Hajlaoui, C. Hallett, and W. Aziz. 2011. Predicting Machine Translation Adequacy. In Proceedings of the 13th Machine Translation Summit (MTSummit 2011).
26. L. Specia. 2011. Exploiting Objective Annotations for Minimising Translation Post-editing Effort. In Proceedings of the 15th Conference of the European Association for Machine Translation (EAMT 2011).
27. D. Xiong, M. Zhang, and H. Li. 2010. Error Detection for Statistical Machine Translation Using Linguistic Features. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL 2010). Association for Computational Linguistics.