Coping with the Subjectivity of Human Judgements in MT Quality Estimation

Proceedings of the Annual Meeting of the Association for Computational Linguistics

Volumn , Issue , 2013, Pages 240-251

  Turchi, Marco ^a   Negri, Matteo ^a   Federico, Marcello ^a  

^a FONDAZIONE BRUNO KESSLER   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

CLASSIFICATION (OF INFORMATION); COMPUTATIONAL LINGUISTICS; COMPUTER AIDED LANGUAGE TRANSLATION; NATURAL LANGUAGE PROCESSING SYSTEMS;

BINARY CLASSIFIERS; DATA ANNOTATION; HIGH QUALITY DATA; HUMAN JUDGMENTS; LABELING PROCEDURES; MACHINE TRANSLATIONS; MANUAL LABELING; QUALITY ESTIMATION; REAL-WORLD; TRANSLATION QUALITY;

MACHINE TRANSLATION;

EID: 85122699929     PISSN: 0736587X     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (30)

1. Nguyen Bach, Fei Huang, and Yaser Al-Onaizan. 2011. Goodness: a Method for Measuring Machine Translation Confidence. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA, pages 211-219. The Association for Computer Linguistics.
2. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: a Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological), pages 289-300.
3. Frédéric Blain, Holger Schwenk, and Jean Senellart. 2012. Incremental Adaptation Using Translation Information and Post-Editing Analysis. In International Workshop on Spoken Language Translation, pages 234-241, Hong-Kong (China).
4. John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2003. Confidence Estimation for Machine Translation. Summer workshop final report, JHU/CLSP.
5. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 Workshop on Statistical Machine Translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation (WMT'12), pages 10-51, Montréal, Canada.
6. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: a Library for Support Vector Machines. ACM Trans. Intell. Syst. Technol., 2(3):27:1-27:27, May.
7. Chien-Ying Chen, Jen-Yuan Yeh, and Hao-Ren Ke. 2010. Plagiarism Detection using ROUGE and WordNet. Journal of Computing, 2(3).
8. José G. C. de Souza, Miquel Esplà-Gomis, Marco Turchi, and Matteo Negri. 2013. Exploiting Qualitative Information from Automatic Word Alignment for Cross-lingual NLP Tasks. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013).
9. Michael Denkowski and Alon Lavie. 2012. Challenges in Predicting Machine Translation Utility for Human Post-Editors. In Proceedings of AMTA 2012.
10. Yifan He, Yanjun Ma, Josef van Genabith, and Andy Way. 2010. Bridging SMT and TM with Translation Recommendation.
11. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pages 177-180. Association for Computational Linguistics.
12. Philip Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X, pages 79-86, Phuket, Thailand.
13. Maarit Koponen. 2012. Comparing Human Perceptions of Post-editing Effort with Post-editing Operations. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 181-190. Association for Computational Linguistics.
14. Michael Lesk. 1986. Automated Sense Disambiguation Using Machine-readable Dictionaries: How to Tell a Pine Cone from an Ice Cream Cone. In Proceedings of the 5th annual international conference on Systems documentation (SIGDOC86).
15. Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Proceedings of the ACL workshop on Text Summarization Branches Out., pages 74-81, Barcelona, Spain.
16. Yanjun Ma, Yifan He, Andy Way, and Josef van Genabith. 2011. Consistent Translation using Discriminative Learning: a Translation Memory-inspired Approach. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1239-1248.
17. Yashar Mehdad, Matteo Negri, and Marcello Federico. 2012a. Detecting semantic equivalence and information disparity in cross-lingual documents. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL'12), pages 120-124, Jeju Island, Korea.
18. Yashar Mehdad, Matteo Negri, and Marcello Federico. 2012b. Match without a Referee: Evaluating MT Adequacy without Reference Translations. In Proceedings of the Seventh Workshop on Statistical Machine Translation, WMT'12, pages 171-180, Stroudsburg, PA, USA. Association for Computational Linguistics.
19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL'02, pages 311-318, Stroudsburg, PA, USA. Association for Computational Linguistics.
20. M. Porter. 2001. Snowball: A language for stemming algorithms.
21. Martin Potthast, Alberto Barrón-Cedeño, Andreas Eiselt, Benno Stein, and Paolo Rosso. 2010. Overview of the 2nd International Competition on Plagiarism Detection. Notebook Papers of CLEF, 10.
22. Christopher B. Quirk. 2004. Training a Sentence-Level Machine Translation Confidence Measure. In In Proceedings of LREC.
23. Matthew Snover, Nitin Madnani, Bonnie J. Dorr, and Richard Schwartz. 2009. Fluency, Adequacy, or HTER?: Exploring Different Human Judgments with a Tunable MT Metric. In Proceedings of the Fourth Workshop on Statistical Machine Translation, StatMT'09, pages 259-268, Stroudsburg, PA, USA. Association for Computational Linguistics.
24. Radu Soricut and Abdessamad Echihabi. 2010. TrustRank: Inducing Trust in Automatic Translations via Ranking. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL'10, pages 612-621, Stroudsburg, PA, USA. Association for Computational Linguistics.
25. Radu Soricut, Nguyen Bach, and Ziyuan Wang. 2012. The SDL language weaver systems in the WMT12 quality estimation shared task. In Proceedings of the Seventh Workshop on Statistical Machine Translation (WMT'12), pages 145-151, Montréal, Canada.
26. Lucia Specia, Nicola Cancedda, Marc Dymetman, Marco Turchi, and Nello Cristianini. 2009. Estimating the Sentence-Level Quality of Machine Translation Systems. In Proceedings of the 13th Annual Conference of the European Association for Machine Translation (EAMT'09), pages 28-35, Barcelona, Spain.
27. Lucia Specia, Dhwaj Raj, and Marco Turchi. 2010. Machine Translation Evaluation versus Quality Estimation. Machine translation, 24(1):39-50.
28. Lucia Specia, Najeh Hajlaoui, Catalina Hallett, and Wilker Aziz. 2011. Predicting machine translation adequacy. In Proceedings of the 13th Machine Translation Summit, pages 513-520, Xiamen, China, September.
29. Lucia Specia. 2011. Exploiting Objective Annotations for Measuring Translation Post-editing Effort. pages 73-80.
30. Marco Turchi and Matteo Negri. 2013. ALTN: Word Alignment Features for Cross-Lingual Textual Entailment. Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013).