Testing for significance of increased correlation with human judgment

EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

Volumn , Issue , 2014, Pages 172-176

  Graham, Yvette ^a   Baldwin, Timothy ^a  

^a UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

NATURAL LANGUAGE PROCESSING SYSTEMS;

AUTOMATIC METRICS; HUMAN ASSESSMENT; HUMAN JUDGMENTS; LANGUAGE PAIRS; MACHINE TRANSLATIONS; OPEN SOURCE IMPLEMENTATION; SIGNIFICANCE TEST; TRANSLATION QUALITY;

TRANSLATION (LANGUAGES);

EID: 84926052089     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.3115/v1/d14-1020     Document Type: Conference Paper

References (17)

1. Julio Castillo and Paula Estrella. 2012. Semantic textual similarity for MT evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 52-58, Montréal, Canada.
2. Boxing Chen, Roland Kuhn, and George Foster. 2012. Improving AMBER, an MT evaluation metric. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 59-63, Montŕeal, Canada.
3. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 85-91, Edinburgh, UK.
4. Mark Fishel, Rico Sennrich, Maja Popović, and Ondřej Bojar. 2012. TerrorCat: A translation error categorization-based MT quality metric. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 64-70, Montréal, Canada.
5. Ulrich Germann. 2003. Greedy decoding for statistical machine translation in almost linear time. In Proceedings of the 2003 Conference of the North American Chapter of the Assoc. Computational Linguistics on Human Language Technology-Volume 1, pages 1-8, Edmonton, Canada.
6. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2014. Randomized significance tests in machine translation. In Proceedings of the ACL 2014 Ninth Workshop on Statistical Machine Translation, pages 266-274, Baltimore, USA.
7. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of Empirical Methods in Natural Language Processing 2004 (EMNLP 2004), pages 388-395, Barcelona, Spain.
8. Shankar Kumar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the 4th International Conference on Human Language Technology Research and 5th Annual Meeting of the NAACL (HLT-NAACL 2004), pages 169-176, Boston, USA.
9. Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2003. A novel string-to-string distance measure with applications to machine translation evaluation. In Proceedings 9th Machine Translation Summit (MT Summit IX), pages 240-247, New Orleans, USA.
10. Matouš Macháxěk and Ondřej Bojar. 2011. Approximating a deep-syntactic metric for MT evaluation and tuning. In Proceedings of the SixthWorkshop on Statistical Machine Translation, pages 92-98, Edinburgh, UK.
11. Dan Melamed, Ryan Green, and Joseph Turian. 2003. Precision and recall of machine translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (HLTNAACL 2003) - Short Papers, pages 61-63, Edmonton, Canada.
12. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 160-167, Sapporo, Japan.
13. Kishore Papineni, Salim Roukos, Todd Ward, andWei- Jing Zhu. 2001. BLEU: A method for automatic evaluation of machine translation. Technical Report RC22176 (W0109-022), IBM Research, Thomas J. Watson Research Center.
14. Maja Popovic. 2012. Class error rates for evaluation of machine translation output. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 71-75, Montréal, Canada.
15. Stefan Riezler and John T. Maxwell. 2005. On some pitfalls in automatic evaluation and significance testing for mt. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 57-64, Ann Arbor, USA.
16. James H. Steiger. 1980. Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2):245.
17. Evan J. Williams. 1959. Regression Analysis, volume 14. Wiley, New York, USA.