Human evaluation of machine translation through binary system comparisons

Proceedings of the Annual Meeting of the Association for Computational Linguistics

Volumn , Issue , 2007, Pages 96-103

  Vilar, David ^a   Leusch, Gregor ^a   Ney, Hermann ^a   Banchs, Rafael E ^b  

^a RWTH AACHEN UNIVERSITY   (Germany)

^b UNIVERSITAT POLITÈCNICA DE CATALUNYA   (Spain)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; COMPUTER AIDED LANGUAGE TRANSLATION; WELL TESTING;

AUTOMATIC EVALUATION; BINARY SYSTEMS; EVALUATION GUIDELINES; EVALUATION SCHEME; HUMAN EVALUATION; MACHINE TRANSLATION EVALUATIONS; MACHINE TRANSLATION SYSTEMS; MACHINE TRANSLATIONS; STATISTICAL PROCESS; SYSTEMS COMPARISON;

MACHINE TRANSLATION;

EID: 85120056873     PISSN: 0736587X     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (17)

1. M. Bisani and H. Ney. 2004. Bootstrap estimates for confidence intervals in ASR performance evaluationx. IEEE ICASSP, pages 409-412, Montreal, Canada, May.
2. T. Bui and M. Thanh. 1985. Significant improvements to the Ford-Johnson algorithm for sorting. BIT Numerical Mathematics, 25(1):70-75.
3. C. Callison-Burch, M. Osborne, and P. Koehn. 2006. Reevaluating the role of BLEU in machine translation research. Proceeding of the 11th Conference of the European Chapter of the ACL: EACL 2006, pages 249-256, Trento, Italy, Apr.
4. G. Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. Proc. ARPA Workshop on Human Language Technology.
5. B. Efron and R. J. Tibshirani. 1993. An Introduction to the Bootstrap. Chapman & Hall, New York and London.
6. L. Ford Jr and S. Johnson. 1959. A Tournament Problem. The American Mathematical Monthly, 66(5):387-389.
7. D. E. Knuth. 1973. The Art of Computer Programming, volume 3. Addison-Wesley, 1st edition. Sorting and Searching.
8. P. Koehn and C. Monz. 2006. Manual and automatic evaluation of machine translation between european languages. Proceedings of the Workshop on Statistical Machine Translation, pages 102-121, New York City, Jun.
9. G. Leusch, N. Ueffing, D. Vilar, and H. Ney. 2005. Preprocessing and normalization for automatic evaluation of machine translation. 43rd ACL: Proc. Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, pages 17-24, Ann Arbor, Michigan, Jun.
10. F. J. Och. 2003. Minimum error rate training in statistical machine translation. Proc. of the 41st ACL, pages 160-167, Sapporo, Japan, Jul.
11. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. Proc. of the 40th ACL, pages 311-318, Philadelphia, PA, Jul.
12. M. Peczarski. 2002. Sorting 13 elements requires 34 comparisons. LNCS, 2461/2002:785-794, Sep.
13. M. Snover, B. J. Dorr, R. Schwartz, J. Makhoul, L. Micciulla, and R. Weischedel. 2005. A study of translation error rate with targeted human annotation. Technical Report LAMP-TR-126, CS-TR-4755, UMIACS-TR-2005-58, University of Maryland, College Park, MD.
14. L. Thurstone. 1927. The method of paired comparisons for social values. Journal of Abnormal and Social Psychology, 21:384-400.
15. D. Vilar, E. Matusov, S. Hasan, R. Zens, and H. Ney. 2005. Statistical Machine Translation of European Parliamentary Speeches. Proceedings of MT Summit X, pages 259-266, Phuket, Thailand, Sep.
16. M. Wells. 1971. Elements of combinatorial computing. Pergamon Press.
17. Y. Zhang and S. Vogel. 2004. Measuring confidence intervals for the machine translation evaluation metrics. Proceedings of the 10th International Conference on Theoretical and Methodological Issues in Machine Translation, pages 4-6, Baltimore, MD.