Overview of the IWSLT 2011 Evaluation Campaign

2011 International Workshop on Spoken Language Translation, IWSLT 2011

Volumn , Issue , 2011, Pages 11-27

  Federico, Marcello ^a   Bentivogli, Luisa ^b   Paul, Michael ^c   Stüker, Sebastian ^d  

^a FONDAZIONE BRUNO KESSLER   (Italy)

^b CELCT   (Italy)

^c NATIONAL INSTITUTE OF INFORMATION AND COMMUNICATIONS TECHNOLOGY   (Japan)

^d KARLSRUHE INSTITUTE OF TECHNOLOGY   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

CHARACTER RECOGNITION; COMPUTATIONAL LINGUISTICS;

AUTOMATIC TRANSLATION; EVALUATION RESULTS; PREVIOUS YEAR; SPEECH TRANSLATION; SUBJECTIVE EVALUATIONS; SYSTEM COMBINATION; SYSTEM DESCRIPTION;

SPEECH RECOGNITION;

EID: 84865515782     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a Method for Automatic Evaluation of Machine Translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, USA, 2002, pp. 311-318.
2. A. Lavie and A. Agarwal, “METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments,” in Proceedings of the Second Workshop on Statistical Machine Translation (WMT), Prague, Czech Republic, 2007, pp. 228-231.
3. S. Niessen, F. J. Och, G. Leusch, and H. Ney, “An Evaluation Tool for Machine Translation: Fast Evaluation for Machine Translation Research,” in Proceedings of the Second International Conference on Language Resources & Evaluation (LREC), Athens, Greece, 2000, pp. 39-45.
4. F. J. Och, “Minimum Error Rate Training in SMT,” in Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL), Sapporo, Japan, 2003, pp. 160-167.
5. M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul, “A Study of Translation Edit Rate with Targeted Human Annotation,” in Proceedings of the The Seventh Conference of the Association for Machine Translation in the Americas (AMTA), Cambridge, USA, 2006, pp. 223-231.
6. J. P. Turian, L. Shen, and I. D. Melamed, “Evaluation of Machine Translation and its Evaluation,” in Proceedings of the MT Summmit IX, New Orleans, USA, 2003, pp. 386-393.
7. G. Doddington, “Automatic Evaluation of Machine Translation Quality using N-gram Co-Occurrence Statistics,” in Proceedings of the Second International Conference on Human Language Technology (HLT), San Diego, USA, 2002, pp. 257-258.
8. Y. Zhang, S. Vogel, and A. Waibel, “Interpreting Bleu/NIST Scores: How Much Improvement do We Need to Have a Better System?” in Proceedings of the Second International Conference on Language Resources & Evaluation (LREC), 2004, pp. 2051-2054.
9. J. G. Fiscus, “A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (ROVER),” in Proceedings of the IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU), Santa Barbara, USA, 1997, pp. 347-354.
10. E. Matusov, G. Leusch, R. E. Banchs, N. Bertoldi, D. Dechelotte, M. Federico, M. Kolss, Y.-S. Lee, J. B. Marino, M. Paulik, S. Roukos, H. Schwenk, and H. Ney, “System Combination for Machine Translation of Spoken and Written Language,” IEEE Transactions on Audio, Speech and Language Processing, vol. 14(5), pp. 1222-1237, 2008.
11. P. Banerjee, H. Almaghout, S. Naskar, J. Jiang, A. Way, J. Roturier, and J. van Genabith, “The DCU Machine Translation Systems for IWSLT 2011,” in Proceedings of the eight International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA, 2011.
12. D. Vilar, E. Avramidis, M. Popović, and S. Hunsicker, “DFKI's SC and MT Submissions to IWSLT 2011,” in Proceedings of the eight International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA, 2011.
13. N. Ruiz, A. Bisazza, F. Brugnara, D. Falavigna, D. Giuliani, S. Jaber, R. Gretter, and M. Federico, “FBK @ IWSLT 2011,” in Proceedings of the eight International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA, 2011.
14. M. Mediani, E. Cho, J. Nieheus, T. Herrmann, and A. Waibel, “The KIT English-French Translation system for IWSLT 2011,” in Proceedings of the eight International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA, 2011.
15. B. Lecouteux, L. Besancier, and H. Blanchon, “LIG English-French Spoken Language Translation System for IWSLT 2011,” in Proceedings of the eight International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA, 2011.
16. T. Lavergne, A. Allauzen, H.-S. Le, and F. Yvon, “LIMSI's experiments in domain adaptation for IWSLT11,” in Proceedings of the eight International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA, 2011.
17. A. Rousseau, F. Bougares, P. Deléglise, H. Schwenk, and Y. Estève, “LIUM's system for the IWSLT 2011 Speech Translation Tasks,” in Proceedings of the eight International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA, 2011.
18. R. Aminzadeh, T. Anderson, R. Slyh, B. Ore, E. Hansen, W. Shen, J. Drexler, and T. Gleason, “The MIT-LL/AFRL IWSLT-2011 MT System,” in Proceedings of the eight International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA, 2011.
19. X. He, A. Axelrod, L. Deng, A. Acero, M.-Y. Hwang, A. Nguyen, A. Wang, and X. Huang, “The MSR System for IWSLT 2011 Evaluation,” in Proceedings of the eight International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA, 2011.
20. K. Abe, Y. Wu, C. lin Huang, P. R. Dixon, S. Matsuda, C. Hori, and H. Kashioka, “The NICT ASR System for IWSLT 2011,” in Proceedings of the eight International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA, 2011.
21. A. Finch, C.-L. Goh, G. Neubig, and E. Sumita, “The NICT Translation System for IWSLT 2011,” in ProceedingsoftheeightInternationalWorkshoponSpoken Language Translation (IWSLT), San Francisco, CA, 2011.
22. J. Wuebker, M. Huck, S. Mansour, M. Freitag, M. Feng, S. Peitz, C. Schmidt, and H. Ney, “The RWTH Aachen Machine Translation System for IWSLT 2011,” in Proceedings of the eight International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA, 2011.
23. L. Bentivogli, M. Federico, G. Moretti, and M. Paul, “Getting Expert Quality from the Crowd for Machine Translation Evaluation,” in Proceedings of the MT Summmit XIII, Xiamen, China, 2011, pp. 521-528.
24. C. Callison-Burch, C. Fordyce, P. Koehn, C. Monz, and J. Schroeder, “(Meta-) Evaluation of Machine Translation,” in Proceedings of the Second Workshop on Statistical Machine Translation (WMT), Prague, Czech Republic, 2007, pp. 136-158.
25. C. Callison-Burch, “Fast, cheap, and creative: Evaluating translation quality using Amazon's Mechanical Turk,” in Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, August 2009, pp. 286-295. [Online]. Available: http://www.aclweb.org/anthology/D/D09/D09-1030
26. S. Siegel and N. J. Castellan, Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill, 1988.
27. J. L. Fleiss, “Measuring nominal scale agreement among many raters,” Psychological Bulletin, vol. 76(5), 1971.
28. J. Landis and G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33 (1), pp. 159-174, 1977.