Study and correlation analysis of linguistic, perceptual, and automatic machine translation evaluations

Journal of the American Society for Information Science and Technology

Volumn 63, Issue 1, 2012, Pages 174-184

  Farreús, Mireia ^a   Costa Jussà, Marta R ^b   Morse, Maja Popović ^c  

^a UNIVERSITAT OBERTA DE CATALUNYA   (Spain)

^b Barcelona Media Innovation Center   (Spain)

^c GERMAN RESEARCH CENTER FOR ARTIFICIAL INTELLIGENCE DFKI   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATIC MACHINES; AUTOMATIC METRICS; CORRELATION ANALYSIS; EVALUATION METHOD; HUMAN EVALUATION; MACHINE TRANSLATIONS; OBJECTIVE EVALUATION; OUTPUT ERRORS; PERCEPTUAL EVALUATION; RULE BASED; SEMANTIC LEVELS; SPEARMAN CORRELATION; TARGET LANGUAGE;

ERRORS; INFORMATION THEORY; LINGUISTICS; SEMANTICS;

TRANSLATION (LANGUAGES);

EID: 83655201280     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.21674     Document Type: Article

References (32)

1. Arnold, D., &, Balkan, L., (1995). Machine translation: An introductory guide. Computational Linguistics, 21 (4), 577-578.
2. Bangalore, S., &, Riccardi, G., (2000). Stochastic finite-state models for spoken language machine translation. In Proceedings of the Workshop on Embedded Machine Translation Systems (pp. 52-59). Stroudsburg, PA: Association for Computational Linguistics.
3. Brown, P., Della Pietra, S., Della Pietra, V., &, Mercer, R., (1993). The mathematics of statistical machine translation. Computational Linguistics, 19 (2), 263-311.
4. Callison-Burch, C., Kohen, P., Monz, C., &, Schroeder, J., (2009). Findings of the 2009 Workshop on Statistical Machine Translation. Proceedings of the Fourth Workshop on Statistical Machine Translation (pp. 1-28). Stroudsburg, PA: Association for Computational Linguistics.
5. Callison-Burch, C., Osborne, M., &, Kohen, P., (2006). Re-evaluating the role of BLEU in machine translation research. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics (pp. 249-256). Stroudsburg, PA: Association for Computational Linguistics.
6. Casacuberta, F., (2001). Finite-state transducers for speech-input translation. In Proceedings of the Automatic Speech Recognition and Understanding Workshop, ASRU (pp. 375-380). Washington, DC: IEEE Press.
7. Chen, S.F., &, Goodman, J.T., (1998). An empirical study of smoothing techniques for language modelling (Tech. Rep. No. TR-10-98). Cambridge, MA: Harvard University, Department of Computer Science.
8. Cohen, J., (1968). Weighed kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70 (4), 23-220.
9. Dorr, B.J., (1994). Machine translation: A view from the lexicon. Computational Linguistics, 20 (4), 670-676.
10. Edwards, A.L., (1976). The correlation coefficient. An introduction to linear regression and correlation (chap. 4, pp. 36-46). San Francisco: W.H. Freeman.
11. Farrús, M., Costa-jussà, M., Mariño, J., &, Fonollosa, J.A.R., (2010). Linguistic-based evaluation criteria to identify statistical machine translation errors. Proceedings of the 14th Annual Meeting of the European Association for Machine Translation (pp. 167-173). Saint-Raphaël, France.
12. Farrús, M., Costa-jussà, M.R., Poch, M., Hernández, A., Mariño, J.B., &, Fonollosa, J.A.R., (2009). Improving a Catalan-Spanish statistical translation system using morphosyntactic knowledge. In Proceedings of the 13th Annual Meeting of the European Association for Machine Translation (pp. 52-57). Berlin, Germany: Springer.
13. Farrús, M., Costa-jussà, M., Mariño, J., Poch, M., Hernández, A., Henríquez, C., &, Fonollosa, J.A.R., (2011). Overcoming statistical machine translation limitations: Error analysis and proposed solutions for the Catalan-Spanish language pair. Language Resources and Evaluation (forthcoming).
14. Flanagan, M.A., (1994). Error classification for MT evaluation. In Proceedings of the Association for Machine Translation in the Americas (pp. 65-72). Columbia, MD: Association for Machine Translation in the Americas.
15. Forcada, M.L., Tyers, F.M., &, Ramírez, G., (2009). The apertium machine translation platform: Five years on. First International Workshop on Free/Open-Source Rule-Based Machine Translation. Alacant, Spain.
16. Giménez, J., &, Màrquez, L., (2007). Linguistic features for automatic evaluation of heterogeneous MT systems. Proceedings of the Second Workshop on Statistical Machine Translation (pp. 256-264). Stroudsburg, PA: Association for Computational Linguistics.
17. Hutchins, W.J., (1986). Machine translation: Past, present, future. Chichester, UK: Ellis Horwood.
18. Koehn, P., Och, F., &, Marcu, D., (2003). Statistical phrase-based translation. Proceedings of the Human Language Technology Conference (HLT-NAACL) (pp. 127-133). Stroudsburg, PA: Association for Computational Linguistics.
19. Landis, J.R., &, Koch, G.G., (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.
20. Lavie, A., &, Agarwal, A., (2007). METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation (pp. 65-72). Stroudsburg, PA: Association for Computational Linguistics.
21. Lopez, A., (2007). A survey of statistical machine translation. A survey of statistical machine translation. (Tech, Rep. No. A033664). Maryland University College Park, Department of Computer Science.
22. Mariño, J., Banchs, R.E., Crego, J.M., de Gispert, A., Lambert, P., Fonollosa, J.A.R., &, Costa-jussà, M.R., (2006). N-gram based machine translation. Computational Linguistics, 32 (4), 527-549.
23. McCowan, I., Moore, D., Dines, J., Gatica-Pérez, D., Flynn, M., Wellner, P., &, Bourlard, H., (2004). On the use of information retrieval measures for speech recognition evaluation. (Tech. Rep. No RR 04-73). Martigny, Switzerland: IDIAP.
24. Och, F., (2003). Minimum error rate training in statistical machine translation. Proceedings of the 41th Annual Meeting of the Association for Computational Linguistics (pp. 160-167). Stroudsburg, PA: Association for Computational Linguistics.
25. Olive, J., (2005). Global autonomous language exploitation (GALE), DARPA/IPTO Proposer Information Pamphlet. Retrieved from
26. Papineni, K., Roukos, S., Ward, T., &, Zhu, W.-J., (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (pp. 311-318). Stroudsburg, PA: Association for Computational Linguistics.
27. Popović, M., (2009). Machine translation: Statistical approach with additional linguistic knowledge. Unpublished doctoral dissertation, RWTH Aachen University, Aachen, Germany.
28. Popović, M., &, Ney, H., (2009). Syntax-oriented evaluation measures for machine translation output. In Proceedings of the Fourth Workshop on Statistical Machine Translation (pp. 29-32). Stroudsburg, PA: Association for Computational Linguistics.
29. Snover, M., Madnani, N., Dorr, J., &, Schwartz, R., (2010). TER-plus-paraphrase, semantic, and alignment enhancements to translation edit rate. Machine Translation, 23 (2-3), 117-127. Berlin, Germany: Springer.
30. Spearman, C., (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72-101.
31. Vilar, D., Xu, J., Fernando-D'Haro, L., &, Ney, H., (2006). Error analysis of statistical machine translation output. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2006) (pp. 697-702), Genoa, Italy. Retrieved from
32. Zens, R., Och, F., &, Ney, H., (2002). Phrase-based statistical machine translation. In, Jarke, M., Koehler, J., &, Lakemeyer, (Eds.), KI-2002: Advances in Artificial Intelligence (Vol. 2479, pp. 18-32). Aachen, Germany: Springer Verlag.