Adequacy-fluency metrics: Evaluating MT in the continuous space model framework

IEEE Transactions on Audio, Speech and Language Processing

Volumn 23, Issue 3, 2015, Pages 472-482

  Banchs, Rafael E ^a   D'Haro, Luis F ^a   Li, Haizhou ^a,b  

^a Human Language Technology Department   (Singapore)

^b NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

Author keywords

Machine translation; natural language processing; system evaluation

Indexed keywords

COMPUTATIONAL LINGUISTICS; COMPUTER AIDED LANGUAGE TRANSLATION; LEARNING ALGORITHMS; MODELING LANGUAGES; NATURAL LANGUAGE PROCESSING SYSTEMS; SEMANTICS; TRANSLATION (LANGUAGES);

AUTOMATIC EVALUATION; EUROPEAN LANGUAGES; MACHINE TRANSLATIONS; N-GRAM LANGUAGE MODELS; NATURAL LANGUAGE PROCESSING; SYSTEM EVALUATION; TRANSLATION PROCESS; TRANSLATION QUALITY;

TRANSLATION (LANGUAGES); NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 84923899901     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASLP.2015.2405751     Document Type: Article

References (40)

1. G. Hinton and R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504-507, Jul. 2006.
2. R. E. Banchs and M. R. Costa-jussà, "Cross-language document retrieval by using nonlinear semantic mapping," Appl. Artif. Intell., vol. 27, no. 9, pp. 1-22, Oct. 2013.
3. T. Mikolov et al., "Distributed representations of words and phrases and their compositionality," in Proc. Adv. Neural Inf. Process. Syst., Lake Tahoe, NV, USA, Dec. 2013, vol. 26, pp. 3111-3119.
4. J. Gao et al., "Learning continuous phrase representations for translation modelling," in Proc. 52nd Annu. Meeting Assoc. Comput. l Linguist., Baltimore, MD, USA, Jun. 2014, vol. 1, pp. 699- 709.
5. J. Zhang et al., "Bilingually-constrained phrase embeddings for machine translation," in Proc. 52nd Annu. Meeting Assoc.Computat. Linguist., Baltimore, MD, USA, Jun. 2014, vol. 1, pp. 111-121.
6. R. Socher et al., "Semi-supervised recursive autoencoders for predicting sentiment distributions," in Proc. Conf. Empir. Meth. Nat. l Lang. Process., Edinburgh, U.K., Jul. 2011, pp. 151-161.
7. T. Mikolov et al., "Recurrent neural network based language model,"in Proc. Interspeech, Florence, Italy, Aug. 2010, pp. 1045-1048.
8. J. Blatz et al., "Confidence estimation for machine translation," Johns Hopkins Univ., Baltimore, MD, USA, Final Report, 2003, WS2003 CLSP Summer Workshop.
9. J. S. White, T. O'Cornell, and F. O'Nava, "The ARPA MT evaluation methodologies: Evolution, lessons and future approaches," in Proc. Assoc. Mach. Translat. Amer., Oct. 1994, pp. 193-205.
10. th Annu. Meeting Assoc. Comput. Linguist., Portland, OR, USA, Jun. 2011, pp. 153-158.
11. th Annu. Meeting Assoc. Comput. Linguist., Philadelphia, PA, USA, Jul. 2002, pp. 311-318.
12. th Eur. Conf. Speech Commun. Technol., Rhodos, Greece, Sep. 1997, pp. 2667-2670.
13. G. Doddington, "Automatic evaluation of machine translation quality using n-gram co-occurrence statistics," in Proc. Human Lang. Tech. Conf., San Diego, CA, USA, Mar. 2002.
14. A. Lavie and M. J. Denkowski, "The meteor metric for automatic evaluation of machine translation," Mach. Translat., vol. 23, pp. 105-115, May 2009.
15. M. Snover et al., "Study of translation edit rate with targeted human annotation," in Proc. 7th Biennial Conf. Assoc. Mach. Translat. Amer., Cambridge, MA, USA, Aug. 2006.
16. C. K. Lo and D. Wu, "MEANT: An inexpensive, high-accuracy, semiautomatic metric for evaluating translation utility based on semantic roles," in Proc. 49th Annu.Meeting Assoc. for Comput. Linguist., Portland, OR, USA, Jun. 2011, pp. 220-229.
17. C. K. Lo and D. Wu, "MEANT at WMT 2013: A tunable, accurate yet inexpensive semantic frame based MT evaluation metric," in Proc. 8th Workshop Statist. Mach. Translat., Sofia, Bulgaria, Aug. 2013, pp. 422-428.
18. E. Hovy, M. King, and A. Popescu-Belis, "Principles of context-based machine translation evaluation," Mach. Translat., vol. 17, no. 1, pp. 43-75, Mar. 2002.
19. C. Callison-Burch et al., "(Meta-) evaluation of machine translation,"in Proc. 2nd Workshop Statist. Mach. Translat., Prague, Czech Republic, Jun. 2007, pp. 136-158.
20. A. Lopez, "Statistical machine translation," ACM Comput. Surveys, vol. 40, no. 3, pp. 8-49, Aug. 2008.
21. H. Somers, "Round-trip translation: What is it good for?," in Proc. Workshop Australasian Lang. Technol. Assoc, Sydney, Australia, Dec. 2005, pp. 127-133.
22. R. Rapp, "The back-translation score: Automatic MT evaluation at the sentences level without reference translations," in Proc. 47th Annu. Meeting Assoc. Comput. Linguist., Singapore, Aug. 2009, pp. 133-136.
23. C. K. Lo et al., "XMEANT: Better semantic MT evaluation without reference translations," in Proc. 52nd Annu. Meeting Assoc. Comput. Linguist., Baltimore, MD, USA, Jun. 2014, pp. 765-771.
24. C. B. Quirk, "Training a sentence-level machine translation confidence measure," in Proc. 4th Int. Conf. Lang. Resources Eval., Lisbon, Portugal, May 2004, pp. 825-828.
25. M. Gamon, A. Aue, and M. Smets, "Sentence-level MT evaluation without reference translations: Beyond language modeling," in Proc. 10th Annu. Conf. Eur. Assoc. Mach. Translat., Budapest, Hungary, May 2005, pp. 103-111.
26. J. S. Albrecht and R. Hwa, "Regression for sentence-level MT evaluation with pseudo references," in Proc. 45th Annu. Meeting Assoc. Comput. Linguist., Prague, Czech Republic, Jun. 2007, pp. 296-303.
27. L. Specia et al., "Improving the confidence of machine translation quality estimates," in Proc. MT Summit XII, Ottawa, ON, Canada, Aug. 2009.
28. A. L. F. Han et al., "LEPOR: A robust evaluation metric for machine translation with augmented factors," in Proc. 24th Int. Conf. Comput. Linguist., Mumbai, India, Dec. 2012.
29. M. Macháèek and O. Bojar, "Results of the WMT13 metrics shared task," in Proc. 8th Workshop Statist. Mach. Translat., Sofia, Bulgaria, Aug. 2013, pp. 45-51.
30. C. K. Lo and D. Wu, "On the reliability and inter-annotator agreement of human semantic MT evaluation via HMEANT," in Proc. 14th Int. Conf. Lang. Resources Eval., Reykjavik, Iceland, May 2014, pp. 602-607.
31. A. Birch et al., "The feasibility of HMEANT as a human MT evaluation metric," in Proc. 8th Workshop Statist. Mach. Translat., Assoc. for Comput. Linguist., Sofia, Bulgaria, Aug. 2013, pp. 52-61.
32. D. Vilar et al., "Human evaluation of machine translation through binary system comparisons," in Proc. 2nd Workshop Statist. Mach. Translat., Prague, Czech Republic, Jun. 2007, pp. 96-103.
33. T. K. Landauer, P. W. Foltz, and D. Laham, "Introduction to latent semantic analysis," Discourse Processes, vol. 25, pp. 259-284, 1998.
34. S. Dumais, T. K. Landauer, and M. L. Littman, "Automatic cross-linguistic information retrieval using latent semantic indexing," in Proc. SIGIR Workshop Cross-Lingual Inf. Retrieval, Philadelphia, PA, USA, Jul. 1997, pp. 16-23.
35. G. M. Salton, A. K. Wong, and C. S. Yang, "A vector space model for automatic indexing," Commun. ACM, vol. 18, no. 11, pp. 613-620, Nov. 1975.
36. C. D. Manning and H. Schutze, Foundations of Statistical Natural Language Processing. Cambridge, MA: The MIT Press, 1999, ch. 6.
37. G. H. Golub and W. Kahan, "Calculating the singular values and pseudo-inverse of a matrix," J. Soc. Ind. Appl. Math.: Numer. Anal., vol. 2, no. 2, pp. 205-224, 1965.
38. A. Stolcke, "SRILM - an extensible language modeling toolkit," in Proc. 7th Int. Conf. Spoken Lang. Process., Denver, CO, USA, Sep. 2002.
39. M. Snover et al., "Fluency, adequacy, or HTER? exploring different human judgments with a tunable MT metric," in Proc. 4th Workshop Statist. Mach. Translat., Athens, Greece, Mar. 2009.
40. F. A. Gers and J. Schmidhuber, "LSTM recurrent networks learn simple context free and context sensitive languages," IEEE Trans. Neural Netw., vol. 12, no. 6, pp. 1333-1340, Nov. 2001.