Comparative evaluation of online machine translation systems with legal texts

Law Library Journal

Volumn 100, Issue 2, 2008, Pages 299-321

  Kit, Chunyu ^a   Tak, Ming Wong ^a  

^a CITY UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 44649164693     PISSN: 00239283     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (86)

1. AUTOMATIC LANGUAGE PROCESSING ADVISORY COMMITTEE, LANGUAGE AND MACHINES: COMPUTERS IN TRANSLATION AND LINGUISTICS (1966) (Publication 1416, National Academy of Sciences National Research Council), available at http://www.nap.edu/openbook.php?isbn=ARC000005.
2. Sarah Yates, Scaling the Tower of Babel Fish: An Analysis of the Machine Translation of Legal Information, 98 LAW LIBR. J. 481, 491, 2006 LAW LIBR. J. 27, ¶ 42.
3. AltaVista, Babel Fish Translation, http://babelfish.altavista.com (last visited Dec. 21, 2007).
4. AltaVista, Babel Fish FAQ, Why Are Some Translations Lesser Quality?, http://www.altavista.com/help/babelfish/babel_faq (last visited Jan. 4, 2008).
5. Yates, supra note 2, at 500, ¶ 67.
6. Harold Somers, The Use of Machine Translation by Law Librarians - A Reply to Yates, 99 LAW LIBR. J. 611, 618, 2007 LAW LIBR. J. 35, ¶ 34
7. (citing Harold L. Somers & Nuria Prieto-Alvarez, Multiple-Choice Reading Comprehension Tests for Comparative Evaluation of MT Systems, in WORKSHOP ON MACHINE TRANSLATION EVALUATION AMTA-2000, at 107 (2000)).
8. Id. at 618-19, ¶ 35.
9. See id. at 619, ¶ 35.
10. See id. at 616, ¶ 23.
11. Readers interested in MT history may refer to John Hutchins, The History of Machine Translation in a Nutshell (Nov. 2005), http://www.hutchinsweb.me.uk/Nutshell-2005.pdf.
12. Kurt Ament, Babel Fish, http://www.infotektur.com/demos/babelfish/ en.html (last visited Jan. 4, 2008).
13. John Hutchins, Machine Translation: A Concise History, in COMPUTER AIDED TRANSLATION: THEORY AND PRACTICE (forthcoming 2008), available at http://www. hutchinsweb.me.uk/CUHK-2006.pdf, at 7 (last visited Jan. 4, 2008).
14. John Hutchins, Has Machine Translation Improved? Some Historical Comparisons, in IAMT, MT SUMMIT IX 7-8 (2003), http://www.amtaweb. org/summit/MTSummit/FinalPapers/12-Hutchins-final .pdf.
15. Martin Kay, The Proper Place of Men and Machines in Language Translation, 12 MACHINE TRANSLATION 3 (1997).
16. Jin Yang & Elke Lange, Going Live on the Internet, in COMPUTERS AND TRANSLATION 191, 198-99 (Harold Somers ed., 2003).
17. See generally Yates, supra note 2.
18. See Yates, supra note 2, at 484, ¶ 14.
19. This MT pyramid diagram (with some modifications) was first used by Vauquois in 1968, according to Doug Arnold, Why Translation is Difficult for Computers, in COMPUTERS AND TRANSLATION, supra note 15, at 119, 122.
20. The transfer approach is also known as the rule-based approach.
21. As shown in figure 1, there is no single way of transfer; some MT systems may adopt a synthesis-light approach (e.g., Systran) while some may adopt a synthesis-heavy approach (e.g., Google).
22. Arnold, supra note 18, at 122.
23. Id. at 124-25.
24. Id. at 129.
25. Id. at 133-34.
26. Id. at 133.
27. See John Hutchins, Current Commercial Machine Translation Systems and Computer-Based Translation Tools: System Types and Their Uses, 17 INTERNATIONAL JOURNAL OF TRANSLATION 5, 34 (2005) (discussing reasons why MT fails and how it can be improved using feedback).
28. See Free Translation.com Help and FAQ, What Should Machine Translation Be Used For?, http:// www.freetranslation.com/help/#uses (last visited Jan. 10, 2008).
29. AltaVista Help, Translation, http://www.altavista.com/help/babelfish/ babel_help (last visited Jan. 10, 2008).
30. Yates, supra note 2, at 486-88, ¶¶ 21-29.
31. Id. at 486, ¶ 25.
32. John Hutchins, Machine Translation and Computer-based Translation Tools: What's Available and How It's Used, in A NEW SPECTRUM OF TRANSLATION STUDIES 13, 42-43 (José Maria Bravo ed., 2004).
33. Terence Lewis, When Not to Use MT and Other Translation Tools 36-38 (1997), http://www.mt-archive.info/EAMT-1997-Lewis.pdf (paper presented at EAMT Workshop, Copenhagen, May 1997).
34. Hutchins, supra note 26, at 34.
35. D. Verne Morland, Nutzlos, Bien Pratique, or Muy Util? Business Users Speak Out on the Value of Pure Machine Translation, http://www.roi-learning. com/dvm/pubs/articles/tatc-24 (last visited Mar. 18, 2008).
36. Id.
37. MIKE DILLINGER & ARLE LOMMEL, LISA BEST PRACTICE GUIDES: IMPLEMENTING MACHINE TRANSLATION 4 (2004), available at http://www.lisa.org/products/bestPractice/index.html (site registration required).
38. See John Hutchins, The Development and Use of Machine Translation Systems and Computer-based Translation Tools, 15 INTERNATIONAL JOURNAL OF TRANSLATION 5-6 (2003).
39. DILLINGER & LOMMEL, supra note 36, at 8.
40. John S. White, How to Evaluate Machine Translation, in COMPUTERS AND TRANSLATION, supra note 15, at 211, 213.
41. Id. at 218-19.
42. Harold Somers, Round-Trip Translation: What Is It Good For?, PROC. AUSTRALASIAN LANGUAGE TECH. WORKSHOP 127, 127 (2005), available at http://www.alta.asn.au/ events/altw2005/cdrom/pdf/ ALTA200519.pdf.
43. Biomedical Translation, More Machine Translation: Fun with Computer Generated Translation! (Oct. 2003), http://www.biomedical.com/news.html.
44. Somers, supra note 41, at 127-28 (quoting Theresa A. O'Connell, Preparing Your Website for Machine Translation: How to Avoid Losing (or Gaining) Something in the Translation (2001), http:// www-128.ibm.com/developerworks/ web/library/us-mt) (alterations in original).
45. Free Translation.com Help and FAQ, Why Does the Translation Not Make Sense If I Translate My Text into Another Language and Then Back Again?, http://www.freetranslation.com/help/#sense (last visited Jan. 9, 2008).
46. See Somers, supra note 41, at 131.
47. See generally, Papineni, et al., BLEU: A Method for Automatic Evaluation of Machine Translation, in PROCEEDINGS OF THE 40TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS 311. But see Somers, supra note 7, at 616, ¶ 23 for a critique of this method.
48. Papineni et al., supra note 46, at 311 (emphasis omitted).
49. White, supra note 39, at 240.
50. Papineni et al., supra note 46, at 312.
51. The details of the BLEU metric, which involves some more complex mathematical calculation such as n-gram precision and sentence brevity penalty, are not described here.
52. George Doddington, Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurrence Statistics, 2 HUMAN LANGUAGE TECHNOLOGY CONFERENCE 138, 138 (2002), available at http://portal.acm.org/ft_gateway.cfm?id=1289273&type=pdf&coll= GUIDE&dl=GUIDE&CFID=11617123&CFTOKEN=54214794 (site registration required).
53. NIST 2005 Machine Translation Evaluation Official Results (Aug. 1, 2005), http://www.nist.gov/speech/tests/mt/2005/doc/ mt05eval_official_results_release_20050801_v3.html.
54. See NIST, The 2006 NIST Machine Translation Evaluation Plan (Mar. 16, 2006), http://www.nist.gov/speech/tests/mt/2006/doc/mt06_evalplan.v4.pdf
55. Doddington, supra note 51, at 142, fig. 6.
56. Treaty of Nice, Mar. 10, 2001, 2001 O.J. (C 80) 1; Consolidated Version of the Treaty Establishing the European Community, Nov. 10, 1997, 1997 O.J. (C 340) 3; Consolidated Text of the Treaty on European Union, Dec. 24, 2002, 2002 O.J. (C 325) 5; Treaty Establishing a Constitution for Europe, Oct. 29, 2004, 2004 O.J. (C 310) 1.
57. EUR-Lex, http://eur-lex.europa.eu/en/index.htm (last visited Jan. 4, 2008).
58. The Universal Declaration of Human Rights, http://www.unhchr.ch/udhr/ index.htm (last visited Jan. 4, 2008).
59. The thirteen languages are Dutch, French, German, Greek, Italian, Portuguese, Russian, Spanish, Swedish, Arabic, Chinese, Japanese, and Korean.
60. Babel Fish, supra note 3.
61. Google Translate, http://www.google.com/language_tools (last visited Jan. 10, 2008).
62. ProMT, http://www.online-translator.com (last visited Jan. 10, 2008).
63. SDL free translator, http://www.freetranslation.com (last visited Jan. 10, 2008).
64. Systran, http://www.systransoft.com (last visited Jan. 10, 2008).
65. WorldLingo, http://www.worldlingo.com/en/products_services/ worldlingo_translator.html (last visited Jan. 10, 2008).
66. See Mary Flanagan & Steve McClure, SYSTRAN and the Reinvention of MT 3 (Jan. 2002), http://www.systransoft.com/DocDownload/ industry-reports/2002.01.SYSTRAN.And.The.Reinvention.Of.MT.pdf.
67. See Posting of Franz Och to Google Research Blog, http://googleresearch.blogspot.com/2006/04/ statistical-machine-translation- live.html (Apr. 28, 2006).
68. For example, Reverso (www.reverso.net/text_translation.asp) has a limit of 400 characters for each translation, and Amikai (www.amikai.com/demo.jsp) a limit of 100 characters.
69. See supra ¶ 6.
70. Sarah Yates, I Need This in English, AALL SPECTRUM, Apr. 2005, at 8.
71. In MT developers' terminology, how well a text or a language pair can be translated is termed "MT Translatability."
72. The extraction of textual content from a web page was carried out by trimming off all HTML tags and other noncontent codes in the page.
73. MT Scoring Software (mteval-v11b.pl) (released May 20, 2005), http://www.nist.gov/speech/tests/mt/ 2008/scoring.html.
74. Traditional and Simplified Chinese are encoded by two different Chinese encoding schemes, namely, Big5 and GB codes respectively. In our test, we found that Google Translate is capable of selecting the proper encoding between the two automatically.
75. In fact, WorldLingo stands out significantly for its language coverage over all language pairs, by not restricting to a small number of fixed language pairs. It provides separate lists of source and target languages in its Web interface to allow its users to select any combination of them. This offers a total of 182 language pairs, far exceeding any other online MT system. Worldlingo, supra note 64.
76. Data as of Jan. 13, 2007.
77. Somers, supra note 7, at 616 n. 29.
78. NIST, supra note 52; NIST, supra note 53, at 1.
79. See Jan Krikke, Machine Translation Inching Toward Human Quality, IEEE INTELLIGENT SYSTEMS, Mar./Apr. 2006, at 4.
80. Hutchins, supra note 26, at 30.
81. See Kay, supra note 14, at 3, 20-22.
82. An evaluation similar to the one performed by Yates. See Yates, supra note 2.
83. White, supra note 39, at 230-31.
84. See Hutchins, supra note 26, at 28.
85. See id. at 11.
86. Somers, supra note 7, at 616, n. 29.