Spoken Dialogue Translation Systems Evaluation: Results, New Trends, Problems and Proposals

2004 International Workshop on Spoken Language Translation, IWSLT 2004

Volumn , Issue , 2004, Pages 95-102

  Blanchon, Hervé ^a   Boitet, Christian ^a   Besacier, Laurent ^a  

^a Laboratoire CLIPS   (France)

Author keywords

[No Author keywords available]

Indexed keywords

'CURRENT; EVALUATION METHODS; EVALUATION RESULTS; OBJECTIVE METHODS; SPOKEN DIALOGUE; STATE OF THE ART; SUBJECTIVE METHODS; SYSTEM EVALUATION; TRANSLATION SYSTEMS;

TRANSLATION (LANGUAGES);

EID: 85120080645     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. Akiba, Y., Imamura, K., et al. (2001) Using Multiple Edit Distances to Automatically Rank Machine Translation Output. Proc. MT Summit VIII. Santiago de Compostela, Spain. 18-22 September, 2001. vol. 1/1: pp. 15-20.
2. Blanchon, H. and Boitet, C. (2000) Speech Translation for French within the C-STAR II Consortium and Future Perspectives. Proc. ICSLP 2000. Beijing, China. Oct. 16-20, 2000. vol. 4/4: pp. 412-417.
3. Blanchon, H., Boitet, C., et al. (2004) Towards Fairer Evaluation of Commercial MT Systems on Basic Travel Expressions Corpora. Proc. IWSLT'2004. Kyoto, Japan. September 30 - October 1, 2004. vol. 1/1: pp. 6 p.
4. Boitet, C., Blanchon, H., et al. (2000) A way to integrate context processing in the MT component of spoken, task-oriented translation systems. Proc. MSC-2000. Kyoto, Japan. October 11-13, 2000. vol. 1/1: pp. 83-87.
5. Burger, S., Besacier, L., et al. (2001) The NESPOLE! VoIP Dialog Database. Proc. Eurospeech. Aalborg, Denmark. September 3-7, 2001. 4 p.
6. Coughlin, D. (2003) Correlating Automated and Human Assessments of Machine Translation Quality. Proc. MT Summit IX. New Orleans, USA. September 23-27, 2003. 8 p.
7. Doddington, G. (2002) Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurrence Statistics. Proc. HLT 2002. San Diego, California. March 24-27, 2002. vol. 1/1: pp. 128-132 (note book proceedings).
8. Hovy, E., King, M., et al. (2002) Principles of Context-Based Machine Translation Evaluation. in Machine Translation. vol. 17(1): pp. 43-75.
9. King, M. (1996) Evaluating Natural Language Processing Systems. in Communication of the ACM. vol. 29(1): pp. 73-79.
10. Lazzari, G. (2000) Spoken Translation: Challenges and Opportunities. Proc. ICSLP 2000. Beijing, China. Oct. 16-20, 2000. vol. 4/4: pp. 430-435.
11. Leusch, G., Ueffing, N., et al. (2003) A Novel StringtoString Distance Measure with Application to Machine Translation Evaluation. Proc. MT Summit IX. New Orleans, U.S.A. September 23-27, 2003. 8 p.
12. Lin, C.-Y. (2004) ROUGE: A Package for Automatic Evaluation of Summaries. Proc. ACL 2004. Barcelona, Spain. July 21-26, 2004. 8 p.
13. Lin, C.-Y. and Och, F. J. (2004) Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics. Proc. ACL 2004. Barcelona, Spain. July 21-26, 2004. 8 p.
14. Lin, C.-Y. and Och, F. J. (2004) ORANGE: a Method for Evaluation Automatic Evaluation metrics for Machine Translation. Proc. COLING 2004. Geneva, Switzerland. August 23-27, 2004. 8 p.
15. Lowrence, R. and Wagner, C.-K. (1975) An Extension of the String-to-String Correction Problem. in Journal of the ACM. vol. 22(2): pp. 177-183.
16. Mana, N., Burger, S., et al. (2003) The Nespole! VoIP Corpora in Tourism and Medical Domains. Proc. EUROSPEECH 2003. Geneva, Switzerland. September 1-4, 2003. 4 p.
17. Melamed, I. D., Green, R., et al. (2003) Precision and Recall of Machine Translation. Proc. HLT-NAACL 2003, companion Volume. Edmonton, Canada. May 27 - June 1, 2003. 3 p.
18. Nießen, S., Och, F. J., et al. (2000) An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research. Proc. LREC 2000. Athens, Greece. 31 May - 2 June 2000. vol. 1/3: pp. 39-45.
19. Papineni, K., Roukos, S., et al. (2002) BLEU: a Method for Automatic Evaluation of Machine Translation. Proc. ACL-02. Philadelphia, USA. July 7-12, 2002. vol. 1/1: pp. 311-318.
20. Su, K.-Y., Wu, M.-W., et al. (1992) A New Quantitative Quality Measure for Machine Translation System. Proc. COLING-92. Nantes, France. 23-28 août 1992. vol. 2/4: pp. 433-439.
21. Sugaya, F., Takezawa, T., et al. (2000) Evaluation of ATR-MATRIX Speech Translation System with Pair Comparison Method Between Human and System. Proc. ICSLP 2000. Beijing, China. October 16-20, 2000. vol. 3/4: pp. 1105-1108.
22. Takezawa, T., Sumita, E., et al. (2002) Towards a Broad-coverage Bilingual Corpus for Speech Translation of Travel Conversation in the Real World. Proc. LREC-2002. Las Palmas, Spain. May 29-31, 2002. vol. 1/3: pp. 147-152.
23. Tessiore, L. and Hahn, W. (2000) Functional Evaluation of a Machine Interpretation System: Verbmobil. in Wahlster, W. (ed.), Verbmobil: Foundation of Speech-to-Speech Translation. Springer-Verlag. Berlin. pp. 611-631.
24. Turian, J. P., Shen, L., et al. (2003) Evaluation of Machine Translation and its Evaluation. Proc. MT Summit IX. New Orleans, U.S.A. September 23-27, 2003. pp. 386-393.
25. Vu Minh, Q., Besacier, L., et al. (2004) Interchange format-based language model for automatic speech recognintion in speech-to-speech translation. Proc. RIVF'04 (Recherche Informatique Vietnam-Francophonie). To be published in a special issue of Studia Informatica Universalis [Suger Editor]. February 2-5, 2004. vol. 1/1: pp. 47-50.
26. Wagner, C.-K. and Fisher, M.-J. (1974) The String-to-String Correction Problem. in Journal of the ACM. vol. 21(1): pp. 168-173.
27. Wahlster, W. ed., (2000) Verbmobil: Foundations of Speech-to-Speech Translation. Springer-Verlag. Berlin. 677 p.