Applying automated metrics to speech translation dialogs

Proceedings of the 6th International Conference on Language Resources and Evaluation, LREC 2008

Volumn , Issue , 2008, Pages 2086-2093

  Condon, Sherri ^a   Phillips, Jon ^a   Doran, Christy ^a   Aberdeen, John ^a   Parvaz, Dan ^a   Oshika, Beatrice ^a   Sanders, Greg ^b   Schlenoff, Craig ^b  

^a MITRE CORPORATION   (United States)

^b NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BLEU SCORES; DEFENSE ADVANCED RESEARCH PROJECTS AGENCIES; HUMAN JUDGMENTS; RESEARCH PROGRAMS; SPEECH TRANSLATION; SPEECH TRANSLATION SYSTEMS; SYSTEM EVALUATION; TEST DATA;

TRANSLATION (LANGUAGES);

EID: 84870248019     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. Banerjee, S. and A. Lavie. (2005). METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL 2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, pp. 65-73.
2. Belvin, R., Riehemann, S., and K. Precoda. (2004). A Fine-Grained Evaluation Method for Speech-to-Speech Machine Translation Using Concept Annotations. In Proceedings of LREC 2004, pp. 1427-1430.
3. Callison-Burch, C., Osborne, M., and P., Koehn. (2006). Re-evaluating the role of BLEU in Machine Translation Research. In Proceedings of EACL 2006, pp. 249-256.
4. Chatterjee, N., Johnson, A., and M. Krishna. (2007). Some Improvements over the BLEU Metric for Measuring Translation Quality for Hindi. In Proceedings of the International Conference on Computing: Theory and Applications 2007, pp. 485-90.
5. Coughlin, D. (2003). Correlating automated and human assessments of machine translation quality. In Proceedings of MT Summit IX, pp. 63-70.
6. Culy, C. and S. Riehemann. (2003). The Limits of N-gram Translation Evaluation Metrics. In Proceedings of the MT Summit IX, AMTA, pp. 71-18.
7. Doddington, G.. (2002). Automatic Evaluation of Machine Translation Quality Using N-Gram Co-Occurrence Statistics. In Proceeding of the Second International Conference on Human Language Technology, pp. 138-145.
8. Gates, D., Lavie, A., Levin, L., Waibel, A., Gavalda, M. Mayfield, L., Woszczyna, M., and P. Zhan. (1996). End-to-End Evaluation in JANUS: a Speech-to-speech Translation System. In Proceedings of the ECAI Workshop on Dialogue Processing in Spoken Language Systems, pp. 195-206.
9. King, M. (1996). Evaluating natural language processing systems. Communications of the ACM, 39, pp. 73-79.
10. Koehn, P. (2004). Statistical Significance Tests for Machine Translation Evaluation. In Proceedings of EMNLP 2004.
11. Lavie, A., Sagae, S., and S. Jayaraman. (2004). The Significance of Recall in Automatic Metrics for MT Evaluation. In Proceedings of the 6th Conference of the Association for Machine Translation in the Americas (AMTA-2004), pp. 134-143.
12. Levin, L. Gates, D., Lavie, A., Pianesi, F., Wallace, D., Watanabe, T., and M. Woszczyna. (2000). Evaluation of a practical interlingua for task-oriented dialogue. In Proceedings of the NAACL-ANLP 2000 Workshop on Applied interlinguas: practical applications of interlingual approaches to NLP, Volume 2, pp. 18-23.
13. Lita, L. V., Rogati, M., and A. Lavie. (2005). BLANC: Learning Evaluation Metrics for MT. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pp. 740-747.
14. Nübel, R. (1997). "End-to-End Evaluation in Verbmobil I", In Proceedings of MT Summit VI, pp. 232-239.
15. Owczarzak, K., van Genabith, J., and A. Way. (2007). Dependency-Based Automatic Evaluation for Machine Translation. In Proceedings of HLT-NAACL 2007 AMTA Workshop on Syntax and Structure in Statistical Translation, pp. 80-87.
16. Papineni, K., Roukos, S., Ward, T., and W-J. Zhu. (2002). Bleu: A method for automatic evaluation of machine translation. In Proceedings of ACL 2002, pp. 311-318.
17. Sanders, G., Bronsart, S., Condon, S., and C. Schlenoff, (2008). Odds of successful transfer of low-level concepts: A key metric for bidirectional speech-to-speech machine translation in DARPA's TRANSTAC program. In Proceedings of LREC 2008.
18. Snover, M., Dorr, B., Schwartz, R., Makhoul, J., and L. Micciula. (2006). A Study of Translation Error Rate with Targeted Human Annotation. In Proceedings of AMTA 2006, pp. 223-231.
19. Turian, J. P., Shen, L. and I. D. Melamed. (2003). Evaluation of Machine Translation and Its Evaluation. In Proceedings of MT Summit 2003, pp. 386-393.
20. Weiss, B., Schlenoff, C., Sanders, G., Steves, M., Condon, S., Phillips, J., and Parvaz, D. (2008). Performance Evaluation of Speech Translation Systems. In Proceedings of LREC 2008.
21. Zhang, Y. and S. Vogel. (2004). Measuring confidence intervals for the machine translation evaluation metrics. In Proceedings of TMI 2004, pp. 85-94.