Continuous measurement scales in human evaluation of machine translation

LAW 2013 and ID 2013 - 7th Linguistic Annotation Workshop and Interoperability with Discourse, Proceedings of the Workshop

Volumn , Issue , 2020, Pages 33-41

  Graham, Yvette ^a   Baldwin, Timothy ^a   Moffat, Alistair ^a   Zobel, Justin ^a  

^a UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; COMPUTER AIDED LANGUAGE TRANSLATION; INTEROPERABILITY;

AMAZON'S MECHANICAL TURKS; CONTINUOUS MEASUREMENTS; CONTINUOUS SCALE; CONTROL TECHNIQUES; HUMAN EVALUATION; KAPPA COEFFICIENT; MACHINE TRANSLATION EVALUATIONS; MACHINE TRANSLATIONS;

QUALITY CONTROL;

EID: 85084339089     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (16)

1. A. Alexandrov. 2010. Characteristics of single-item measures in Likert scale format. The Electronic Journal of Business Research Methods, 8:1-12.
2. C. Callison-Burch, C. Fordyce, P. Koehn, C. Monz, and J. Schroeder. 2007. (Meta-) evaluation of machine translation. In Proc. 2nd Wkshp. Statistical Machine Translation, pages 136-158, Prague, Czech Republic.
3. C. Callison-Burch, C. Fordyce, P. Koehn, C. Monz, and J. Schroeder. 2008. Further meta-evaluation of machine translation. In Proc. 3rd Wkshp. Statistical Machine Translation, pages 70-106, Columbus, Ohio.
4. C. Callison-Burch, P. Koehn, C. Monz, and J. Schroeder. 2009. Findings of the 2009 Workshop on Statistical Machine Translation. In Proc. 4th Wkshp. Statistical Machine Translation, pages 1-28, Athens, Greece.
5. C. Callison-Burch, P. Koehn, C. Monz, K. Peterson, M. Przybocki, and O. Zaidan. 2010. Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation. In Proc. 5th Wkshp. Statistical Machine Translation, pages 17-53, Uppsala, Sweden.
6. C. Callison-Burch, P. Koehn, C. Monz, and O. Zaidan. 2011. Findings of the 2011 Workshop on Statistical Machine Translation. In Proc. 6th Wkshp. Statistical Machine Translation, pages 22-64, Edinburgh, Scotland.
7. C. Callison-Burch, P. Koehn, C. Monz, M. Post, R. Soricut, and L. Specia. 2012. Findings of the 2012 Workshop on Statistical Machine Translation. In Proc. 7th Wkshp. Statistical Machine Translation, pages 10-51, Montreal, Canada.
8. C. Callison-Burch. 2009. Fast, cheap, and creative: Evaluating translation quality using Amazon's Mechanical Turk. In Proc. Conf. Empirical Methods in Natural Language Processing, pages 286-295, Singapore.
9. M. Denkowski and A. Lavie. 2010. Choosing the right evaluation for machine translation: An examination of annotator and automatic metric performance on human judgement tasks. In Proc. 9th Conf. Assoc. Machine Translation in the Americas (AMTA), Denver, Colorado.
10. K. Fort, G. Adda, and K. B. Cohen. 2011. Amazon Mechanical Turk: Gold mine or coal mine? Computational Linguistics, 37(2):413-420.
11. E. Gibson, S. Piantadosi, and K. Fedorenko. 2011. Using Mechanical Turk to obtain and analyze English acceptability judgments. Language and Linguistics Compass, 5/8:509-524.
12. Y. Graham, T. Baldwin, A. Harwood, A. Moffat, and J. Zobel. 2012. Measurement of progress in machine translation. In Proc. Australasian Language Technology Wkshp., pages 70-78, Dunedin, New Zealand.
13. LDC. 2005. Linguistic data annotation specification: Assessment of fluency and adequacy in translations. Technical report, Linguistic Data Consortium. Revision 1.5.
14. R. A. Seymour, J. M. Simpson, J. E. Charlton, and M. E. Phillips. 1985. An evaluation of length and end-phrase of visiual analogue scales in dental pain. Pain, 21:177-185.
15. R. Snow, B. O'Connor, D. Jursfsky, and A. Y. Ng. 2008. Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. In Proc. Conf. Empirical Methods in Natural Language Processing, pages 254-263, Honolulu, Hawaii.
16. D. L. Streiner and G. R. Norman. 1989. Health Measurement Scales: A Practical Guide to their Development and Use. Oxford University Press, fourth edition.