A Grain of Salt for the WMT Manual Evaluation

WMT 2011 - 6thWorkshop on Statistical Machine Translation, Proceedings of the Workshop

Volumn , Issue , 2011, Pages 1-11

  Bojar, Ondřej ^a   Ercegovčević, Miloš ^a   Popel, Martin ^a   Zaidan, Omar F ^b  

^a CHARLES UNIVERSITY   (Czech Republic)

^b JOHNS HOPKINS UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; MACHINE TRANSLATION; NATURAL LANGUAGE PROCESSING SYSTEMS;

RESEARCH COMMUNITIES; STATISTICAL MACHINE TRANSLATION;

COMPUTER AIDED LANGUAGE TRANSLATION;

EID: 84884921357     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (14)

1. R. Artstein and M. Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555-596.
2. John J. Bartko and William T. Carpenter. 1976. On the methods and theory of reliability. Journal of Nervous and Mental Disease, 163(5):307-317.
3. E. M. Bennett, R. Alpert, and A. C. Goldstein. 1954. Communications through limited questioning. Public Opinion Quarterly, 18(3):303-308.
4. Jan Berka, Martin Černý, and Ondřej Bojar. 2011. Quiz-Based Evaluation of Machine Translation. Prague Bulletin of Mathematical Linguistics, 95:77-86, March.
5. Ondřej Bojar. 2011. Analyzing Error Types in English-Czech Machine Translation. Prague Bulletin of Mathematical Linguistics, 95:63-76, March.
6. Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 17-53, Uppsala, Sweden, July. Association for Computational Linguistics.
7. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46.
8. Barbara Di Eugenio and Michael Glass. 2004. The kappa statistic: A second look. Computational linguistics, 30(1):95-101.
9. J. L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378-382.
10. Klaus Krippendorff. 1980. Content Analysis: An Introduction to Its Methodology. Sage Publications, Beverly Hills, CA. Chapter 12.
11. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33:159-174.
12. José B. Mariño Mireia Farrús, Marta R. Costa-jussà and José A. R. Fonollosa. 2010. Linguistic-based evaluation criteria to identify statistical machine translation errors. In Proceedings of the 14th Annual Conference of the Euoropean Association for Machine Translation (EAMT'10), pages 167-173, May.
13. William A. Scott. 1955. Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3):321-325.
14. David Vilar, Jia Xu, Luis Fernando D'Haro, and Hermann Ney. 2006. Error Analysis of Machine Translation Output. In International Conference on Language Resources and Evaluation, pages 697-702, Genoa, Italy, May.