Modelling annotator bias with multi-task Gaussian processes: An application to machine translation quality estimation

ACL 2013 - 51st Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference

Volumn 1, Issue , 2013, Pages 32-42

  Cohn, Trevor ^a   Specia, Lucia ^a  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER AIDED LANGUAGE TRANSLATION; GAUSSIAN DISTRIBUTION; GAUSSIAN NOISE (ELECTRONIC);

GAUSSIAN PROCESSES; HUMAN SUBJECTS; LEVELS OF CONSISTENCY; LINGUISTIC DATA; MACHINE TRANSLATIONS; MULTITASK LEARNING; NOVEL TECHNIQUES; TWO MACHINES;

COMPUTATIONAL LINGUISTICS;

EID: 84905714139     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (30)

1. Mauricio A. Alvarez, Lorenzo Rosasco, and Neil D. Lawrence. 2011. Kernels for vector-valued functions: A review. Foundations and Trends in Machine Learning, 4(3):195-266.
2. Nguyen Bach, Fei Huang, and Yaser Al-Onaizan. 2011. Goodness: a method for measuring machine translation confidence. In the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 211-219, Portland, Oregon.
3. John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2004. Confidence Estimation for Machine Translation. In the 20th International Conference on Computational Linguistics (Coling 2004), pages 315-321, Geneva.
4. Edwin Bonilla, Kian Ming Chai, and Christopher Williams. 2008. Multi-task gaussian process prediction. In Advances in Neural Information Processing Systems (NIPS).
5. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In the Sixth Workshop on Statistical Machine Translation, pages 22-64, Edinburgh, Scotland.
6. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 workshop on statistical machine translation. In the Seventh Workshop on Statistical Machine Translation, pages 10-51, Montréal, Canada.
7. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In the 45th Annual Meeting of the Association for Computational Linguistics, Prague, Czech Republic.
8. Theodoros Evgeniou, Charles A. Micchelli, and Massimiliano Pontil. 2006. Learning multiple tasks with kernel methods. Journal of Machine Learning Research, 6(1):615.
9. Peter A. Flach, Sebastian Spiegler, Bruno Golénia, Simon Price, John Guiver, Ralf Herbrich, Thore Graepel, and Mohammed J. Zaki. 2010. Novel tools to streamline the conference review process: experiences from SIGKDD'09. SIGKDD Explor. Newsl., 11(2):63-67, May.
10. Mehmet Gönen and Ethem Alpaydin. 2011. Multiple kernel learning algorithms. Journal of Machine Learning Research, 12:2211-2268.
11. Perry Groot, Adriana Birlutiu, and Tom Heskes. 2011. Learning from multiple annotators with gaussian processes. In Proceedings of the 21st international conference on Artificial neural networks - Volume Part II, ICANN'11, pages 159-164, Espoo, Finland.
12. Yifan He, Yanjun Ma, Josef van Genabith, and Andy Way. 2010. Bridging smt and tm with translation recommendation. In the 48th Annual Meeting of the Association for Computational Linguistics, pages 622-630, Uppsala, Sweden.
13. Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP'10, pages 64-67, Washington DC.
14. Maarit Koponen, Wilker Aziz, Luciana Ramos, and Lucia Specia. 2012. Post-editing time as a measure of cognitive effort. In Proceedings of the AMTA 2012 Workshop on Post-editing Technology and Practice, WPTP 2012, San Diego, CA.
15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania.
16. Mirko Plitt and François Masselot. 2010. A productivity test of statistical machine translation post-editing in a typical localisation context. Prague Bull. Math. Linguistics, 93:7-16.
17. Tamara Polajnar, Simon Rogers, and Mark Girolami. 2011. Protein interaction detection in sentences via gaussian processes; a preliminary evaluation. Int. J. Data Min. Bioinformatics, 5(1):52-72, February.
18. Christopher B. Quirk. 2004. Training a sentence-level machine translation confidence metric. In Proceedings of the International Conference on Language Resources and Evaluation, volume 4 of LREC 2004, pages 825-828, Lisbon, Portugal.
19. Carl E. Rasmussen and Christopher K. I. Williams. 2006. Gaussian processes for machine learning, volume 1. MIT press Cambridge, MA.
20. Vikas C. Raykar and Shipeng Yu. 2012. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. J. Mach. Learn. Res., 13:491-518.
21. Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. J. Mach. Learn. Res., 99:1297-1322.
22. Simon Rogers, Mark Girolami, and Tamara Polajnar. 2010. Semi-parametric analysis of multi-rater data. Statistics and Computing, 20(3):317-334.
23. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD, KDD'08, pages 614-622, Las Vegas, Nevada.
24. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast-but is it good? Evaluating non-expert annotations for natural language tasks. In the 2008 Conference on Empirical Methods in Natural Language Processing, pages 254-263, Honolulu, Hawaii.
25. Radu Soricut and Abdessamad Echihabi. 2010. Trustrank: Inducing trust in automatic translations via ranking. In the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 612-621, Uppsala, Sweden, July.
26. Lucia Specia, Marco Turchi, Nicola Cancedda, Marc Dymetman, and Nello Cristianini. 2009. Estimating the Sentence-Level Quality of Machine Translation Systems. In the 13th Annual Meeting of the European Association for Machine Translation (EAMT'2009), pages 28-37, Barcelona.
27. Lucia Specia, Dhwaj Raj, and Marco Turchi. 2010. Machine translation evaluation versus quality estimation. Machine Translation, pages 39-50.
28. Lucia Specia. 2011. Exploiting Objective Annotations for Measuring Translation Post-editing Effort. In the 15th Annual Meeting of the European Association for Machine Translation (EAMT'2011), pages 73-80, Leuven.
29. Peter Welinder, Steve Branson, Serge Belongie, and Pietro Perona. 2010. The Multidimensional Wisdom of Crowds. In Advances in Neural Information Processing Systems, volume 23, pages 2424-2432.
30. Jacob Whitehill, Paul Ruvolo, Ting-fan Wu, Jacob Bergsma, and Javier Movellan. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in Neural Information Processing Systems, 22:2035-2043.