Aggregating and predicting sequence labels from crowd annotations

ACL 2017 - 55th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)

Volumn 1, Issue , 2017, Pages 299-309

  Nguyen, An T ^a   Wallace, Byron C ^b   Li, Junyi Jessy ^c   Nenkova, Ani ^c   Lease, Matthew ^a  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

^b NORTHEASTERN UNIVERSITY   (United States)

^c UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; FORECASTING; HIDDEN MARKOV MODELS; LINGUISTICS; MARKOV PROCESSES;

BIOMEDICAL ABSTRACTS; NAMED ENTITY RECOGNITION; NEWS ARTICLES; SOURCE CODES; TEXT GENRE; TRAINING DATA; UNANNOTATED TEXTS;

CHARACTER RECOGNITION;

EID: 85040925942     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.18653/v1/P17-1028     Document Type: Conference Paper

References (41)

1. Matthew James Beal. 2003. Variational algorithms for approximate Bayesian inference. University of London United Kingdom.
2. Wei Bi, Liwei Wang, James T. Kwok, and Zhuowen Tu. 2014. Learning to predict from crowdsourced data. In Uncertainty in Artificial Intelligence.
3. Daniel M Bikel, Scott Miller, Richard Schwartz, and Ralph Weischedel. 1997. Nymble: a high-performance learning name-finder. In Proceedings of the fifth conference on Applied natural language processing. Association for Computational Linguistics, pages 194-201. https://doi.org/10.3115/974557.974586.
4. Léon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010, Springer, pages 177-186.
5. Jonathan Bragg, Andrey Kolobov, Mausam Mausam, and Daniel S Weld. 2014. Parallel task routing for crowdsourcing. In Second AAAI Conference on Human Computation and Crowdsourcing.
6. Hai Leong Chieu and Hwee Tou Ng. 2002. Named entity recognition: a maximum entropy approach using global information. In Proceedings of the 19th international conference on Computational linguistics-Volume 1. Association for Computational Linguistics, pages 1-7. http://aclweb.org/anthology/C02-1025.
7. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research 12(Aug):2493-2537.
8. Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics pages 20-28.
9. Arthur P Dempster, Nan M Laird, and Donald B Rubin. 1977. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society. Series B (methodological) pages 1-38.
10. Mark Dredze, Partha Pratim Talukdar, and Koby Crammer. 2009. Sequence learning from data with multiple labels. In ECML-PKDD 2009 workshop on Learning from Multi- Label Data.
11. Paul Felt, Eric Ringger, Kevin Seppi, and Robbie Haer-tel. 2015. Early gains matter: A case for preferring generative over discriminative crowdsourcing models. In Conference of the North American Chapter of the Association for Computational Linguistics. https://doi.org/10.3115/v1/N15-1089.
12. Tim Finin, Will Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. Annotating named entities in twitter data with crowdsourcing. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk. Association for Computational Linguistics, pages 80-88.
13. Sharon Goldwater and Tom Griffiths. 2007. A fully bayesian approach to unsupervised part-of-speech tagging. In Annual meeting-association for computational linguistics. Citeseer, volume 45, page 744. http://aclweb.org/anthology/P07-1094.
14. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735-1780.
15. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with mace. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pages 1120-1130. http://aclweb.org/anthology/N13-1132.
16. Dirk Hovy, Barbara Plank, and Anders Søgaard. 2014. Experiments with crowdsourced re-annotation of a pos tagging data set. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, pages 377-382. https://doi.org/10.3115/v1/P14-2062.
17. Xiaoli Huang, Jimmy Lin, and Dina Demner-Fushman. 2006. PICO as a Knowledge Representation for Clinical Questions. In AMIA 2006 Symposium Proceedings. pages 359-363.
18. Ziheng Huang, Jialu Zhong, and Rebecca J. Passon-neau. 2015. Estimation of discourse segmentation labels from crowd data. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 2190-2200. http://aclweb.org/anthology/D15-1261.
19. Mark Johnson. 2007. Why doesn't em find good hmm pos-taggers? In EMNLP-CoNLL. pages 296-305. http://aclweb.org/anthology/D07-1031.
20. Hiroshi Kajino, Yuta Tsuboi, and Hisashi Kashima. 2012. A convex formulation for learning from crowds. In Twenty-Sixth AAAI Conference on Artificial Intelligence.
21. Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. In International conference on artificial intelligence and statistics. pages 619-627.
22. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pages 1746-1751. http://www.aclweb.org/anthology/D14-1181.
23. John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the eighteenth international conference on machine learning, ICML. volume 1, pages 282-289.
24. Guillaume Lample, Miguel Ballesteros, Sandeep Sub-ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pages 260-270. https://doi.org/10.18653/v1/N16-1030.
25. Chao Liu and Yi-min Wang. 2012. Truelabel+ confusions: A spectrum of probabilistic models in analyzing multiple ratings. In Proceedings of the 29th International Conference on Machine Learning (ICML-12). pages 225-232.
26. Qiang Liu, Jian Peng, and Alex T Ihler. 2012. Variational inference for crowdsourcing. In Advances in Neural Information Processing Systems. pages 692-700.
27. Andrew McCallum and Wei Li. 2003. Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, chapter Early results for Named Entity Recognition with Conditional Random Fields, Feature Induction and Web-Enhanced Lexicons. http://aclweb.org/anthology/W03-0430.
28. An T Nguyen, Byron C Wallace, and Matthew Lease. 2016. A correlated worker model for grouped, imbalanced and multitask data. In Uncertainty in Artificial Intelligence.
29. Naoaki Okazaki. 2007. Crfsuite: a fast implementation of conditional random fields (crfs). http://www.chokkan.org/software/crfsuite/.
30. Lawrence Rabiner and B Juang. 1986. An introduction to hidden markov models. ieee assp magazine 3(1):4-16.
31. Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research 11(Apr):1297-1322.
32. Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. 2014. Sequence labeling with multiple annotators. Machine learning 95(2):165-181.
33. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1985. Learning internal representations by error propagation. Technical report, DTIC Document.
34. Connie Schardt, Martha B Adams, Thomas Owens, Sheri Keitz, and Paul Fontelo. 2007. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC medical informatics and decision making 7(1):16.
35. Aashish Sheshadri and Matthew Lease. 2013. Square: A benchmark for research on computing crowd consensus. In First AAAI Conference on Human Computation and Crowdsourcing.
36. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 254-263. http://aclweb.org/anthology/D081027.
37. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition, pages 142-147. http://aclweb.org/anthology/W030419.
38. Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, and Milad Shokouhi. 2014. Community-based bayesian aggregation models for crowdsourcing. In Proceedings of the 23rd international conference on World wide web. ACM, pages 155-164.
39. Byron C Wallace, Joël Kuiper, Aakash Sharma, Mingxi Brian Zhu, and Iain J Marshall. 2016. Extracting pico sentences from clinical trial reports using supervised distant supervision. Journal of Machine Learning Research 17(132):1-25.
40. Hui Yang, Anton Mityagin, Krysta M Svore, and Sergey Markov. 2010. Collecting high quality overlapping labels at low cost. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval. ACM, pages 459-466.
41. Ye Zhang and Byron Wallace. 2015. A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification. arXiv preprint arXiv:1510.03820.