Generalisation in named entity recognition: A quantitative analysis

Computer Speech and Language

Volumn 44, Issue , 2017, Pages 61-83

  Augenstein, Isabelle ^a   Derczynski, Leon ^a   Bontcheva, Kalina ^a  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Entity drift; Generalisation; Information extraction; Named entity recognition; Natural language processing; Quantitative show 146 study; Social media

Indexed keywords

INFORMATION RETRIEVAL; SOCIAL NETWORKING (ONLINE);

ENTITY DRIFT; GENERALISATION; NAMED ENTITY RECOGNITION; NATURAL LANGUAGE PROCESSING; QUANTITATIVE SHOW $146 STUDY; SOCIAL MEDIA;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 85014527750     PISSN: 08852308     EISSN: 10958363     Source Type: Journal    
DOI: 10.1016/j.csl.2017.01.012     Document Type: Article

References (62)

1. Al-Onaizan, Y., Knight, K., Translating named entities using monolingual and bilingual resources. Proceedings of the Forty Annual Meeting on Association for Computational Linguistics, 2002, Association for Computational Linguistics, 400–408.
2. Attardi, G., DeepNL: a deep learning NLP pipeline. Proceedings of the First Workshop on Vector Space Modeling for Natural Language Processing, 2015, Association for Computational Linguistics, Denver, Colorado, 109–115.
3. Augenstein, I., 2014. Joint information extraction from the web using linked data. In Proceedings of the International Semantic Web Conference, Springer, pp. 505–512.
4. Baldwin, T., Cook, P., Lui, M., MacKinlay, A., Wang, L., How noisy social media text, how different social media sources?. Proceedings of the 2013 International Joint Conference on Natural Language Processing (IJCNLP), 2013, 356–364.
5. Baldwin, T., Kim, Y.-B., de Marneffe, M.C., Ritter, A., Han, B., Xu, W., Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normalization and named entity recognition. Xu, W., Han, B., Ritter, A., (eds.) Proceedings of the 2015 Workshop on Noisy User-generated Text, 2015, Association for Computational Linguistics, Beijing, China, 126–135.
6. Bengio, Y., Deep learning of representations for unsupervised and transfer learning. Proceedings of the 2012 Workshop on Unsupervised and Transfer Learning Challenges in Machine Learning, vol. 7, 2012, 19.
7. Bollacker, K., Evans, C., Paritosh, P., Sturge, T., Taylor, J., Freebase: a collaboratively created graph database for structuring human knowledge. Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, 2008, ACM, New York, NY, USA, 1247–1250.
8. Bontcheva, K., Derczynski, L., TwitIE: An Open-Source Information Extraction Pipeline for Microblog Text. Proceedings of Recent Advances in Natural Language Processing, 2014, Hissar, Bulgaria, 83–90.
9. Brown, P.F., Desouza, P.V., Mercer, R.L., Pietra, V.J.D., Lai, J.C., Class-based n-gram models of natural language. Comput. Linguist. 18:4 (1992), 467–479.
10. Cherry, C., Guo, H., The unreasonable effectiveness of word representations for Twitter named entity recognition. Mihalcea, R., Chai, J., Sarkar, A., (eds.) Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2015, Association for Computational Linguistics, Denver, Colorado, 735–745.
11. Chinchor, N.A., Proceedings of the seventh message understanding conference (MUC-7) named entity task definition. Proceedings of the Seventh Message Understanding Conference (MUC-7). Fairfax, VA, 1998.
12. Chiticariu, L., Krishnamurthy, R., Li, Y., Reiss, F., Vaithyanathan, S., Domain adaptation of rule-based annotators for named-entity recognition tasks. Li, H., M‘arquez, L., (eds.) Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Cambridge, MA, 2010, 1002–1012.
13. Chiu, J.P., Nichols, E., Named entity recognition with bidirectional LSTM-SNNs. Trans. Assoc. Comput. Linguist. 6 (2016), 357–370.
14. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P., Natural language processing (almost) from scratch. J. Mach. Learn. Res. (JMLR) 999888 (2011), 2493–2537.
15. Daumé, H.I., Frustratingly Easy Domain Adaptation. Zaenen, A., van den Bosch, A., (eds.) Proceedings of the Forty-fifth Annual Meeting of the Association of Computational Linguistics, 2007, Association for Computational Linguistics, Prague, Czech Republic, 256–263.
16. Derczynski, L., Augenstein, I., Bontcheva, K., 2015. USFD: Twitter NER with drift compensation and linked data. Proceedings of the Workshop on Noisy User-generated Text (W-NUT), pp. 48–53.
17. Derczynski, L., Chester, S., Generalised Brown clustering and roll-up feature generation. Proceedings of the 2016 Annual Conference of the Association for Advancement of Artificial Intelligence, 2016, AAAI.
18. Derczynski, L., Maynard, D., Aswani, N., Bontcheva, K., Microblog-genre noise and impact on semantic annotation accuracy. Proceedings of the Twenty-fourth ACM Conference on Hypertext and Social Media, 2013, ACM, New York, NY, USA, 21–30.
19. Derczynski, L., Maynard, D., Rizzo, G., van Erp, M., Gorrell, G., Troncy, R., Bontcheva, K., Analysis of named entity recognition and linking for Tweets. Inf. Process. Manag. 51 (2015), 32–49.
20. Eisenstein, J., What to do about bad language on the internet. Proceedings of the 2013 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL), 2013, 359–369.
21. Finin, T., Murnane, W., Karandikar, A., Keller, N., Martineau, J., Dredze, M., Annotating named entities in Twitter data with crowdsourcing. Callison-Burch, C., Dredze, M., (eds.) Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, 2010, Association for Computational Linguistics, Los Angeles, 80–88.
22. Finkel, J.R., Grenager, T., Manning, C., Incorporating non-local information into information extraction systems by Gibbs sampling. Blunsom, P., Cohen, S., Dhillon, P., Liang, P., (eds.) Proceedings of the Forty-third Annual Meeting of the Association for Computational Linguistics (ACL’05), 2005, Association for Computational Linguistics, Ann Arbor, Michigan, 363–370.
23. Forman, G., Cohen, I., Learning from little: comparison of classifiers given little training. Proceedings of the Eighth European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD ’04), 2004, Springer-Verlag New York, Inc., New York, NY, USA, 161–172.
24. Fromreide, H., Hovy, D., Søgaard, A., Crowdsourcing and annotating NER for Twitter #drift. Calzolari, N., Choukri, K., Declerck, T., Loftsson, H., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., Piperidis, S., (eds.) Proceedings of the 2014 International Conference on Language Resources and Evaluation Conference (LREC), 2014, European Language Resources Association, 2544–2547.
25. Gella, S., Cook, P., Baldwin, T., One sense per tweeter.. and other lexical semantic tales of Twitter. Proceedings of the 2014 Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2014, 215.
26. Glorot, X., Bordes, A., Bengio, Y., Domain adaptation for large-scale sentiment classification: a deep learning approach. Getoor, L., Scheffer, T., (eds.) Proceedings of the Twenty-eighth International Conference on Machine Learning (ICML), 2011, Omnipress, Bellevue, Washington, USA, 513–520.
27. Grishman, R., Sundheim, B., Message understanding conference-6: a brief history. Proceedings of the 1995 International Conference on Computational Linguistics (COLING), 1995, Association for Computational Linguistics.
28. Guo, H., Zhu, H., Guo, Z., Zhang, X., Wu, X., Su, Z., Domain adaptation with latent semantic association for named entity recognition. Ostendorf, M., Collins, M., Narayanan, S., Oard, D.W., Vanderwende, L., (eds.) Proceedings of the 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics of the Human Language Technologies, 2009, Association for Computational Linguistics, Boulder, Colorado, 281–289.
29. Hovy, D., Johannsen, A., Søgaard, A., User review sites as a resource for large-scale sociolinguistic studies. Gangemi, A., Leonardi, S., Panconesi, A., (eds.) Proceedings of the Twenty-fourth International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2015, ACM, 452–461.
30. Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., Weischedel, R., OntoNotes: the 90% solution. Moore, R.C., Bilmes, J., Chu-Carroll, J., Sanderson, M., (eds.) Proceedings of the 2006 Human Language Technology Conference of the NAACL, 2006, Association for Computational Linguistics, New York City, USA, 57–60 Companion Volume: Short Papers.
31. Hu, Y., Talamadupula, K., Kambhampati, S., et al. Dude, srsly?: the surprisingly formal nature of Twitter's language. Proceedings of the 2013 International Conference on Weblogs and Social Media (ICWSM), 2013.
32. Lafferty, J.D., McCallum, A., Pereira, F.C.N., Conditional random fields: probabilistic models for segmenting and labeling sequence data. Proceedings of the Eighteenth International Conference on Machine Learning, 2001, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 282–289.
33. Lewis, D.D., Yang, Y., Rose, T.G., Li, F., RCV1: a new benchmark collection for text categorization research. J. Mach. Learn. Res., 5, 2004, 361–397.
34. Liu, X., Zhang, S., Wei, F., Zhou, M., Recognizing named entities in Tweets. Matsumoto, Y., Mihalcea, R., (eds.) Proceedings of the Forty-ninth Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (HLT-ACL), 2011, Association for Computational Linguistics, Portland, Oregon, USA, 359–367.
35. Locke, B., Martin, J., Named entity recognition: adapting to microblogging. University of Colorado, 2009.
36. Masud, M.M., Chen, Q., Khan, L., Aggarwal, C., Gao, J., Han, J., Thuraisingham, B., Addressing concept-evolution in concept-drifting data streams. Proceedings of the IEEE Tenth International Conference on Data Mining (ICDM), 2010, IEEE, 929–934.
37. Maynard, D., Bontcheva, K., Augenstein, I., 2016. Natural language processing for the semantic web. Synthesis Lectures on the Semantic Web: Theory and Technology. Morgan & Claypool.
38. Maynard, D., Bontcheva, K., Cunningham, H., Towards a semantic extraction of named entities. Proceedings of the 2003 Recent Advances in Natural Language Processing. Bulgaria, 2003.
39. Mooney, R.J., Bunescu, R.C., Subsequence kernels for relation extraction. Proceedings of the Nineteenth Conference on Neural Information Processing Systems (NIPS). Vancouver, BC, 2005, 171–178.
40. Nadeau, D., Sekine, S., A survey of named entity recognition and classification. J. Linguist. Invest. 30:1 (2007), 1–20.
41. Newman, D., Chemudugunta, C., Smyth, P., Statistical entity-topic models. Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2006, ACM, 680–686.
42. Okazaki, N., 2007. CRFsuite: a fast implementation of Conditional Random Fields (CRFs). http://www.chokkan.org/software/crfsuite/.
43. Palmer, D.D., Day, D.S., A statistical profile of the named entity task. Proceedings of the Fifth Conference on Applied Natural Language Processing, 1997, Association for Computational Linguistics, Stroudsburg, PA, USA, 190–193.
44. Pavalanathan, U., Eisenstein, J., Audience-modulated variation in online social media. Am. Speech 90:2 (2015), 187–213.
45. Plank, B., Hovy, D., Personality traits on Twitter–or–How to get 1,500 personality tests in a week. Proceedings of the Sixth Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, 2015, 92–98.
46. Plank, B., Hovy, D., McDonald, R., Søgaard, A., Adapting taggers to Twitter with not-so-distant supervision. Tsujii, J., Hajic, J., (eds.) Proceedings of the 2014 International Conference on Computational Linguistics (COLING, 2014, Dublin City University and Association for Computational Linguistics, Dublin, Ireland, 1783–1792 Technical Papers.
47. Preoţiuc-Pietro, D., Volkova, S., Lampos, V., Bachrach, Y., Aletras, N., Studying user income through language, behaviour and affect in social media. PloS One, 10(9), 2015, e0138717.
48. Ratinov, L., Roth, D., Design challenges and misconceptions in named entity recognition. Stevenson, S., Carreras, X., (eds.) Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), 2009, Association for Computational Linguistics, Boulder, Colorado, 147–155.
49. Recasens, M., de Marneffe, M.-C., Potts, C., The life and death of discourse entities: identifying singleton mentions. Vanderwende, L., III, H.D., Kirchhoff, K., (eds.) Proceedings of the NAACL HLT 2013 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, 2013, Association for Computational Linguistics, Atlanta, Georgia, 627–633.
50. Ritter, A., Clark, S., Mausam, Etzioni, O., Named entity recognition in tweets: an experimental study. Barzilay, R., Johnson, M., (eds.) Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2011, Association for Computational Linguistics, Edinburgh, Scotland, UK., 1524–1534.
51. Rowe, M., Stankovic, M., Dadzie, A., Nunes, B., Cano, A., Making sense of microposts (#MSM2013): big things come in small packages. Proceedings of the 2013 Workshops on World Wide Web Conference (WWW), 2013.
52. Rowe, M., Stankovic, M., Dadzie, A.-S., #Microposts2015 – 5th Workshop on ‘Making Sense of Microposts’: Big things come in small packages. Proceedings of the Twenty-fourth International Conference on World Wide Web Companion, 2015, International World Wide Web Conferences Steering Committee, 1551–1552.
53. Schiffman, B., Nenkova, A., McKeown, K., Experiments in multidocument summarization. Proceedings of the Second International Conference on Human Language Technology Research, 2002, Morgan Kaufmann Publishers Inc., 52–58.
54. Socher, R., Lin, C.C.-Y., Ng, A.Y., Manning, C.D., Parsing natural scenes and natural language with recursive neural networks. Getoor, L., Scheffer, T., (eds.) Proceedings of the Twenty-eighth International Conference on Machine Learning (ICML), 2011, Omnipress, Bellevue, Washington, USA, 129–136.
55. Steinberger, R., Pouliquen, B., Kabadjov, M., Belyaeva, J., van der Goot, E., JRC-NAMES: a freely available, highly multilingual named entity resource. Proceedings of the 2011 Conference on Recent Advances in Natural Language Processing, 2011, 104.
56. Sutton, C., McCallum, A., Composition of conditional random fields for transfer learning. Mooney, R., Brew, C., Lee-Feng Chien, P.C.-c., Academia Sinica, Katrin Kirchhoff, P.C.-c., (eds.) Proceedings of the 2005 Conference on Human Language Technology and Empirical Methods in Natural Language Processing, 2005, University of Washington, Association for Computational Linguistics, Vancouver, British Columbia, Canada, 748–754.
57. Tjong Kim Sang, E.F., De Meulder, F., Introduction to the CoNLL-2003 shared task: language-independent named entity recognition. Daelemans, W., Osborne, M., (eds.) Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL. Edmonton, Canada, 2003, 142–147.
58. Toda, H., Kataoka, R., A search result clustering method using informatively named entities. Proceedings of the Seventh Annual ACM International Workshop on Web Information and Data Management, 2005, ACM, 81–86.
59. Walker, C., Strassel, S., Medero, J., Maeda, K., ACE 2005 Multilingual Training Corpus. 2006, Linguistic Data Consortium, Philadelphia.
60. Whitelaw, C., Kehlenbeck, A., Petrovic, N., Ungar, L., Web-scale named entity recognition. Proceedings of the Seventeenth ACM Conference on Information and Knowledge Management, 2008, ACM, New York, NY, USA, 123–132.
61. Wu, D., Lee, W.S., Ye, N., Chieu, H.L., Domain adaptive bootstrapping for named entity recognition. Koehn, P., Mihalcea, R., (eds.) Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, 2009, Association for Computational Linguistics, Singapore, 1523–1532.
62. Yang, Y., Eisenstein, J., 2017. Overcoming language variation in sentiment analysis with social attention. In: Transaction of the Association for Computational Linguistics (TACL), to appear.