Evaluating entity linking: An analysis of current benchmark datasets and a roadmap for doing a better job

Proceedings of the 10th International Conference on Language Resources and Evaluation, LREC 2016

Volumn , Issue , 2016, Pages 4373-4379

  Van Erp, Marieke ^a   Mendes, Pablo N ^b   Paulheim, Heiko ^c   Ilievski, Filip ^a   Plu, Julien ^d   Rizzo, Giuseppe ^e   Waitelonis, Jörg ^f  

^a VU UNIVERSITY AMSTERDAM   (Netherlands)

^b IBM RESEARCH   (United States)

^c UNIVERSITY OF MANNHEIM   (Germany)

^d EURECOM   (France)

^e ISMB   (Italy)

^f UNIVERSITY OF POTSDAM   (Germany)

Author keywords

Benchmark; Entity linking; Evaluation

Indexed keywords

BENCHMARKING;

BENCHMARK DATASETS; ENTITY LINKING; EVALUATION; ROADMAP;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 85030118708     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. Adida, B., Herman, I., Sporny, M., and Birbeck, M. (2012). RDFa 1.1 Primer. Technical report, World Wide Web Consortium, http://www.w3.org/TR/2012/NOTE-rdfa-primer-20120607/, June.
2. Andrea Giovanni Nuzzolese, Anna Lisa Gentile, Valentina Presutti, Aldo Gangemi, Darõ Garigliotti, and Roberto Navigli. (2015). Open knowledge extraction challenge. In Semantic Web Evaluation Challenges, page 3.
3. Cano, A. E., Rizzo, G., Varga, A., Rowe, M., Stankovic Milan, and Dadzie, A.-S. (2014). Making Sense of Microposts (#Microposts2014) Named Entity Extraction & Linking Challenge. In 4th International Workshop on Making Sense of Microposts, #Microposts.
4. Carmel, D., Chang, M.-W., Gabrilovich, E., Hsu, B.-J., and Wang, K. (2014). ERD 2014: Entity recognition and disambiguation challenge. In 37th Annual ACM SIGIR CONFERENCE.
5. Doddington, G., Mitchell, A., Przybocki, M. A., Ramshaw, L., Strassel, S., and Weischedel, R. M. (2004). The Automatic Content Extraction (ACE) Program-Tasks, Data, and Evaluation. In 4th International Conference On Language Resources and Evaluation, LREC.
6. Gerber, D., Hellmann, S., Bühmann, L., Soru, T., Usbeck, R., and Ngomo, A.-C. N. (2013). Real-time rdf extraction from unstructured data streams. In The Semantic Web-ISWC 2013, pages 135-150. Springer.
7. Hoffart, J., Yosef, M. A., Bordin, I., Fürstenau, H., Pinkal, M., Spaniol, M., Taneva, B., Thater, S., and Weikum, G. (2011). Robust Disambiguation of Named Entities. In Conference on Empirical Methods in Natural Language Processing, EMNLP.
8. Ling, X., Singh, S., and Weld, D. S. (2015). Design Challenges for Entity Linking. Transactions of the Association for Computational Linguistics, 3:315-28.
9. McNamee, P. (2009). Overview of the TAC 2009 knowledge base population track.
10. Minard, A.-L., Speranza, M., Urizar, R., na Altuna, B., van Erp, M., Schoen, A., and van Son, C. (2016). MEAN-TIME, the newsreader multilingual event and time corpus. In Proceedings of the 10th edition of the Language Resources and Evaluation Conference (LREC 2016).
11. Moro, A. and Navigli, R. (2015). SemEval-2015 Task 13: Multilingual All-Words Sense Disambiguation and Entity Linking. In 9th International Workshop on Semantic Evaluation, SemEval.
12. Page, L., Brin, S., Motwani, R., and Winograd, T. (1999). The PageRank Citation Ranking: Bringing Order to the Web.
13. Paulheim, H. and Bizer, C. (2014). Improving the Quality of Linked Data Using Statistical Distributions. International Journal on Semantic Web and Information Systems (IJSWIS), 10(2):63-86.
14. Ristoski, P., Bizer, C., and Paulheim, H. (2015). Mining the web of linked data with rapidminer. Journal of Web Semantics, 35(3):142-151.
15. Rizzo, G., Cano Amparo E, Pereira, B., and Varga, A. (2015). Making sense of Microposts (#Microposts2015) named entity recognition & linking challenge. In 5th International Workshop on Making Sense of Microposts, #Microposts.
16. Röder, M., Usbeck, R., Hellmann, S., Gerber, D., and Both, A. (2014). N3-a collection of datasets for named entity recognition and disambiguation in the nlp interchange format. In 9th Language Resources and Evaluation Conference, LREC.
17. Steinmetz, N., Knuth, M., and Sack, H. (2013). Statistical analyses of named entity disambiguation benchmarks. In Proceedings of NLP & DBpedia 2013 workshop in conjunction with 12th International Semantic Web Conference (ISWC2013), CEUR Workshop Proceedings.
18. Tjong Kim Sang, E. F. and De Meulder, F. (2003). Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. In Conference on Computational Natural Language Learning, CoNLL.
19. Tristram, F., Walter, S., Cimiano, P., and Unger, C. (2015). Weasel: a machine learning based approach to entity linking combining different features. In 3rd International Workshop on NLP&DBpedia.
20. Usbeck, R., Röder, M., Ngonga Ngomo, A.-C., Baron, C., Both, A., Brümmer, M., Ceccarelli, D., Cornolti, M., Cherix, D., Eickmann, B., Ferragina, P., Lemke, C., Moro, A., Navigli, R., Piccinno, F., Rizzo, G., Sack, H., Speck, R., Troncy, R., Waitelonis, J., and Wesemann, L. (2015). GERBIL: General Entity Annotator Benchmarking Framework. In 24th International Conference on World Wide Web, WWW.
21. Waitelonis, J., Exeler, C., and Sack, H. (2015). Linked Data Enabled Generalized Vector Space Model to Improve Document Retrieval. In Proceedings of NLP & DBpedia 2015 workshop in conjunction with 14th International Semantic Web Conference (ISWC2015), CEUR Workshop Proceedings.