Repeatable and reliable semantic search evaluation

Journal of Web Semantics

Volumn 21, Issue , 2013, Pages 14-29

  Blanco, Roi ^b   Halpin, Harry ^c   Herzig, Daniel M ^a   Mika, Peter ^b   Pound, Jeffrey ^d   Thompson, Henry S ^c   Tran, Thanh ^a  

^a KARLSRUHE INSTITUTE OF TECHNOLOGY   (Germany)

^b YAHOO RESEARCH   (Spain)

^c UNIVERSITY OF EDINBURGH   (United Kingdom)

^d UNIVERSITY OF WATERLOO   (Canada)

Author keywords

RDF; Semantic search; Semantic search evaluation; Structured data; Web data; Web search

Indexed keywords

RDF; SEMANTIC SEARCH; STRUCTURED DATA; WEB DATA; WEB SEARCHES;

INFORMATION RETRIEVAL; SEMANTICS;

SEMANTIC WEB;

EID: 84884285189     PISSN: 15708268     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.websem.2013.05.005     Document Type: Article

References (52)

1. J. Chu-Carroll, J.M. Prager, K. Czuba, D.A. Ferrucci, P.A. Duboué, Semantic search via XML fragments: a high-precision approach to IR, in: SIGIR, 2006, pp. 445-452.
2. J. Chu-Carroll, J.M. Prager, An experimental study of the impact of information extraction accuracy on semantic search performance, in: CIKM, 2007, pp. 505-514.
3. P. Castells, M. Fernández, and D. Vallet An adaptation of the vector-space model for ontology-based information retrieval IEEE Trans. Knowl. Data Eng. 19 2 2007 261 272
4. T. Tran, S. Bloehdorn, P. Cimiano, P. Haase, Expressive resource descriptions for ontology-based information retrieval, in: ICTIR, 2007, pp. 55-68.
5. R. Guha, R. McCool, and E. Miller Semantic search WWW 2003 ACM 700 709 http://doi.acm.org/10.1145/775152.775250
6. T. Tran, H. Wang, and P. Haase Hermes: data web search on a pay-as-you-go integration infrastructure J. Web Sem. 7 3 2009 189 203
7. E. Oren, R. Delbru, M. Catasta, R. Cyganiak, H. Stenzhorn, and G. Tummarello Sindice.com: a document-oriented lookup index for open linked data International Journal of Metadata, Semantics and Ontologies (IJMSO) 3 1 2008 37 52
8. E.M. Voorhees, Query expansion using lexical-semantic relations, in: SIGIR, 1994, pp. 61-69.
9. T. Tran, D.M. Herzig, and G. Ladwig SemSearchPro - using semantics throughout the search process J. Web Sem. 9 4 2011 349 364
10. J. Pound, P. Mika, H. Zaragoza, Ad-hoc object ranking in the web of data, in: Proceedings of the WWW, Raleigh, United States of America, 2010, pp. 771-780.
11. R. Blanco, H. Halpin, D.M. Herzig, P. Mika, J. Pound, H.S. Thompson, D.T. Tran, Repeatable and reliable search system evaluation using crowdsourcing, in: SIGIR, 2011, pp. 923-932.
12. H. Halpin, D.M. Herzig, P. Mika, R. Blanco, J. Pound, H.S. Thompson, D.T. Tran, Evaluating ad-hoc object retrieval, in: Int. Workshop on Evaluation of Semantic Technologies, IWEST 2010, at ISWC, 2010. URL: http://people.csail.mit. edu/pcm/tempISWC/workshops/IWEST2010/paper9.pdf.
13. R. Blanco, H. Halpin, D.M. Herzig, P. Mika, J. Pound, H.S. Thompson, D.T. Tran, Entity search evaluation over structured web data, in: Proc. of the 1st Int. Workshop on Entity-Oriented Search, EOS 2011, at SIGIR, 2011. URL: http://research.microsoft.com/en-us/um/beijing/events/eos2011/20.pdf.
14. K. Balog, A.P. de Vries, P. Serdyukov, P. Thomas, T. Westerveld, Overview of the TREC 2009 entity track, in: NIST Special Publication: SP 500-278, 2009.
15. O. Alonso, D.E. Rose, and B. Stewart Crowdsourcing for relevance evaluation SIGIR Forum 42 2 2008 9 15 http://doi.acm.org/10.1145/1480506.1480508
16. O. Alonso, R. Schenkel, M. Theobald, Crowdsourcing assessments for XML ranked retrieval, in: ECIR, 2010, pp. 602-606.
17. C. Callison-Burch Fast, cheap, and creative: evaluating translation quality using Amazon's mechanical turk Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing 2009 Association for Computational Linguistics Singapore 286 295 URL: http://www.aclweb.org/ anthology/D/D09/D09-1030
18. S. Nowak, and S.M. Rüger How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation Multimedia Information Retrieval 2010 557 566
19. J. Waitelonis, N. Ludwig, M. Knuth, and H. Sack Whoknows? Evaluating linked data heuristics with a quiz that cleans up DBPedia International Journal of Interactive Technology and Smart Education (ITSE) 8 4 2011 236 248
20. L. Von Ahn, B. Maurer, C. McMillen, D. Abraham, and M. Blum reCAPTCHA: human-based character recognition via web security measures Science 321 5895 2008 1465
21. D.M. Herzig, T. Tran, Heterogeneous web data search using relevance-based on the fly data integration, in: WWW, 2012, pp. 141-150.
22. J. Kamps, S. Geva, A. Trotman, A. Woodley, M. Koolen, Overview of the INEX 2008 ad hoc track, in: Advances in Focused Retrieval: 7th International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2008, 2009, pp. 1-28.
23. Y. Luo, W. Wang, X. Lin, SPARK: a keyword search engine on relational databases, in: ICDE, 2008, pp. 1552-1555.
24. K. Balog, P. Serdyukov, A. de Vries, Overview of the TREC 2010 entity track, in: TREC 2010 Working Notes, 2010.
25. G. Demartini, T. Iofciu, and A.P. De Vries Overview of the INEX 2009 entity ranking track Proceedings of the Focused Retrieval and Evaluation, and 8th International Conference on Initiative for the Evaluation of XML Retrieval INEX'09 2010 Springer-Verlag Berlin, Heidelberg 254 264
26. C. Unger, P. Cimiano, V. Lopez, E. Motta, QALD-1 open challenge, 2011. http://www.sc.cit-ec.uni-bielefeld.de/sites/www.sc.cit-ec.uni-bielefeld.de/ files/sharedtask.pdf.
27. Proceedings of the Second International Workshop on Keyword Search on Structured Data, KEYS 2010 2010 ACM Indianapolis, Indiana, USA
28. J. Coffman, and A.C. Weaver A framework for evaluating database keyword search strategies Proceedings of the 19th ACM International Conference on Information and Knowledge Management CIKM'10 2010 ACM New York, NY, USA 729 738
29. C. Cleverdon, M. Kean, Factors determining the performance of indexing systems, 1966. URL: http://dspace.lib.cranfield.ac.uk/handle/1826/863.
30. C.W. Cleverdon, The significance of the Cranfield tests on index languages, in: SIGIR, 1991, pp. 3-12.
31. C. Bizer, and P. Mika The semantic web challenge, 2009 J. Web Sem. 8 4 2010 341
32. H. Halpin, A query-driven characterization of linked data, in: Proceedings of the WWW Workshop on Linked Data on the Web, Madrid, Spain, 2009.
33. E. Voorhees The philosophy of information retrieval evaluation Proceedings of the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems 2001 Springer-Verlag 355 370
34. S.P. Harter Variations in relevance assessments and the measurement of retrieval effectiveness J. Am. Soc. Inf. Sci. Technol. 47 1 1996 37 49 10.1002/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.3.CO;2-I
35. E.M. Voorhees, and D.K. Harman TREC: Experiment and Evaluation in Information Retrieval 2005 Digital Libraries and Electronic Publishing, MIT Press URL: http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid= 10667&mode=toc
36. P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A.P. de Vries, and E. Yilmaz Relevance assessment: are judges exchangeable and does it matter SIGIR 2008 ACM New York, NY, USA 667 674 http://doi.acm.org/10.1145/1390334.1390447
37. H. Halpin, R. Blanco, Machine-learning for spammer detection in crowd-sourcing, in: Workshop on Human Computation at AAAI, Technical Report WS-12-08, 2012, pp. 85-86. URL: https://www.aaai.org/ocs/index.php/WS/AAAIW12/ paper/viewPDFInterstitial/5321/5606.
38. W. Mason, and D.J. Watts Financial incentives and the "performance of crowds" Proc. of the ACM SIGKDD Workshop on Human Computation HCOMP'09 2009 ACM New York, NY, USA 77 85 URL: http://doi.acm.org/10.1145/1600150.1600175
39. B. Carpenter Multilevel Bayesian Models of Categorical Data Annotation, Technical Report, Tech. Rep., Alias-I 2008 http://lingpipe.files.wordpress.com/ 2008/11/carp-Bayesian-multilevel-annotation.pdf
40. S. Elbassuoni, M. Ramanath, R. Schenkel, M. Sydow, and G. Weikum Language-model-based ranking for queries on RDF-graphs CIKM 2009 ACM 977 986 http://doi.acm.org/10.1145/1645953.1646078
41. I. Soboroff, and D. Harman Novelty detection: the trec experience HLT'05 2005 ACL USA 10.3115/1220575.1220589
42. C.D. Manning, P. Raghavan, and H. Schütze Introduction to Information Retrieval 2008 Cambridge University Press
43. G. Salton, A. Wong, and C.S. Yang A vector space model for automatic indexing Commun. ACM 18 1975 613 620 URL: http://doi.acm.org/10.1145/361219. 361220
44. S. Robertson, S. Walker, Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval, in: Proceedings of the SIGIR Conference on Research and Development in Information Retrieval (1).
45. C. Zhai, and J. Lafferty A study of smoothing methods for language models applied to information retrieval ACM Trans. Inf. Syst. 22 2 2004 179 214 10.1145/984321.984322
46. S. Robertson, and H. Zaragoza The probabilistic relevance framework: BM25 and beyond Foundations and Trends in Information Retrieval 3 4 2010 333 389 10.1561/1500000019
47. R. Blanco, P. Mika, S. Vigna, Effective and efficient entity search in RDF data, in: International Semantic Web Conference (1), 2011, pp. 83-97.
48. F. McSherry, and M. Najork Computing information retrieval performance measures efficiently in the presence of tied scores Proceedings of the 30th ECIR 2008 Springer-Verlag Berlin, Heidelberg
49. H. Fang, C. Zhai, An exploration of axiomatic approaches to information retrieval, in: SIGIR, 2005, pp. 480-487.
50. C. Zhai, J.D. Lafferty, Model-based feedback in the language modeling approach to information retrieval, in: CIKM, 2001, pp. 403-410.
51. D. Milne, I.H. Witten, An open-source toolkit for mining wikipedia, in: Proc. New Zealand Computer Science Research Student Conf., Vol. 9, 2009.
52. I. Ounis, G. Amati, V. Plachouras, B. He, C. Macdonald, C. Lioma, Terrier: a high performance and scalable information retrieval platform, in: Proceedings of ACM SIGIR'06 Workshop on Open Source Information Retrieval, OSIR 2006, 2006.