Harnessing the scientific data produced by the experimental evaluation of search engines and information access systems

Procedia Computer Science

Volumn 4, Issue , 2011, Pages 740-749

  Ferro, Nicola ^a   Hanbury, Allan ^b   Müller, Henning ^c   Santucci, Giuseppe ^d  

^a UNIVERSITY OF PADOVA   (Italy)

^b Information Retrieval Facility   (Austria)

^c University of Applied Sciences   (Switzerland)

^d SAPIENZA UNIVERSITY OF ROME   (Italy)

Author keywords

DIRECT; Experimental evaluation; Information retrieval; Large scale evaluation campaigns; Multilingual information access; Scientific data

Indexed keywords

INFORMATION RETRIEVAL;

DIRECT; EXPERIMENTAL EVALUATION; INFORMATION ACCESS; LARGE-SCALE EVALUATION CAMPAIGNS; SCIENTIFIC DATA;

SEARCH ENGINES;

EID: 79958274190     PISSN: None     EISSN: 18770509     Source Type: Conference Proceeding    
DOI: 10.1016/j.procs.2011.04.078     Document Type: Conference Paper

References (32)

1. M. Dussin, N. Ferro, Managing the Knowledge Creation Process of Large-Scale Evaluation Campaigns, in: M. Agosti, J. Borbinha, S. Kapidakis, C. Papatheodorou, G. Tsakonas (Eds.), Proc. 13th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2009), Lecture Notes in Computer Science (LNCS) 5714, Springer, Heidelberg, Germany, 2009, pp. 63-74.
2. N. Ferro, D. Harman, CLEF 2009: Grid@CLEF Pilot Track Overview, in: C. Peters, G. M. Di Nunzio, M. Kurimo, T. Mandl, D. Mostefa, A. Pẽnas, G. Roda (Eds.), Multilingual Information Access Evaluation Vol. I Text Retrieval Experiments - Tenth Workshop of the Cross- Language Evaluation Forum (CLEF 2009). Revised Selected Papers, Lecture Notes in Computer Science (LNCS) 6241, Springer, Heidelberg, Germany, 2010, pp. 552-565.
3. A. Hanbury, H. Müller, Automated Component-Level Evaluation: Present and Future, in: Agosti et al. [32], pp. 124-135.
4. C. W. Cleverdon, The Cranfield Tests on Index Languages Devices, in: K. Sp̈arck Jones, P. Willett (Eds.), Readings in Information Retrieval, Morgan Kaufmann Publisher, Inc., San Francisco, CA, USA, 1997, pp. 47-60.
5. B. R. Rowe, D. W. Wood, A. L. Link, D. A. Simoni, Economic Impact Assessment of NIST's Text REtrieval Conference (TREC) Program, RTI Project Number 0211875, RTI International, USA. http://trec.nist.gov/pubs/2010.economic. impact.pdf, 2010.
6. Commission of the European Communities, Communication from the Commission to the European Parliament, the Council and the European Economic and Social Committee on scientific information in the digital age: access, dissemination and preservation, COMM(2008) 56 Final.
7. National Science Board, Long-Lived Digital Data Collections: Enabling Research and Education in the 21st Century (NSB-05-40), National Science Foundation (NSF). http://www.nsf.gov/pubs/2005/nsb0540/2005.
8. Working Group on Data for Science, FROM DATA TO WISDOM: Pathways to Successful Data Management for Australian Science, Report to Ministerís Science, Engineering and Innovation Council (PMSEIC), http://www.innovation.gov. au/Science/PMSEIC/Documents/FromDatatoWisdom.pdf, 2006.
9. Economics and Statistics Division, WIPO, World intellectual property indicators 2010, Tech. Rep. 941, WIPO (September 2010).
10. D. Hunt, L. B. Nguyen, M. Rodgers (Eds.), Patent searching: tools & techniques, John Wiley & Sons, 2007.
11. J. List, How drawings could enhance retrieval in mechanical and device patent searching, World Patent Information 29 (3) (2007) 210-218.
12. M. Agosti, G. M. Di Nunzio, N. Ferro, D. Harman, C. Peters, The Future of Large-scale Evaluation Campaigns for Information Retrieval in Europe, in: N. Fuhr, L. Kov́acs, C. Meghini (Eds.), Proc. 11th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2007), Lecture Notes in Computer Science (LNCS) 4675, Springer, Heidelberg, Germany, 2007, pp. 509-512.
13. N. Ferro, CLEF, CLEF 2010, and PROMISEs: Perspectives for the Cross-Language Evaluation Forum, in: N. Kando, K. Kishida (Eds.), Proc. 8th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Cross-Lingual Information Access, National Institute of Informatics, Tokyo, Japan, 2010, pp. 2-12.
14. M. Agosti, G. M. Di Nunzio, N. Ferro, A Proposal to Extend and Enrich the Scientific Data Curation of Evaluation Campaigns, in: T. Sakay, M. Sanderson, D. K. Evans (Eds.), Proc. 1st International Workshop on Evaluating Information Access (EVIA 2007), National Institute of Informatics, Tokyo, Japan, 2007, pp. 62-73.
15. M. Agosti, N. Ferro, Towards an Evaluation Infrastructure for DL Performance Evaluation, in: G. Tsakonas, C. Papatheodorou (Eds.), Evaluation of Digital Libraries: An insight into useful applications and methods, Chandos Publishing, Oxford, UK, 2009, pp. 93-120.
16. G. M. Di Nunzio, N. Ferro, DIRECT: a System for Evaluating Information Access Components of Digital Libraries, in: A. Rauber, S. Christodoulakis, A. Min Tjoa (Eds.), Proc. 9th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2005), Lecture Notes in Computer Science (LNCS) 3652, Springer, Heidelberg, Germany, 2005, pp. 483-484.
17. M. Dussin, N. Ferro, Design of a Digital Library System for Large-Scale Evaluation Campaigns, in: B. Christensen-Dalsgaard, D. Castelli, J. K. Lippincott, B. Ammitzbøll Jurik (Eds.), Proc. 12th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2008), Lecture Notes in Computer Science (LNCS) 5173, Springer, Heidelberg, Germany, 2008, pp. 400-401.
18. M. Dussin, N. Ferro, The Role of the DIKW Hierarchy in the Design of a Digital Library System for the Scientific Data of Large-Scale Evaluation Campaigns, in: R. Larsen, A. Paepcke, J. L. Borbinha, M. Naaman (Eds.), Proc. 8th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2008), ACM Press, New York, USA, 2008, p. 450.
19. M. Agosti, G. M. Di Nunzio, M. Dussin, N. Ferro, 10 Years of CLEF Data in DIRECT: Where We Are and Where We Can Go, in: T. Sakay, M. Sanderson, W. Webber (Eds.), Proc. 3rd International Workshop on Evaluating Information Access (EVIA 2010), National Institute of Informatics, Tokyo, Japan, 2010, pp. 16-24.
20. M. Agosti, N. Ferro, A Formal Model of Annotations of Digital Content, ACM Transactions on Information Systems (TOIS) 26 (1) (2008) 3:1-3:57.
21. D. A. Keim, J. Kohlhammer, G. Santucci, F. Mansmann, F. Wanner, M. Schaefer, Visual Analytics Challenges, in: IEEE eChallenges 2009, 2009, pp. 21-23.
22. M. Braschler, K. Choukri, N. Ferro, A. Hanbury, J. Karlgren, H. Müller, V. Petras, E. Pianta, M. de Rijke, G. Santucci, A PROMISE for Experimental Evaluation, in: Agosti et al. [32], pp. 140-144.
23. N. Ferro, PROMISE: Advancing the Evaluation of Multilingual and Multimedia Information Systems, ERCIM News 84 (2011) 49.
24. N. Paskin (Ed.), The DOI Handbook - Edition 4.4.1, International DOI Foundation (IDF). http://dx.doi.org/10.1000/186, 2006.
25. N. Ferro, G. Sabetta, G. Santucci, G. Tino, F. Veltri, Visual Comparison of Ranked Result Cumulated Gains, in: S. Miksch, G. Santucci (Eds.), EuroVa - International Workshop on Visual Analytics, The Eurographics Association, Switzerland, 2011.
26. R. T. Fielding, R. N. Taylor, Principled Design of the Modern Web Architecture, ACM Transactions on Internet Technology (TOIT) 2 (2) (2002) 115-150.
27. R. Fielding, Y. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, T. Berners-Lee, Hypertext Transfer Protocol - HTTP/1.1, RFC 2616 (June 1999).
28. T. Berners-Lee, R. Fielding, L. Masinter, Uniform Resource Identifier (URI): Generic Syntax, RFC 3986 (January 2005).
29. W3C, Extensible Markup Language (XML) 1.0 (Fifth Edition) - W3C Recommendation 26 November 2008, http://www.w3.org/TR/xml/(November 2008).
30. W3C, XMLHttpRequest - W3C Candidate Recommendation 3 August 2010, http://www.w3.org/TR/XMLHttpRequest/(August 2010).
31. N. Freed, N. Borenstein, Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies, RFC 2045 (November 1996).
32. M. Agosti, N. Ferro, C. Peters, M. de Rijke, A. Smeaton (Eds.), Multilingual and Multimodal Information Access Evaluation. Proceedings of the International Conference of the Cross-Language Evaluation Forum (CLEF 2010), Lecture Notes in Computer Science (LNCS) 6360, Springer, Heidelberg, Germany, 2010.