VIRTUE: A visual tool for information retrieval performance evaluation and failure analysis

Journal of Visual Languages and Computing

Volumn 25, Issue 4, 2014, Pages 394-413

  Angelini, Marco ^a   Ferro, Nicola ^b   Santucci, Giuseppe ^a   Silvello, Gianmaria ^b  

^a SAPIENZA UNIVERSITY OF ROME   (Italy)

^b UNIVERSITY OF PADOVA   (Italy)

Author keywords

Experimental evaluation; Failure analysis; Information retrieval; Performance analysis; Visual analytics

Indexed keywords

EID: 84903121367     PISSN: 1045926X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jvlc.2013.12.003     Document Type: Article

References (71)

1. Agosti M., Berendsen R., Bogers T., Braschler M., Buitelaar P., Choukri K., Di Nunzio G.M., Ferro N., Forner P., Hanbury A., Friberg Heppin K., Hansen P., Järvelin A., Larsen B., Lupu M., Masiero I., Müller H., Peruzzo S., Petras V., Piroi F., de Rijke M., Santucci G., Silvello G., Toms E. PROMISE retreat report-prospects and opportunities for information access evaluation. SIGIR Forum 2012, 46(December (2)).
2. M. Agosti, E. Di Buccio, N. Ferro, I. Masiero, S. Peruzzo, G. Silvello, DIRECTions: design and specification of an IR evaluation infrastructure, in: T. Catarci, P. Forner, D. Hiemstra, A. Peñas, G. Santucci (Eds.), Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics. Proceedings of the Third International Conference of the CLEF Initiative (CLEF 2012), Lecture Notes in Computer Science (LNCS), vol. 7488, Springer, Heidelberg, Germany, 2012, pp. 88-99.
3. M. Agosti, G.M. Di Nunzio, M. Dussin, N. Ferro, 10Years of CLEF Data in DIRECT: where we are and where we can go, in: T. Sakay, M. Sanderson, W. Webber (Eds.), Proceedings of 3rd International Workshop on Evaluating Information Access (EVIA 2010), National Institute of Informatics, Tokyo, Japan, 2010, pp. 16-24.
4. Agosti M., Ferro N. Towards an evaluation infrastructure for DL performance evaluation. Evaluation of Digital Libraries. An Insight into Useful Applications and Methods 2009, 93-120. Chandos Publishing, Oxford, UK. G. Tsakonas, C. Papatheodorou (Eds.).
5. M. Angelini, N. Ferro, K. Järvelin, H. Keskustalo, A. Pirkola, G. Santucci, G. Silvello, Cumulated relative position: a metric for ranking evaluation, in: T. Catarci, P. Forner, D. Hiemstra, A. Peñas, G. Santucci (Eds.), Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics. Proceedings of the 3rd International Conference of the CLEF Initiative (CLEF 2012), Lecture Notes in Computer Science (LNCS), vol. 7488, Springer, Heidelberg, Germany, 2012, pp. 112-123.
6. M. Angelini, N. Ferro, G. Santucci, G. Silvello, Visual interactive failure analysis: supporting users in information retrieval evaluation, in: J. Kamps, W. Kraaij, N. Fuhr (Eds.), Proceedings of 4th Symposium on Information Interaction in Context (IIiX 2012), ACM Press, New York, USA, 2012, pp. 195-203.
7. Balog K., Fang Y., de Rijke M., Serdyukov P., Si L. Expertise Retrieval. Found. Trends Inf. Retr. 2012, 6(2-3):127-256.
8. Banks D., Over P., Zhang N.-F. Blind men and elephants. six approaches to TREC data. Inf. Retr. 1999, 1(May):7-34.
9. M. Behrisch, J. Davey, S. Simon, T. Schreck, D. Keim, J. Kohlhammer, Visual comparison of orderings and rankings, in: M. Pohl, H. Schumann (Eds.), Proceedings of 4th International Workshop on Visual Analytics (EuroVA 2013), Eurographics Association, Goslar, Germany, 2013.
10. C. Buckley, Why Current IR Engines Fail, in: M. Sanderson, K. Järvelin, J. Allan, P. Bruza (Eds.), Proceedings of 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2004), ACM Press, New York, USA, 2004, pp. 584-585.
11. C. Buckley, E.M. Voorhees, Evaluating Evaluation Measure Stability, in: E. Yannakoudakis, N.J. Belkin, M.-K. Leong, P. Ingwersen (Eds.), Proceedings of 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2000), ACM Press, New York, USA, 2000, pp. 33-40.
12. Buettcher S., Clarke C.L.A., Cormack G.V. Information Retrieval. Implementing and Evaluating Search Engines 2010, The MIT Press, Cambridge, MA, USA.
13. Burnett S., Clarke S., Davis M., Edwards R., Kellett A. Enterprise Search and Retrieval. Unlocking the Organisation's Potential 2006, Butler Direct Limited, Hull, UK.
14. L. Candela, D. Castelli, N. Ferro, Y. Ioannidis, G. Koutrika, C. Meghini, P. Pagano, S. Ross, D. Soergel, M. Agosti, M. Dobreva, V. Katifori, H. Schuldt, The DELOS Digital Library Reference Model, Foundations for Digital Libraries, ISTI-CNR at Gruppo ALI, Pisa, Italy, December 2007. http://www.delos.info/files/pdf/ReferenceModel/DELOS_DLReferenceModel_0.98.pdf.
15. B. Carterette, On Rank Correlation and the Distance Between Rankings, in: J. Allan, J.A. Aslam, M. Sanderson, C. Zhai, J. Zobel (Eds.), Proceedings of 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2009), ACM Press, New York, USA, 2009, pp. 436-443.
16. T. Catarci, P. Forner, D. Hiemstra, A. Peñas, G. Santucci (Eds.), Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics. Proceedings of the Third International Conference of the CLEF Initiative (CLEF 2012), Lecture Notes in Computer Science (LNCS), vol. 7488, Springer, Heidelberg, Germany, 2012.
17. Cleverdon C.W. The Cranfield tests on index languages devices. Readings in Information Retrieval 1997, 47-60. Morgan Kaufmann Publisher, Inc., San Francisco, CA, USA. K. Spärck Jones, P. Willett (Eds.).
18. Crestani F., Vegas J., de la Fuente P. A graphical user interface for the retrieval of hierarchically structured documents. Inf. Process. Manag. 2004, 40(2):269-289.
19. Croft W.B., Metzler D., Strohman T. Search Engines. Information Retrieval in Practice 2009, Addison-Wesley, Reading, MA, USA.
20. S. Cronen-Townsend, Y. Zhou, W.B. Croft, Predicting query performance, in: K. Järvelin, M. Beaulieu, R. Baeza-Yates, S. Hyon Myaeng (Eds.), Proceedings of 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2002), ACM Press, New York, USA, 2002, pp. 299-306.
21. M. Derthick, M.G. Christel, A.G. Hauptmann, H.D. Wactlar, Constant density displays using diversity sampling, in: Proceedings of the 9th Annual IEEE Conference on Information Visualization (INFOVIS'03), IEEE Computer Society, Washington, DC, USA, 2003, pp. 137-144. URL: http://dl.acm.org/citation.cfm?id=1947368.1947395.
22. E. Di Buccio, M. Dussin, N. Ferro, I. Masiero, G. Santucci, G. Tino, Interactive analysis and exploration of experimental evaluation results, in: M.L. Wilson, T. Russell-Rose, B. Larsen, J. Kalbach (Eds.), Proceedings of 1st European Workshop on Human-Computer Interaction and Information Retrieval (EuroHCIR 2011), 2011, pp. 11-14. URL: http://ceur-ws.org/Vol-763/.
23. E. Di Buccio, M. Dussin, N. Ferro, I. Masiero, G. Santucci, G. Tino, To re-rank or to re-query: can visual analytics solve this dilemma?, in: P. Forner, J. Gonzalo, J. Kekäläinen, M. Lalmas, M. de Rijke, (Eds.), Multilingual and Multimodal Information Access Evaluation. Proceedings of the Second International Conference of the Cross-Language Evaluation Forum (CLEF 2011), Lecture Notes in Computer Science (LNCS), vol. 6941, Springer, Heidelberg, Germany, 2011, pp. 119-130.
24. Fan W., Gordon M.D., Pathak P. A generic ranking function discovery framework by genetic programming for information retrieval. Inf. Process. Manag. 2004, 40(4):587-602.
25. Ferro N., Hanbury A., Müller H., Santucci G. Harnessing the scientific data produced by the experimental evaluation of search engines and information access systems. Proc. Comput. Sci. 2011, 4:740-749.
26. N. Ferro, A. Sabetta, G. Santucci, G. Tino, Visual comparison of ranked result cumulated gains, in: S. Miksch, G. Santucci (Eds.), Proceedings of 2nd International Workshop on Visual Analytics (EuroVA 2011), Eurographics Association, Goslar, Germany, 2011, pp. 21-24.
27. R.H. Fowler, W.A. Lawrence-Fowler, B.A. Wilson, Integrating query, thesaurus, and documents through a common visual representation, in: E.A. Fox (Ed.), Proceedings of 14th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1991), ACM Press, New York, USA, 1991, pp. 142-151.
28. Fox E.A., Gonçalves M.A., Shen R. Theoretical Foundations for Digital Libraries. The 5S (Societies, Scenarios, Spaces, Structures, Streams) Approach 2012, Morgan & Claypool Publishers, USA.
29. P. Hansen, C. Bartolini, G. Convertino, G. Santucci, M. Angelini, G. Granato, Collaborative Environment of the PROMISE Infrastructure: an "ELEGantt" Approach, in: M.L. Wilson, T. Russell-Rose, B. Larsen, J. Kalbach (Eds.), Proceedings of the 2nd European Workshop on Human-Computer Interaction and Information Retrieval. CEUR Workshop Proceedings. CEUR-WS.org, vol. 909, 2012, pp. 55-58.
30. Harman D., Buckley C. Overview of the reliable information access workshop. Inf. Retr. 2009, 12(6):615-641.
31. D.K. Harman, Some thoughts on failure analysis for noisy data, in: D. Lopresti, S. Roy, K. Schulz, L. Venkata Subramaniam (Eds.), Proceedings of 2nd Workshop on Analytics for Noisy unstructured text Data (AND 2008), ACM Press, New York, USA, 2008, p. 1.
32. Harman D.K. Information Retrieval Evaluation 2011, Morgan & Claypool Publishers, USA.
33. D.K. Harman, E.M. Voorhees (Eds.), TREC. Experiment and Evaluation in Information Retrieval, MIT Press, Cambridge, MA, USA, 2005.
34. Hearst M.A. "Natural" search user interfaces. Commun. ACM 2011, 54(11):60-67.
35. D.A. Hull, Using statistical testing in the evaluation of retrieval experiments, in: R. Korfhage, E. Rasmussen, P. Willett (Eds.), Proceedings of 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1993), ACM Press, New York, USA, 1993, pp. 329-338.
36. Järvelin K., Kekäläinen J. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. (TOIS) 2002, 20(October):422-446.
37. N. Kando, D. Ishikawa, M. Sugimoto (Eds.), Proceedings of 9th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Cross-Lingual Information Access, National Institute of Informatics, Tokyo, Japan, 2011.
38. E.M. Keen, Evaluation parameters, in: G. Salton, (Ed.), The SMART Retrieval System. Experiments in Automatic Document Processing, Prentice-Hall, Inc., Englewood Cliff, NJ, USA, 1971, pp. 74-111.
39. Kekäläinen J. Binary and graded relevance in IR evaluations-comparison of the effects on ranking of IR systems. Inf. Process. Manag. 2005, 41(5):1019-1033.
40. Kekäläinen J., Järvelin K. Using graded relevance assessments in IR evaluation. J. Am. Soc. Inf. Sci. Technol. 2002, 53(November (13)):1120-1129.
41. Kendall M. Rank Correlation Methods 1948, Griffin, Oxford, UK.
42. H. Keskustalo, K. Järvelin, A. Pirkola, J. Kekäläinen, Intuition-supporting visualization of user's performance based on explicit negative higher-order relevance, in: T.-S. Chua, M.-K. Leong, D.W. Oard, F. Sebastiani (Eds.), Proceedings of 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2008), ACM Press, New York, USA, 2008, pp. 675-681.
43. Koshman S. Testing user interaction with a prototype visualization-based information retrieval system. J. Am. Soc. Inf. Sci. Technol. 2005, 56(8):824-833. URL: http://dx.doi.org/10.1002/asi.20175.
44. Lupu M., Hanbury A. Patent retrieval. Found. Trends Inf. Retr. (FnTIR) 2013, 7(1):1-97.
45. Manning C.D., Raghavan P., Schütze H. Introduction to Information Retrieval 2008, Cambridge University Press.
46. McGill R., Tukey J.W., Larsen W.A. Variations of box plots. Am. Statist. 1978, 32(February (1)):12-16. (American Statistical Association, 732 North Washington Street, Alexandria, VA 22314-1943).
47. Mizzaro S. Relevance. the whole history. J. Am. Soc. Inf. Sci. Technol. (JASIST) 1997, 48(September (9)):810-832.
48. Morse E.L., Lewis M., Olsen K.A. Testing visual information retrieval methodologies case study. comparative analysis of textual, icon, graphical, and spring displays. J. Am. Soc. Inf. Sci. Technol. (JASIST) 2002, 53(1):28-40.
49. Robertson S.E. The methodology of information retrieval experiment. Information Retrieval Experiment 1981, 9-31. Butterworths, London, UK. K. Spärck Jones (Ed.).
50. Robertson S.E. On the history of evaluation in IR. J. Inf. Sci. 2008, 34(4):439-456.
51. J.J. Rocchio, Relevance feedback in information retrieval, in: G. Salton, (Ed.), The SMART Retrieval System. Experiments in Automatic Document Processing. Prentice-Hall, Inc., Englewood Cliff, NJ, USA, 1971, pp. 313-323.
52. B.R. Rowe, D.W. Wood, A.L. Link, D.A. Simoni, Economic Impact Assessment of NIST's Text REtrieval Conference (TREC) Program, RTI Project Number 0211875, RTI International, USA. URL, July 2010. http://trec.nist.gov/pubs/2010.economic.impact.pdf.
53. T. Sakai, Evaluating evaluation metrics based on the bootstrap, in: E.N. Efthimiadis, S. Dumais, D. Hawking, K. Järvelin (Eds.), Proceedings of 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006), ACM Press, New York, USA, 2006, pp. 525-532.
54. Sakai T. On the reliability of information retrieval metrics based on graded relevance. Inf. Process. Manag. 2007, 43(2):531-548.
55. M. Sanderson, J. Zobel, Information retrieval system evaluation: effort, sensitivity, and reliability, in: R. Baeza-Yates, N. Ziviani, G. Marchionini, A. Moffat, J. Tait (Eds.), Proceedings of 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2005), ACM Press, New York, USA, 2005, pp. 162-169.
56. Savoy J. Statistical inference in retrieval effectiveness evaluation. Inf. Process. Manag. 1997, 33(44):495-512.
57. J. Savoy, Why do successful search systems fail for some topics, in: Y. Cho, Y. Wan Koo, R.L. Wainwright, H.M. Haddad, S.Y. Shin (Eds.), Proceedings of 2007 ACM Symposium on Applied Computing (SAC 2007), ACM Press, New York, USA, 2007, pp. 872-877.
58. Seo J., Shneiderman B. A rank-by-feature framework for interactive exploration of multidimensional data. Inf. Vis. 2005, 4(July (2)):96-113. URL: http://dx.doi.org/10.1057/palgrave.ivs.9500091.
59. E. Sormunen, Liberal relevance criteria of TREC-counting on negligible documents?, in: K. Järvelin, M. Beaulieu, R. Baeza-Yates, S. Hyon Myaeng (Eds.), Proceedings of 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2002), ACM Press, New York, USA, 2002, pp. 324-330.
60. E. Sormunen, Liberal relevance criteria of TREC: counting on negligible documents?, in: Proceedings of the 25th International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM Press, 2002, pp. 324-330.
61. E. Sormunen, S. Hokkanen, P. Kangaslampi, P. Pyy, B. Sepponen, Query performance analyser-a web-based tool for IR research and instruction, in: K. Järvelin, M. Beaulieu, R. Baeza-Yates, S. Hyon Myaeng (Eds.), Proceedings of 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2002), ACM Press, New York, USA, 2002, p. 450.
62. Teevan J., Dumais S.T., Horvitz E. Potential for personalization. ACM Trans. Comput. Hum. Interact. 2010, 17(1):1-31.
63. Tukey J.W. Exploratory data analysis 1970, Addison-Wesley, USA. preliminary edition.
64. Tukey J.W. Exploratory Data Analysis 1977, Addison-Wesley, USA.
65. E. Voorhees, Evaluation by highly relevant documents, in: D.H. Kraft, W.B. Croft, D.J. Harper, J. Zobel (Eds.), Proceedings of 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2001), ACM Press, New York, USA, 2001, pp. 74-82.
66. E.M. Voorhees, D.K. Harman (Eds.), The 7th Text REtrieval Conference (TREC-7), Gaithersburg, Maryland, 9-11 November, 1998, National Institute of Standards and Technology (NIST), 1998 (Special Publication 500-242).
67. K. Vredenburg, J.-Y. Mao, P.W. Smith, T. Carey, A survey of user-centered design practice, In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'02), ACM, New York, NY, USA, 2002, pp. 471-478. URL: http://doi.acm.org/10.1145/503376.503460.
68. Witten I.H., Bainbridge D., Nichols D.M. How to Build a Digital Library 2009, Morgan Kaufmann Publishers, San Francisco, CA, USA. 2nd ed.
69. Zhang J. TOFIR. a tool of facilitating information retrieval-introduce a visual retrieval model. Inf. Process. Manag. 2001, 37(4):639-657.
70. Zhang J. Visualization for Information Retrieval 2008, Springer-Verlag, Heidelberg, Germany.
71. Angelini Marco, Ferro Nicola, Santucci Giuseppe, Silvello Gianmaria Improving ranking evaluation employing visual analytics. Proceedings of Information Access Evaluation. Multilinguality, Multimodality, and Visualization - Forth International Conference of the Cross-Language Evaluation Forum (CLEF 2013), Lecture Notes in Computer Science (LNCS) 2013, vol. 8138:29-40. Springer, Heidelberg, Germany.