Evaluation of contextual information retrieval effectiveness: Overview of issues and research

Knowledge and Information Systems

Volumn 24, Issue 1, 2010, Pages 1-34

  Tamine Lechani, Lynda ^a   Boughanem, Mohand ^a   Daoud, Mariam ^a  

^a UNIVERSITÉ DE TOULOUSE   (France)

Author keywords

Context; Empirical evaluation; Profile; Relevance

Indexed keywords

INFORMATION RETRIEVAL; INFORMATION RETRIEVAL SYSTEMS;

CONTEXT; CONTEXTUAL INFORMATION; EMPIRICAL EVALUATIONS; EVALUATION METHODOLOGIES; PROFILE; RELEVANCE; RETRIEVAL EFFECTIVENESS; RETRIEVAL EVALUATION;

SEARCH ENGINES;

EID: 77953915372     PISSN: 02191377     EISSN: 02193116     Source Type: Journal    
DOI: 10.1007/s10115-009-0231-1     Document Type: Article

References (124)

1. Agichtein E, Brill E, Dumais ST (2006) Improving web search ranking by incorporating user behavior information. In: Proceedings of the 29th international SIGIR conference on research and development in information retrieval, pp 19-26.
2. Allan J (2002) Challenges in information retrieval and langage modelling. In: Report of a workshop held at the Center for Intelligent Information Retrieval, University of Massachusetts, Amherst.
3. Allan J (2003) Hard track overview in trec 2003 high accuracy retrieval from documents. In: Proceedings of the 12th text retrieval conference (TREC-12), National Institute of Standards and Technology, NIST special publication, pp 24-37.
4. Anand SS, Mobasher B (2007) Introduction to intelligent techniques for web personalization. ACM Trans Internet Technol 7(4): 18.
5. Anderson C, Domingos P, Weld D (2001) Personalizing web sites for mobile users. In: Anderson CR, Domingos P, Weld DS (eds) Personalizing web sites for mobile users. Proceedings of the 10th international WWW conference, pp 565-575.
6. Baeza-Yates R, Ribeiro-Neto B (1999) Modern information retrieval. ACM Press, Addison-Wesley.
7. Beard K, Sharma V (1997) Multidimensional ranking for data in digital spatial libraries. Int J Digit Libr 1(2): 153-160.
8. Belew RK, Hatton J (1996) Rave reviews: acquiring relevance assessments from multiple users. Working notes of the AAAI Spring symposium on Machine learning in information access, Standford, CA, USA.
9. Bierig R, Göker A (2006) Time, location and interest: an empirical and user-centred study. In: Proceedings of the 1st international conference on information interaction in context (IIiX). ACM, New York, pp 79-87.
10. Bilal D (2000) Children's use of the yahooligans! web search engine: cognitive, physical, and affective behaviors on fact-based search tasks. J Am Soc Inf Sci 51(7): 646-665.
11. Borlund P (2003a) The concept of relevance in IR. J Am Soc Inf Sci Technol 54(10): 913-925.
12. Borlund P (2003b) The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. J Inf Res 8(3): 152.
13. Borlund P, Ingwersen P (1998) Measures of relative relevance and ranked half-life: performance indicators in interactive IR. In: Croft WB et al (eds) Proceedings of the 21st ACM SIGIR international conference on research and development, pp 324-331.
14. Bouvin NO, Christensen B, Gronbaeck K, Hansen FA (2003) Hycon: a framework for context-aware mobile hypermedia. Hypermedia 9(1): 59-88.
15. Buckley C, Voorhees EM (2004) Retrieval evaluation with incomplete information. In: Proceedings of the 27th annual international ACM SIGIR conference on research and development in information retrieval. ACM, New York, pp 25-32.
16. Budzik J, Hammond K (2000) User interactions with every day applications as context for just-in-time information access. In: Proceedings of the 5th international conference on intelligent user interfaces, pp 44-51.
17. Bylund M, Espinoza F (2002) Testing and demonstrating context-aware services with quake iii arena. Commun ACM 45(1): 46-48.
18. Bystrom K, Jarvelin K (1995) Task complexity affects information seeking and use. Inf Process Manag 31(2): 191-213.
19. Chandrasekaran P, Joshi A (2002) Mobileiq: a framework for mobile information access. In: Proceedings of the 3rd international conference on mobile data management. IEEE Computer Society, Washington, DC, USA, p 43.
20. Cheverst K, Davies N, Mitchell K, Friday A, Efstratiou C (2000) Developing a context-aware electronic tourist guide: some issues and experiences. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, pp 17-24.
21. Chin DN (2001) Empirical evaluation of user models and user-adapted systems. User Model User-Adapted Interact 11(1-2): 181-194.
22. Chittaro, L. (eds) (2003) Human-computer interaction with mobile devices and services. Lecture Notes in Computer Science, vol 2795. Springer, Berlin.
23. Clarcke C, Craswell N, Soboroff I (2004) Overview of the TREC 2004 terabyte track. In: The text retrieval conference. NIST.
24. Cleverdon C (1967) The cranfield test on index language devices. In: Aslib, pp 173-194.
25. Crestani F, Ruthven I (2007) Introduction to special issue on contextual information retrieval systems. Inf Retr 10: 829-847.
26. Daoud M, Tamine-Lechani L, Boughanem M (2008) Learning user interests for a session-based personalized search. In: Proceedings of the 2nd international IIiX symposium (IIiX'08), London, UK, pp 57-64.
27. Davies N, Mitchell K, Cheverest K, Blair G (1998) Developing a context sensitive tourist guide. In: First workshop on human computer interaction with mobile devices, GIST Technical Report G98-1.
28. Dervin B, Nilan M (1986) Information needs and uses. In: William ME (ed) ARIST, pp 3-33.
29. Dey AK (2001) Understanding and using context. Pers Ubiquitous Comput 5(1): 4-7.
30. Diaz A, Garcia A, Gervas P (2008) User-centred versus system-centred evaluation of a personalization system. Inf Process Manag 44(3): 1293-1307.
31. Ding C, Patra J (2007) User modeling for personalized web search with self-organizing map. J Am Soc Inf Sci Technol 58(4): 494-507.
32. Dumais S, Cuttrell E, Cadiz J, Jancke G, Sarin R, Robbins D (2003) Stuff i've seen: a system for a personal information retrieval and re-use. In: Proceedings of the 26th ACM SIGIR', Toronto, pp 72-79.
33. Ellis D (1996) The dilemma of measurement in information retrieval research. J Am Soc Inf Sci 47(1): 23-36.
34. Frias-Martinez E, Chen SY, Macredie RD, Liu X (2007) The role of human factors in stereotyping behavior and perception of digital library users: a robust clustering approach. User Model User-Adapted Interact 17(3): 1391-1573.
35. Göker A, Myrhaug H (2008) Evaluation of a mobile information system in context. Inf Process Manag 44(1): 39-65.
36. Göker A, Myrhaug HI (2002) User context and personalisation. In: ECCBR workshop on case based reasoning and personalisation, Aberdeen.
37. Göker A, Watt S, Myrhaug H, Whitehead N, Yakici M, Bierig R, Kanth S, Cumming H (2004) An ambient, personalised, and context-sensitive information system for mobile users. In: Proceedings of the 2nd European union symposium on ambient intelligence (EUSAI), ACM, New York, NY, USA, pp 19-24.
38. Gowan JM (2003) A multiple model approach to personalised information access. Master Thesis in computer science, Faculty of science, University College Dublin.
39. Gwizdka J, Chignell M (1999) Towards information retrieval measures for evaluation of web search engines. Unpublished manuscript (1999). Available at http://www. imedia. mie. utoronto. ca/jacekg/pubs. html.
40. Harman D (1995) Overview of the 4th text retrieval conference (TREC-4). In: Proceedings of the 4th text retrieval conference (TREC-4). National Institute of Standards and Technology, NIST special publication, pp 1-24.
41. Harter S, Hert C (1997) Evaluation of information retrieval systems: approaches, issues, and methods. Ann Rev Inf Sci Technol 32(1): 3-94.
42. Hattori S, Tezuka T, Tanaka K (2007) Context-aware query refinement for mobile web search. In: Proceedings of the 2007 international symposium on applications and the internet workshops, IEEE Computer Society, Washington, DC, USA, p 15.
43. Haveliwala T (2002) Topic-sensitive page rank. In: International ACM world wide web conference, pp 727-736.
44. Held A, Buchholz S, Schill A (2002) (n. d.) Modeling of context information for pervasive computing applications. In: Proceedings of the 6th world multiconference on systemics, cybernetics and informatics (SCI2002).
45. Herlocker JL, Konstan JA, Terveen LG, Riedl JT (2004) Evaluating collaborative filtering recommender systems. ACM Trans Inf Syst 22(1): 5-53.
46. Hersh WR, Elliot DL, Hickam DH, Wolf SL, Molnar A (1995) Towards new measures of information retrieval evaluation. In: Proceedings of the 18th annual international ACM SIGIR conference on research and development in information retrieval, ACM, New York, NY, USA, pp 164-170.
47. Hupfer ME, Detlor B (2006) Gender and web information seeking: a self-concept orientation model. J Am Soc Inf Sci Technol 57(8): 1105-1115.
48. Ingwersen P (1994) Polyrepresentation of information needs and semantic entities: elements of a cognitive theory for information retrieval interaction. In: Proceedings of the 17th annual international ACM SIGIR conference on research and development in information retrieval. Springer, New York, pp 101-110.
49. Ingwersen P (1996) Cognitive perspectives of information retrieval interaction: Elements of a cognitive theory. J Doc 52(1): 3-50.
50. Ingwersen P, Jarvelin K (2005) The turn: integration of information seeking and information retrieval in context. Springer, Berlin.
51. Iqbal R, Sturm J, Kulyk O, Wang J, Terken J (2005) User-centred design and evaluation of ubiquitous services. In: Proceedings of the 23rd annual international conference on design of communication (SIGDOC '05), ACM, New York, NY, USA, pp 138-145.
52. Jansen BJ, Booth DL, Spink A (2007a) Determining the user intent of web search engine queries. In: Proceedings of the 16th international conference on World Wide Web, ACM, New York, NY, USA, pp 1149-1150.
53. Jansen BJ, Booth DL, Spink A (2007b) Determining the user intent of web search engine queries. In: WWW '07: proceedings of the 16th international conference on World Wide Web, ACM, New York, NY, USA, pp 1149-1150.
54. Jansen BJ, Booth DL, Spink A (2008) Determining the informational, navigational, and transactional intent of web queries. Inf Process Manag 44(3): 1251-1266.
55. Jarvelin K, Kekalainen J (2000) Ir Evaluation methods for highly relevant documents. In: Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval, Belkin and al, pp 41-48.
56. Jarvelin K, Kekalainen J (2002) Cumulative gain-based evaluation of IR techniques. ACM Trans Inf Syst 20(4): 422-446.
57. Joachims T, Granka L, Hembrooke H, Radlinski F, Gay G (2007) Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Trans Inf Syst 25(2): 7.
58. John R, Mooney G (2001) Fuzzy user modelling for information retrieval on the world wide web. Knowl Inf Syst 3(1): 81-95.
59. Kekalainen J, Jarvelin K (2004) Evaluating information retrieval systems under the challenges of interaction and multidimensional dynamic relevance. In: Ingwersen P, Vakkari P (eds) Proceedings of the 4th CoLIS conference, pp 253-270.
60. Kelly D, Fu X (2007) Eliciting better information need descriptions from users of information search systems. Inf Process Manag 43(1): 30-46.
61. Kepler J (2005) Context-awarness in mobile tourism guides-a comprehensive survey. Technical Report, University. Linz, IFS/TK.
62. Kim K (2008) Effects of emotion control and task on web searching behavior. Inf Process Manag 44(1): 373-385.
63. Kim K, Allen B (2002) Cognitive and task influences on web searching behavior. J Am Soc Inf Sci Technol 53(2): 109-119.
64. Kjeldskov J, Graham C (2003) A review of mobile hci research methods. In: Proceedings mobile HCI, Lecture notes in computer science, pp 317-335.
65. Kraft R, Maghoul F, Chang C (2005) Y!q: contextual search at the point of inspiration. In: CIKM '05: proceedings of the 14th ACM international conference on information and knowledge management, ACM Press, New York, NY, USA, pp 816-823.
66. Lang K (1995) NewsWeeder: learning to filter netnews. In: Proceedings of the 12th international conference on machine learning, Morgan Kaufmann, San Mateo, pp 331-339.
67. Law E, Klobucar T, Pipan M (2006) User effect in evaluating personalized information retrieval systems. In: EC-TEL, Springer, Berlin, pp 257-271.
68. Lee J, Hu X, Downie J (2005) Qa websites: rich research resources for contextualizing information retrieval behaviors. In: Proceedings of the 28th international SIGIR conference on research and development in information retrieval, Workshop on information retrieval in context, pp 33-366.
69. Leung CW, Chan SC, Chung F (2006) A collaborative filtering framework based on fuzzy association rules and multiple-level similarity. Knowl Inf Syst 10(3): 357-381.
70. Liu F, Yu C (2004) Personalized web search for improving retrieval effectiveness. IEEE Trans Knowl Data Eng 16(1): 28-40.
71. Marchionini G (1995) Information seeking in electronics environments. Cambridge university press, Cambridge Series on Human-Computer Interaction.
72. Micarelli A, Sicarrone F (2004) Anatomy and empirical evaluation of an daptive web-based information filtering system. User Model User-Adapted Interact 14(2-3): 159-200.
73. Mitchell T, Chen S, Macredie R (2005) Hypermedia learning and prior knowledge: domain expertise vs. system expertise. J Comput Assist Learn 21(12): 53-64.
74. Mizzaro S. (1998) How many relevances in information retrieval?. Interact Comput 10(3): 303-320.
75. Mobasher B (2007) Data mining for Web personalization. In: Brusilovsky P, Kobsa A, Nejdl W (eds) Lecture notes in computer science. Springer, Berlin.
76. Mostafa J, Mukhopadhyay S, Palakal M (2003) Simulation studies of different dimensions of users' interests and their impact on user modeling and information filtering. Inf Retr 6(2): 199-223.
77. Navarro-Prieto R, Scaife M, Rogers Y (2006) Cognitive strategies in web searching. In: 5th conference on human factors & the web.
78. Petrelli D (2008) On the role of user-centred evaluation in the advancement of interactive information retrieval. Inf Process Manag 44(1): 22-38.
79. Ramesh V, Glass L, Vessey I (2004) Research in computer science: an empirical study. J Syst Softw 70(1-2): 165-176.
80. Rigaux P (2002) Spatial database: with applications to GIS. Morgan Kauffmann, San Francisco.
81. Robertson S (2002) Comparing the performance of adaptive filtering and ranked output systems. Inf Retr 5(2-3): 257-268.
82. Robertson SE (1997) The probability ranking principle in IR. Readings in information retrieval, pp 281-286.
83. Robertson SE, Hancock-Beaulieu MM (1992) On the evaluation of IR systems. Inf Process Manag 28(4): 457-466.
84. Robertson S, Jones KS (1976) Relevance weighting for search terms. J Am Soc Inf Sci 27(3): 129-146.
85. Rocchio J (1971) Relevance feedback in information retrieval. In: Salton G (eds) The SMART retrieval system-experiments in automatic document processing, Prentice Hall, Englewood Cliffs.
86. Ryan N, Pascoe J, Morse D (1997) Enhanced reality fieldwork: the context-aware archaeological assistant. In: Gaffney V, van Leusen M, Exxon S (eds) Computer Applications in Archeology.
87. Salton G (1971) The SMART information retrieval system. Prentice-Hall, Englewood Cliffs.
88. Schilit B, Adams N, Want R (1994) Context-aware computing applications. In: Proceedings of the workshop on mobile computing systems and applications, IEEE Computer Society, Santa Cruz, CA, pp 85-90.
89. Schilit BN, LaMarca A, Borriello G, Griswold WG, McDonald D, Lazowska E, Balachandran A, Hong J, Iverson V (2003) Ubiquitous location-aware computing and the place lab initiative challenge. In: WMASHE'03, The first ACM international workshop on wireless mobile applications and services on WLAN (WMASH 2003), San Diego, CA, September 19, 2003, ACM, New York, NY, USA.
90. Shamber L (1994) Relevance and information behaviour. In: William ME (ed) ARIST, pp 3-48.
91. Shen X, Tan B, Zhai C (2005a) Context-sensitive information retrieval using implicit feedback. In: Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval, ACM, New York, NY, USA, pp 43-50.
92. Shen X, Tan B, Zhai C (2005b) Implicit user modeling for personalized search. In: Proceedings of the 14th ACM international conference on Information and knowledge management, ACM, New York, NY, USA, pp 824-831.
93. Sieg A, Mobasher B, Burke R (2004) User's information context: integrating user profiles and concept hierarchies. In: Proceedings of the 2004 meeting of the International Federation of Classification Societies.
94. Sieg A, Mobasher B, Burke R (2007) Web search personalization with ontological user profiles. In: Proceedings of the 16th ACM conference on conference on information and knowledge management, ACM, New York, NY, USA, pp 525-534.
95. Smyth B, Balfe E (2006) Anonymous personalization in collaborative web search. Inf Retr 9(2): 165-190.
96. Solomon P (1988) Children's information retrieval behavior: a case analysis of opac. J Am Soc Inf Sci 44(5): 245.
97. Sonnenwald D, Pejtersen A (1994) Towards a framework to support information needs in design: a concurrent engineering example. Knowledge organization and mangement, pp 161-172.
98. Sormunen E (2002) Liberal relevance criteria of TREC-: counting on negligible documents? In: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, ACM, New York, NY, USA, pp 324-330.
99. Sparck-Jones K, van Rijsbergen C (1976) Information retrieval test collections. J Doc 32(1): 59-72.
100. Speretta M, Gauch S (2005) Personalized search based on user search histories. In: Proceedings of the IEEE/WIC/ACM international conference on web intelligence, pp 622-628.
101. Su LT (1992) Evaluation measures for interactive information retrieval. Inf Process Manag 28(4): 503-516.
102. Tamine L, Boughanem M (2001) Applying heuristics to improve a genetic query optimisation process in information retrieval. In: 23 European colloquium on information retrieval, ECIR'2001, Darmstadt, Germany, 04/04/01-06/04/01, GMD-IPSI, pp 15-23.
103. Tamine L, Boughanem M, Zemirli WN (2008) Personalized document ranking: exploiting evidence from multiple user interests for profiling and retrieval. J Digit Inf Manag (in press).
104. Tamine L, Chrisment C, Boughanem M (2003) Multiple query evaluation based on an enchanced genetic algorithm IP& M. Inf Process Manag 39(2): 215-231.
105. Tan A, Ong H, Pan H, Ng J, Li Q (2004) Towards personalised web intelligence. Knowl Inf Syst 6(5): 595-616.
106. Tan B, Shen X, Zhai C (2006) Mining long-term search history to improve search accuracy. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, New York, NY, USA, pp 718-723.
107. Tao Y, Mamoulis N, Papadias D (2003) Validity information retrieval for spatio-temporal queries. In: SSTD, 2003.
108. Teevan J, Dumais S (2005) Personalizing search via automated analysis of interests and activities. In: Proceedings of the 28th international SIGIR conference on research and development in information retrieval, pp 449-456.
109. Tombros A, Ruthven I, Jose JM (2005) How users assess web pages for information seeking. J Am Soc Inf Sci Technol 56(4): 327-344.
110. Trajkova J, Gauch S (2004) Improving ontology-based user profiles. In: Proceedings of RIAO 2004, Vaucluse, France.
111. Turpin H, Hersh W (2001) Why batch and user evaluations do not give the same results. In: ACM conference on research and development in information retrieval, pp 225-231.
112. Uzuner O, Katz B, Yuret D (1999) Word sense disambiguation for information retrieval. In: AAAI '99/IAAI '99: proceedings of the sixteenth national conference on artificial intelligence and the eleventh innovative applications of artificial intelligence conference innovative applications of artificial intelligence, American Association for Artificial Intelligence, Menlo Park, CA, USA, p 985.
113. Vakkari P, Sormunen E (2004) The influence of relevance levels on the effectiveness of interactive information retrieval. J Am Soc Inf Sci Technol 55(11): 963-969.
114. Vieira V, Tedesco P, Salgado AC, Brézillon P (2007) Investigating the specifics of contextual elements management: the cemantika approach. In: CONTEXT, pp 493-506.
115. Voorhees E (2000) Variations in relevance judgments and the measurement of retrieval effectiveness. Inf Process Manag 36(1): 697-716.
116. Webb G, Pazzani M, Billsus D (2001) Machine learning for user modeling. User Model User-Adapted Interact 11(1-2): 19-29.
117. White R, Ruthven I, Jose J, van Rijsbergen C (2005) Evaluating implicit feedback models using searcher simulations. ACM Trans Inf Syst 23(3): 325-361.
118. Woodruff A, Plaunt C (1994) GIPSY: automated geographic indexing of text documents. J Am Soc Inf Sci 45(9): 645-655.
119. Wu X, Kumar V, Ross Quinlan J, Ghosh J, Yang Q, Motoda H, McLachlan GJ, Ng A, Liu B, Yu PS, Zhou Z-H, Steinbach M, Hand DJ, Steinberg D (2007) Top 10 algorithms in data mining. Knowl Inf Syst 14(1): 1-37.
120. Xie HI (2008) 'Users' evaluation of digital libraries (dls): their uses, their criteria, and their assessment. Inf Process Manag 44(3): 1346-1373.
121. Yau S, Huan L, Huang D, Yao Y (2003) Situation-aware personalized information retrieval for mobile internet. In: Proceedings of the 27th annual international computer software and applications conference (COMPSAC)'.
122. Yilmaz E, Aslam JA (2008) Estimating average precision when judgments are incomplete. Knowl Inf Syst 16(2): 173-211.
123. Zhou Y, Croft WB (2008) Measuring ranked list robustness for query performance prediction. Knowl Inf Syst 16(2): 155-171.
124. Zobel J (1998) How reliable are the results of large-scale information retrieval experiments?. In: In ACM conference on research and development in information retrieval, pp 307-314.