Testing a decision-theoretic approach to the evaluation of information retrieval systems

Journal of Information Science

Volumn 34, Issue 6, 2008, Pages 861-876

  Wang, Ye Diana ^a   Forgionne, Guisseppi ^b  

^a Krasnow Institute for Advanced Study   (United States)

^b UNIVERSITY OF MARYLAND BALTIMORE COUNTY   (United States)

Author keywords

Analytic hierarchy process; Decision making; Evaluation; Information retrieval system; Information search process; Multi criteria model

Indexed keywords

ANALYTIC HIERARCHY PROCESS; HIERARCHICAL SYSTEMS; INFORMATION MANAGEMENT; INFORMATION RETRIEVAL; INFORMATION RETRIEVAL SYSTEMS; INFORMATION SERVICES; MANAGEMENT INFORMATION SYSTEMS;

ANALYTIC HIERARCHIES; EVALUATION; EVALUATION MEASURES; EVALUATION METHODOLOGIES; EVALUATION MODELS; INFORMATION OVERLOADS; INFORMATION RETRIEVAL APPROACHES; INFORMATION SEARCH PROCESS; INTEGRATED EVALUATIONS; INTERACTIVE IR; MODEL TRANSFORMS; MULTI-CRITERIA DECISION MAKINGS; MULTI-CRITERIA MODEL; REAL PROBLEMS; RESEARCH QUESTIONS;

DECISION THEORY;

EID: 57249085901     PISSN: 01655515     EISSN: 17416485     Source Type: Journal    
DOI: 10.1177/0165551508091308     Document Type: Article

References (42)

1. C.W. Cleverdon, Aslib Cranfield Research Project: Report on the Testing and Analysis of an Investigation into the Comparative Efficiency of Indexing Systems (1962). Available at: https://aerade.cranfield.ac.uk/handle/1826/836 (accessed 16 March 2008).
2. C.W. Cleverdon, J. Mills and E.M. Keen, Aslib Cranfield Research Project: Factors Determining the Performance of Indexing Systems, Vol.1: Design. Available (text and appendix) at: https://aerade.cranfield.ac.uk/handle/1826/861 and https://aerade.cranfield.ac.uk/handle/1826/862 (accessed 16 March 2008 ).
3. P. Borlund, The concept of relevance in IR, Journal of the American Society for Information Science and Technology 54 (10) (2003) 913-925.
4. T. Saracevic, Relevance reconsidered. In P. Ingwersen and N.O. Pors (Eds), Information Science: Integration in Perspective: Proceedings of the 2nd International Conference on Conceptions of Library and Information Science (Elsevier, Amsterdam, 1996) 201-218.
5. T. Saracevic, Relevance: a review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance, Journal of the American Society for Information Science and Technology 58 (13) (2007) 1915-1933.
6. G. Salton, The state of retrieval system evaluation, Information Processing & Management 28 (4) (1992) 441-449.
7. L.T. Su, Evaluation measures for interactive information retrieval, Information Processing & Management 28 (4) (1992) 503-516.
8. L.T. Su, Value of search results as a whole as the best single measure of information retrieval performance, Information Processing & Management 34 (5) (1998) 557-579.
9. L.T. Su, A comprehensive and systematic model of user evaluation of web search engines: I. Theory and background, Journal of the American Society for Information Science and Technology 54 (13) (2003) 1175-1192.
10. L.T. Su, A comprehensive and systematic model of user evaluation of web search engines: II. An evaluation by undergraduates, Journal of the American Society for Information Science and Technology 54 (13) (2003) 1193-1223.
11. F.C. Johnson, J.R. Griffiths and R.J. Hartley, Task dimensions of user evaluations of information retrieval systems, Information Research 8 (4) (2003). Available at: http://informationr.net/ir/8-4/paper157.html (accessed 15 November 2007).
12. N. Wacholder, D. Kelly, P. Kantor, R. Rittman, Y. Sun, B. Bai, S. Small, B. Yamrom and T. Strzalkowski, A model for quantitative evaluation of an end-to-end question-answering system, Journal of the American Society for Information Science and Technology 58 (8) (2007) 1082-1099.
13. S.P. Harter, Variations in relevance assessments and the measurement of retrieval effectiveness, Journal of the American Society for Information Science 47 (1) (1996) 37-49.
14. C. Kuhlthau, Seeking Meaning: a Process Approach to Library and Information Services ( Ablex Publishing Corporation, Norwood, NJ, 1993 ).
15. G.A. Forgionne, An AHP model of DSS effectiveness, European Journal of Information Systems 8 (1999) 95-106.
16. L. Donohew and L. Tipton. A conceptual model of information seeking, avoiding and processing. In: Clarke, P. (ed.), Models for Mass Communication Research (Sage, Beverly Hills, CA, 1973) 243-269.
17. D. Soergel, Organizing Information: Principles of Data Base and Retrieval Systems ( Academic Press, San Diego, CA, 1985 ).
18. D. Soergel, Is user satisfaction a hobgoblin ? Journal of the American Society for Information Science 27 (4) (1976) 256-259.
19. Y.D. Wang and G.A. Forgionne, A decision-theoretic approach to the evaluation of information retrieval systems, Information Processing & Management 42 (4) (2006) 863-874.
20. S.E. Robertson, Process and outcome: on the evaluation of IR systems in the age of interaction, GUIs and multimedia. In: Proceedings of Mira 99: Evaluating Interactive Information Retrieval (1999). Available at: http://www.bcs.org/upload/pdf/ewic-mi99-paper7.pdf (accessed 12 November 2007).
21. P. Vakkari, Task based information searching. In: Cronin, B. (ed.), Annual Review of Information Science and Technology 37 (Information Today, Medford, NJ, 2003) 413-464.
22. C.J. van Rijsbergen, Information Retrieval (2 nd edn.) ( Butterworths, London, 1979 ).
23. G.A. Forgionne, Decision-making support system effectiveness: the process to outcome link, Information Knowledge Systems Management 2 (2000) 169-188.
24. W. Cooper, On selecting a measure of retrieval effectiveness, Journal of the American Society for Information Science 24 (2) (1973) 87-100.
25. W. Cooper, On selecting a measure of retrieval effectiveness. Part II. Implementation of the philosophy, Journal of the American Society for Information Science 24 (6) (1973) 413-424.
26. T. Saracevic, Evaluation of evaluation in information retrieval. In: Proceedings of the Eighteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (ACM, New York, 1995) 138-146.
27. P. Borlund, Evaluation of interactive information retrieval systems (PhD Thesis, a...bo Akademi University, a...bo, Finland, 2000 ).
28. P. Borlund, The IIR evaluation model: a framework for evaluation of interactive information retrieval systems, Information Research 8 (3) (2003). Available at: http://informationr.net/ir/8-3/paper152.html (accessed 11 May 2007).
29. K. JaCurrency signrvelin and P. Ingwersen, Information seeking research needs extension toward tasks and technology, Information Research 10 (1) (2004). Available at: http://informationr.net/ir/10-1/paper212.html (accessed 10 April 2007)
30. N.J. Belkin, Anomalous states of knowledge as a basis for information retrieval, Canadian Journal of Information Science 5 (1980) 133-143.
31. N.J. Belkin, R.N. Oddy and H.M. Brooks, ASK for information retrieval: part I, background and theory, Journal of Documentation 38 (1982) 61-71.
32. P. Ingwersen, Information Retrieval Interaction ( Taylor Graham, London, 1992 ).
33. P. Ingwersen, Cognitive perspectives of information retrieval interaction: elements of a cognitive IR theory, Journal of Documentation 52 (1) (1996) 3-50.
34. T.L. Saaty, The analytic hierarchy process - what it is and how it is used, Mathematical Modelling 9 (3-5) (1987) 161-176.
35. P. Harker. The art and science of decision making: the analytic hierarchy process (Working Paper 88-06-03 Decision Sciences Department, Wharton School, University of Pennsylvania, PA, 1988 ).
36. T.L. Saaty and L.G. Vargas. Decision Making in Economic, Political, Social and Technological Environments with the Analytic Hierarchy Process ( RWS Publications, Pittsburg, PA, 1994 ).
37. G.E. Phillips-Wren, E.D. Hahn and G.A. Forgionne, A multiple-criteria framework for evaluation of decision support systems, OMEGA - The International Journal of Management Science 32 (4) (2004) 323-332.
38. SAS Institute, SAS System Release 8.2 (TS2M0) for Microsoft Windows ( Cary, NC, 2001 ).
39. H. Motulsky, Analyzing Data with GraphPad Prism (1999). Available at: http://www.graphpad.com/articles/AnalyzingData.pdf (accessed 16 April 2007).
40. G. Peng, Testing normality of data using SAS. In: Proceedings of PharmaSUG 04 ( PharmaSUG, Chapel Hill, NC, 2004 ).
41. P. Borlund and P. Ingwersen, Measures of relative relevance and ranked half-life: performance indicators for interactive IR. In: B.W Croft, A. Moffat, C.J. van Rijsbergen, R. Wilkinson and J. Zobel (eds), Proceedings of the 21st ACM SIGIR Conference on Research and Development of Information Retrieval (ACM, New York, 1998) 324-331.
42. K. JaCurrency signrvelin and J. KekaCurrency signlaCurrency signinen. IR evaluation methods for retrieving highly relevant documents. In: N.J. Belkin, P. Ingwersen and M.-K. Leong (eds), Proceedings of the 23rd ACM SIGIR Conference on Research and Development of Information Retrieval (ACM, New York, 2000) 41-48.