A systematic review of interactive information retrieval evaluation studies, 1967-2006

Journal of the American Society for Information Science and Technology

Volumn 64, Issue 4, 2013, Pages 745-770

  Kelly, Diane ^a   Sugimoto, Cassidy R ^b  

^a UNIVERSITY OF NORTH CAROLINA   (United States)

^b INDIANA UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EVALUATION METHODS; EVALUATION RESEARCH; INCLUSION AND EXCLUSIONS; INTERACTIVE INFORMATION RETRIEVAL; MEASUREMENT AND ANALYSIS; RESEARCH METHODS; SYSTEMATIC REVIEW; USER PERSPECTIVES;

INFORMATION RETRIEVAL;

RESEARCH;

EID: 84876681435     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.22799     Document Type: Review

References (73)

1. Belkin, N.J., Oddy, R.N., & Brooks, H.M. (1982). Ask for information retrieval: Part 1. Journal of Documentation, 38(2), 61-71.
2. Belkin, N.J., & Vickery, A. (1985). Interaction in information systems: A review of research from document retrieval to knowledge-based systems. London, UK: The British Library.
3. Bennett, J.L. (1971). Interactive bibliographic search as a challenge to interface design. In D.E. Walker (Ed.), Interactive bibliographic search: The user/computer interface (pp. 1-16). Montvale, NJ: AFIPS Press.
4. Bernal, J.D. (1948). Preliminary analysis of pilot questionnaires on the use of scientific literature. In Proceedings of the Royal Society Scientific Information Conference (pp. 589-637).
5. Borlund, P. (2003). The IIR evaluation model: A framework for evaluation of interactive information retrieval systems. Information Research, 8(3), 152.
6. Bourne, C.P. (1966). Evaluation of indexing systems. Annual Review of Information Science & Technology, 1, 171-190.
7. Bourne, C.P. & Hahn, T.B. (2003). Ahistory of online information services, 1963-1976. Cambridge, MA: MIT Press.
8. Carroll, J.M. (2011). Human computer interaction (HCI). Retrieved from http://www.interaction-design.org/encyclopedia/human-computer-interaction-hci. html
9. Case, D.O. (2002). Looking for information: A survey of research on information seeking, needs and behavior. Lexington, KY: Academic Press.
10. Cleverdon, C.W. (1960). Report on the first stage of an investigation onto the comparative efficiency of indexing systems. Retrieved from http://www.sigir.org/museum/contents.html
11. Cool, C., & Belkin, N.J. (2011). Interactive information retrieval: History and background. In I. Ruthven & D. Kelly (Eds.), Interactive information seeking, behaviour and retrieval. London, UK: Facet Publishing.
12. Cooper, H.M. (1988). Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society, 1(1), 104-126.
13. Cooper, H.M. (1989). Integrating research: A guide for literature reviews (2nd ed.). Sage Publications: Newbury Park, CA.
14. Cooper, H., & Hedges, L.V. (1994). The handbook of research synthesis. New York, NY: Russell Sage Foundation.
15. Cooper,W.S. (1973). On selecting a measure of retrieval effectiveness, part 1: The "subjective" philosophy of evaluation. Journal of the American Society for Information Science, 24, 87-100.
16. Davis, R.M. (1966). Man-machine communication. Annual Review of Information Science & Technology, 1, 221-254.
17. DeMey, M. (1977). The cognitive viewpoint: Its development and its scope. In M. De Mey et al. (Eds.), CC77: InternationalWorkshop on the CognitiveViewpoint (pp. xvi-xxxi). Ghent, Belgium:University of Ghent Press.
18. Dervin, B., & Nilan, M. (1986). Information needs and uses. Annual Review of Information Science & Technology, 21, 3-33.
19. Dumais, S.T., & Belkin, N.J. (2005). The TREC Interactive Tracks: Putting the user into search. In E.M. Voorhees & D.K. Harman (Eds.), TREC: Experiment and evaluation in information retrieval (pp. 123-153). Cambridge, MA: MIT Press.
20. Dumais, S.T., Jeffries, R., Russell, D., Tang, D. & Teevan, J. (2011). Design of large-scale log analysis studies. Retrieved from http://research.microsoft.com/~sdumais/CHI2011-LogCourse-Share.pdf
21. Feldman, K.A. (1971). Using the work of others: Some observations on reviewing and integrating. Sociology of Education, 44(1), 86-102.
22. Fidel, R. (2011). Approaches to investigating information interaction and behaviour. In I. Ruthven & D. Kelly (Eds.), Interactive information seeking, behaviour and retrieval. London, UK: Facet Publishing.
23. Fisher, K., & Julien, H. (2009). Information behavior. Annual Review of Information Science & Technology, 43, 317-358.
24. Furnas, G.W., Landauer, T.K., Gomez, L.M., & Dumais, S.T. (1987). The vocabulary problem in human-system communication. Communications of the ACM, 30(11), 964-971.
25. Glass, G.V. (1976). Primary, secondary, and meta-analysis. Educational Researcher, 5, 3-8.
26. Harman, D.K. (2011). Information retrieval evaluation. San Rafael, CA: Morgan & Claypool Publishers.
27. Hornbæk, K., & Law, E.L.-C. (2007). Meta-analysis of correlations among usability measures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (SIGCHI '07), San Jose, CA (pp. 617-626). New York, NY: Association for Computing Machinery. (Pubitemid 47581650)
28. Ide, E. (1969). Relevance feedback in an automatic document retrieval system. Retrieved from http://www.sigir.org/museum/contents.html
29. Ingwersen, P., & Järvelin, K. (2005). The turn: Integration of information seeking and retrieval in context. Dordrecht, The Netherlands: Springer.
30. Jansen, B.J. (2009). Understanding user-web interactions via web analytics. San Rafael, CA: Morgan & Claypool Publishers.
31. Järvelin, K. (2007). An analysis of two approaches in information retrieval: From frameworks to study designs. Journal of the American Society for Information Science & Technology, 58(7), 971-986. (Pubitemid 46826214)
32. Järvelin, K. (2011). Evaluation. In I. Ruthven & D. Kelly (Eds.), Interactive information seeking, behaviour and retrieval. London, UK: Facet Publishing.
33. Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005). Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th AnnualACM International Conference on Research and Development in Information Retrieval (SIGIR '05), Salvador, Brazil (154-161).
34. Kanoulas, E., Hall, M., Clough, P., Carterette, B., & Sanderson, M. (2011). Overview of the TREC 2011 session track. In E. M. Voorhees & L. P. Buckland (Eds.), Proceedings of the 20th Text Retrieval Conference (TREC 2011). NIST Special Publication: SP 500-295.
35. Kelly, D. (2009). Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval, 3(1-2), 1-224.
36. Kelly, D., Dumais, S., & Pedersen, J. (2009). Evaluation challenges and directions for information seeking support systems. IEEE Computer, 42(3), 60-66.
37. King, D.W., & Bryant, E.C. (1971). The evaluation of information services and products. Washington, DC: Information Resources Press.
38. Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R.M. (2009). Controlled experiments on the Web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140-181.
39. Kuhn, T.S. (1996). The structure of scientific revolutions. Chicago, IL: University of Chicago Press. (Original work published 1962)
40. Li, Y., & Belkin, N.J. (2008). A faceted approach to conceptualizing tasks in information seeking. Information Processing & Management, 44(6), 1822-1837.
41. Nielsen, J., & Levy, J. (1994). Measuring usability: Preference vs. performance. Communications of the ACM, 37, 4, 66-75.
42. O'Brien, H.L., & Toms, E.G. (2010). Measuring interactive information retrieval: The case of the User Engagement Scale. In Proceedings of Information Interaction in Context (IIiX) (pp. 335-340). New Brunswick, NJ: ACM Digital Library. doi: 10.1145/1840784.1840835.
43. Radlinski, F., Kurup, M., & Joachims, T. (2008). How does clickthrough data reflect retrieval quality? In Proceedings of the ACM Conference on Information and Knowledge Management (CIKM '08), Napa Valley, CA (pp. 43-52). New York, NY: Association for Computing Machinery.
44. Robertson, S.E. (2008). On the history of evaluation in IR. Journal of Information Science, 34(4), 439-456.
45. Ruthven, I., & Kelly, D. (2011). Interactive information seeking, behaviour and retrieval. London, UK: Facet Publishing.
46. Salton, G. (1970). Evaluation problems in interactive information retrieval. Information Storage & Retrieval, 6, 29-44.
47. Salton, G. (1971). The SMART retrieval system; experiments in automatic document processing. Englewood Cliffs, NJ: Prentice-Hall.
48. Sanderson, M. (2010). Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval, 4, 247-375.
49. Saracevic, T. (1995). Evaluation of evaluation in information retrieval. In Proceedings of the 18th Annual ACM SIGIR Conference on Research and Development of Information Retrieval, Seattle, WA (pp. 138-146). New York, NY: Association for Computing Machinery.
50. Saracevic, T. (2007). Relevance:Areview of the literature and a framework for thinking on the notion in information science. Part II: Nature and manifestations of relevance. Journal of the American Society for Information Science and Technology, 58(3), 1915-1933. (Pubitemid 350229698)
51. Savage-Knepshield, P.A., & Belkin, N.J. (1999). Interaction in information retrieval: Trends over time. Journal of the American Society for Information Science and Technology, 50(12), 1067-1082. (Pubitemid 30510995)
52. Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage Publications.
53. Siatri, R. (1999). The evolution of user studies. Libri, 49, 132-141.
54. Spärck-Jones, K. (1981). Information retrieval experiment. London, UK: Butterworths & Co. Ltd.
55. Spärck-Jones, K., & Willett, P. (1997). Readings in information retrieval. San Francisco, CA: Morgan Kaufmann Publishers, Inc.
56. Swanson, D.R. (1971). Some unexplained aspects of the Cranfield test of indexing performance factors. Library Quarterly, 41(3), 223-228.
57. Tague, J.M. (1981). The pragmatics of information retrieval experimentation. In K. Spärck Jones (Ed.), Information retrieval experiment (pp. 59-104). London, UK: Butterworths & Co. Ltd.
58. Tague-Sutcliffe, J.M. (1992). The pragmatics of information retrieval experimentation, revised. Information Processing &Management, 28(4), 467-490.
59. Tao, T., & Zhai, C.-X. (2007). An exploration of proximity measures for information retrieval. In Proceedings of the 30th annual international ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '07), AmsteRDAm, The Netherlands (pp. 295-302). New York, NY: Association for Computing Machinery. (Pubitemid 350164973)
60. Thompson, D.A. (1971). Interface design for an interactive information retrieval system: A literature survey and a research system description. Journal of theAmerican Society for Information Science, 22(6), 361-373.
61. Urquhart, D.J. (1948). The distribution and use of scientific and technical information. In Proceedings of the Royal Society Scientific Information Conference (pp. 408-419).
62. Vakkari, P. (2001). Changes in search tactics and relevance judgments when preparing a research proposal: A summary of the findings of a longitudinal study. Information Retrieval, 4(3-4), 295-310. (Pubitemid 33642153)
63. Vakkari, P. (2008). Trends and approaches in information behavior research. Information Research, 13(4), 361.
64. Vakkari, P. (2010). Exploratory searching as conceptual exploration. In Proceedings of the Fourth Human Computer Information Retrieval Workshop, New Brunswick, NJ (pp. 24-27).
65. Vakkari, P., & Sormunen, E. (2004). The influence of relevance levels on the effectiveness of interactive information retrieval. Journal of the American Society for Information Science and Technology, 55(11), 963-969.
66. Valenza, R. (2009). Literature, language, and the rise of the intellectual disciplines in Britain, 1680-1820. Cambridge University Press.
67. Voorhees, E.M. (2007). TREC: Continuing information retrieval's tradition of experimentation. Communications of the ACM, 50(11), 51-54. (Pubitemid 350083512)
68. Voorhees, E.M., & Harman, D.K. (2005). TREC: Experiment and evaluation in information retrieval. Cambridge, MA: MIT Press.
69. Walker, D.E. (1971). Interactive bibliographic search: The user/computer interface. Montvale, NJ: AFIPS Press.
70. Wagner, D.G., & Berger, J. (1985). Do sociological theories grow. American Journal of Sociology, 90(4), 697-728.
71. White, H.D., & McCain, K.W. (1998). Visualizing a discipline: An author co-citation analysis of information science, 1972-1995. Journal of the American Society for Information Science, 49(4), 327-355. (Pubitemid 128612354)
72. Whitley, R. (2000). The intellectual and social organization of the sciences (2nd ed). New York, NY: Oxford University Press.
73. Wolf, F.M. (1986). Meta-analysis: Quantitative methods for research synthesis. Thousand Oaks, CA: Sage Publications.