An evaluation framework of user interaction with metadata surrogates

Journal of Information Science

Volumn 35, Issue 3, 2009, Pages 321-339

  Balatsoukas, Panos ^a   Morris, Anne ^a   O'Brien, Ann ^a  

^a LOUGHBOROUGH UNIVERSITY   (United Kingdom)

Author keywords

Information retrieval; Interface design; Search results; User centred metadata

Indexed keywords

EVALUATION FRAMEWORK; FULL-TEXT DOCUMENTS; INTERFACE DESIGN; LITERATURE REVIEWS; MAIN COMPONENT; SEARCH RESULTS; TEXTUAL METADATA; USER INTERACTION; USER-CENTRED METADATA;

INFORMATION RETRIEVAL; INFORMATION SERVICES;

METADATA;

EID: 67449131567     PISSN: 01655515     EISSN: 17416485     Source Type: Journal    
DOI: 10.1177/0165551508099090     Document Type: Article

References (81)

1. N. Fuhr, P. Hansen, M. Mabe, A. Micsik and T. Solvberg, Digital libraries: a generic classification and evaluation scheme. In: ECDL, Proceedings of the 5th European Conference on Research and Advanced Technology for Digital Libraries (Springer, London, 2001). (Pubitemid 33347047) (Pubitemid 33347047)
2. G. Tsakonas and C. Papatheodorou, Analyzing and evaluating usefulness and usability in electronic information services, Journal of Information Science 32 (5) (2006) 400-419. (Pubitemid 44564376)
3. T. Saracevic, P. Kantor, A. Chamis and D. Trivison, A study of information seeking and retrieving. I. Background and methodology, Journal of the American Society for Information Science 39 (3) (1988) 161-176. (Pubitemid 18618158) (Pubitemid 18618158)
4. L. Su, A comprehensive and systematic model of user evaluation of Web search engines, Journal of the American Society for Information Science and Technology 54 (13) (2003) 1175-1192.
5. F. Johnson, J. Griffiths and R. Hartley, Task dimensions of user evaluations of information retrieval systems, Information Research 8 (4) (2003). Available at: http://informationr.net/ir/8-4/paper157.html (accessed 23 October 2007). (Pubitemid 38816900)
6. D. Petrelli, On the role of user-centred evaluation in the advancement of interactive information retrieval. Information Processing and Management 44 (1) (2008) 22-38. (Pubitemid 350007895) (Pubitemid 350007895)
7. G. Marchionini, Toward multimedia surrogates (2007). Available at: http://sils.unc.edu/~march/multimedia-surrogates.pdf (accessed 12 March 2008).
8. B. Wildemuth, G. Marchionini, T. Wilkens, M.M. Yang, G. Geisler, B. Fowler, A. Hughes, and X. Mu, Alternative surrogates for video objects in a digital library: users' perspectives on their relative usability. In: M. Agosti and C. Thanos (eds), Proceedings of the 6th European conference on Research and Advanced Technology for Digital Libraries (Springer, London, 2002).
9. H. Joho and J. Jose, A comparative study of the effectiveness of search result presentation on the web. In: Advances in Information Retrieval (Springer, Berlin, 2006).
10. B. Fraser and M. Gluck, Usability of geospatial metadata or space-time matters, Bulletin of the American Society for Information Science, 25 (6) (1999). Available at: http://www.asis.org/Bulletin/Aug-99/fraser-gluck.html (accessed 12 March 2008).
11. The Oxford English Dictionary (Oxford University Press, Oxford, 2008). Available at: http://www.oed.com/ (accessed 22 July 2008).
12. J. Greenberg, Metadata and the world wide web. In: Encyclopaedia of Library and Information Science (Marcel Dekker, New York, 2005).
13. O. Drori, Improving display of search results in information retrieval systems - users' study. Technical report no.200034 (2000). Available at: www.global-report.com/drori/?l=he&a=3234 (accessed 23 July 2006).
14. O. Drori, How to display search results in digital libraries: user study. In: Proceedings of the New Developments in Digital Library, NDDL 2003, 3rd International Workshop (2003). Available at: www.global-report.com//drori/?l= he&a=3330 (accessed 20 April 2008).
15. S. Dziadosz and R. Chandrasekar, Do thumbnail previews help users make better relevance decisions about web search results? In: K. Jarvelin, M. Beaulieu, R. Baeza-Yates and S.H. Myaeng, Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (ACM Press, New York, 2002).
16. M.L. Zeng and J. Qin, Metadata (Facet Publishing, London, 2008).
17. M.M. Sebrechts, J. Vasilakis, M.S. Miller, J.V. Cugini and S.J. Laskowski, Visualisation of search results: a comparative evaluation of text, 2D and 3D Interfaces. In: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (ACM Press, New York, 1999).
18. S. Green, G. Marchionini, C. Plaisant and B. Shneiderman, Previews and overviews in digital libraries: designing surrogates to support visual information seeking, Journal of the American Society for Information Science and Technology 51 (4) (2000) 380-393. (Pubitemid 30552658)
19. Y. Song and G. Marchionini, Effects of audio and visual surrogates for making sense of digital video. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (ACM, New York, 2007). (Pubitemid 47581674) (Pubitemid 47581674)
20. T. Saracevic, Extension and application of the stratified model of information retrieval interaction. In: Proceedings of the Annual Meeting of the American Society for Information Science (JASIS, Medford, NJ, 1997).
21. P. Ingwersen, The Turn: Integration of Information Seeking and Retrieval in Context (Springer, Dordrecht, 2005).
22. M.J. Bates, The Design of Browsing and Berrypicking Techniques for the Online Search Interface (UCLA, Los Angeles, 1998). Available at: www.gseis.ucla.edu/faculty/bates/berrypicking.html (accessed 22 July 2008).
23. A. Spink, T. Wilson, D. Ellis and N. Ford, Modeling users' successive searches in digital environments, D-Lib Magazine (April 1998). Available at: www.dlib.org/dlib/april98/04spink.html (accessed 22 July 2008).
24. N.J. Belkin, C..Cool, A. Stein and U. Thiel, Cases, scripts and information-seeking strategies: on the design of interactive information retrieval systems, Expert Systems with Applications 9 (3) (1995) 379-395.
25. C. Cool and N.J. Belkin, A classification of interactions with information. In: Proceedings of the Fourth International Conference on Conceptions of Library and Information Science (Libraries Unlimited, Greenwood Village, 2002).
26. B. Shneiderman, D. Byrd and B. Croft, Clarifying search: a user-interface framework for text searches, D-Lib Magazine (January 1997). Available at: www.dlib.org/dlib/january97/retrieval/01shneiderman.html (accessed 11 June 2006).
27. G. Marchionini, Information Seeking in Electronic Environments (Cambridge University Press, Cambridge, 1995).
28. I. Ruthven, M. Baillie, L. Azzopardi, R. Bierig, E. Nicol, S. Sweeney and M. Yaciki, Contextual factors affecting the utility of surrogates within exploratory search, Information Processing and Management 44 (2) (2008) 437-462.
29. C. Borgman, Designing digital libraries for usability. In: A. Bishop, N. Van House and P. Butterfield (eds), Digital Library Use (MIT Press, Cambridge, MA, 2003).
30. D. Norman, The Design of Everyday Things (Basic Books, New York, 2002).
31. B. Shneiderman and C. Plaisant, Designing the User Interface (Pearson, London, 2004).
32. J. Nielsen, Ten Usability Heuristics. Available at: www.useit.com/papers/ heuristic/heuristic-list.html (accessed 12 May 2008).
33. P. Balatsoukas, A. Morris and A. O'Brien, Designing metadata surrogates for search result interfaces of learning object repositories: linear versus clustered metadata design. In: L. Chan and B. Martens (eds), Proceedings of the 11th International Conference on Electronic Publishing held in Vienna, Austria 13-15 June 2007 (TUW, Vienna, 2007).
34. R.S. Rele and A.T. Duchowski, Using eye tracking to evaluate alternative search results interfaces. In: Proceedings of the 49th Annual Meeting of Human Factors and Ergonomics Society, September 26-30 2005, Orlando, FL, HFES (HFES, Orlando, FL, 2005).
35. M.L. Resnick, C.A. Maldonado, J.M. Santos and R. Lergier, Modeling on-line search behavior using alternative output structures. In: Proceedings of the Human Factors and Ergonomics Society 45th Annual Conference, Minneapolis, MN, USA (HFES, Minneapolis, MN, 2001).
36. L. Granka, H. Hembrooke, G. Gay and M. Feunser M, Eye-tracking analysis of user behavior in WWW search. In: M. Sanderson, K. Jarvelin, J. Allan and P. Bruza (eds), Proceedings of the 27th annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, July 25-29, 2004 (ACM, Sheffield, 2004).
37. R. Fidel and D. Soergel, Factors affecting online bibliographic retrieval: a conceptual framework for research, Journal of the American Society for Information Science 34 (3) (1981) 163-180. (Pubitemid 13589535) (Pubitemid 13589535)
38. C. Kuhlthau, Inside the search process: information seeking from the user's perspective, Journal of the American Society for Information Science 42 (5) (1991) 361-371.
39. O. Drori and E. Tamir, Display of search results in Hebrew: a comparison study between Google and LCC&K interface, Journal of Information Science 31 (3) (2005) 164-177. (Pubitemid 43087411)
40. J.W. Janes, Relevance judgments and the incremental presentation of document representations, Information Processing and Management 27 (6) (1991) 629-646.
41. R.S. Marcus, P. Kugel and A.R. Benenfeld, Catalog information and text as indicators of relevance, Journal of the American Society for Information Science 29 (1) (1978) 15-30. (Pubitemid 9420942) (Pubitemid 9420942)
42. A. Taylor, C. Cool, N. Belkin and W. Amadio, Relationships between categories of relevance criteria and stage in task completion, Information Processing and Management 43 (4) (2007) 1071-1084. (Pubitemid 46400291) (Pubitemid 46400291)
43. L. Wen, I. Ruthven and P. Borlund, The effects of topic familiarity on online search behaviour and use of relevance criteria. In: Proceedings of the 28th European Conference on Information Retrieval (Springer, Berlin, 2006). (Pubitemid 44029486) (Pubitemid 44029486)
44. X. Yuan, N. Belkin and J. Kim, The relationship between ASK and relevance criteria. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 11-15, 2002, Tampere, Finland (ACM Press, Tampere, Finland, 2002).
45. A. Morris and P. Balatsoukas, Usability evaluation of ebrary and overdrive online systems. In: P. Brophy, J. Craven and M. Markland, Libraries Without Walls 6: Evaluating the Distributed Delivery of Library Services (Facet Publishing, London, 2006).
46. S. Kerr, Wayfinding in an electronic database: the relative importance of navigational; cues vs mental models, Information Processing and Management 26 (4) (1990) 511-523.
47. N. Ford, D. Miller and N. Moss, Web search strategies and approaches to studying, Journal of the American Society for Information Science and Technology 54 (6) (2003) 473-489.
48. M. Graff, Individual differences in hypertext browsing strategies, Behaviour & Information Technology 24 (2) (2005) 93-99.
49. A. Spink, A user-centered approach to evaluating human interaction with web search engines: an exploratory study, Information Processing and Management 38 (3) (2002) 401-426. (Pubitemid 34040561) (Pubitemid 34040561)
50. P. Wang and D. Soergel, A cognitive model of document use during a research project. Study I. Document selection, Journal of the American Society for Information Science and Technology 49 (2) (1998) 115-133. (Pubitemid 128594911) (Pubitemid 128594911)
51. A. Crystal and J. Greenberg, Relevance criteria identified by health information users during Web searches, Journal of the American Society for Information Science and Technology 57 (10) (2006) 1368-2882.
52. B. Kules and B. Shneiderman, Categorised graphical overviews for web search results: an exploratory study using U.S. government agencies as a meaningful and stable structure. In: Proceedings of the Third Annual Workshop on HCI Research in MIS (AIS, Washington, DC, 2004).
53. P. Pirolli, S. Card, and M. Wege, The effects of information scent on visual search in the Hyperbolic Tree browser, ACM Transactions on Computer Human Interaction 10 (1) (2003) 20-53.
54. P. Borlund, The concept of relevance in IR, Journal of the American Society of Information Science and Technology 54 (10) (2003) 913-925.
55. C. Barry, User-defined relevance criteria: an exploratory study, Journal of the American Society for Information Science and Technology 45 (3) (1994) 149-159.
56. C. Barry and L. Schamber, Users' criteria for relevance evaluation: a cross-situational comparison, Information Processing and Management 34 (2 / 3) (1998) 219-236. (Pubitemid 128392791) (Pubitemid 128392791)
57. R. Small, S. Sutton, M. Miwa, C. Urfels and M. Eisenberg, Information seeking for instructional planning: an exploratory study, Journal of Research on Computing in Education 31 (2) (1998) 204-219.
58. R. Tang and P. Solomon, Use of relevance criteria across stages of document evaluation: on the complementarity of experimental and naturalistic studies, Journal of the American Society for Information Science and Technology 52 (8) (2001) 676-685.
59. S.Y. Rieh, Judgment of information quality and cognitive authority in the Web, Journal of the American Society for Information Science and Technology 53 145-161. (Pubitemid 34138427)
60. R. Pearson and P. van Schaik, The effect of spatial layout of and link colour in web pages on performance in a visual search task and an interactive task, International Journal of Human Computer Studies 59 (3) (2003) 327-353.
61. J. Ling and P. van Schaik, The influence of font type and line length on visual search and information retrieval in web pages, International Journal of Human Computer Studies 64 (5) (2006) 395-404.
62. J. Nielsen, Designing Web Usability (New Riders, Indianapolis, 2000).
63. S. Mizzaro, Relevance the whole history, Journal of the American Society for Information Science and Technology 48 (9) (1997) 810-832. (Pubitemid 127628564) (Pubitemid 127628564)
64. T. Paek, S. Dumais and R. Logan, WaveLens: a new view onto Internet search results. In: Proceedings of SIGCHI Conference on Human Factors in computing systems CHI2004, April 24-29, 2004, Vienna, Austria (ACM Press, New York, 2004).
65. S. Dumais, E. Cutrell and H. Chen, Optimising search by showing results. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2001 (ACM Press, New York, 2001).
66. Failte SearchLT, Search LT Evaluation Report (2002). Available at: www.failte.ac.uk/documents/eval-report.rtf (accessed 12 July 2006).
67. S.L. Shreeves and C.M. Kirkham, Experiences of educators using a portal of aggregated metadata, Journal of Digital Information 5 (3) (2004). Available at: http://jodi.tamu.edu/Articles/v05/i03/Shreeves/ (accessed 11 May 2006).
68. S.A. Ozel, I.S. Altingovde, O. Ulusoy, G. Ozsoyoglu and Z.M. Ozsoyoglu, Metadata-based modelling of information resources on the web, Journal of the American Society for Information Science and Technology 55 (2) (2004) 97-110.
69. Y. Zhang, J. Jeng and Y. Li, IFLA FRBR as user-centred metadata evaluation framework for moving image collections. In: Proceedings of the American Society for Information Science and Technology, annual meeting, November 12-17 2004 (ASIST, Maryland, 2004). Available at: www.scils.rutgers. edu/~miceval/progress/asist04-poster.doc (accessed 11 July 2006).
70. J. Clark, A usability study of the Belgian American Research Collection: measuring the functionality of a digital library, OCLC Systems and Services 20 (3) (2004) 115-127.
71. V. Mordechai, O. Drori and A.J. Frank, TTLS: a grouped display of search results based on organisational taxonomy using the LCC&K interface. In: Proceedings of the Tenth International Conference on Enterprise Information Systems (ICEIS Press, 2008). Available at: www.global-report.com/drori/?l= he&a=291330 (accessed 22 July 2008).
72. B. Kules, J. Kustanowitz and B. Shneiderman, Categorizing web search results into meaningful and stable categories using fast-feature techniques. In: Proceedings of the 6th ACM/IEEE Joint Conference on Digital Libraries-CS, 1-15 June 2006 (ACM Press, New York, 2006). (Pubitemid 46609260) (Pubitemid 46609260)
73. J. Kim, Finding documents in a digital institutional repository: Dspace and Eprints. In: G. Andrew (ed), Proceedings of the 68th Annual Meeting of the American Society for Information Science and Technology (ASIST, Maryland, 2005).
74. M. Hearst, Clustering versus faceted categories for information exploration, Communications of the ACM 49 (4) (2006) 59-61. (Pubitemid 43909066) (Pubitemid 43909066)
75. E.D. Liddy, E.E. Allen, C.M. Finneran, G. Gay, H. Hembrooke and L.A. Granka, MetaTest: evaluation of metadata from generation to use. In: Proceedings of the 3rd ACM/IEEE-CS Joint Conference on Digital Libraries (IEEE Computer Society, Washington, DC, 2003).
76. J. Preece, Y. Rogers and H. Sharp, Interaction Design: Beyond Human Computer Interaction (Wiley, Hoboken, NJ, 2006).
77. D. Nicholas, P. Huntington, H.R. Jamali and A. Watkinson, The information seeking behaviour of the users of digital scholarly journals, Information Processing and Management 42 (2006) 1345-1365.
78. S. Jones, S.J. Cunningham and P. McNab, An analysis of usage of a digital library. In: Research and Advanced Technology for Digital Libraries (Springer, Berlin, 1998). (Pubitemid 128145529) (Pubitemid 128145529)
79. T.C. Reeves, X. Apedoe and Y.H. Woo, Evaluating Digital Libraries: a User-friendly Guide (University Corporation for Atmospheric Research, 2003). Available at: http://dlist.sir.arizona.edu/398 (accessed 22 July 2008).
80. Y. Choi and E. Rasmussen, Users' relevance criteria in image retrieval in American history, Information Processing and Management, 38 (5) (2002) 695-726.
81. P. Pirolli, Information Foraging Theory: Adaptive Interaction with Information (Oxford University Press, Oxford, 2007).