Developing a holistic model for digital library evaluation

Journal of the American Society for Information Science and Technology

Volumn 61, Issue 1, 2010, Pages 88-110

  Zhang, Ying ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CARD-SORTING; EVALUATION MODELS; FLEXIBLE FRAMEWORK; HOLISTIC MODEL; LITERATURE REVIEWS; ONLINE SURVEYS; RESEARCH APPROACH; STAKEHOLDER GROUPS; STATISTICAL TECHNIQUES;

FLUORINE CONTAINING POLYMERS; FUNCTION EVALUATION; SURVEYS;

DIGITAL LIBRARIES;

EID: 72849107223     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.21220     Document Type: Article

References (88)

1. Abbas, J. (2002). Middle school children's use of the ARTEMIS Digital Library. In G. Marchionini & W.R. Hersh (Eds.), Proceedings of the Second ACM/IEEE-CS Joint Conference on Digital Libraries (pp. 98-105). New York: ACM Press.
2. Adams A., & Blandford, A. (2001). Digital libraries in a clinical setting: Friend or foe? In P. S. Constantopoulos & I. Sølvberg (Eds.), Research and advanced technology for digital libraries. Proceedings of Fifth European Conference (ECDL 2001) (pp. 214-224). Berlin, Germany: Springer.
3. Auerbach, C.F., & Silverstein, L.B. (2003). Qualitative data: An introduction to coding and analysis. New York: New York. University Press.
4. Basefky, S. (2009). The end of institutional repositories & the beginning of social academic research service: An enhanced role for libraries. Retrieved on June 17, 2009, from http://www.11rx.com/node/2177/print
5. Bearman, D. (2007). Digital libraries. Annual Review of Information Science and Technology, 41, 223-272.
6. Bergmark, D., Lagoze, C., & Sbityakov, A. (2002). Focused crawls, tunneling, and digital libraries. In M. Agosti & C. Thanos (Eds.), Research and Advanced Technology for Digital Libraries: Proceedings of the Sixth European Conference (ECDL '02) (pp. 9.1-106). Berlin, Germany: Springer.
7. Bishop, A.P. (1998). Logins and bailouts: Measuring access, use, and success in digital libraries. The Journal of Electronic Publishing, 4(2). Retrieved September 29, 2009, from http://quod.lib.umich.edu/cgi/t/text/text- idx?c=jep;view=text;rgn=main;idno=3336451.0004.207
8. Bishop, A.P. (1999). Making digital libraries go: comparing use across genres. In N. Rowe & E.A. Fox (Eds.), Proceedings of the Fourth ACM Conference on Digital Libraries (pp. 94-103). New York: ACM Press.
9. Bishop, A.P, Neumann, LJ., Star, S.L., Merkel, C., Ignacia, E., & Sandusky, R.J. (2000). Digital libraries: Situating use in changing information infrastructure. Journal of the American Society for Information Science, 51(4), 394-413.
10. Blixrud, J.C. (2002). Measures for electronic use: The ARL E-Metrics project. Retrieved September 29, 2009, from http://www.lboro.co.uk/departments/ dis/lisu/downloads/statsimpracticepdfs/blixrud.pdf
11. Bollen, J., & Luce, R. (2002). Evaluation of digital library impact and user communities by analysis of usage patterns. D-Lib Magazine, 8(6). Retrieved September 29, 2009, from http://www.dlib.org/dlib/june02/bollen/ 06bollen.html
12. Borghuis, M., Elsevier Science, Carnegie-Mellon University, Cornell University, Georgia Institute of Technology, Massachusetts Institute of Technology, et al. (1996). Tulip: Final report. New York: Elsevier Science.
13. Borgman, C.L. (1999). What are digital libraries? Competing visions. Information Processing and Management, 35(3), 227-243.
14. Borgman, C.L. (2002). Final report to the National Science Foundation. In Proceedings of the Fourth DELOS Workshop, evaluation of digital libraries: Testbeds, measurements, and metrics. Retrieved September 29, 2009, from http://www.sztaki.hu/confe.rences/deval/presentations/final-report.html
15. Borgman, C.L., Leazer, G.H., Gilliand-Swetland, A.J., & Gazan, R. (2001). Iterative design and evaluation of a geographic digital library for university students: A case study of the Alexander Digital Earth Prototype (ADEPT). In P.S. Constantopoulos & I. Sølvberg (Eds.), Research and advanced technology for digital libraries: Proceedings of the Fifth European Conference (pp. 390-401). Berlin, Germany: Springer.
16. Bosman, F.J.M., Bruza, P.D., Van de Weide, T.P., & Weusten, L.V.M. (1998). Documentation, cataloging, and query by navigation: A practical and sound approach. In C. Nikolaou & C. Stephanidis (Eds.), Research and advanced technology for digital libraries: Proceedings of the Second European Conference (pp. 459-478). Berlin, Germany: Springer.
17. Brophy, P., Clarke, Z., Brinkley, M., Mundt, S., & Poll, R. (2000). EQUINOX: Library performance measurement and equality management system-Performance indicators for electronic library services. Retrieved September 29, 2009, from http://equ.inox.dcu.ie/reports/pilist.html
18. Browne, P., & Gurrin, C. (2001). Dublin City University video track experiments for TREC 200.1. Retrieved September 29, 2009, from http://trec.nist.gov/pubs/trec.10/papers/dcu-trec01-final.pdf
19. Carter, D.S., & Janes, J. (2000). Unobtrusive data analysis of digital reference questions and service at the Internet Public Library: An exploratory study. Library Trends, 49(2), 251-265.
20. Champeny, L., Borgman, C.L., Leazer, G.H., Gilliand-Swetland, A.J., Millwood, K.A., D'Avolio, L. et al. (2004). Developing a digital learning environment: An evaluation of design and implementation processes. In H.C. Chen, H. Wactlar, C.C., Chen, E.P. Lim, & M. Christel (Eds.), Proceedings of the 2004 Joint ACM/IEEE Conference on Digital Libraries (pp. 37-46). New York: ACM Press.
21. Chowdhury, G.G., Chowdhury, S. (2003). Introduction to digital libraries. London: Facet.
22. Computer Science & Telecommunication Board, National Research Council (1998, July/August). Design and evaluation. A review of the state-of-the-art. D-Lib Magazine, 4. Retrieved September 29, 2009, from http://www.dlib.org/dlib/ july98/nrc/07nrc.html
23. Cullen, R. (2001). Perspectives on user satisfaction surveys. Library Trends, 49(4), 662-686.
24. Dillon, A. (1999). TIME-A multi-level framework for the design and evaluation of digital libraries. International Journal of Digital Libraries, 2 (2/3), 170-177.
25. Ding, W., Marchionini, G., & Soergel, D. (1999). Multimodel surrogates for video browsing. Proceedings of Digital Libraries '99. The Fourth Annual ACM Conference on Digital Libraries, Berkeley, CA, 85-93.
26. Entlich, R., Garson, L., Lesk, M., Normore, L., Olsen, J., & Weibel, S. (1996). Testing a digital library: User response to the CORE project. Library Hi Tech, 14(4), 99-118.
27. Fuhr, N., Hansen, P., Mabe, M., & Micsik, A. (2001). Digital libraries: A generic classification and evaluation scheme. In P.S. Constantopoulos & I. Sølvberg (Eds.), Research and advanced technology for digital libraries: Proceedings of the Fifth European Conference (pp. 187-199). Berlin, Germany: Springer.
28. Fuhr N., Klas, C.P., Schaefer, A., & Mutschke, P. (2002). DAFFODIL: An integrated desktop for supporting high-level search, activities in federated digital libraries. Research and advanced technology for digital libraries. Proceedings of 6th European Conference, Paris, France, 597-612.
29. Glass, G.V., Peckham, P.D., & Sanders, J.R. (1972). Consequences of failure to meet the assumptions underlying the fixed effects analysis of variance and covariance. Review of Educational Research, 42(2), 237-288.
30. Goncalves, M.A., Moreira, B.L., Fox, E.A., & Watson, L.T. (2007). "What is a good digital library?" A quality model for digital libraries. Information Processing and Management, 43(5), 1416-1437.
31. Goodrum, A.A. (2001). Multidimensional scaling of video surrogates. Journal of American Society for Information Science, 52(2), 174-182.
32. Harter, S.P., & Hert, C.A. (1997). Evaluation of information retrieval systems: Approaches, issues, and methods. In M. E. William (Ed.), Annual Review of Information Science and Technology (pp. 3-94). Medford, NJ: Information Today.
33. Hauptmann, A., & Jin, R. (2001). Video retrieval with the Informedia Digital Video System. Retrieved September 29, 2009, from http://trec.nist.gov/ pubs/trec10/papers/CMU-VideoTrack.pdf
34. Hee, M., Yoon, Y.I., & Kim, K.C. (1999). Unified video retrieval systems supporting similarity retrieval. In T.J.M. Bench-Capon, G. Soda, & A.M. Tjoa (Eds.), Proceedings of the Tenth International Workshop on Database and Expert System Applications (pp. 884-888). New York: Springer.
35. Hill, L.L., Carver, L., & Larsgaard, M., Dolin, R., Smith, T.R., Frew, J., et al. (2000). Alexandria Digital Library: User evaluation studies and system design. Journal of the American Society for Information Science, 51(3), 246-259. (Pubitemid 30540239)
36. Huxley, L. (2002). Renardus: Following the Fox from project to service. In M. Agosti & C. Thanos (Eds.), Research and advanced technology for digital libraries: Proceedings of the Sixth European Conference (pp. 157-171). Berlin, Germany: Springer.
37. Isfandyari-Moghaddam, A., & Bayat, B. (2008). Digital libraries in the mirror of the literature: Issues and considerations. Electronic Library, 26(6), 844-862.
38. Jones, G.J.F., & Lam-Adesina, A.M. (2002). An investigation of mixed-media information retrieval. Research and Advanced Technology for Digital Libraries. Proceedings of the 2nd European Conference, Paris, France, 463-478.
39. Jones, M.L.W., Gay, G.K., & Rieger, R.H. (1999). Project soup: Comparing evaluations of digital collection efforts. D-Lib Magazine, 5(11). Retrieved September 29, 2009, from http://dlib.org/dlib/november99/11jones.html
40. Jones, S., Cunningham, S.J., Mcnab, R., & Boddie, S. (2000). A transaction log analysis of a digital library. International Journal of Digital Library, 3(2), 152-169.
41. Jones, S., & Paynter, G.W. (2002). Automatic extraction of document keyphrases for use in digital libraries: Evaluation and application. Journal of the American Society for Information Science & Technology, 53(8), 653-677.
42. Kengeri, R., Seals, C.D., Reddy, H.P., Harley, H.D., & Fox, E.A. (1999). Usability study of digital libraries: ACM, IEEE-CS, NCSTRL, and NDLTD. International Journal on Digital Libraries, 2(2/3), 157-169.
43. Kenney, A.R., Sharpe, L.H., & Berger, B. (1998). Illustrated book study: Digital conversion requirements of printed illustration. In C. Nikolaou & C. Stephanidis (Eds.), Research and advanced technology for digital libraries: Proceedings of the Second European Conference (pp. 279-293). Berlin, Germany: Springer.
44. Khalil, M.A., & Jayatilleke, R. (2000). Digital libraries: Their usage for the end user point of view. Proceedings of the National Online Meeting, New York, NY, 179-187.
45. Khoo, M. (2001). Ethnography, evaluation, and design as integrated strategies: A case study from WES. In P.S. Constantopoulos & I. Sølvberg (Eds.), Research and advanced technology for digital libraries: Proceedings of the Fifth European Conference (pp. 263-274). Berlin, Germany: Springer.
46. Khoo, M., Devaul, H., & Sumner, T. (2002). Functional requirements for online tools to support community-led collections building. In M. Agosti & C. Thanos (Eds.), Research and advanced technology for digital libraries: Proceedings of the Sixth European Conference (pp. 190-203). Berlin, Germany: Springer.
47. Kwak, B.H., Jun, W., & Gruenwald, L. (2002). A study on the evaluation model for university libraries in digital environments. In M. Agosti & C. Thanos (Eds.), Research and advanced technology for digital libraries: Proceedings of the Sixth European Conference (pp. 204-217). Berlin, Germany: Springer.
48. Kyrillidou, M., & Giersch, S. (2005). Developing the DigiQUAL protocol for digital library evaluation. In M. Marlino, T. Summer, & F. Shipman (Eds.), Proceedings of the Fifth ACM/IEEE-CS Joint Conference on Digital Libraries (pp. 172-173). New York: ACM Press.
49. Lankes, R.D., Gross, M., & McClure, C.R. (2003). Cost, statistics, measures, and standards for digital reference services: A preliminary view. Library Trends, 51(3), 401-413. (Pubitemid 36703705)
50. Larsen, R. (2000). The DLib test suite and metrics working group: Harvesting the experience from the digital library initiative. Retrieved on April 13, 2004, from http://www.dlib.org/metrics/public/papers/The-Dlib-Test- Suite-and-Metrics.pdf
51. Lindlof, T.R. (1995). Qualitative communication research methods. Thousand Oaks, CA: Sage.
52. Lynch, C. (2003). Colliding with the real world: Heresies and unexplored questions about audience, economics, and control of digital libraries. In A.P. Bishop, A.A. Van House, & B.P. Buttenfield (Eds.), Digital library use: Social practice in design and evaluation (pp. 191-216). Cambridge MA: The MIT Press.
53. Marchionini, G. (2000). Evaluation digital libraries: A longitudinal and multifaceted view. Library Trends, 49(2), 304-333.
54. Marchionini, G., Plaisant, C., & Komlodi, A. (2003). The people in digital libraries: Multifaceted approaches to assessing needs and impact. In P. Ann Biship et al. (Eds.), Digital library use: Social practice in design and evaluation (pp.119-160). Cambridge MA: The MIT Press.
55. Mead, J.P., & Gay, G. (1995). Concept mapping: An innovative approach to digital library design and evaluation. ACM SIGOIS Bulletin, 16(2), 10-14.
56. Meyyappan, N., Foo, S., & Chowdhury, G.G. (2004). Design and evaluation of a task-based digital library for the academic community. Journal of Documentation, 60(4), 449-475. (Pubitemid 39182229)
57. Nicholson, S. (2004). A conceptual framework for the holistic measurement and cumulative evaluation of library services. Journal of Documentation, 60(2), 164-182.
58. Nielsen, J. (1993). Usability engineering. Boston MA: Academic Press.
59. O'Day, V.L., & Nardi, B.A. (2003). An ecological perspective on digital libraries. In A.P. Bishop, A.A. Van House, & B.P. Buttenfield (Eds.), Digital library use: Social practice in design and evaluation (pp. 65-82). Cambridge MA: The MIT Press.
60. Papadakis, I., Andreou, I., & Chrissikopoulos, V. (2002). Interactive search results. In M. Agosti & C. Thanos (Eds.), Research and advanced technology for digital libraries: Proceedings of the Sixth European Conference (pp. 448-462). Berlin, Germany: Springer.
61. Park, S. (2000). Usability, user preference, effectiveness, and user behaviors when searching individual and integrated full-text databases: Implication for digital libraries. Journal of the American Society for Information Science, 51(5), 456-468.
62. Peng, L.K., Ramaiah, C.K., & Foo, S. (2004). Heuristic-based user interface evaluation at Nanyang Technological University in Singapore. Program-Electronic Library and Information System, 38(1), 42-59.
63. Places, A.S., Brisaboa, N.R., & Farina, A., Luaces, M.R., Parama, J.R., & Penabad, M.R. (2007). The Galician virtual library. Online Information Review, 31(3), 333-352.
64. Prown, S. (1999). "Detecting 'broke': usability testing of library Web sites." Retrieved on December 12, 2004, at http://www.library.yale.edu/ ~prowns/nebic/nebictalk.html
65. Rui, Y., Gupta, A., & Acero, A. (2000). Automatically extracting high-lights for TV baseball programs. In S. Ghandeharizadeh, S.F. Chang, S. Fischer, J.A. Konstan, & N. Nahrstedt (Eds.), Proceedings of the Eight ACM International Conference on Multimedia (pp. 105-115). New York: ACM Press.
66. Salampasis, M., & Diamantaras, K.I. (2002). Experimental user-centered evaluation of an open hypermedia system and Web information seeking environments. Journal of Digital Information, 2(4). Retrieved September 29, 2009, from http://journals.td1.org/jodi/article/viewArticle/57/60
67. Sanderson, M., & Crestani, F. (1998). Mixing and merging for spoken document retrieval. In C. Nikolaou & C. Stephanidis (Eds.), Research and advanced technology for digital libraries: Proceedings of the Second European Conference (pp. 397-407). Berlin, Germany: Springer.
68. Saracevic, T. (1996). Modeling Interaction in information retrieval (IR) - A review and proposal. Proceedings of the 59th Annual Meeting of American Society for Information Science, 33, 3-9. (Pubitemid 126730536)
69. Saracevic, T. (1997). The stratified model of information retrieval interaction. Extension and approaches. Proceedings of the 60th Annual Meeting of the American Society for Information Science, 313-327.
70. Saracevic, T. (2000). Digital library evaluation: Toward an evolution of concepts. Library Trends, 49(2), 350-369. (Pubitemid 33242547)
71. Shachaf, P., Oltmann, S.M., & Horowitz, S.M. (2008). Service equality in virtual reference. Journal of the American Society for Information Science and Technology, 59(4), 535-550.
72. Shim, W. (2000). Measuring services, resources, users, and use in the networked environment. Retrieved September 29, 2009, from http://www.ar1.org/ bm~doc/emetrics-2/pdf
73. Sumner, T., & Dawe, M. (2001). Looking at digital library usability from a reuse perspective. In E.A. Fox & C.L. Borgman (Eds.), Proceedings of the First ACM/IEEE Joint Conference on Digital Libraries (pp. 416-425). New York: ACM Press.
74. Thong, J.Y.L., Hong, W.Y., & Tam, K.Y. (2002). Understanding user acceptance of digital libraries: What are the roles of interface characteristics, organizational context, and individual differences? International Journal of Human-Computer Studies, 57(3), 215-242.
75. Tsakonas, G., & Papatheodorou, C. (2008). Exploring usefulness and usability in the evaluation of open access digital libraries. Information Processing and Management, 44(3), 1234-1250.
76. Van House, N.A., Butler, M., & Schiff, L. (1996). Needs assessment and evaluation of a digital environmental library: The Berkeley experience. Retrieved September 29, 2009, from http://people.ischool.berkeley.edu/~vanhouse/ d196.html
77. Waters, D. (1998). DLF Annual Report 1998-1999: Introduction. Retrieved on September 1, 2005, from http://www.diglib.org/AR9899p1.html
78. Wesson, J., & Greunen, D.V. (2002). Visualization of usability data: Measuring task efficiency. In P. Kotzé, L. Venter, & J. Barrow (Eds.), Proceedings of the 2002 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists on Enablement through Technology (pp. 1.1-18). New York: ACM Press.
79. White, M.D. (2001). Digital reference services: framework for analysis and evaluation. Library & Information Science Research, 23(3), 211-231. (Pubitemid 33622958)
80. Wildemuth, B., Marchionini, G., Yang, M., Geisler, G., Wilkens, T., Hughes, A., et al. (2003). How fast is too fast? Evaluating fast forward surrogates for digital video. Proceedings of the ACM/IEEE Joint Conference on Research on Digital Libraries, Los Alamitos, CA: IEEE. pp. 221-230.
81. Wilson, R., & Landoni, M. (2001). Evaluating electronic textbooks: A methodology. In P.S. Constantopoulos & I. Sølvberg (Eds.), Research and advanced technology for digital libraries: Proceedings of the Fifth European Conference (pp. 1-12). Berlin, Germany: Springer.
82. Xie, H.I., & Wolfram, D. (2002). State digital library usability: Contributing organizational factors. Journal of the American Society for Information Science & Technology, 53(13), 1085-1097.
83. Xie, H.I. (2006). Evaluation of digital libraries: Criteria and problems from users' perspectives. Library and Information Science Research, 28(3), 433-452.
84. Xie, H.I. (2008). Users' evaluation of digital libraries (DLs): Their uses, their criteria, and their assessment. Information Processing and Management, 44(3), 1346-1373.
85. Zhang, H.J., Low, C.Y., Smoliar, S.W., & Wu, J.H. (1995). Video parsing, retrieval and browsing: And integrated and content-based solution. Proceedings of the Third ACM International Conference on Multimedia, CA: San Francisco, 15-24.
86. Zhang, Y. (2004). Moving image collections evaluation-Final report. Retrieved September 29, 2009, from http://comminfo.rutgers.edu/miceval/progress/ final-report/doc
87. Zhang, Y., & Li, Y.L. (2008). A user-centered functional metadata evaluation on moving image collections. Journal of American Society for Information Science and Technology, 59(8), 1331-1346.
88. Zhang, X.M., Li, Y.L., Liu, J.J., & Zhang, Y. (2008). Effects of interaction design in digital libraries on user interactions. Journal of Documentation, 64(3), 438-463.