Evaluation of interactive knowledge-based systems: Overview and design for empirical testing

Journal of the American Society for Information Science

Volumn 47, Issue 1, 1996, Pages 57-69

  Lancaster, F W ^a   Ulvila, Jacob W ^b   Humphrey, Susanne M ^c   Smith, Linda C ^a   Allen, Bryce ^a   Herner, Saul ^d  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b Decision Science Associates   (United States)

^c NATIONAL INSTITUTES OF HEALTH   (United States)

^d Herner and Company   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DESIGN; EXPERT SYSTEMS; INDEXING (OF INFORMATION); INFORMATION SERVICES; INTERACTIVE COMPUTER SYSTEMS; PERFORMANCE; TESTING;

EMPIRICAL TESTING; INDEXING SYSTEMS;

KNOWLEDGE BASED SYSTEMS;

EID: 0029755208     PISSN: 00028231     EISSN: None     Source Type: Journal    
DOI: 10.1002/(SICI)1097-4571(199601)47:1<57::AID-ASI6>3.0.CO;2-1     Document Type: Review

References (52)

1. Adelman, L. (1992). Evaluating decision support and expert systems. New York: Wiley.
2. Adelman, L., & Ulvila, J. W. (1991). Evaluating expert system technology. In S. J. Andriole & S. M. Halpin (Eds.), Information technology for command and control: Methods and tools for systems development and evaluation (pp. 537-547). New York: IEEE Press.
3. 3 I systems that employ AI, Volume 1: Handbook for testing expert systems (Technical Report 90-9). Reston, VA: Decision Science Consortium. (NTIS No. AD-A266 358/1)
4. Alberico, R., & Micco, M. (1990). Expert systems for reference and information retrieval. Westport, CT: Meckler.
5. Bahill, A.T. (1991). Verifying and validating personal computer-based expert systems. Englewood Cliffs, NJ: Prentice Hall.
6. Bailey, C. W., Jr. (1992). The intelligent reference information system project: A merger of CD-ROM LAN and expert system technologies. Information Technology and Libraries, 11, 237-243.
7. Berner, E. S., Webster, G. D., Shugerman, A. A., Jackson, J. R., Algina, J., Baker, A. L., Ball, E. V., Cobbs, C. G., Dennis, V. W., Frenkel, E. P., Hudson, L. D., Mancall, E. L., Rackley, C. E., & Taundon, O. D. (1994). Performance of four computer-based diagnostic systems. The New England Journal of Medicine, 330, 1792-1796.
8. Brachman, R. J., Amarel, S., Engelman, C., Engelmore, R. S., Feigenbaum. E. A., & Wilkins, D. E. (1983). What are expert systems? In F. Hayes-Roth, D. A. Waterman & D. B. Lenat (Eds.), Building expert systems (pp. 31-57). Reading, MA: Addison-Wesley.
9. Constantine, M. M., & Ulvila, J. W. (1990). Testing knowledge-based systems: The state of the practice and suggestions for improvement. Expert Systems with Applications, 1, 237-248.
10. Cosgrove, S. J., & Weimann, J. M. (1992). Expert system technology applied to item classification. Library Hi Tech, 10(1-2), 33-40.
11. Davies, R., Smith, A. G., & Morris, A. (1992). Expert systemsin reference work. In A. Morris (Ed.), The application of expert systems in libraries and information centres (pp. 91-132). London: Bowker-Saur.
12. DeMillo, R. A., McCracken, W. M., Martin, R. J., & Passafiume, J. F. (1987). Software testing and evaluation. Menlo Park, CA: Benjamin/Cummings.
13. Fairley, R. E. (1985). Software engineering concepts. New York: McGraw-Hill.
14. Franklin, W. R., Bansal, R., Gilbert, E., & Shroff, G. (1988). Debugging and tracing expert systems. In B. R. Konsynski (Ed.), Decision support and knowledge based systems track: Proceedings of the Twenty-First Annual Hawaii International Conference on System Sciences, Vol. 3 (pp. 159-167). Washington, DC: IEEE Computer Society.
15. Hawks, C. P. (1994). Expert systems in technical services and collection management. Information Technology and Libraries, 13, 203-212.
16. Haves, P. J. (1981). The logic of frames. In B. L. Webber & N. J. Nilsson (Eds.), Readings in artificial intelligence (pp. 451-458). Palo Alto, CA: Tioga, 1981. [Also: (1985). Los Altos, CA: Morgan Kaufmann].
17. Hayes-Roth, F. (1992). Knowledge systems: An introduction. Library Hi Tech, 10(1-2), 15-32. [Also: (1992). Expert systems. In S. C. Shapiro (Ed.), Encyclopedia of artificial intelligence (2nd ed.), Vol. 1 (pp. 477-489). New York: Wiley.]
18. Hayes-Roth, F. (1992). Knowledge systems: An introduction. Library Hi Tech, 10(1-2), 15-32. [Also: (1992). Expert systems. In S. C. Shapiro (Ed.), Encyclopedia of artificial intelligence (2nd ed.), Vol. 1 (pp. 477-489). New York: Wiley.]
19. Herner, S., Lancaster, F. W., Smith, L. C., Allen, B., & Ulvila, J. (1993). Design for a study to evaluate the MedIndEx approach to subject indexing (LHNCBC 93-3). Bethesda, MD: National Library of Medicine. (NTIS No. PB93-236016)
20. Hu, C. (1987). An evaluation of online database selection by a gateway system with artificial intelligence techniques. Unpublished doctoral dissertation, Graduate School of Library and Information Science, University of Illinois at Urbana-Champaign.
21. Humphrey, S. M. (1992a). Interactive knowledge-based systems for improved subject analysis and retrieval. In F. W. Lancaster & L. C. Smith (Eds.), Artificial intelligence and expert systems: Will they change the library? Proceedings of the 27th Annual Clinic on Library Applications of Data Processing (pp. 83-117). Urbana, IL: University of Illinois at Urbana-Champaign, Graduate School of Library and Information Science.
22. Humphrey, S. M. (1992b). Indexing biomedical documents: From thesaural to knowledge-based retrieval systems. Artificial Intelligence in Medicine, 4, 343-371.
23. Humphrey, S. M. (1994). Knowledge-based systems for indexing. In R. Fidel, T. Bellardo Hahn, E. M. Rasmussen & P. J. Smith (Eds.), Challenges in indexing electronic text and images (pp. 161-175). Medford, NJ: Learned Information.
24. Jeng, L.-H. (1986). An expert system for determining title proper in descriptive cataloging: A conceptual model. Cataloging & Classification Quarterly, 7(2), 55-70.
25. Johnston, M., & Weckert, J. (1990). Selection advisor: An expert system for collection management. Information Technology and Libruries, 9, 219-225.
26. Kang, Y., & Bahill, A. T. (1990). A tool for detecting expert system errors. AI Expert, 5(2), 46-51.
27. Kassirer, J. P. (1994). A report card on computer-assisted diagnosis - the grade: C. The New England Journal of Medicine, 330, 1824-1825.
28. Kirk, R. E. (1982). Experimental design: Procedures for the behavioral sciences (2nd ed.). Monterey, CA: Brooks/Cole.
29. Leonard, L. E. (1975). Inter-indexer consistency and retrieval effectiveness: Measurement of relationships. Unpublished doctoral dissertation, Graduate School of Library and Information Science, University of Illinois at Urbana-Champaign.
30. Meador, J. M., & Cline, L. (1992). Displaying and utilizing selection tools in a user-friendly electronic environment. Library Acquisitions: Practice & Theory, 16, 289-294.
31. Miller, P. L., & Sittig, D. F. (1990). The evaluation of clinical decision support systems: What is necessary versus what is interesting. Medical informatics [London], 15, 185-190.
32. Nguyen, T. A., Perkins, W. A., Laffey, T. J., & Pecora, D. (1987). Knowledge base verification. AI Magazine, 8(2), 69-75.
33. O'Keefe, R. M. (1989). The evaluation of decision-aiding systems: Guidelines and methods. Information & Management, 17, 217-226.
34. Parrott, J. R. (1986). Expert systems for reference work. Microcomputers for Information Management, 3, 155-171.
35. Parrott, J. R. (1989). Simulation of the reference process, Part 2: REFSIM, an implementation with expert system and ICAI modes. Reference Librarian, 23, 153-176.
36. Parrott, J. R. (1992). Reference expert systems: Foundations in reference theory. In F. W. Lancaster & L. C. Smith (Eds.), Artificial intelligence and expert systems: Will they change the library? Proceedings of the 27th Annual Clinic on Library Applications of Data Processing (pp. 118-160). Urbana, IL: University of Illinois at Urbana-Champaign, Graduate School of Library and Information Science.
37. Pontigo, J., Tovar-Reyes, E., Rodriguez, G., & Ortiz-Gama, S. (1992). Expert systems in document delivery: The feasibility of learning capabilities. In F. W. Lancaster & L. C. Smith (Eds.), Artificial intelligence and expert systems: Will they change the library? Proceedings of the 27th Annual Clinic on Library Applications of Data Processing (pp. 255-268). Urbana, IL: University of Illinois at Urbana-Champaign, Graduate School of Library and Information Science.
38. Poulter, A., Morris, A., & Dow, J. (1994). LIS professionals as knowledge engineers. Annual Review of Information Science and Technology, 29, 305-350.
39. Rada, R., Backus, J., Giampa, T., Gibbs, C. & Goel, S. (1987). Computerized guides to journal selection. Information Technology and Libraries, 6, 173-184.
40. Richardson, J., Jr. (1989). Toward an expert system for reference service: A research agenda for the 1990s. College & Research Libraries, 50, 231-248.
41. Rolling, L. (1981). Indexing consistency, quality and efficiency. Information Processing & Management, 17, 69-76.
42. Rushby, J. (1988). Quality measures and assurance for AI software (NASA Contractor Report 4187). Washington, DC: National Aeronautics and Space Administration (Code NTT-4). (NTIS No. N88-30304/5)
43. Shortlifle, E. H., & Clancey, W. J. (1984). Anticipating the second decade. In W. J. Clancey & E. H. Shortliffe (Eds.), Readings in medical artificial intelligence: The first decade (pp. 463-472). Reading, MA: Addison-Wesley.
44. Sowell, S. L. (1989). Expanding horizons in collection development with expert systems: Development and testing of a demonstration prototype. Special Libraries, 80, 45-50.
45. Stachowitz, R. A., & Combs, J. B. (1987). Validation of expert systems. In E. A. Stohr, L. Hoevel, L. Haynes, Y. Chu, A. Speckhard, B. D. Shriver, R. R. Grams, & R. H. Sprague, Jr. (Eds.), Proceedings of the Twentieth Annual Hawaii International Conference on System Sciences, Vol. 1 (pp. 686-695). Honolulu, HI: Hawaii International Conference on System Sciences.
46. 3 I systems that employ AI (Tech. Rep. No. 87-7). Falls Church, VA: Decision Science Consortium.
47. van Brakel, P. A. (1988). Evaluating an intelligent gateway: A methodology. South African Journal of Library and Information Science, 56, 277-288.
48. Vedder, R. G., Fortin, M. G., Lemmermann, S. A., & Johnson, R. N. (1989). Five PC-based expert systems for business reference: An evaluation. Information Technology and Libraries, 8, 42-54.
49. Vickery, B. C. (1992). Intelligent interfaces to online databases. In F. W. Lancaster & L. C. Smith (Eds.), Artificial intelligence and expert systems: Will they change the library? Proceedings of the 27th Annual Clinic on Library Applications of Data Processing (pp. 240-254). Urbana, IL: University of Illinois at Urbana-Champaign, Graduate School of Library and Information Science.
50. Vinze, A. S., Vogel, D. R., & Nunamaker, J. F., Jr. (1991). Performance evaluation of a knowledge-based system: A validation study. Information & Management, 21, 225-235.
51. Weibel, S. (1992). Automated cataloging: Implications for libraries and patrons. In F. W. Lancaster & L. C. Smith (Eds.), Artificial intelligence and expert systems: Will they change the library? Proceedings of the 27th Annual Clinic on Library Applications of Data Processing (pp. 69-82). Urbana, IL: University of Illinois at Urbana-Champaign, Graduate School of Library and Information Science.
52. Wyatt, J., & Spiegelhalter, D. (1990). Evaluating medical expert systems: What to test and how? Medical Informatics [London], 15, 205-217.