Validation of intelligent systems: A critical study and a tool

Expert Systems with Applications

Volumn 18, Issue 1, 2000, Pages 1-16

  Mosqueira Rey, E ^a   Moret Bonillo, V ^a  

^a UNIVERSITY OF A CORUÑA   (Spain)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER AIDED SOFTWARE ENGINEERING; COMPUTER SYSTEMS PROGRAMMING;

INTELLIGENT SYSTEMS; SOFTWARE PACKAGE SHIVA;

ARTIFICIAL INTELLIGENCE;

EID: 0033691443     PISSN: 09574174     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0957-4174(99)00045-7     Document Type: Article

References (45)

1. Alonso-Betanzos A., Devoe L.D., Castillo R.A., Moret-Bonillo V., Hernández-Sande C., Searle N.S. FOETOS in clinical practice: a retrospective analysis of its performance. Artificial Intelligence in Medicine. 1:1989;93-99.
2. Alonso-Betanzos A., Guijarro-Berdiñas B., Moret-Bonillo V., López-González S. The NST-EXPERT project: the need to evolve. Artificial Intelligence in Medicine. 7:1995;297-313.
3. Alonso-Betanzos A., Mosqueira-Rey E., Moret-Bonillo V., Baldonedo del Río B. Applying statistical, uncertainty-based and connectionist approaches to the prediction of fetal outcome: a comparative study. Artificial Intelligence in Medicine. 1998.
4. Bahill A.T., Bharathan K., Curlee R.F. How the testing techniques for a decision support system changed over nine years. IEEE Transactions on Systems, Man and Cybernetics. 25:(12):1995;1533-1542.
5. Boehm B.W. Software engineering economics. 1981;Prentice-Hall, Englewood Cliffs, NJ.
6. Boehm B.W. A spiral model of software development and enhancement. Computer. May:1988;61-72.
7. Borg I., Groenen P. Modern multidimensional scaling. 1997;Springer, New York.
8. Cardeñosa J., Juristo N., Morant J.L., Pazos J. The knowledge engineer in the knowledge industry. The Journal of Knowledge Engineering. 4:(2):1991;43-53.
9. Chandon J.L., De Soete G. Fitting a least squares ultrametric to dissimilarity data: approximation versus optimization. Diday E. Data analysis and informatics. III:1984;213-221 Elsevier, Amsterdam.
10. Chandon J.L., Lemaire J., Pouget J. Construction de l'ultramétrique la plus proche d'une dissimilarité au sens des moindres carrés. R.A.I.R.O. Recherche opérationelle. 14:1980;157-170.
11. Chandrasekaran B. On evaluating AI systems for medical diagnosis. Artificial Intelligence Magazine. 4:(2):1983;34-37.
12. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 20:1960;37-46.
13. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin. 70:(4):1968;213-220.
14. De Soete G. A least squares algorithm for fitting an ultrametric tree to a dissimilarity matrix. Pattern Recognition Letters. 2:1984;133-137.
15. Donker D.K., Hasman A., Van Geijin H.P. Kappa statistics: what does it say? Lun K.C. MEDINFO'92. 1992;Elsevier, Amsterdam.
16. Dubes R.C. Cluster analysis and related issues. Chen C.H., Pau L.F., Wang P.S.P. Handbook of pattern recognition and computer vision. 1993;3-32 World Scientific, Singapore.
17. Everitt B.S. Cluster analysis. 1993;Edward Arnold, London.
18. Ginsberg A., Weiss S. SEEK2: a generalized approach to automatic knowledge base refinement. Proceedings of the IJCAI. 1985;367-374.
19. Gupta U.G. Validation and verification of knowledge-based systems: a survey. Journal of Applied Intelligence. 3:1993;343-363.
20. Hubert, L., Arabie, P., & Meulman, J. (1999). Combinatorial data analysis: optimization by dynamic programming [On-line]. URL: ftp://www.psych.uiuc.edu/pub/lhubert/.
21. Kendall M.G., Gibbons J.D. Rank correlation methods. 1990;Edward Arnold, London.
22. Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics. 33:1977;159-174.
23. Lee S., O'Keefe R.M. Developing a strategy for expert system verification and validation. IEEE Transactions on Systems, Man and Cybernetics. 24:(4):1994;643-655.
24. Liebowitz J. Useful approach for evaluating expert systems. Expert Systems. 3:(2):1986;86-96.
25. López, B., Meseguer, P., & Plaza, E. (1990). Knowledge based systems validation: a state of the art, AICOM, 3(2).
26. Lozano J.A., Larrañaga P. Applying genetic algorithms to search for the best hierarchical clustering of a dataset. Pattern Recognition Letters. 1999.
27. Mengshoel O.J. Knowledge validation: principles and practice. IEEE Expert. 8:1993;62-68.
28. Moret-Bonillo V., Alonso-Betanzos A., García E., Cabrero-Canosa M., Guijarro-Berdiñas B. The PATRICIA project: a semantic-based methodology for intelligent monitoring in the ICU. IEEE Engineering and Medical Biology Magazine. 14:1993;59-68.
29. Moret-Bonillo V., Mosqueira-Rey E., Alonso-Betanzos A. Information analysis and validation of intelligent monitoring systems in intensive care units. IEEE Transactions on Information Technology in Biomedicine. 1:(2):1997;87-99.
30. Morris J. Software engineering and AI. ACM SIGART Newsletter. 92:1985;1985.
31. Mosqueira-Rey, E. (1998). SHIVA: Un Sistema Heurístico e Integrado para la Validación de Sistemas Inteligentes, PhD dissertation, University of A Coruña, Spain.
32. Murrel S., Plant R.T. A survey of tools for the validation and verification of knowledge-based systems: 1985-1995. Decision Support Systems. 21:1997;307-323.
33. Nguyen T.A., Walton A.P., Laffey T.J., Pecora D. Knowledge base verification. Artificial Intelligence Magazine. 8:(2):1987;69-75.
34. Noblett L.M., Jones C.M. Knowledge-based system development: the need for a methodology. The Journal of Knowledge Engineering. 4:(4):1991;39-46.
35. O'Keefe R.M., Balci O., Smith E.P. Validating expert system performance. IEEE Expert. 2:(4):1987;81-89.
36. O'Leary D.E. Validation of expert systems. Decision Sciences. 18:(3):1987;468-486.
37. Politakis P. Empirical analysis of expert systems. Research Notes in Artificial Intelligence. 6:1985;Pitman, London.
38. Preece A.D., Shinghal R., Batarekh A. Verifying expert systems: a logical framework and a practical tool. Expert Systems with Applications. 5:1992;421-436.
39. Sackman H. Delphi assessment: expert opinion, forecasting and group process. 1974;Rand Corporation, Santa Monica, CA.
40. Shapiro A. The evaluation of clinical predictions. New England Journal of Medicine. 296:1977;1509-1514.
41. Stachowitz R.A., Combs J.B. Validation of expert systems. The Proceedings of the 20th Hawaii International Conference on System Sciences (HICSS). 1:1987;689-695.
42. Swets J.A. Measuring the accuracy of diagnostic systems. Science. 240:1988;1285-1293.
43. Torgerson W.S. Theory and methods of scaling. 1958;Wiley, New York.
44. Williams G.W. Comparing the joint agreement of several raters with another rater. Biometrics. 32:1976;619-627.
45. Zlatareva N.P. A refinement framework to support validation and maintenance of knowledge-based systems. Expert Systems with Applications. 15:1998;245-252.