Reference standards, judges, and comparison subjects: Roles for experts in evaluating system performance

Journal of the American Medical Informatics Association

Volumn 9, Issue 1, 2002, Pages 1-15

  Hripcsak, George ^a   Wilcox, Adam ^a  

^a Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EXPERT SYSTEM; MEDICAL INFORMATICS; MEDICAL INFORMATION SYSTEM; PERFORMANCE; RELIABILITY; REVIEW; STANDARD; VALIDATION PROCESS;

EID: 0036177840     PISSN: 10675027     EISSN: None     Source Type: Journal    
DOI: 10.1136/jamia.2002.0090001     Document Type: Review

References (87)

1. Evaluation of ECG interpretation results obtained by computer and cardiologists
2. Issues in the evaluation of artificial intelligence systems in medicine
3. The diagnostic performance of computer programs for the interpretation of electrocardiograms
4. Reference standards for software evaluation
5. Performance of four computer-based diagnostic systems
6. The Delphi method used to validate diagnostic knowledge in a computerised ECG interpreter
7. Computer-aided assessment of transient ischemic attacks
8. The evaluation of medical expert systems: Experience with the AI/RHEUM knowledge-based consultant system in rheumatology
9. User variability in abstracting and entering printed case histories with Quick Medical Reference (QMR)
10. A computer protocol to predict myocardial infarction in emergency department patients with chest pain
11. Unlocking clinical data from narrative reports: A study of natural language processing
12. Extracting findings from narrative reports: Software transferability and sources of physician disagreement
13. Comparison of computer-aided and human review of general practitioners' management of hypertension
14. Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence
15. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests
16. The meaning and use of the area under the receiver operating characteristic (ROC) curve
17. Measuring the accuracy of diagnostic systems
18. The area under the ROC curve and its competitors
19. Basic principles of ROC analysis
20. Evaluating the yield of medical tests
21. Online access to MEDLINE in clinical settings
22. A performance and failure analysis of SAPHIRE with a MEDLINE test collection
23. Bayesian modeling for linking causally related observations in chest X-ray reports
24. Automatic extraction of PIOPED interpretations from ventilation/perfusion lung scan reports
25. Comparing expert systems for identifying chest X-ray reports that support pneumonia
26. Natural language processing of asthma discharge summaries for the monitoring of patient care
27. Natural language processing and the representation of clinical data
28. An automated indexing system utilizing semantic net expansion
29. Evaluating message understanding systems: An analysis of the Third Message Understanding Conference (MUC-3)
30. A free-text processing system to capture physician findings: Canonical phrase identification system (CAPIS)
31. Computerized extraction of coded findings from free-text radiologic reports
32. MUC-4 evaluation metrics
33. Evaluation of medical expert systems: A case study in performance assessment
34. Two applications of statistical modelling to natural language processing
35. The measurement of performance in probabilistic diagnosis, part V: General recommendations
36. Short-term mortality predictions for critically ill hospitalised patients: Science and ethics
37. Measuring agreement between two judges on the presence or absence of a trait
38. Assessing agreement on classification tasks: The kappa statistic
39. Evaluation of the quality of information retrieval of clinical findings from a computerized patient database using a semantic terminological model
40. Automated linkage of free-text descriptions of patients with a practice guideline
41. Evaluating consensus among physicians in medical knowledge base construction
42. Validation of the medical expert system PNEUMON-IA
43. Evaluation of a new method for cardiovascular reasoning
44. The treatment advice of a computer-based cancer chemotherapy protocol advisor
45. Antimicrobial selection by a computer: A blinded evaluation by infectious diseases experts
46. A performance evaluation of the expert system ANEMIA
47. Validation of the AI/RHEUM knowledge base with data from consecutive rheumatological outpatients
48. Identifying adverse drug events: Development of a computer-based monitor and comparison with chart review and stimulated voluntary report
49. Performances of 27 MEDLINE systems tested by searches with clinical questions
50. An experiment comparing lexical and statistical methods for extracting MeSH terms from clinical free text
51. Phase II evaluation of clinical coding schemes: Completeness, taxonomy, mapping, definitions, and clarity
52. Evaluation of a semantic data model for chest radiology: Application of a new methodology
53. Automatic detection of acute bacterial pneumonia from chest x-ray reports
54. INTERNIST-1, an experimental computer-based diagnostic consultant for general practice internal medicine
55. Monitoring free-text data using medical language processing
56. Human and computer-aided diagnosis of abdominal pain: Further report with emphasis on performance of clinicians
57. Use of an artificial neural network for the diagnosis of myocardial infarction
58. Validation of the medical expert system RENOIR
59. Controlling for chance agreement in the validation of medical expert systems with no gold standard: PNEUMON-IA and RENOIR revisited
60. A reliability study for evaluating information extraction from radiology reports
61. Generalizability theory
62. The statistical assessment of the variability in observer perception and description of roentgenographic pulmonary shadows
63. Comparing human and machine performance for natural language information extraction: Results for English microelectronics from the MUC-5 evaluation
64. Predicting one-year outcome following acute myocardial infarction: Physicians versus computers
65. Clinical prediction rules: Applications and methodological standards
66. On the heuristic nature of medical decision-support systems
67. Development and evaluation of a computerized admission diagnosis encoding system
68. Evaluation of clinical decision aids-more to think about
69. Representing expertise in a computer-program: The serum-protein diagnostic program
70. Effects of a decision support system on the diagnostic accuracy of users: A preliminary report
71. Automatic prediction of trauma registry procedure codes from emergency room dictations
72. Toward a measured approach to medical informatics
73. Optimal document-indexing vocabulary for MEDLINE
74. Information needs of health care professionals in an AIDS outpatient clinic as determined by chart review
75. An overview of the objective structured clinical examination
76. The objective structured clinical examination: A step in the direction of competency-based evaluation
77. Analyzing the sources of unreliability in fixed prosthodontics mock board examinations