The diagnostic performance of computer programs for the interpretation of electrocardiograms

New England Journal of Medicine

Volumn 325, Issue 25, 1991, Pages 1767-1773

  Willems, Jos L ^a   Abreu Lima, Cassiano ^b   Arnaud, Pierre ^c   Van Bemmel, Jan H ^d   Brohet, Christian ^e   Degani, Rosanna ^f   Denis, Bernard ^g   Gehring, Jürgen ^h   Graham, Ian ⁱ   Van Herpen, Gerard ^j   Machado, Hilario ^k   Macfarlane, Peter W ^l   Michaelis, Jörg ^m   Moulopoulos, Spyridon D ⁿ   Rubel, Paul ^o   Zywietz, Christoph ^p  

^a UNIVERSITY OF LEUVEN   (Belgium)

^b UNIVERSITY OF PORTO   (Portugal)

^c INSERM   (France)

^d ERASMUS UNIVERSITY ROTTERDAM   (Netherlands)

^e UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

^f CNR   (Italy)

^g UNIV GRENOBLE ALPES   (France)

^h Klinik Hohenried   (Germany)

ⁱ ADELAIDE AND MEATH HOSPITAL   (Ireland)

^j LEIDEN UNIVERSITY   (Netherlands)

^k HOSPITAL DE SANTA MARIA   (Portugal)

^l UNIVERSITY OF GLASGOW   (United Kingdom)

^m JOHANNES GUTENBERG UNIVERSITY   (Germany)

ⁿ UNIVERSITY OF ATHENS   (Greece)

^o UNIVERSITÉ DE LYON   (France)

^p HANNOVER MEDICAL SCHOOL   (Germany)

more..

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; COMPUTER ANALYSIS; COMPUTER PROGRAM; CONTROLLED STUDY; ELECTROCARDIOGRAM; FEMALE; HEART INFARCTION; HEART LEFT VENTRICLE HYPERTROPHY; HEART RIGHT VENTRICLE HYPERTROPHY; HUMAN; INTERMETHOD COMPARISON; MAJOR CLINICAL STUDY; MALE; PHYSICIAN; PRIORITY JOURNAL;

EID: 0026410610     PISSN: 00284793     EISSN: 15334406     Source Type: Journal    
DOI: 10.1056/NEJM199112193252503     Document Type: Article

References (48)

1. See NAPS document no. 04914 for six pages of supplementary material
2. Drazen E., Mann N., Borun R., Laks M., Berson A. Survey of computer assisted electrocardiography in the United States. J Electrocardiol 1988; 21: Suppl:S98-S104.
3. Bruce R.A., Yarnall S.R. Reliability and normal variations of computer analysis of Frank electrocardiograms by the Smith-Hyde program (1968 version). Am J Cardiol 1972; 29:389-396
4. Caceres C.A., Hochberg H.M. Performance of the computer and physician in the analysis of the electrocardiogram. Am Heart J 1970; 79:439-443
5. Pipberger H.V., Cornfield J. What ECG-computer program to choose for clinical application: the need for consumer protection. Circulation 1973; 47:918-920
6. Bailey J.J., Itscoitz S.B., Hirshfield J.W. Jr., Grauer L.E., Horton M.B. A method for evaluating computer programs for electrocardiographic interpretation. I. Application to the experimental IBM program of 1971. Circulation 1974; 50:73-79
7. Meyer J., Heinrich K.W., Merx W., Effert S. Computeranalyse des Elektrocardiograms mit verschiedenen Programmen. Dtsch Med Wochenschr 1974; 23:1213-1223
8. Smets P., Willems J.L., Talmon J., De Maertelaer V., Kornreich F. Methodology for the comparison of various diagnostic procedures. Biom Praximetrie 1975; 15:3-35.
9. How reliable is the computerized electrocardiographic interpretation? JAMA 1975; 231:1090.
10. Pipberger H.V. Comparative evaluation of ECG computer programs. In: Ostrow HG, Ripley KL, eds. Computers in cardiology. Long Beach, Calif.: IEEE Computer Society, 1976: 85-8.
11. Willems J.L., Piessens J., Ector J., Kesteloot H., De Geest H. Comparison of computer and visual ECG analysis results: correlation with independent cineangiographic data. In: Ostrow HG, Ripley KL, eds. Computers in cardiology. Long Beach, Calif.: IEEE Computer Society, 1976: 89-94.
12. Feruglio G.A., Feraco E., Mirolo R. Experience of a user of an ECG interpretation system. In: van Bemmel JH, Willems JL, eds. Trends in computer-processed electrocardiograms. Amsterdam: North-Holland, 1977: 239-47.
13. Rautaharju P.M., Ariet M., Pryor T.A., et al. The quest for optimal electrocardiography. Task Force III: computers in diagnostic electrocardiography. Am J Cardiol 1978; 41:158-170
14. Bourdillon P.J., Kilpatrick D. Clinicians, the Mount Sinai program and the Veterans#x0027; Administration program evaluated against clinicopathological data derived independently of the electrocardiogram. Eur J Cardiol 1978; 8:395-412.
15. Rautaharju P.M., Smets P. Evaluation of computer-ECG programs: the strange case of the Golden Standard. Comput Biomed Res 1979; 12:3946.
16. Bonner R., Jackson L.K., Hooper J.K., Smith C.H., Hsieh R.C. A method for testing performance in electrocardiographic interpretation. In: Pryor AT, Bailey JJ, eds. Computerized interpretation of the ECG IV. New York: Engineering Foundation, 1979: 126-44.
17. Brohet C., Richman H.G. Clinical evaluation of automated processing of electrocardiograms by the Veterans Administration Program (AVA 3.4). Am J Cardiol 1979; 43:1167-1174
18. Bailey J.J., Harris E.K. Evaluation of ECG interpretation: truth versus beauty. In: Tolan GD, Pryor TA, eds. Computerized interpretation of the electrocardiogram V. New York: Engineering Foundation, 1980: 179-82.
19. Macfarlane P.W., Melville D.I., Horton M.R., Bailey J.J. Comparative evaluation of the IBM (12-lead) and Royal Infirmary (orthogonal three-lead) ECG computer programs. Circulation 1981; 63:354-359
20. Laks M.M. “Gold standard” for ECG diagnosis — revisited 1981. In: Bonner RE, Pryor TA, Laks MM, Cole SS, eds. Computerized interpretation of the electrocardiogram VI. New York: Engineering Foundation, 1982: 26775.
21. Bailey J.J., Berson A., Jackson L., et al. Evaluation methodologies for ECG diagnostic systems. In: Bonner RE, Pryor TA, Laks MM, Cole SS, eds. Computerized interpretation of the electrocardiogram VI. New York: Engineering Foundation, 1981: 53-62.
22. Wolf H.K., Milliken J.A., Gregor R.D., et al. Accuracy of type “A” statements of human and computer ECG classifiers. In: Laks MM, Cole SS, eds. Computerized interpretation of the electrocardiogram VII. New York: Engineering Foundation, 1982: 335-40.
23. Bernard P., Chaitman B.R., Scholl J.M., Val P.G., Chabot M. Comparative diagnostic performance of the Telemed computer ECG program. J Electrocardiol 1983; 16:97-103.
24. Graham I.M., Reardon B., Mulcahy D., Mulcahy R. User evaluation of a commercially available computerized electrocardiographic interpretation system. In: Willems JL, van Bemmel JH, Zywietz C, eds. Computer ECG analysis: towards standardisation. Amsterdam: North-Holland, 1986: 191-3.
25. Willems J.L. The CSE project: goals and achievements. In: Willems JL, van Bemmel JH, Zywietz C, eds. Computer ECG analysis: towards standardisation. Amsterdam: North-Holland, 1986: 11-21.
26. Pipberger H.V. ECG computer analysis: past, present and future. In: Willems JL, van Bemmel JH, Zywietz C, eds. Computer ECG analysis: towards standardisation. Amsterdam: North-Holland, 1986: 3-10.
27. Harris L.K., Blackburn H.W., Prineas R.J., Wolf H.K. Electrocardiographic diagnosis of myocardial infarction by human readers and two computer programs. In: European Congress on Cardiology, August 20—24, 1976, Paris. Abstract book I. Paris: European Society of Cardiology, 1976: 725. abstract.
28. Richman H.G., Brohet C.R. Evaluation of a statistical decision program for automated ECG processing. In: Hofman I, Hamby RE, eds. Vectorcardiography 3. Amsterdam: North-Holland, 1976: 199-207.
29. Kornreich F., Block P., Bourgain R., et al. Multigroup diagnostic classification with a new “maximal” lead system. Adv Cardiol 1976; 16:121-127
30. Khadr N.E., Bray C.L., Beton D.C., et al. Diagnosis of left ventricular hypertrophy and myocardial infarction by Bonner/IBM Program verified by ECGindependent evidence. In: Ripley KL, Ostrow HG, eds. Computers in cardiology. Long Beach, Calif.: IEEE Computer Society, 1979: 93-7.
31. Willems J.L., Lesaffre E., Pardaens J., DeSchreye D. Multigroup logistic classification of the 12- and 3-lead ECG. In: Willems JL, van Bemmel JH, Zywietz C, eds. Computer ECG analysis: towards standardisation. Amsterdam: North-Holland, 1986: 203-10.
32. Willems J.L., Lesaffre E., Pardaens J. Comparison of the classification ability of the electrocardiogram and vectorcardiogram. Am J Cardiol 1987; 59:119-124
33. Willems J.L., Arnaud P., van Bemmel J.H., Degani R., Macfarlane P.W., Zywietz C. Common standards for quantitative electrocardiography: goals and main results. Methods Inf Med 1990; 20:263-271
34. van Bemmel J.H., Willems J.L. Standardization and validation of medical decision-support systems: the CSE project. Methods Inf Med 1990; 29:261-262
35. Willems J.L. Common standards for quantitative electrocardiography — 6th CSE progress report. Leuven, Belgium: Acco, 1986.
36. Willems J.L., Abreu-Lima C., Arnaud P., et al. Effect of combining electrocardiographic interpretation results on diagnostic accuracy. Eur Heart J 1988; 9:1348-1355
37. Austen W.G., Edwards J.E., Frye R.L., et al. A reporting system on patients evaluated for coronary artery disease. AHA Committee Report. Circulation 1975; 51:Suppl:l-40.
38. Willems J.L., Robles de Medina E.O., Bernard R., et al. Criteria for intraventricular conduction disturbances and pre-excitation. J Am Coll Cardiol 1985; 5:1261-1275
39. Feinstein A.R. Clinical biostatistics. St. Louis: C.V. Mosby, 1977: 214.
40. Michaelis J., Wellek S., Willems J.L. Reference standards for software evaluation. Methods Inf Med 1990; 29:289-297
41. Bailey J.J., Campbell G., Horton M.R., Shrager R.I., Willems J.L. Determination of statistically significant differences in the performance of ECG diagnostic algorithms: an improved method. J Electrocardiol 1988; 21:Suppl:S188-S192.
42. Willems J.L., Campbell G., Bailey J.J. Progress on the CSE diagnostic study: application of McNemar#x0027;s test revisited. J Electrocardiol 1989; 22:Suppl: 135-40.
43. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960; 20:37-46.
44. Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics 1977; 33:159-174
45. Simonson E., Tuna N., Okamoto N., Toshima H. Diagnostic accuracy of the vectorcardiogram and electrocardiogram: a cooperative study. Am J Cardiol 1966; 17:829-878
46. Milliken J.A., Burggraf G., McCans J., Gregor R.D., Wolf H.K., Rautaharju P.M. Evaluating ECG interpreters against expert majority and clinical data. In: Laks M, ed. Computerized interpretation of the electrocardiogram VII. New York: Engineering Foundation, 1982: 341-6.
47. Willems J.L. Common standards for quantitative electrocardiography —10th CSE progress report. Leuven, Belgium: Acco, 1990.
48. Milliken J.A., Burggraf G.W., McCans J.L. The impact of the computer-reported vectorcardiogram on the cardiologist interpreter of the scalar electrocardiogram. J Electrocardiol 1977; 10:375-378