Evaluation of diagnostic tests when there is no gold standard. A review of methods

Health Technology Assessment

Volumn 11, Issue 50, 2007, Pages

  Rutjes, A W S ^a   Reitsma, J B ^a   Coomarasamy, A ^b   Khan, K S ^b   Bossuyt, P M M ^a  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b UNIVERSITY OF BIRMINGHAM   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

CONSENSUS; DATA BASE; DIAGNOSTIC ACCURACY; DIAGNOSTIC ERROR; DIAGNOSTIC TEST; DIAGNOSTIC VALUE; GOLD STANDARD; INFORMATION PROCESSING; MEDICAL RESEARCH; METHODOLOGY; RELIABILITY; REVIEW; SENSITIVITY AND SPECIFICITY; STANDARDIZATION; STATISTICAL MODEL; STATISTICS; STRATEGIC PLANNING; VALIDATION PROCESS; DIAGNOSTIC PROCEDURE; HEALTH CARE QUALITY; HUMAN; STANDARD;

DIAGNOSTIC TECHNIQUES AND PROCEDURES; HUMANS; PROCESS ASSESSMENT (HEALTH CARE); REFERENCE STANDARDS; SENSITIVITY AND SPECIFICITY;

EID: 38849166655     PISSN: 13665278     EISSN: None     Source Type: Journal    
DOI: 10.3310/hta11500     Document Type: Review

References (118)

1. Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 1999; 282:1061-6.
2. Whiting P, Rutjes AW, Reitsma JB, Glas AS, Bossuyt PM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med 2004;140:189-202.
3. Rutjes AW, Reitsma JB, Di NM, Smidt N, van Rijn JC, Bossuyt PM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ 2006; 174:469-76.
4. Knottnerus JA, van Weel G. General introduction: evaluation of diagnostic procedures. In Knottnerus JA, editor. The evidence base of clinical diagnosis. London: BMJ Books; 2002. pp. 1-18.
5. Sackett DL, Haynes RB. The architecture of diagnostic research. In Knottnerus JA editor. The evidence base of clinical diagnosis. London: BMJ Books; 2002. pp. 19-38.
6. Lijmer JG, Mol BWJ, Bonsel GJ, Prins MH, Bossuyt PM. Strategies for the evaluation of diagnostic technologies. Evaluation of diagnostic tests: from accuracy to outcome (Thesis). Amsterdam; 2001. pp. 15-28.
7. Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991;11: 88-94.
8. Knottnerus JA, Muris JW. Assessment of the accuracy of diagnostic tests: the cross-sectional study. In Knottnerus JA, editor. The evidence base of clinical diagnosis. London: BMJ Books; 2002. pp. 39-59.
9. Bossuyt PM, Irwig L, Craig J, Glasziou P. Comparative accuracy: assessing new tests against existing diagnostic pathways. BMJ 2006;332: 1089-92.
10. Lord SJ, Irwig L. Simes RJ. When is measuring sensitivity and specificity sufficient to evaluate a diagnostic test, and when do we need randomized trials? Ann Intern Med 2006;144:850-5.
11. Boyko EJ, Alderman BW, Baron AE. Reference test errors bias the evaluation of diagnostic tests for ischemic heart disease. J Gen Intern Med 1988; 3:476-81.
12. Deneef P. Evaluating rapid tests for streptococcal pharyngitis: the apparent accuracy of a diagnostic test when there are errors in the standard of comparison. Med Decis Making 1987;7:92-6.
13. Pepe MS. Incomplete data and imperfect reference tests. In Pepe MS, editor. The statistical evaluation of medical tests for classification and prediction. Oxford: Oxford University Press; 2004. pp. 168-213.
14. Vacek PM. The effect of conditional dependence on the evaluation of diagnostic tests. Biometrics 1985;41:959-68.
15. Thibodeau L. Evaluating diagnostic tests. Biometrics 1981; 37:801-4.
16. International Organization for Standardization. International vocabulary of basic and general terms in metrology. Geneva: ISO; 1993.
17. Yerushalmy J. Statistical problems in assessing methods of medical diagnosis, with special reference to X-ray techniques. Public Health Rep 1947;62:1432-49.
18. Ledley RS, Lusted LB. Reasoning foundations of medical diagnosis; symbolic logic, probability, and value theory aid our understanding of how physicians reason. Science 1959;130:9-21.
19. Whiting P, Rutjes AW, Reitsma JB, Clas AS, Bossuyt PM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med 2004;140:189-202.
20. Rutjes AW, Reitsma JB, IrWig L, Bossuyt PM. Partial and differential bias in diagnostic accuracy studies. Sources of bias and variation in diagnostic accuracy studies (Thesis), Amsterdam; 2005. pp. 31-44.
21. Gustafson P. The utility of prior information and stratification for parameter estimation with two screening tests but no gold standard. Stat Med 2005;24:1203-17.
22. Begg CB, Greenes RA. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics 1983; 39:207-15
23. Harel O, Zhou XH. Multiple imputation for correcting verification bias. Stat Med 2006;25: 3769-86.
24. Pepe MS. The statistical evaluation of medical tests for classification and prediction. Oxford: Oxford University Press; 2003.
25. Alonzo TA, Pepe MS. Using a combination of reference tests to assess the accuracy of a new diagnostic test. Stat Med 1999; 18:2987-3003.
26. Hui SL, Zhou XH. Evaluation of diagnostic tests without gold standards. Stat Methods Med Res 1998; 7:354-70.
27. Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. Chichester Wiley; 2006.
28. Lilford RJ, Richardson A, Stevens A, et al. Issues in methodological research: perspectives from researchers and commissioners. Health Technol Assess 2001;5(8).
29. Little R, Rubin D. Statistical analysis with missing data. 2nd ed. New York Wiley; 2002.
30. Paliwal P, Gelfand AE. Estimating measures of diagnostic accuracy when some covariate information is missing. Stat Med 2006; 25:2981-93.
31. Choi BC. Sensitivity and specificity of a single diagnostic test in the presence of work-up bias. J Clin Epidemiol 1992;45:581-6.
32. Diamond GA. Off Bayes: effect of verification bias on posterior probabilities calculated using Bayes' theorem. Med Decis Making l992;12:22-31.
33. Cecil MP, Kosinski AS, Jones MT, et al. The importance of work-up (verification) bias correction in assessing the accuracy of SPECT thallium-201 testing for the diagnosis of coronary artery disease. J Clin Epidemiol 1996;49:735-42.
34. Danias PG, Parker JA. Novel Internet-based tool for correcting apparent sensitivity and specificity of diagnostic tests to adjust for referral (verification) bias. Radiographics 2002;22(2):e4.
35. Held L, Ranyimbo AO. A Bayesian approach to estimate and validate the false negative fraction in a two-stage multiple screening test. Methods Inf Med 2004;43:461-4.
36. Kosinski AS, Barnhart HX. Accounting for nonignorable verification bias in assessment of diagnostic tests. Biometrics 2003; 59:163-71.
37. Kosinski AS, Barnhart HX. A global sensitivity analysis of performance of a medical diagnostic test when verification bias is present. Stat Med 2003;22:2711-21.
38. Toledano AY, Gatsonis C. Generalized estimating equations for ordinal categorical data: arbitrary patterns of missing responses and missingness in a key covariate. Biometrics 1999;55:488-96.
39. Zhou XH. Correcting for verification bias in studies of a diagnostic test's accuracy. Stat Methods Med Res 1998;7:337-53.
40. Zhou XH, Higgs RE. COMPROC and CHECKNORM: computer programs for comparing accuracies of diagnostic tests using ROC curves in the presence of verification bias. Comput Methods Programs Biomed 1998;57:179-86.
41. Zhou XH, Higgs RE. Assessing the relative accuracies of two screening tests in the presence of verification bias. Stat Med 2000; 19:1697-705.
42. Molenberghs G, Goetghebeur EJ, Lipsitz SR,. Kenward MG, Lesaffre E, Michiels B. Missing data perspectives of the fluvoxamine data set: a review. Stat Med 1999;18:2449-64.
43. Engels JM, Diehr P. Imputation of missing longitudinal data: a comparison of methods. J Clin Epidemiol 2003;56:968-76.
44. Liu M, Taylor JM, Belin TR. Multiple imputation and posterior simulation for multivariate missing data in longitudinal studies. Biometrics 2000;56: 1157-63.
45. Rubin DB. Multiple imputation after 18+ years. J Am Stat Assoc 1996;91:473-89.
46. Schafer JL. Multiple imputation: a primer. Stat Methods Med Res 1999; 8: 3-15.
47. Zhou X-H, Obuchowski NA, McClish DK. Statistical methods in diagnostic medicine. New York: Wiley; 2002.
48. Greenes RA, Begg CB. Assessment of diagnostic technologies. Methodology for unbiased estimation from samples of selectively verified patients. Invest Radiol 1985;20:751-6.
49. Pisano ED, Gatsonis C, Hendrick E, et al. Diagnostic performance of digital versus film mammography for breast-cancer screening. N Engl J Med 2005;353:1773-83.
50. Harrell FE, Jr., Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996; 15:361-87.
51. Drum D, Christacopoulos J. Hepatic scintigraphy in clinical decision making. J Nucl Med 1969; 13:908-15.
52. Marshall V, Williams DC, Smith KD. Diaphanography as a means of detecting breast dincer. Radiology 1981;150:339-43.
53. Zhou X-H, Obuchowski NA, McClish DK. Methods for correcting imperfect standard bias. Statistical methods diagnostic medicine. New York: Wiley; 2002. pp. 359-95.
54. Hadgu A, Dendukuri N, Hilden J. Evaluation of nucleic acid amplification tests in the absence of a perfect gold-standard test: a review of the statistical and epidemiologic issues. Epidemiology 2005; 16:604-12.
55. Staquet M, Rozencweig M, Lee YJ, et al. Methodology for the assessment of new dichotomous diagnostic tests. J Chronic Dis 1981;34:599-610.
56. Valenstein PN. Evaluating diagnostic tests with imperfect standards. Am J Clin Pathol 1996;93: 252-8.
57. Brenner H. Correcting for exposure misclassification using an alloyed gold standard. Epidemiology 1996;7:406-10.
58. Wacholder S, Armstrong B, Hartge P. Validation studies using an alloyed gold standard. Am J Epidemiol 1993;137:1251-8.
59. Schneeweiss S, Kriegmair M, Stepp H. Is everything all right if nothing seems wrong? A simple method of assessing the diagnostic value of encloscopic procedures when a gold standard is absent. J Urol 1999;161:1116-19.
60. Schneeweiss S. Sensitivity analysis of the diagnostic value of endoscopies; in cross-sectional studies in the absence of a gold standard. Int J Technol Assess Health Care 2000;16:834-41.
61. van Belle A., Buller HR, Huisman MV, et al. Effectiveness of managing suspected pulmonary embolism using an algorithm combining clinical probability, D-dimer testing, and computed tomography. JAMA 2006;295:172-9.
62. Ransohoff DE, Feinstein AR. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med 1978; 299:926-30.
63. Kline JA, Israel EG, Michelson EA, O'Neil BJ, Plewa MC, Portelli DC. Diagnostic accuracy of a bedside D-dimer assay and alveolar dead-space measurement for rapid exclusion of pulmonary embolism: a multicenter study. JAMA 2001;285:761-8.
64. Alonzo TA, Pepe MS. Assessing the accuracy of a new diagnostic test when a gold standard does not exist. UW Biostatistics Working Paper Series 1998; paper 156.
65. Hadgu A. The discrepancy in discrepant analysis. Lancet 1996; 348:592-3.
66. Miller WC. Can we do better than discrepant analysis for new diagnostic test evaluation? Clin Infect Dis 1998;27:1186-93.
67. Miller WC. Bias in discrepant analysis: when two wrongs don't make a right. J Clin Epidemiol 1998; 51:219-31.
68. Hadgu A. Bias in the evaluation of DNA-amplification tests for detecting Chlamydia trachomatis. Stat Med 1997;16:1391-9.
69. Lipman H, Asdes J. Quantifying the bias associated with use of discrepant analysis. Clin Chem 1998;44:108-15.
70. Hadgu A. Discrepant analysis: a biased and an unscientific method for estimating test sensitivity and specificity. J Clin Epidemiol 1999;52:1231-7.
71. Hadgu A. Discrepant analysis is an inappropriate and unscientific method. J Clin Microbiol 2000;38:4301-2.
72. Green TA, Black CM, Johnson RE. Evaluation of bias in diagnostic-test sensitivity and specificity estimates computed by discrepant analysis. J Clin Microbiol 1998;36:375-81.
73. Diamond GA. Affirmative actions: can the discriminant accuracy of a test be determined in the face of selection bias? Med Decis Making 1991;11:48-56.
74. Wu CH, Lee MF, Yin SC, Yang DM, Cheng SF. Comparison of polymerase chain reaction, monoclonal antibodv based enzyme immunoassay and cell culture for detection of Chlamydia trachomatis in genital specimens. Sex Transm Dis 1992;19:393-7.
75. Martin DH, Nsuami M, Schachter J, et al. Use of multiple nucleic acid amplification tests to define the infected-patient "gold standard" in clinical trials of new diagnostic tests for Chlamydia trachomatis infections. J Clin Microbiol 2004; 42:4749-58.
76. Jang D, Sellors JW, Mahony JB, Pickard L, Chernesky MA. Effects of broadening the gold standard on the performance of a chemiluminometric immunoassay to detect Chlamydia trachomatis antigens in centrifuged first void urine and urethral swab samples from men. Sex Transm Dis 1992;19:315-19.
77. Das R, Kiicullen N, Morrell C, Robinson MB, Barth JH, Hall AS. The British Cardiac Society Working Group definition of myocardial infarction: implications for practice. Heart 2006;92:21-6.
78. Thijs JC, Van Zwet AA, Thijs WJ, et al. Diagnostic tests for Helicobacter pylori: a prospective evaluation of their accuracy, without selecting a single test as the gold standard. Am J Gostroenterol 1996: 91:2125-9.
79. Madan K, Ahuja V, Gupta SD, Bal C, Kapoor A, Sharma MP. Impact of 24-h esophageal pH monitoring on the diagnosis of gastroesophageal reflux disease: defining the gold standard. J Gostroenterol Hepatol 2005;20:30-7.
80. Gagnon R, Charlin B, Coletti M, Sauve E, Van DV. Assesment in the context of uncertainty: how many members are-needed on the panel of reference of d script concordance test? Med Educ 2005; 39:284-91.
81. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995;311:376-80.
82. Rutten FH, Moons KG, Cramer MJ, et al. Recognising heart failure in elderly patients with stable chronic obstructive pulmonary disease in primary care: cross sectional diagnostic study. BMJ 2005; 331:1379.
83. Hui SL, Zhou XH. Evaluation of diagnostic tests without gold standards. Stat Methods Med Res 1998;7:354-70.
84. Walter SD, Irwig LM. Estimation of test error rates, disease prevalence and relative risk from misclassified data: a review. J Clin Epidemiol 1988;41:923-37.
85. Pepe MS, Janes H. Insights into latent class analysis. UW Biostatistics Working Paper Series 2005; paper 236.
86. Formann AK. Measurement errors in caries diagnosis: some further latent class models. Biometrics 1994;50:865-71.
87. Committee on Quality Management, Japan Society of Clinical Chemistry. Proposed standard for matrix reference materials for quantitative laboratory methods. JPN J Clin Chem 2003; 32:180-5.
88. Qu Y, Tan M, Kutner MH. Random effects models in latent class analysis for evaluating accuracy of diagnostic tests. Biometrics 1996; 52:797-810.
89. Bertrand P, Benichou J, Grenier P, Chastang C. Hui and Walter's latent-class reference-free approach may be more useful in assessing agreement than diagnostic performance. J Clin Epidemiol 2005; 58:688-700.
90. Vermunt JK, Magidson J. Latent class analysis. In Lewis-Beck M, Bryman A, Liao TF, editors. The Sage Encyclopedia of Social Sciences Research Methods. Thousand Oaks, CA Sage Publications; 2004. pp. 549-53.
91. Joseph L, Gyorkos TW, Coupal L. Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard. Am J Epidemiol 1995;141:263-72.
92. Black MA, Craig BA. Estimating disease prevalence in the absence of a gold standard. Stat Med 2002;21:2653-69.
93. Hadgu A, Qu Y. A biomedical application of latent class models with random effects. Appl Stat 1998;47:603-16.
94. Gustafson P. Measurement error and misclassification in statistics and epidemiology: impacts and Bayesian adjustments. Boca Raton, FL: Chapman and Hall/CRC; 2003.
95. Dendukuri N, Joseph L. Bayesian approaches to modelling the conditional dependence between multiple diagnostic tests. Biometrics 2001; 57:158-67.
96. Georgiadis MP, Johnson WO, Gardner IA, Singh R. Correlation-adjusted estimation of sensitivity and specificity of two diagnostic tests. Appl Stat 2003;52:63-76.
97. Bertrand P, Benichou J, Grenier P, Chastang C. Hui and Walter's latent-class reference-free approach may be more useful in assessing agreement than diagnostic performance. J Clin Epidemiol 2005; 58:688-700.
98. Black CM. Current methods of laboratory diagnosis of Chlamydia trachomatis infections. Clin Microbiol Rev 1997; 10:160-84.
99. Barnes RC. Laboratory diagnosis of human chlamydial infections. Clin Microbiol Rev 1989;2:119-36.
100. Winter G. A comparative discussion of the notion of 'validity' in qualitative and quantitative research. The qualitative report. 2000; 4. URL: http://www.nova.edu/ssss/QR/OR4-3/Winter.html. Accessed 21 September 2007.
101. Streiner DL, Norman GR. Validity. In Streiner DL, Norman GR, editors. Health measurement scales. a practical guide to their development and use. Oxford: Oxford University Press; 1995. pp. 144-62.
102. Bland JM, Altman DG. Statistics notes: validating, scales and indexes. BMJ 2002;324:606-7.
103. Fletcher RH, Fletcher SW, Wagner EH. Abnormality. In Satterfield TS, editor. Clinical epidemiology: the essentials. Baltimore, MD: Williams and Wilkins; 1996. pp. 22-24.
104. Cronbach LJ, Meehl PE. Construct validity in psychological tests. Psychoi Bull 1955;52:281-302.
105. Lijmer JG, Bossuyt PM. Diagnostic testing and prognosis: the randomised controlled trial in diagnostic research. In Knottnerus JA, editor. The evidence base of clinical diagnosis. London: BMJ Books; 2002. pp. 61-80.
106. Bossuyt PM, Lijmer JG, Mol BW. Randomised comparisons of medical tests: sometimes invalid, not always efficient. Lancet 2000; 356:1844-7.
107. Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991;11: 88-94.
108. Subtil D, Goeusse P, Houfflin-Debarge V, et al. Randomised comparison of uterine artery Doppler and aspirin (100 mg) with placebo in nulliparous women: the Essai Regional Aspirine Mere-Enfant Study (Part 2). BJOG 2003;110:485-91.
109. Duley L, Henderson-Smart D, Knight M, King J. Antiplatelet drugs for prevention of pre-eclampsia and its consequences: systematic review. BMJ 2001;322:329-33.
110. Collinson PO, Stubbs PJ, Kessler AC. Multicentre evaluation of the diagnostic value of cardiac troponin T, CK-MB mass, and myoglobin for assessing patients with suspected acute coronary syndromes in routine clinical practice. Heart 2003;89:280-6.
111. Kontos MC, Shah R, Fritz LM, et al. Implication of different cardiac troponin I levels for clinical outcomes and prognosis of acute chest pain patients. J Am Coll Cardiol 2004;43:958-65.
112. Jaffe AS, Ravkilde J, Roberts R, et al. It's time for a change to a troponin standard. Circulation 2000;102:1216-20.
113. Apple FS, Wu AH, Mair J, et al. Future biomarkers for detection of ischemia and risk stratification in acute coronary syndrome. Clin Chem 2005;51:810-24.
114. Ewer K, Deeks J, Alvarez L, et al. Comparison of T-cell-based assay with tuberculin skin test for diagnosis of Mycobacterium tuberculosis infection in a school tuberculosis outbreak. Lancet 2003;361:1168-73.
115. Bossuyt PM, Reitsma JB, Bruns DE, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem 2003;49:7-18.
116. Moons KG, Biesheuvel CJ, Grobbee DE. Test research versus diagnostic research. Clin Chem 2004;50:473-6.
117. Lee TH, Goldman L. Evaluation of the patient with acute chest pain. N Engl J Med 2000; 342:1187-95.
118. Ankum WM, Van der Veen F, Hamerlynck JV, Lammes FB. Suspected ectopic pregnancy. What to do when human chorionic gonadotropin levels are below the discriminatory zone. J Reprod Med 1995;40:525-8.