A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard

Journal of Clinical Epidemiology

Volumn 62, Issue 8, 2009, Pages 797-806

  Reitsma, Johannes B ^a   Rutjes, Anne W S ^a   Khan, Khalid S ^b   Coomarasamy, Arri ^b   Bossuyt, Patrick M ^a  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b UNIVERSITY OF BIRMINGHAM   (United Kingdom)

Author keywords

Diagnosis; Gold standard; Misclassification; Partial verification; Reference standard; Sensitivity and specificity

Indexed keywords

CLINICAL EVALUATION; COMPUTER ASSISTED TOMOGRAPHY; CONSENSUS; DIAGNOSTIC ACCURACY; HUMAN; METHODOLOGY; PRIORITY JOURNAL; REVIEW; STANDARD; STATISTICAL MODEL; VALIDATION STUDY;

DATA INTERPRETATION, STATISTICAL; DIAGNOSTIC TESTS, ROUTINE; HUMANS; REFERENCE STANDARDS; REPRODUCIBILITY OF RESULTS; SOFTWARE DESIGN;

EID: 67649313500     PISSN: 08954356     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2009.02.005     Document Type: Review

References (70)

1. Fryback D.G., and Thornbury J.R. The efficacy of diagnostic imaging. Med Decis Making 11 (1991) 88-94
2. Sackett D., and Haynes R. The architecture of diagnostic research. In: Knottnerus J.A. (Ed). The evidence base of clinical diagnosis (2002), BMJ Books, London, UK 19-38
3. Knottnerus J., and Muris J. Assessment of the accuracy of diagnostic tests: the cross-sectional study. In: Knottnerus J.A. (Ed). The evidence base of clinical diagnosis (2002), BMJ Books, London, UK 39-59
4. Knottnerus J.A., and van Weel C. General introduction: evaluation of diagnostic procedures. In: Knottnerus J.A. (Ed). The evidence base of clinical diagnosis (2002), BMJ Books, London, UK 1-18
5. Pepe M.S. Incomplete data and imperfect reference tests. In: Pepe M.S. (Ed). The statistical evaluation of medical tests for classification and prediction (2004), Oxford University Press, Oxford, UK 168-213
6. Valenstein P.N. Evaluating diagnostic tests with imperfect standards. Am J Clin Pathol 93 (1990) 252-258
7. Rutjes A.W., Reitsma J.B., Coomarasamy A., Khan K.S., and Bossuyt P.M. Evaluation of diagnostic tests when there is no gold standard. A review of methods. Health Technol Assess 11 (2007) 50
8. Bossuyt P.M., Reitsma J.B., Bruns D.E., Gatsonis C.A., Glasziou P.P., Irwig L.M., et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med 138 (2003) W1-W12
9. Birtle A.J., Freeman A., Masters J.R., Payne H.A., and Harland S.J. Clinical features of patients who present with metastatic prostate carcinoma and serum prostate-specific antigen (PSA) levels < 10 ng/mL: the "PSA negative" patients. Cancer 98 (2003) 2362-2367
10. Doshi T., Rusinak D., Halvorsen R.A., Rockey D.C., Suzuki K., and Dachman A.H. CT colonography: false-negative interpretations. Radiology 244 (2007) 165-173
11. Whiting P., Rutjes A.W., Reitsma J.B., Glas A.S., Bossuyt P.M., and Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med 140 (2004) 189-202
12. Jones T.D. The diagnosis of rheumatic fever. JAMA 126 (1944) 481-484
13. Guidelines for the diagnosis of rheumatic fever. Jones Criteria, 1992 update. Special Writing Group of the Committee on Rheumatic Fever, Endocarditis, and Kawasaki Disease of the Council on Cardiovascular Disease in the Young of the American Heart Association. JAMA 268 (1992) 2069-2073
14. Boyko E.J., Alderman B.W., and Baron A.E. Reference test errors bias the evaluation of diagnostic tests for ischemic heart disease. J Gen Intern Med 3 (1988) 476-481
15. Lijmer J.G., Mol B.W., Heisterkamp S., Bonsel G.J., Prins M.H., van der Meulen J.H., et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 282 (1999) 1061-1066
16. Rutjes A.W., Reitsma J.B., Di Nisio M., Smidt N., van Rijn J.C., and Bossuyt P.M. Evidence of bias and variation in diagnostic accuracy studies. CMAJ 174 (2006) 469-476
17. Bossuyt P.M., Reitsma J.B., Bruns D.E., Gatsonis C.A., Glasziou P.P., Irwig L.M., et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD Initiative. Ann Intern Med 138 (2003) 40-44
18. Whiting P., Rutjes A.W., Dinnes J., Reitsma J.B., Bossuyt P.M., and Kleijnen J. Development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess 8 (2004) 25
19. Begg C.B., and Greenes R.A. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics 39 (1983) 207-215
20. Harel O., and Zhou X.H. Multiple imputation for correcting verification bias. Stat Med 25 (2006) 3769-3786
21. Little R.J., and Rubin D.B. Statistical analysis with missing data (2002), Wiley, New York
22. Molenberghs G., Goetghebeur E.J., Lipsitz S.R., Kenward M.G., Lesaffre E., and Michiels B. Missing data perspectives of the fluvoxamine data set: a review. Stat Med 18 (1999) 2449-2464
23. Zhou X.H. Correcting for verification bias in studies of a diagnostic test's accuracy. Stat Methods Med Res 7 (1998) 337-353
24. Diamond G.A. Off Bayes: effect of verification bias on posterior probabilities calculated using Bayes' theorem. Med Decis Making 12 (1992) 22-31
25. Kosinski A.S., and Barnhart H.X. Accounting for nonignorable verification bias in assessment of diagnostic tests. Biometrics 59 (2003) 163-171
26. Hadgu A., Dendukuri N., and Hilden J. Evaluation of nucleic acid amplification tests in the absence of a perfect gold-standard test: a review of the statistical and epidemiologic issues. Epidemiology 16 (2005) 604-612
27. Brenner H. Correcting for exposure misclassification using an alloyed gold standard. Epidemiology 7 (1996) 406-410
28. Zhou X., Obuchowski N., McClish D., and e. Methods for correcting imperfect standard bias. Statistical methods in diagnostic medicine (2002), John Wiley & Sons, New York 359-395
29. Staquet M., Rozencweig M., Lee Y.J., and Muggia F.M. Methodology for the assessment of new dichotomous diagnostic tests. J Chronic Dis 34 (1981) 599-610
30. Wacholder S., Armstrong B., and Hartge P. Validation studies using an alloyed gold standard. Am J Epidemiol 137 (1993) 1251-1258
31. Ransohoff D.F., and Feinstein A.R. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med 299 (1978) 926-930
32. Rutjes AW, Reitsma JB, Irwig L, Bossuyt PM. Partial and differential verification in diagnostic accuracy studies. In: Sources of bias and variation in diagnostic accuracy studies (Thesis). Amsterdam, The Netherlands: 2005:31-44.
33. Bowler J.V., Munoz D.G., Merskey H., and Hachinski V. Fallacies in the pathological confirmation of the diagnosis of Alzheimer's disease. J Neurol Neurosurg Psychiatr 64 (1998) 18-24
34. Alonzo T.A., and Pepe M.S. Using a combination of reference tests to assess the accuracy of a new diagnostic test. Stat Med 18 (1999) 2987-3003
35. Miller W.C. Can we do better than discrepant analysis for new diagnostic test evaluation?. Clin Infect Dis 27 (1998) 1186-1193
36. Martin D.H., Nsuami M., Schachter J., Hook III E.W., Ferrero D., Quinn T.C., et al. Use of multiple nucleic acid amplification tests to define the infected-patient "gold standard" in clinical trials of new diagnostic tests for Chlamydia trachomatis infections. J Clin Microbiol 42 (2004) 4749-4758
37. Alonzo T.A., and Pepe M.S. Assessing the accuracy of a new diagnostic test when a gold standard does not exist. UW Biostatistics Working Paper Series (1998) paper 156
38. Gagnon R., Charlin B., Coletti M., Sauve E., and van der Vleuten C. Assessment in the context of uncertainty: how many members are needed on the panel of reference of a script concordance test?. Med Educ 39 (2005) 284-291
39. Pepe M.S., and Janes H. Insights into latent class analysis. UW Biostatistics Working Paper Series (2005) paper 236
40. Hui S.L., and Zhou X.H. Evaluation of diagnostic tests without gold standards. Stat Methods Med Res 7 (1998) 354-370
41. Streiner D.L., and Norman G.R. Validity. In: Streiner D.L., and Norman G.R. (Eds). Health Measurement Scales: a practical guide to their development and use. 2nd edition (1995), Oxford University Press, Oxford, UK 144-162
42. Bland J.M., and Altman D.G. Statistics notes: validating scales and indexes. BMJ 324 (2002) 606-607
43. Winter G. A comparative discussion of the notion of 'validity' in qualitative and quantitative research. The Qualitative Report 4(3/4) [on-line serial]. Available at: http://www.nova.edu/ssss/QR/QR4-3/winter.html.
44. Drum D.E., and Christacopoulos J.S. Hepatic scintigraphy in clinical decision making. J Nucl Med 13 (1969) 908-915
45. Kline J.A., Israel E.G., Michelson E.A., O'Neil B.J., Plewa M.C., and Portelli D.C. Diagnostic accuracy of a bedside D-dimer assay and alveolar dead-space measurement for rapid exclusion of pulmonary embolism: a multicenter study. JAMA 285 (2001) 761-768
46. Schneeweiss S., Kriegmair M., and Stepp H. Is everything all right if nothing seems wrong? A simple method of assessing the diagnostic value of endoscopic procedures when a gold standard is absent. J Urol 161 (1999) 1116-1119
47. Schneeweiss S. Sensitivity analysis of the diagnostic value of endoscopies in cross-sectional studies in the absence of a gold standard. Int J Technol Assess Health Care 16 (2000) 834-841
48. Jang D., Sellors J.W., Mahony J.B., Pickard L., and Chernesky M.A. Effects of broadening the gold standard on the performance of a chemiluminometric immunoassay to detect Chlamydia trachomatis antigens in centrifuged first void urine and urethral swab samples from men. Sex Transm Dis 19 (1992) 315-319
49. Rutten F.H., Moons K.G., Cramer M.J., Grobbee D.E., Zuithoff N.P., Lammers J.W., et al. Recognising heart failure in elderly patients with stable chronic obstructive pulmonary disease in primary care: cross sectional diagnostic study. BMJ 331 (2005) 1379
50. Ewer K., Deeks J., Alvarez L., Bryant G., Waller S., Andersen P., et al. Comparison of T-cell-based assay with tuberculin skin test for diagnosis of Mycobacterium tuberculosis infection in a school tuberculosis outbreak. Lancet 361 (2003) 1168-1173
51. Kontos M.C., Shah R., Fritz L.M., Anderson F.P., Tatum J.L., Ornato J.P., et al. Implication of different cardiac troponin I levels for clinical outcomes and prognosis of acute chest pain patients. J Am Coll Cardiol 43 (2004) 958-965
52. Jaffe A.S., Ravkilde J., Roberts R., Naslund U., Apple F.S., Galvani M., et al. It's time for a change to a troponin standard. Circulation 102 (2000) 1216-1220
53. Lord S.J., Irwig L.M., and Simes R.J. When is measuring sensitivity and specificity sufficient to evaluate a diagnostic test, and when do we need randomized trials?. Ann Intern Med 144 (2006) 850-855
54. Lijmer J.G., and Bossuyt P.M. Diagnostic testing and prognosis: the randomised controlled trial in diagnostic research. In: Knottnerus J.A. (Ed). The evidence base of clinical diagnosis (2002), BMJ Books, London, UK 61-93
55. Mueller C., Laule-Kilian K., Schindler C., Klima T., Frana B., Rodriquez D., et al. Cost-effectiveness of B-type natriuretic peptide testing in patients with acute dyspnea. Arch Intern Med 160 (2006) 1081-1087
56. Taskforce of the European Society of Cardiology. Guidelines for the diagnosis and treatment of chronic heart failure: executive summary. Eur Heart J 26 (2005) 1115-1140
57. Irwig L., Glasziou P.P., Berry G., Chock C., Mock P., and Simpson J.M. Efficient study designs to assess the accuracy of screening tests. Am J Epidemiol 140 (1994) 759-769
58. Hadgu A. The discrepancy in discrepant analysis. Lancet 348 (1996) 592-593
59. Miller W.C. Bias in discrepant analysis: when two wrongs don't make a right. J Clin Epidemiol 51 (1998) 219-231
60. Lipman H.B., and Astles J.R. Quantifying the bias associated with use of discrepant analysis. Clin Chem 44 (1998) 108-115
61. Hadgu A. Discrepant analysis: a biased and an unscientific method for estimating test sensitivity and specificity. J Clin Epidemiol 52 (1999) 1231-1237
62. Paulus W.J., Tschope C., Sanderson J.E., Rusconi C., Flachskampf F.A., Rademakers F.E., et al. How to diagnose diastolic heart failure: a consensus statement on the diagnosis of heart failure with normal left ventricular ejection fraction by the Heart Failure and Echocardiography Associations of the European Society of Cardiology. Eur Heart J 28 (2007) 2539-2550
63. Jones J., and Hunter D. Consensus methods for medical and health services research. BMJ 311 (1995) 376-380
64. Vermunt J.K., and Magidson J. Latent class analysis. In: Lewis-Beck M., Bryman A., and Liao T.F. (Eds). The sage encyclopedia of social sciences research methods (2004), Sage Publications, Thousand Oaks, CA 549-553
65. Walter S.D., and Irwig L.M. Estimation of test error rates, disease prevalence and relative risk from misclassified data: a review. J Clin Epidemiol 41 (1988) 923-937
66. Qu Y., Tan M., and Kutner M.H. Random effects models in latent class analysis for evaluating accuracy of diagnostic tests. Biometrics 52 (1996) 797-810
67. Dendukuri N., and Joseph L. Bayesian approaches to modelling the conditional dependence between multiple diagnostic tests. Biometrics 57 (2001) 158-167
68. Joseph L., Gyorkos T.W., and Coupal L. Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard. Am J Epidemiol 141 (1995) 263-272
69. Collinson P.O., Stubbs P.J., and Kessler A.C. Multicentre evaluation of the diagnostic value of cardiac troponin T, CK-MB mass, and myoglobin for assessing patients with suspected acute coronary syndromes in routine clinical practice. Heart 89 (2003) 280-286
70. Bossuyt P.M., Lijmer J.G., and Mol B.W. Randomised comparisons of medical tests: sometimes invalid, not always efficient. Lancet 356 (2000) 1844-1847