A systematic review classifies sources of bias and variation in diagnostic test accuracy studies

Journal of Clinical Epidemiology

Volumn 66, Issue 10, 2013, Pages 1093-1104

  Whiting, Penny F ^a,b   Rutjes, Anne W S ^c,d   Westwood, Marie E ^a   Mallett, Susan ^e  

^a KLEIJNEN SYSTEMATIC REVIEWS LTD   (United Kingdom)

^b UNIVERSITY OF BRISTOL   (United Kingdom)

^c UNIVERSITY OF BERN   (Switzerland)

^d UNIVERSITY OF CHIETI   (Italy)

^e UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

Bias; Sensitivity; Specificity; Systematic review; Test accuracy; Variation

Indexed keywords

ARTICLE; DEMOGRAPHY; DIAGNOSTIC TEST ACCURACY STUDY; DISEASE SEVERITY; HUMAN; OBSERVER BIAS; OBSERVER VARIATION; PREVALENCE; PRIORITY JOURNAL; STANDARD; STUDY DESIGN; SYSTEMATIC REVIEW;

EID: 84883452445     PISSN: 08954356     EISSN: 18785921     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2013.05.014     Document Type: Review

References (118)

1. P.F. Whiting, A.W. Rutjes, M.E. Westwood, S. Mallett, J.J. Deeks, and J.B. Reitsma QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies Ann Intern Med 155 2011 529 536
2. S.J. Lord, L.P. Staub, P.M. Bossuyt, and L.M. Irwig Target practice: choosing target conditions for test accuracy studies that are relevant to clinical practice BMJ 343 2011 d4684
3. P. Whiting, A.W. Rutjes, J.B. Reitsma, A.S. Glas, P.M. Bossuyt, and J. Kleijnen Sources of variation and bias in studies of diagnostic accuracy: a systematic review [Review] [61 refs] Ann Intern Med 140 2004 189 202
4. C.B. Begg Biases in the assessment of diagnostic tests Stat Med 6 1987 411 423
5. J.G. Lijmer, B.W. Mol, S. Heisterkamp, G.J. Bonsel, M.H. Prins, and J.H. van der Meulen Empirical evidence of design-related bias in studies of diagnostic tests JAMA 282 1999 1061 1066
6. K. Boyer, J. Wies, and C.M. Turkelson Effects of bias on the results of diagnostic studies of carpal tunnel syndrome [Review] [41 refs] J Hand Surg Am 34 6 2009 1006 1013
7. F.A. Medeiros, D. Ng, L.M. Zangwill, P.A. Sample, C. Bowd, and R.N. Weinreb The effects of study design and spectrum bias on the evaluation of diagnostic accuracy of confocal scanning laser ophthalmoscopy in glaucoma Invest Ophthalmol Vis Sci 48 2007 214 222
8. J. Burch, M. Westwood, and K. Soares-Weiser Should data from case-controlled studies be included in systematic reviews alongside diagnostic cohort studies? [abstract] 2006 XIV Cochrane Colloquium Dublin, Ireland 86
9. A.W. Rutjes, J.B. Reitsma, N.M. Di, N. Smidt, J.C. van Rijn, and P.M. Bossuyt Evidence of bias and variation in diagnostic accuracy studies CMAJ 174 4 2006 469 476
10. A.W.S. Rutjes, N. Smidt, N.M. Di, J.G. Lijmer, B.W.J. Mol, and J.C. van Rijn Study design features affect estimates of sensitivity and specificity, but effects may vary. Sources of bias and variation in diagnostic accuracy studies, Thesis 2005 University of Amsterdam The Netherlands 121 136
11. C.J. Biesheuvel, Y. Vergouwe, R. Oudega, A.W. Hoes, D.E. Grobbee, and K.G. Moons Advantages of the nested case-control design in diagnostic research BMC Med Res Methodol 8 2008
12. P. Whiting, A.W. Rutjes, J.B. Reitsma, P.M. Bossuyt, and J. Kleijnen The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews BMC Med Res Methodol 3 2003 25
13. R.K. Morris, T.J. Selman, J. Zamora, and K.S. Khan Methodological quality of test accuracy studies included in systematic reviews in obstetrics and gynaecology: sources of bias BMC Womens Health 11 2011 2011 [Article Number]
14. S. Michaud, S. Suzuki, and S. Harbarth Effect of design-related bias in studies of diagnostic tests for ventilator-associated pneumonia [Review] [50 refs] Am J Respir Crit Care Med 166 2002 1320 1325
15. T.J. Clark, R.G. ter, A. Coomarasamy, and K.S. Khan Bias associated with delayed verification in test accuracy studies: accuracy of tests for endometrial hyperplasia may be much higher than we think! [Review] [36 refs] BMC Med 2 2004 18
16. S. Goodacre, F.C. Sampson, A.J. Sutton, S. Mason, and F. Morris Variation in the diagnostic performance of D-dimer for suspected deep vein thrombosis QJM 98 2005 513 527
17. N. Houssami, and S. Ciatto Design-related bias in estimates of accuracy when comparing imaging tests: examples from breast imaging research Eur Radiol 20 9 2010 2061 2066
18. J.T. Philbrick, R.I. Horwitz, A.R. Feinstein, R.A. Langou, and J.P. Chandler The limited spectrum of patients studied in exercise test research. Analyzing the tip of the iceberg JAMA 248 1982 2467 2470
19. R. Detrano, A. Janosi, K.P. Lyons, G. Marcondes, N. Abbassi, and V.F. Froelicher Factors affecting sensitivity and specificity of a diagnostic test: the exercise thallium scintigram Am J Med 84 1988 699 710
20. M.L. Geleijnse, B.J. Krenning, B.M. van Dalen, A. Nemes, O.I. Soliman, and J.G. Bosch Factors affecting sensitivity and specificity of diagnostic testing: dobutamine stress echocardiography [Review] [82 refs] J Am Soc Echocardiogr 22 11 2009 1199 1208
21. T.P. Haines, K. Hill, W. Walsh, and R. Osborne Design-related bias in hospital fall risk screening tool predictive accuracy evaluations: systematic review and meta-analysis [Review] [70 refs] J Gerontol A Biol Sci Med Sci 62 6 2007 664 672
22. D. Stengel, K. Bauwens, G. Rademacher, S. Mutze, and A. Ekkernkamp Association between compliance with methodological standards of diagnostic research and reported test accuracy: meta-analysis of focused assessment of US for trauma Radiology 236 2005 102 111
23. R. Detrano, R. Gianrossi, D. Mulvihill, K. Lehmann, P. Dubach, and A. Colombo Exercise-induced ST segment depression in the diagnosis of multivessel coronary disease: a meta analysis J Am Coll Cardiol 14 1989 1501 1508
24. F. Curtin, A. Morabia, C. Pichard, and D.O. Slosman Body mass index compared to dual-energy x-ray absorptiometry: evidence for a spectrum bias J Clin Epidemiol 50 1997 837 843
25. M.A. Hlatky, D.B. Pryor, F.E. Harrell Jr., R.M. Califf, D.B. Mark, and R.A. Rosati Factors affecting sensitivity and specificity of exercise electrocardiography. Multivariable analysis Am J Med 77 1984 64 71
26. D. Levy, S. Labib, and K. Anderson Determinant of sensitivity and specificity of electrocardiographic criteria for left ventricular hypertrophy Circulation 81 1990 815 820
27. K.G. Moons, G.A. van Es, J.W. Deckers, J.D. Habbema, and D.E. Grobbee Limitations of sensitivity, specificity, likelihood ratio, and Bayes' theorem in assessing diagnostic probabilities: a clinical example Epidemiology 8 1997 12 17
28. A.P. Morise, and G.A. Diamond Comparison of the sensitivity and specificity of exercise electrocardiography in biased and unbiased populations of men and women Am Heart J 130 1995 741 747
29. I.S. Syed, T. Miller, R.J. Gibbons, J.W. Askew, D. Hodge, and P. Chareonthaitawee Effect of referral bias on the diagnostic accuracy of N-13 ammonia and rubidium-82 myocardial perfusion imaging with positron emission tomography in the detection of coronary artery disease Circulation 118 18 Suppl 2 2008
30. I.M. Thompson, C. Chi, D.P. Ankerst, P.J. Goodman, C.M. Tangen, and S.M. Lippman Effect of finasteride on the sensitivity of PSA for detecting prostate cancer J Natl Cancer Inst 98 2006 1128 1133
31. S.T. Van Turenhout, F.A. Oort, J.S.T.S. Droste, A.P. Visscher, V.M. Coupe, and R.W. Van Der Hulst Gender disparities in performance of a fecal immunochemical test for detection of advanced neoplasia Gastroenterology 2011 S405 S406 Conference(var.pagings)
32. R.N. Siccama, K.J.M. Janssen, N.A.F. Verheijden, R. Oudega, L. Bax, and J.J.M. van Delden Systematic review: diagnostic accuracy of clinical decision rules for venous thromboembolism in elderly Ageing Res Rev 10 2 2011 304 313
33. C. Elie, J. Coste French Society of Clinical Cytology Study Group A methodological framework to distinguish spectrum effects from spectrum biases and to assess diagnostic and screening test accuracy for patient populations: application to the Papanicolaou cervical cancer smear test BMC Med Res Methodol 8 2008 7
34. R.S. Punglia, A.V. D'Amico, W.J. Catalona, K.A. Roehl, and K.M. Kuntz Effect of verification bias on screening for prostate cancer by measurement of prostate-specific antigen N Engl J Med 349 2003 335 342
35. J.R. Steinbauer, S.B. Cantor, C.E. Holzer, and R.J. Volk Ethnic and sex bias in primary care screening tests for alcohol use disorders Ann Intern Med 129 1998 353 362
36. L.M. Bachmann, R.G. ter, W.E. Weber, and A.G. Kessels Multivariable adjustments counteract spectrum and test review bias in accuracy studies J Clin Epidemiol 62 2009 357 361
37. D.L. Gilbert, G. Sethuraman, U. Kotagal, and C.R. Buncher Meta-analysis of EEG test performance shows wide variation among studies (Provisional abstract) Neurology 60 2003 564 570
38. P. Mastandrea Some heterogeneity factors affecting the B-type natriuretic peptides outcome: a meta-analysis [Review] [77 refs] Clin Chem Lab Med 46 12 2008 1687 1695
39. L. Gaffikin, J. McGrath, M. Arbyn, and P.D. Blumenthal Avoiding verification bias in screening test evaluation in resource poor settings: a case study from Zimbabwe Clin Trials 5 2008 496 503
40. T.D. Miller, D.O. Hodge, T.F. Christian, J.J. Milavetz, K.R. Bailey, and R.J. Gibbons Effects of adjustment for referral bias on the sensitivity and specificity of single photon emission computed tomography for the diagnosis of coronary artery disease Am J Med 112 2002 290 297
41. V.L. Roger, P.A. Pellikka, M.R. Bell, C.W. Chow, K.R. Bailey, and J.B. Seward Sex and test verification bias. Impact on the diagnostic value of exercise echocardiography Circulation 95 1997 405 410
42. C. Santana-Boado, J. Candell-Riera, J. Castell-Conesa, S. Aguade-Bruix, A. Garcia-Burillo, and T. Canela Diagnostic accuracy of technetium-99m-MIBI myocardial SPECT in women and men J Nucl Med 39 5 1998 751 755
43. A. Shoaibi, D.R. Tavris, and S. McNulty Gender differences in correlates of troponin assay in diagnosis of myocardial infarction Transl Res 154 5 2009 250 256
44. A.J. Yoon, R.M. Melduni, S.A. Duncan, R.J. Ostfeld, and M.I. Travin The effect of beta-blockers on the diagnostic accuracy of vasodilator pharmacologic SPECT myocardial perfusion imaging J Nucl Cardiol 16 3 2009 358 367
45. M.D. Barber, N.L. Neubauer, and V. Klein-Olarte Can we screen for pelvic organ prolapse without a physical examination in epidemiologic studies? Am J Obstet Gynecol 195 2006 942 948
46. M.S. Lachs, I. Nachamkin, P.H. Edelstein, J. Goldman, A.R. Feinstein, and J.S. Schwartz Spectrum bias in the evaluation of diagnostic tests: lessons from the rapid dipstick test for urinary tract infection Ann Intern Med 117 1992 135 140
47. H. Melbye, and B. Straume The spectrum of patients strongly influences the usefulness of diagnostic tests for pneumonia Scand J Prim Health Care 11 4 1993 241 246
48. P.W. O'Connor, C.M. Tansay, A.S. Detsky, A.I. Mushlin, and W. Kucharczyk The effect of spectrum bias on the utility of magnetic resonance imaging and evoked potentials in the diagnosis of suspected multiple sclerosis Neurology 47 1996 140 144
49. Y.T. van der Schouw, R. Van Dijk, and A.L. Verbeek Problems in selecting the adequate patient population from existing data files for assessment studies of new diagnostic tests J Clin Epidemiol 48 1995 417 422
50. T.K.P. Egglin, and A.R. Feinstein Context bias: a problem in diagnostic radiology JAMA 276 1996 1752 1755
51. W.H. Zhang, S. Levi, S. Alexander, P. Viart, H. Grandjean Eurofetus Study Group Sensitivity of ultrasound screening for congenital anomalies in unselected pregnancies Rev Epidemiol Sante Publique 50 6 2002 571 580
52. A. Rozanski, G.A. Diamond, D. Berman, J.S. Forrester, D. Morris, and H.J. Swan The declining specificity of exercise radionuclide ventriculography N Engl J Med 309 1983 518 522
53. M.J. Tobin, and A. Jubran Variable performance of weaning-predictor tests: role of Bayes' theorem and spectrum and test-referral bias [Review] [67 refs] Intensive Care Med 32 12 2006 2002 2012
54. H. Kittler, H. Pehamberger, K. Wolff, and M. Binder Diagnostic accuracy of dermoscopy [Review] [41 refs] Lancet Oncol 3 3 2002 159 165
55. A.J. Alberg, J.W. Park, B.W. Hager, M.V. Brock, and M. Diener-West The use of "overall accuracy" to evaluate the validity of screening or diagnostic tests J Gen Intern Med 19 2004 460 465
56. M.M. Leeflang, P.M. Bossuyt, and L. Irwig Diagnostic test accuracy may vary with prevalence: implications for evidence-based diagnosis [Review] [38 refs] J Clin Epidemiol 62 2009 5 12
57. D. Ransohoff, and A. Feinstein Problems of spectrum and bias in evaluating the efficacy of diagnostic tests N Engl J Med 299 1978 926 929
58. L.A. Dimatteo, S.R. Lowenstein, B. Brimhall, W. Reiquam, and R. Gonzales The relationship between the clinical features of pharyngitis and the sensitivity of a rapid antigen test: evidence of spectrum bias Ann Emerg Med 38 6 2001 648 652
59. M.C. Hall, B. Kieke, R. Gonzales, and E.A. Belongia Spectrum bias of a rapid antigen detection test for group A beta-hemolytic streptococcal pharyngitis in a pediatric population Pediatrics 114 1 2004 182 186
60. R.G. Pretorius, Y.P. Bao, J.L. Belinson, R.J. Burchette, J.S. Smith, and Y.L. Qiao Inappropriate gold standard bias in cervical cancer screening studies Int J Cancer 121 2007 2218 2224
61. P.D. Stein, A. Gottschalk, J.W. Henry, and K. Shivkumar Stratification of patients according to prior cardiopulmonary disease and probability assessment based on the number of mismatched segmental equivalent perfusion defects. Approaches to strengthen the diagnostic value of ventilation/perfusion lung scans in acute pulmonary embolism Chest 104 1993 1461 1467
62. A. Taube, and B. Tholander Over- and underestimation of the sensitivity of a diagnostic malignancy test due to various selections of the study population Acta Oncol 29 1990 1 5
63. S.D. Brealey, A.J. Scally, S. Hahn, and C. Godfrey Evidence of reference standard related bias in studies of plain radiograph reading performance: a meta-regression [Review] [48 refs] Br J Radiol 80 954 2007 406 413
64. R. Detrano, K.P. Lyons, G. Marcondes, N. Abbassi, V.F. Froelicher, and A. Janosi Methodologic problems in exercise testing research. Are we solving them? Arch Intern Med 148 1988 1289 1295
65. B. Ewald Post hoc choice of cut points introduced bias to diagnostic research [Erratum appears in J Clin Epidemiol. 2007 Jul;60(7):756] J Clin Epidemiol 59 2006 798 801
66. M.M. Leeflang, K.G. Moons, J.B. Reitsma, and A.H. Zwinderman Bias in sensitivity and specificity caused by data-driven selection of optimal cutoff values: mechanisms, magnitude, and solutions Clin Chem 54 2008 729 737
67. W.K. Erly, B.C. Ashdown, R.W. Lucio, R.F. Carmody, J.F. Seeger, and J.N. Alcala Evaluation of emergency CT scans of the head: is there a community standard? AJR Am J Roentgenol 180 6 2003 1727 1730
68. J.H. Moore, D.L. Goss, R.E. Baxter, T.M. DeBerardino, L.T. Mansfield, and D.W. Fellows Clinical diagnostic accuracy and magnetic resonance imaging of patients referred by physical therapists, orthopaedic surgeons, and nonorthopaedic providers J Orthop Sports Phys Ther 35 2005 67 71
69. G. Ciccone, P. Vineis, A. Frigerio, and N. Segnan Inter-observer and intra-observer variability of mammogram interpretation: a field study Eur J Cancer 28A 1992 1054 1058
70. K.S. Berbaum, E.A. Franken Jr., and G.Y. el-Khoury Impact of clinical history on radiographic detection of fractures: a comparison of radiologists and orthopedists AJR Am J Roentgenol 153 6 1989 1221 1224
71. M.B. Cohen, R.P.C. Rodgers, M.S. Hales, J.M. Gonzales, B.M.E. Ljung, and J.H. Beckstead Influence of training and experience in fine-needle aspiration biopsy of breast - receiver operating characteristic curve analysis Arch Pathol Lab Med 111 6 1987 518 520
72. A. Cuaron, A.P. Acero, M. Cardenas, D. Huerta, A. Rodríguez, and R. de Garay Interobserver variability in the interpretation of myocardial images with Tc-99m-labeled diphosphonate and pyrophosphate J Nucl Med 21 1 1980 1 9
73. J. Elmore, C. Wells, and C. Lee Variability in radiologists' interpretation of mammograms N Engl J Med 331 1994 1493 1499
74. S.S. Raab, P.A. Thomas, J.C. Lenel, K. Bottles, K.M. Fitzsimmons, and M.S. Zaleski Pathology and probability: likelihood ratios and receiver operating characteristic curves in the interpretation of bronchial brush specimens Am J Clin Pathol 103 1995 588 593
75. G. Ronco, G. Montanari, V. Aimone, F. Parisio, N. Segnan, and A. Valle Estimating the sensitivity of cervical cytology: errors of interpretation and test limitations Cytopathology 7 3 1996 151 158
76. D.E. Corley, S.H. Kirtland, R.H. Winterbauer, S.P. Hammar, D.H. Dail, and D.E. Bauermeister Reproducibility of the histologic diagnosis of pneumonia among a panel of four pathologists: analysis of a gold standard Chest 112 1997 458 465
77. J.M. Wardlaw, and O. Mielke Early signs of brain infarction at CT: observer reliability and outcome after thrombolytic treatment. Systematic review (Structured abstract) Radiology 235 2005 444 453
78. V.F. Froelicher, K.G. Lehmann, R. Thomas, S. Goldman, D. Morrison, and R. Edson The electrocardiographic exercise test in a population with reduced workup bias: diagnostic performance, computerized interpretation, and multivariable prediction. Veterans Affairs Cooperative Study in Health Services #016 (QUEXTA) Study Group. Quantitative Exercise Testing and Angiography Ann Intern Med 128 12 Pt 1 1998 965 974
79. P. Suri, D.J. Hunter, J.N. Katz, L. Li, and J. Rainville Bias in the physical examination of patients with lumbar radiculopathy BMC Musculoskelet Disord 11 2010 275
80. K.S. Berbaum, G.Y. el-Khoury, E.A. Franken Jr., M. Kathol, W.J. Montgomery, and W. Hesson Impact of clinical history on fracture detection with radiography Radiology 168 1988 507 511
81. P. Doubilet, and P. Herman Interpretation of radiographs: effect of clinical history AJR Am J Roentgenol 137 1981 1055 1058
82. O. Eldevik, G. Dugstad, W. Orrison, and V. Haughton The effect of clinical bias on the interpretation of myelography and spinal computer tomography Radiology 145 1982 85 89
83. E. Potchen, J. Gard, P. Lazar, P. Lahaie, and M. Andary The effect of clinical history data on chest film interpretation: direction or distraction Invest Radiol 14 1979 404
84. M. Schreiber The clinical history as a factor in roentgenogram interpretation JAMA 185 1963 137 139
85. S.S. Raab, T. Oweity, J.H. Hughes, D.R. Salomao, C.M. Kelley, and C.M. Flynn Effect of clinical history on diagnostic accuracy in the cytologic interpretation of bronchial brush specimens Am J Clin Pathol 114 2000 78 83
86. J. Elmore, C. Wells, and D. Howard The impact of clinical history on mammographic interpretations JAMA 277 1997 49 52
87. B. Good, L. Cooperstein, and G. DeMarino Does knowledge of the clinical history affect the accuracy of chest radiograph interpretation? AJR Am J Roentgenol 154 1990 709 712
88. L. Irwig, P. Macaskill, S.D. Walter, and N. Houssami New methods give better estimates of changes in diagnostic accuracy when prior information is provided J Clin Epidemiol 59 2006 299 307
89. S.S. Sonnad, C.P. Langlotz, and J.S. Schwartz Accuracy of MR imaging for staging prostate cancer: a meta-analysis to examine the effect of technologic change Acad Radiol 8 2001 149 157
90. S. Jedrzkiewicz, P. Biaggi, C. Gruner, J. Karski, M. Meineri, and A. Vegas Three-dimensional transesophageal echocardiography accurately predicts mitral valve anatomy in patients undergoing repair for advanced mitral valve prolapse J Am Soc Echocardiogr 24 2011 B44 Conference(var.pagings)
91. J. Mannath, V. Subramanian, E. Telakis, K.W. Lau, and K. Ragunath An inter-observer agreement study of autofluorescence endoscopy in Barrett's oesophagus among expert endoscopists Gastrointest Endosc 26 suppl 1 2011 AB277 Conference(var.pagings)
92. E. Davey, A. Barratt, L. Irwig, S.F. Chan, P. Macaskill, and P. Mannes Effect of study design and quality on unsatisfactory rates, cytology classifications, and accuracy in liquid-based versus conventional cervical cytology: a systematic review Lancet 367 2006 122 132
93. H.M. van Rijkom, and E.H. Verdonschot Factors involved in validity measurements of diagnostic tests for approximal caries - a meta-analysis Caries Res 29 5 1995 364 370
94. J.T. Philbrick, and S. Heim The d-dimer test for deep venous thrombosis: gold standards and bias in negative predictive value Clin Chem 49 2003 570 574
95. G.W. Arana, M.N. Zarzar, and E. Baker The effect of diagnostic methodology on the sensitivity of the TRH stimulation test for depression: a literature review Biol Psychiatry 28 8 1990 733 737
96. A. Cagle, S. Hu, J. Sellors, Y. Bao, J. Lim, and S. Li Use of an expanded gold standard to estimate the accuracy of colposcopy and visual inspection with acetic acid Int J Cancer 126 2010 156 161
97. K.R. Choudhury, D.S. Paik, C.A. Yi, S. Napel, J. Roos, and G.D. Rubin Assessing operating characteristics of CAD algorithms in the absence of a gold standard Med Phys 37 4 2010 1788 1795
98. P. De Neef Evaluating rapid test for streptococcal pharyngitis: the apparent accuracy of a diagnostic test when there are errors in the standard of comparison Med Decis Making 7 1987 92 96
99. E.J. Boyko, B.W. Alderman, and A.E. Baron Reference test errors bias the evaluation of diagnostic tests for ischemic heart disease J Gen Intern Med 3 1988 476 481
100. L. Thibodeau Evaluating diagnostic tests Biometrics 1981 801 804
101. C.E. Phelps, and A. Hutson Estimating diagnostic test accuracy using a "fuzzy gold standard" Med Decis Making 15 1995 44 57
102. M.N. van der Aa, E.W. Steyerberg, C. Bangma, B.W. van Rhijn, E.C. Zwarthoff, and T.H. van der Kwast Cystoscopy revisited as the gold standard for detecting bladder cancer recurrence: diagnostic review bias in the randomized, prospective CEFUB trial J Urol 183 2010 76 80
103. A. Gupta, and C.G. Roehrborn Verification and incorporation biases in studies assessing screening tests: prostate-specific antigen as an example Urology 64 2004 106 111
104. A.S. Rosman, and M.A. Korsten Effect of verification bias on the sensitivity of fecal occult blood testing: a meta-analysis J Gen Intern Med 25 2010 1211 1221
105. B.W. Mol, J.G. Lijmer, J. van der Meulen, E. Pajkrt, C.M. Bilardo, and P.M. Bossuyt Effect of study design on the association between nuchal translucency measurement and Down syndrome Obstet Gynecol 94 5 Pt 2 1999 864 869
106. D. Ransohoff, and W. Muir Diagnostic work-up bias in the evaluation of a test: serum ferritin and hereditary hemochromatosis Med Decis Making 2 1982 139 146
107. M.P. Cecil, A.S. Kosinski, M.T. Jones, A. Taylor, N.P. Alazraki, and R.I. Pettigrew The importance of work-up (verification) bias correction in assessing the accuracy of SPECT thallium-201 testing for the diagnosis of coronary artery disease J Clin Epidemiol 49 1996 735 742
108. R. Panzer, A. Suchman, and P. Griner Workup bias in prediction research Med Decis Making 7 1987 115 119
109. N. Patel, P. Sarkar, K. Omonuwa, B. Babchyck, R. Shah, and A. Talwar Impact of verification bias on pulmonary hypertension using echocardiography and right heart catheterization Chest 2009 136 [Conference(var.pagings)]
110. M.S. Lauer, S.C. Murthy, E.H. Blackstone, I.C. Okereke, and T.W. Rice Fluorodeoxyglucose uptake by positron emission tomography for diagnosis of suspected lung cancer: impact of verification bias Arch Intern Med 167 2007 161 165
111. G.A. Diamond Affirmative actions: can the discriminant accuracy of a test be determined in the face of selection bias? Med Decis Making 11 1991 48 56
112. G.A. Diamond Off Bayes: effect of verification bias on posterior probabilities calculated using Bayes' theorem Med Decis Making 12 1992 22 31
113. H. Nishikawa, Y. Imanaka, M. Sekimoto, and H. Ikai Verification bias in assessment of the utility of MRI in the diagnosis of cruciate ligament tears AJR Am J Roentgenol 195 5 2010 W357 W364
114. J.G. Lijmer, M.G. Hunink, J.J. van den Dungen, J. Loonstra, and A.J. Smit ROC analysis of noninvasive tests for peripheral arterial disease Ultrasound Med Biol 22 4 1996 391 398
115. X.H. Zhou Effect of verification bias on positive and negative predictive values Stat Med 13 1994 1737 1745
116. T.A. Alonzo, J.T. Brinton, B.M. Ringham, and D.H. Glueck Bias in estimating accuracy of a binary screening test with differential disease verification Stat Med 30 2011 1852 1864
117. J.V. Bowler, D.G. Munoz, H. Merskey, and V. Hachinski Fallacies in the pathological confirmation of the diagnosis of Alzheimer's disease J Neurol Neurosurg Psychiatry 64 1 1998 18 24
118. Morise AP, Diamond GA. Does sex discrimination explain the differences in test accuracy among men and women referred for exercise electrocardiography? 67th Scientific Sessions of the American Heart Association, Dallas, Texas, USA, November 1994; 90(4 PART 2).