Use of Expert Panels to Define the Reference Standard in Diagnostic Research: A Systematic Review of Published Methods and Reporting

PLoS Medicine

Volumn 10, Issue 10, 2013, Pages

  Bertens, Loes C M ^a   Broekhuizen, Berna D L ^a   Naaktgeboren, Christiana A ^a   Rutten, Frans H ^a   Hoes, Arno W ^a   van Mourik, Yvonne ^a   Moons, Karel G M ^a   Reitsma, Johannes B ^a  

^a UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; CARDIOVASCULAR DISEASE; CHECKLIST; DATA EXTRACTION; DECISION MAKING; DIAGNOSTIC ACCURACY; FOLLOW UP; HUMAN; MENTAL DISEASE; PANEL STUDY; PATIENT GUIDANCE; PREVALENCE; REPRODUCIBILITY; RESEARCH DIAGNOSTIC CRITERIA; RESPIRATORY TRACT DISEASE; STANDARD; SYSTEMATIC REVIEW;

DECISION MAKING; EXPERT TESTIMONY; HUMANS; PUBLISHING; REFERENCE STANDARDS;

EID: 84886637077     PISSN: 15491277     EISSN: 15491676     Source Type: Journal    
DOI: 10.1371/journal.pmed.1001531     Document Type: Article

References (98)

1. Reitsma JB, Rutjes AW, Khan KS, Coomarasamy A, Bossuyt PM, (2009) A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard. J Clin Epidemiol 62: 797-806.
2. Hadgu A, Dendukuri N, Hilden J, (2005) Evaluation of nucleic acid amplification tests in the absence of a perfect gold-standard test: a review of the statistical and epidemiologic issues. Epidemiology 16: 604-612.
3. Alonzo TA, Pepe MS, (1999) Using a combination of reference tests to assess the accuracy of a new diagnostic test. Stat Med 18: 2987-3003.
4. Pepe MS, Janes H, (2007) Insights into latent class analysis of diagnostic test performance. Biostatistics 8: 474-484.
5. Baughman AL, Bisgard KM, Cortese MM, Thompson WW, Sanden GN, et al. (2008) Utility of composite reference standards and latent class analysis in evaluating the clinical accuracy of diagnostic tests for pertussis. Clin Vaccine Immunol 15: 106-114.
6. Moons KG, Grobbee DE, (2002) When should we remain blind and when should our eyes remain open in diagnostic studies? J Clin Epidemiol 55: 633-636.
7. Magaziner J, Zimmerman SI, German PS, Kuhn K, May C, et al. (1996) Ascertaining dementia by expert panel in epidemiologic studies of nursing home residents. Ann Epidemiol 6: 431-437.
8. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, et al. (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med 6: e1000100 doi:10.1371/journal.pmed.1000100.
9. Lilford RJ, Richardson A, Stevens A, Fitzpatrick R, Edwards S, et al. (2001) Issues in methodological research: perspectives from researchers and commissioners. Health Technol Assess 2001 5: 1-57.
10. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, et al. (2003) The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med 138: W1-W12.
11. Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ JB, et al. (2011) QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 155: 529-536.
12. Baird G, Simonoff E, Pickles A, Chandler S, Loucas T, et al. (2006) Prevalence of disorders of the autism spectrum in a population cohort of children in South Thames: the Special Needs and Autism Project (SNAP). Lancet 368: 210-215.
13. Bienvenu OJ, Samuels JF, Riddle MA, Hoehn-Saric R, Liang KY, et al. (2000) The relationship of obsessive-compulsive disorder to possible spectrum disorders: results from a family study. Biol Psychiatry 48: 287-293.
14. Boustani M, Callahan CM, Unverzagt FW, Austrom MG, Perkins AJ, et al. (2005) Implementing a screening and diagnosis program for dementia in primary care. J Gen Intern Med 20: 572-577.
15. Brugha TS, McManus S, Smith J, Scott FJ, Meltzer H, et al. (2012) Validating two survey methods for identifying cases of autism spectrum disorder among adults in the community. Psychol Med 42: 647-656.
16. Carnero-Pardo C, Espejo-Martinez B, Lopez-Alcalde S, Espinosa-Garcia M, Saez-Zea C, et al. (2011) Diagnostic accuracy, effectiveness and cost for cognitive impairment and dementia screening of three short cognitive tests applicable to illiterates. PLoS One 6: e27069 doi:10.1371/journal.pone.0027069.
17. Class CA, Unverzagt FW, Gao S, Hall KS, Baiyewa O, et al. (1996) Psychiatric disorders in African American nursing home residents. Am J Psychiatry 153: 677-681.
18. de Koning HJ, de Ridder-Sluiter JG, van Agt HM, Reep-van den Bergh CM, van der Stege HA, et al. (2004) A cluster-randomised trial of screening for language disorders in toddlers. J Med Screen 11: 109-116.
19. Drake RE, Osher FC, Noordsy DL, Hurlbut SC, Teague GB, et al. (1990) Diagnosis of alcohol use disorders in schizophrenia. Schizophr Bull 16: 57-67.
20. Duberstein PR, Ma Y, Chapman BP, Conwell Y, McGriff J, et al. (2011) Detection of depression in older adults by family and friends: distinguishing mood disorder signals from the noise of personality and everyday life. Int Psychogeriatr 23: 634-643.
21. Fennig S, Craig TJ, Tanenberg-Karant M, Bromet EJ, (1994) Comparison of facility and research diagnoses in first-admission psychotic patients. Am J Psychiatry 151: 1423-1429.
22. Fladby T, Schuster M, Gronli O, Sjoholm H, Loseth S, et al. (1999) Organic brain disease in psychogeriatric patients: impact of symptoms and screening methods on the diagnostic process. J Geriatr Psychiatry Neurol 12: 16-20.
23. Girard C, Simard M, Noiseux R, Laplante L, Dugas M, et al. (2011) Late-onset-psychosis: cognition. Int Psychogeriatr 23: 1301-1316.
24. Graff-Radford NR, Ferman TJ, Lucas JA, Johnson HK, Parfitt FC, et al. (2006) A cost effective method of identifying and recruiting persons over 80 free of dementia or mild cognitive impairment. Alzheimer Dis Assoc Disord 20: 101-104.
25. Gulevich SJ, Conwell TD, Lane J, Lockwood B, Schwettmann RS, et al. (1997) Stress infrared telethermography is useful in the diagnosis of complex regional pain syndrome, type I (formerly reflex sympathetic dystrophy). Clin J Pain 13: 50-59.
26. Hall KS, Gao S, Baiyewu O, Lane KA, Gureje O, et al. (2009) Prevalence rates for dementia and Alzheimer's disease in African Americans: 1992 versus 2001. Alzheimers Dement 5: 227-233.
27. Johnson S, Hollis C, Hennessy E, Kochhar P, Wolke D, et al. (2011) Screening for autism in preterm children: diagnostic utility of the Social Communication Questionnaire. Arch Dis Child 96: 73-77.
28. Laurila JV, Pitkala KH, Strandberg TE, Tilvis RS, (2004) Delirium among patients with and without dementia: does the diagnosis according to the DSM-IV differ from the previous classifications? Int J Geriatr Psychiatry 19: 271-277.
29. Magaziner J, German P, Zimmerman SI, Hebel JR, Burton L, et al. (2000) The prevalence of dementia in a statewide sample of new nursing home admissions aged 65 and older: diagnosis by expert panel. Epidemiology of Dementia in Nursing Homes Research Group. Gerontologist 40: 663-672.
30. Miller PR, Dasher R, Collins R, Griffiths P, Brown F, (2001) Inpatient diagnostic assessments: 1. Accuracy of structured vs. unstructured interviews. Psychiatry Res 105: 255-264.
31. Ogunniyi A, Daif AK, Al-Rajeh S, AbdulJabbar M, Al-Tahan AR, et al. (1998) Dementia in Saudi Arabia: experience from a university hospital. Acta Neurol Scand 98: 116-120.
32. Ogunniyi A, Lane KA, Baiyewu O, Gao S, Gureje O, et al. (2011) Hypertension and incident dementia in community-dwelling elderly Yoruba Nigerians. Acta Neurol Scand 124: 396-402.
33. Plassman BL, Langa KM, Fisher GG, Heeringa SG, Weir DR, et al. (2007) Prevalence of dementia in the United States: the aging, demographics, and memory study. Neuroepidemiology 29: 125-132.
34. Plassman BL, Langa KM, McCammon RJ, Fisher GG, Potter GG, et al. (2011) Incidence of dementia and cognitive impairment, not dementia in the United States. Ann Neurol 70: 418-426.
35. Potter GG, Plassman BL, Burke JR, Kabeto MU, Langa KM, et al. (2009) Cognitive performance and informant reports in the diagnosis of cognitive impairment and dementia in African Americans and whites. Alzheimers Dement 5: 445-453.
36. Sachdev PS, Brodaty H, Valenzuela MJ, Lorentz L, Looi JC, et al. (2006) Clinical determinants of dementia and mild cognitive impairment following ischaemic stroke: the Sydney Stroke Study. Dement Geriatr Cogn Disord 21: 275-283.
37. Steenland NK, Auman CM, Patel PM, Bartell SM, Goldstein FC, et al. (2008) Development of a rapid screening instrument for mild cognitive impairment and undiagnosed dementia. J Alzheimers Dis 15: 419-427.
38. Tanenberg-Karant M, Fennig S, Ram R, Krishna J, Jandorf L, et al. (1995) Bizarre delusions and first-rank symptoms in a first-admission sample: a preliminary analysis of prevalence and correlates. Compr Psychiatry 36: 428-434.
39. Weintraub D, Raskin A, Ruskin PE, Gruber-Baldini AL, Zimmerman SI, et al. (2000) Racial differences in the prevalence of dementia among patients admitted to nursing homes. Psychiatr Serv 51: 1259-1264.
40. Wiener P, Alexopoulos GS, Kakuma T, Meyers BS, Rosenthal E, et al. (1997) The limits of history-taking in geriatric depression. Am J Geriatr Psychiatry 5: 116-125.
41. Williams J, Scott F, Stott C, Allison C, Bolton P, et al. (2005) The CAST (Childhood Asperger Syndrome Test: test accuracy. Autism 9: 45-68.
42. Assomull RG, Shakespeare C, Kalra PR, Lloyd G, Gulati A, et al. (2011) Role of cardiovascular magnetic resonance as a gatekeeper to invasive coronary angiography in patients presenting with heart failure of unknown etiology. Circulation 124: 1351-1360.
43. Bosner S, Haasenritter J, Becker A, Karatolios K, Vaucher P, et al. (2010) Ruling out coronary artery disease in primary care: development and validation of a simple prediction rule. CMAJ 182: 1295-1300.
44. Dao Q, Krishnaswamy P, Kazanegra R, Harrison A, Amirnovin R, et al. (2001) Utility of B-type natriuretic peptide in the diagnosis of congestive heart failure in an urgent-care setting. J Am Coll Cardiol 37: 379-385.
45. Doubal FN, Dennis MS, Wardlaw JM, (2011) Characteristics of patients with minor ischaemic strokes and negative MRI: a cross-sectional study. J Neurol Neurosurg Psychiatry 82: 540-542.
46. Gaikwad AB, Mudalgi BA, Patankar KB, Patil JK, Ghongade DV, (2008) Diagnostic role of 64-slice multidetector row CT scan and CT venogram in cases of cerebral venous thrombosis. Emerg Radiol 15: 325-333.
47. Hoffmann R, Borges AC, Kasprzak JD, von BS, Firschke C, et al. (2007) Analysis of myocardial perfusion or myocardial function for detection of regional myocardial abnormalities. An echocardiographic multicenter comparison study using myocardial contrast echocardiography and 2D echocardiography. Eur J Echocardiogr 8: 438-448.
48. Hoffmann R, von BS, Kasprzak JD, Borges AC, ten CF, et al. (2006) Analysis of regional left ventricular function by cineventriculography, cardiac magnetic resonance imaging, and unenhanced and contrast-enhanced echocardiography: a multicenter comparison of methods. J Am Coll Cardiol 47: 121-128.
49. Hoffmann U, Nagurney JT, Moselewski F, Pena A, Ferencik M, et al. (2006) Coronary multidetector computed tomography in the assessment of patients with acute chest pain. Circulation 114: 2251-2260.
50. Hoffmann U, Pena AJ, Moselewski F, Ferencik M, Abbara S, et al. (2006) MDCT in early triage of patients with acute chest pain. Am J Roentgenol 187: 1240-1247.
51. Kantarci M, Ceviz N, Sevimli S, Bayraktutan U, Ceyhan E, et al. (2007) Diagnostic performance of multidetector computed tomography for detecting aorto-ostial lesions compared with catheter coronary angiography: multidetector computed tomography coronary angiography is superior to catheter angiography in detection of aorto-ostial lesions. J Comput Assist Tomogr 31: 595-599.
52. Kelder JC, Cramer MJ, Verweij WM, Grobbee DE, Hoes AW, (2011) Clinical utility of three B-type natriuretic peptide assays for the initial diagnostic assessment of new slow-onset heart failure. J Card Fail 17: 729-734.
53. Kelder JC, Cramer MJ, Rutten FH, Plokker HW, Grobbee DE, et al. (2011) The furosemide diagnostic test in suspected slow-onset heart failure: popular but not useful. Eur J Heart Fail 13: 513-517.
54. Linn J, Ertl-Wagner B, Seelos KC, Strupp M, Reiser M, et al. (2007) Diagnostic value of multidetector-row CT angiography in the evaluation of thrombosis of the cerebral venous sinuses. AJNR Am J Neuroradiol 28: 946-952.
55. Nordenholz KE, Zieske M, Dyer DS, Hanson JA, Heard K, (2007) Radiologic diagnoses of patients who received imaging for venous thromboembolism despite negative D-dimer tests. Am J Emerg Med 25: 1040-1046.
56. Oudejans I, Mosterd A, Bloemen JA, Valk MJ, van Velzen E, et al. (2011) Clinical evaluation of geriatric outpatients with suspected heart failure: value of symptoms, signs, and additional tests. Eur J Heart Fail 13: 518-527.
57. Remy-Jardin M, Remy J, Masson P, Bonnel F, Debatselier P, et al. (2000) CT angiography of thoracic outlet syndrome: evaluation of imaging protocols for the detection of arterial stenosis. J Comput Assist Tomogr 24: 349-361.
58. Trevelyan J, Needham EW, Smith SC, Mattu RK, (2003) Sources of diagnostic inaccuracy of conventional versus new diagnostic criteria for myocardial infarction in an unselected UK population with suspected cardiac chest pain, and investigation of independent prognostic variables. Heart 89: 1406-1410.
59. Broekhuizen BD, Sachs AP, Hoes AW, Moons KG, van den Berg JW, et al. (2010) Undetected chronic obstructive pulmonary disease and asthma in people over 50 years with persistent cough. Br J Gen Pract 60: 489-494.
60. Broekhuizen BD, Sachs AP, Moons KG, Cheragwandi SA, Damste HE, et al. (2011) Diagnostic value of oral prednisolone test for chronic obstructive pulmonary disorders. Ann Fam Med 9: 104-109.
61. Chavannes N, Schermer T, Akkermans R, Jacobs JE, van de Graaf G, et al. (2004) Impact of spirometry on GPs' diagnostic differentiation and decision-making. Respir Med 98: 1124-1130.
62. Gauvin F, Dassa C, Chaibou M, Proulx F, Farrell CA, et al. (2003) Ventilator-associated pneumonia in intubated children: comparison of different diagnostic methods. Pediatr Crit Care Med 4: 437-443.
63. Guder G, Brenner S, Angermann CE, Ertl G, Held M, et al. (2012) "GOLD or lower limit of normal definition? A comparison with expert-based diagnosis of chronic obstructive pulmonary disease in a prospective cohort-study". Respir Res 13: 13.
64. Mohamed Hoesein FA, Zanen P, Sachs AP, Verheij TJ, Lammers JW, et al. (2012) Spirometric Thresholds for Diagnosing COPD: 0.70 or LLN, Pre- or Post-dilator Values? COPD 9: 338-343.
65. Reinartz P, Wildberger JE, Schaefer W, Nowak B, Mahnken AH, et al. (2004) Tomographic imaging in the diagnosis of pulmonary embolism: a comparison between V/Q lung scintigraphy in SPECT technique and multislice spiral CT. J Nucl Med 45: 1501-1508.
66. Reinartz P, Kaiser HJ, Wildberger JE, Gordji C, Nowak B, et al. (2006) SPECT imaging in the diagnosis of pulmonary embolism: automated detection of match and mismatch defects by means of image-processing techniques. J Nucl Med 47: 968-973.
67. Szucs-Farkas Z, Schaller C, Bensler S, Patak MA, Vock P, et al. (2009) Detection of pulmonary emboli with CT angiography at reduced radiation exposure and contrast material volume: comparison of 80 kVp and 120 kVp protocols in a matched cohort. Invest Radiol 44: 793-799.
68. Thieme SF, Graute V, Nikolaou K, Maxien D, Reiser MF, et al. (2012) Dual Energy CT lung perfusion imaging-correlation with SPECT/CT. Eur J Radiol 81: 360-365.
69. Geirnaerdt MJ, Hermans J, Bloem JL, Kroon HM, Pope TL, et al. (1997) Usefulness of radiography in differentiating enchondroma from central grade 1 chondrosarcoma. AJR Am J Roentgenol 169: 1097-1104.
70. Jorgensen F, Fruergaard P, Launbjerg J, Aggestrup S, Elsborg L, et al. (1998) The diagnostic value of oesophageal radionuclide transit in patients admitted for but without acute myocardial infarction. Clin Physiol 18: 89-96.
71. Marshall JK, Cawdron R, Zealley I, Riddell RH, Somers S, et al. (2004) Prospective comparison of small bowel meal with pneumocolon versus ileo-colonoscopy for the diagnosis of ileal Crohn's disease. Am J Gastroenterol 99: 1321-1329.
72. Martinez FJ, Stanopoulos I, Acero R, Becker FS, Pickering R, et al. (1994) Graded comprehensive cardiopulmonary exercise testing in the evaluation of dyspnea unexplained by routine evaluation. Chest 105: 168-174.
73. Ray P, Birolleau S, Lefort Y, Becquemin MH, Beigelman C, et al. (2006) Acute respiratory failure in the elderly: etiology, emergency diagnosis and prognosis. Crit Care 10: R82.
74. Rutten FH, Cramer MJ, Grobbee DE, Sachs AP, Kirkels JH, et al. (2005) Unrecognized heart failure in elderly patients with stable chronic obstructive pulmonary disease. Eur Heart J 26: 1887-1894.
75. White CS, Kuo D, Kelemen M, Jain V, Musk A, et al. (2005) Chest pain evaluation in the emergency department: can MDCT provide a comprehensive evaluation? AJR Am J Roentgenol 185: 533-540.
76. Amour J, Birenbaum A, Langeron O, Le MY, Bertrand M, et al. (2008) Influence of renal dysfunction on the accuracy of procalcitonin for the diagnosis of postoperative infection after vascular surgery. Crit Care Med 36: 1147-1154.
77. Bisulli F, Vignatelli L, Naldi I, Pittau F, Provini F, et al. (2012) Diagnostic accuracy of a structured interview for nocturnal frontal lobe epilepsy (SINFLE: a proposal for developing diagnostic criteria. Sleep Med 13: 81-87.
78. Gamez-Diaz LY, Enriquez LE, Matute JD, Velasquez S, Gomez ID, et al. (2011) Diagnostic accuracy of HMGB-1, sTREM-1, and CD64 as markers of sepsis in patients recently admitted to the emergency department. Acad Emerg Med 18: 807-815.
79. Ham H, McInnes MD, Woo M, Lemonde S, (2012) Negative predictive value of intravenous contrast-enhanced CT of the abdomen for patients presenting to the emergency department with undifferentiated upper abdominal pain. Emerg Radiol 191: 19-26.
80. Hardie AD, Naik M, Hecht EM, Chandarana H, Mannelli L, et al. (2010) Diagnosis of liver metastases: value of diffusion-weighted MRI compared with gadolinium-enhanced MRI. Eur Radiol 20: 1431-1441.
81. Humphries PD, Simpson JC, Creighton SM, Hall-Craggs MA, (2008) MRI in the assessment of congenital vaginal anomalies. Clin Radiol 63: 442-448.
82. Lin WC, Uppot RN, Li CS, Hahn PF, Sahani DV, (2007) Value of automated coronal reformations from 64-section multidetector row computerized tomography in the diagnosis of urinary stone disease. J Urol 178: 907-911.
83. O'Toole RV, Cox G, Shanmuganathan K, Castillo RC, Turen CH, et al. (2010) Evaluation of computed tomography for determining the diagnosis of acetabular fractures. J Orthop Trauma 24: 284-290.
84. Otte JA, Geelkerken RH, Oostveen E, Mensink PB, Huisman AB, et al. (2005) Clinical impact of gastric exercise tonometry on diagnosis and management of chronic gastrointestinal ischemia. Clin Gastroenterol Hepatol 3: 660-666.
85. Penzkofer AK, Pfluger T, Pochmann Y, Meissner O, Leinsinger G, (2002) MR imaging of the brain in pediatric patients: diagnostic value of HASTE sequences. AJR Am J Roentgenol 179: 509-514.
86. Robin TA, Muller A, Rait J, Keeffe JE, Taylor HR, et al. (2005) Performance of community-based glaucoma screening using Frequency Doubling Technology and Heidelberg Retinal Tomography. Ophthalmic Epidemiol 12: 167-178.
87. Tadros AM, Lunsjo K, Czechowski J, Corr P, bu-Zidan FM, (2007) Usefulness of different imaging modalities in the assessment of scapular fractures caused by blunt trauma. Acta Radiol 48: 71-75.
88. Tepper SJ, Dahlof CG, Dowson A, Newman L, Mansbach H, et al. (2004) Prevalence and diagnosis of migraine in patients consulting their physician with a complaint of headache: data from the Landmark Study. Headache 44: 856-864.
89. Thabut D, D'Amico G, Tan P, De Francis R, Fabricius S, et al. (2010) Diagnostic performance of Baveno IV criteria in cirrhotic patients with upper gastrointestinal bleeding: analysis of the F7 liver-1288 study population. J Hepatol 53: 1029-1034.
90. van Randen A, Lameris W, van Es HW, van Heesewijk HP, van Ramhorst B, et al. (2011) A comparison of the accuracy of ultrasound and computed tomography in common diagnoses causing acute abdominal pain. Eur Radiol 21: 1535-1545.
91. Weih LM, Nanjan M, McCarty CA, Taylor HR, (2001) Prevalence and predictors of open-angle glaucoma: results from the visual impairment project. Ophthalmology 108: 1966-1972.
92. Whiteley WN, Wardlaw JM, Dennis MS, Sandercock PA, (2011) Clinical scores for the identification of stroke and transient ischaemic attack in the emergency department: a cross-sectional study. J Neurol Neurosurg Psychiatry 82: 1006-1010.
93. Gabel MJ, Shipan CR, (2004) A social choice approach to expert consensus panels. J Health Econ 23: 543-564.
94. de Groot JA, Bossuyt PM, Reitsma JB, Rutjes AW, Dendukuri N, et al. (2011) Verification problems in diagnostic accuracy studies: consequences and solutions. BMJ 343: d4770.
95. Bankier AA, Levine D, Halpern EF, Kressel HY, (2010) Consensus interpretation in imaging research: is there a better way? Radiology 257: 14-17.
96. Obuchowski NA, Zepp RC, (1996) Simple steps for improving multiple-reader studies in radiology. AJR Am J Roentgenol 166: 517-521.
97. Ransohoff DF, Feinstein AR, (1978) Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med 299: 926-930.
98. Rutjes AW, Reitsma JB, Coomarasamy A, Khan KS, Bossuyt PM, (2007) Evaluation of diagnostic tests when there is no gold standard. A review of methods. Health Technol Assess 11: iii, ix-51.