Validation, updating and impact of clinical prediction rules: A review

Journal of Clinical Epidemiology

Volumn 61, Issue 11, 2008, Pages 1085-1094

  Toll, D B ^a   Janssen, K J M ^a   Vergouwe, Y ^a   Moons, K G M ^a  

^a UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

Author keywords

Generalizability; Impact analysis; Implementation; Prediction rule; Updating; Validation

Indexed keywords

CLINICAL PRACTICE; COST BENEFIT ANALYSIS; DIAGNOSTIC ACCURACY; EMPIRICISM; GEOGRAPHY; MEDICAL DECISION MAKING; MEDICAL RESEARCH; METHODOLOGY; OUTCOME ASSESSMENT; PREDICTION; PRIMARY MEDICAL CARE; PRIORITY JOURNAL; PROGNOSIS; REVIEW; VALIDATION PROCESS;

BIAS (EPIDEMIOLOGY); DECISION SUPPORT TECHNIQUES; DIFFUSION OF INNOVATION; EVIDENCE-BASED MEDICINE; HUMANS; MENINGITIS, BACTERIAL; PREDICTIVE VALUE OF TESTS; PROGNOSIS; REPRODUCIBILITY OF RESULTS;

EID: 52949100612     PISSN: 08954356     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2008.04.008     Document Type: Review

References (60)

1. Altman D.G., and Royston P. What do we mean by validating a prognostic model?. Stat Med 19 (2000) 453-473
2. Laupacis A., Sekar N., and Stiell I.G. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA 277 (1997) 488-494
3. McGinn T.G., Guyatt G.H., Wyer P.C., Naylor C.D., Stiell I.G., and Richardson W.S. Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA 284 (2000) 79-84
4. Reilly B.M., and Evans A.T. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 144 (2006) 201-209
5. Wasson J.H., Sox H.C., Neff R.K., and Goldman L. Clinical prediction rules. Applications and methodological standards. N Engl J Med 313 (1985) 793-799
6. Ingui B.J., and Rogers M.A. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc 8 (2001) 391-397
7. Harrell Jr. F.E., Lee K.L., and Mark D.B. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 15 (1996) 361-387
8. Justice A.C., Covinsky K.E., and Berlin J.A. Assessing the generalizability of prognostic information. Ann Intern Med 130 (1999) 515-524
9. Copas J.B. Regression, prediction and shrinkage. JR Stat Soc B 45 (1983) 311-354
10. Efron B., and Gong G. A leisurely look at the bootstrap, the jackknife, and cross-validation. Am Stat 37 (1983) 36-48
11. Harrell Jr. F.E. Regression modelling strategies with applications to linear models, logistic regression, and survival analysis (2001), Springer, New York
12. Steyerberg E.W., Bleeker S.E., Moll H.A., Grobbee D.E., and Moons K.G. Internal and external validation of predictive models: a simulation study of bias and precision in small samples. J Clin Epidemiol 56 (2003) 441-447
13. Steyerberg E.W., Eijkemans M.J., Harrell Jr. F.E., and Habbema J.D. Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med 19 (2000) 1059-1079
14. Steyerberg E.W., Harrell Jr. F.E., Borsboom G.J., Eijkemans M.J., Vergouwe Y., and Habbema J.D. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 54 (2001) 774-781
15. Bleeker S.E., Moll H.A., Steyerberg E.W., Donders A.R., Derksen-Lubsen G., Grobbee D.E., et al. External validation is necessary in prediction research: a clinical example. J Clin Epidemiol 56 (2003) 826-832
16. Nashef S.A., Roques F., Michel P., Gauducheau E., Lemeshow S., and Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 16 (1999) 9-13
17. Roques F., Nashef S.A., Michel P., Gauducheau E., de Vincentiis C., Baudet E., et al. Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg 15 (1999) 816-822 discussion 22-23
18. Geissler H.J., Holzl P., Marohl S., Kuhn-Regnier F., Mehlhorn U., Sudkamp M., et al. Risk stratification in heart surgery: comparison of six score systems. Eur J Cardiothorac Surg 17 (2000) 400-406
19. Gogbashian A., Sedrakyan A., and Treasure T. EuroSCORE: a systematic review of international performance. Eur J Cardiothorac Surg 25 (2004) 695-700
20. Kawachi Y., Nakashima A., Toshima Y., Arinaga K., and Kawano H. Risk stratification analysis of operative mortality in heart and thoracic aorta surgery: comparison between Parsonnet and EuroSCORE additive model. Eur J Cardiothorac Surg 20 (2001) 961-966
21. Michel P., Roques F., and Nashef S.A. Logistic or additive EuroSCORE for high-risk patients?. Eur J Cardiothorac Surg 23 (2003) 684-687 discussion 7
22. Nashef S.A., Roques F., Hammill B.G., Peterson E.D., Michel P., Grover F.L., et al. Validation of European System for Cardiac Operative Risk Evaluation (EuroSCORE) in North American cardiac surgery. Eur J Cardiothorac Surg 22 (2002) 101-105
23. Nilsson J., Algotsson L., Hoglund P., Luhrs C., and Brandt J. Early mortality in coronary bypass surgery: the EuroSCORE versus The Society of Thoracic Surgeons risk algorithm. Ann Thorac Surg 77 (2004) 1235-1239 discussion 9-40
24. Yap C.H., Reid C., Yii M., Rowland M.A., Mohajeri M., Skillington P.D., et al. Validation of the EuroSCORE model in Australia. Eur J Cardiothorac Surg 29 (2006) 441-446 discussion 6
25. Nashef S.A. Editorial comment EuroSCORE and the Japanese aorta. Eur J Cardiothorac Surg 30 (2006) 582-583
26. Oostenbrink R., Moons K.G., Derksen-Lubsen G., Grobbee D.E., and Moll H.A. Early prediction of neurological sequelae or death after bacterial meningitis. Acta Paediatr 91 (2002) 391-398
27. Vergouwe Y., Steyerberg E.W., Eijkemans M.J., and Habbema J.D. Validity of prognostic models: when is a model clinically useful?. Semin Urol Oncol 20 (2002) 96-107
28. Peek N., Arts D.G., Bosman R.J., van der Voort P.H., and de Keizer N.F. External validation of prognostic models for critically ill patients required substantial sample sizes. J Clin Epidemiol 60 (2007) 491-501
29. Vergouwe Y., Steyerberg E.W., Eijkemans M.J., and Habbema J.D. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol 58 (2005) 475-483
30. Starmans R., Muris J.W., Fijten G.H., Schouten H.J., Pop P., and Knottnerus J.A. The diagnostic value of scoring models for organic and non-organic gastrointestinal disease, including the irritable-bowel syndrome. Med Decis Making 14 (1994) 208-216
31. Oudega R., Moons K.G., and Hoes A.W. Ruling out deep venous thrombosis in primary care. A simple diagnostic algorithm including D-dimer testing. Thromb Haemost 94 (2005) 200-205
32. Wells P.S., Anderson D.R., Rodger M., Forgie M., Kearon C., Dreyer J., et al. Evaluation of D-dimer in the diagnosis of suspected deep-vein thrombosis. N Engl J Med 349 (2003) 1227-1235
33. Di Nisio M., Squizzato A., Rutjes A.W., Buller H.R., Zwinderman A.H., and Bossuyt P.M. Diagnostic accuracy of D-dimer test for exclusion of venous thromboembolism: a systematic review. J Thromb Haemost 5 (2007) 296-304
34. Bont J., Hak E., Hoes A.W., Schipper M., Schellevis F.G., and Verheij T.J. A prediction rule for elderly primary-care patients with lower respiratory tract infections. Eur Respir J 29 (2007) 969-975
35. Knottnerus J.A. Between iatrotropic stimulus and interiatric referral: the domain of primary care research. J Clin Epidemiol 55 (2002) 1201-1206
36. Wells P.S., Hirsh J., Anderson D.R., Lensing A.W., Foster G., Kearon C., et al. Accuracy of clinical assessment of deep-vein thrombosis. Lancet 345 8961 (1995) 1326-1330
37. Oudega R., Hoes A.W., and Moons K.G. The Wells rule does not adequately rule out deep venous thrombosis in primary care patients. Ann Intern Med 143 (2005) 100-107
38. Stiell I.G., Greenberg G.H., McKnight R.D., Nair R.C., McDowell I., and Worthington J.R. A study to develop clinical decision rules for the use of radiography in acute ankle injuries. Ann Emerg Med 21 (1992) 384-390
39. Myers A., Canty K., and Nelson T. Are the Ottawa ankle rules helpful in ruling out the need for x ray examination in children?. Arch Dis Child 90 (2005) 1309-1311
40. Altman DG. Prognostic models: a methodological framework and review of models for breast cancer. In: Lyman GH, Burstein HJ, editors. Breast cancer translational therapeutic strategies. New York; Informa Healthcare: 2007.
41. Hosmer Dw S.L. Applied logistic regression (1989), John Wiley and Sons Inc., New York
42. Ivanov J., Tu J.V., and Naylor C.D. Ready-made, recalibrated, or Remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery. Circulation 99 (1999) 2098-2104
43. Steyerberg E.W., Borsboom G.J., van Houwelingen H.C., Eijkemans M.J., and Habbema J.D. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med 23 (2004) 2567-2586
44. Janssen K.J., Moons K.G., Kalkman C.J., Grobbee D.E., and Vergouwe Y. Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 61 (2008) 76-86
45. D'Agostino Sr. R.B., Grundy S., Sullivan L.M., and Wilson P. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA 286 (2001) 180-187
46. Liu J., Hong Y., D'Agostino Sr. R.B., Wu Z., Wang W., Sun J., et al. Predictive value for the Chinese population of the Framingham CHD risk assessment tool compared with the Chinese Multi-Provincial Cohort Study. JAMA 291 (2004) 2591-2599
47. Campbell M.K., Elbourne D.R., and Altman D.G. CONSORT statement: extension to cluster randomised trials. BMJ 328 7441 (2004) 702-708
48. Foy R., Penney G.C., Grimshaw J.M., Ramsay C.R., Walker A.E., MacLennan G., et al. A randomised controlled trial of a tailored multifaceted strategy to promote implementation of a clinical guideline on induced abortion care. BJOG 111 (2004) 726-733
49. Marrie T.J., Lau C.Y., Wheeler S.L., Wong C.J., Vandervoort M.K., and Feagan B.G. A controlled trial of a critical pathway for treatment of community-acquired pneumonia. CAPITAL Study Investigators. Community-Acquired Pneumonia Intervention Trial Assessing Levofloxacin. JAMA 283 (2000) 749-755
50. Meyer G., Kopke S., Bender R., and Muhlhauser I. Predicting the risk of falling-efficacy of a risk assessment tool compared to nurses' judgement: a cluster-randomised controlled trial [ISRCTN37794278]. BMC Geriatr 5 (2005) 14
51. Stiell I., Wells G., Laupacis A., Brison R., Verbeek R., Vandemheen K., et al. Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries. Multicentre Ankle Rule Study Group. BMJ 311 7005 (1995) 594-597
52. Sinuff T., Adhikari N.K., Cook D.J., Schunemann H.J., Griffith L.E., Rocker G., et al. Mortality predictions in the intensive care unit: comparing physicians with scoring systems. Crit Care Med 34 (2006) 878-885
53. Blackmore C.C. Clinical prediction rules in trauma imaging: who, how, and why?. Radiology 235 (2005) 371-374
54. Braitman L.E., and Davidoff F. Predicting clinical states in individual patients. Ann Intern Med 125 (1996) 406-412
55. Brehaut J.C., Stiell I.G., Visentin L., and Graham I.D. Clinical decision rules "in the real world": how a widely disseminated rule is used in everyday practice. Acad Emerg Med 12 (2005) 948-956
56. Brehaut J.C., Stiell I.G., and Graham I.D. Will a new clinical decision rule be widely used? The case of the Canadian C-spine rule. Acad Emerg Med 13 (2006) 413-420
57. Stiell I.G., Wells G.A., Vandemheen K.L., Clement C.M., Lesiuk H., De Maio V.J., et al. The Canadian C-spine rule for radiography in alert and stable trauma patients. JAMA 286 (2001) 1841-1848
58. Altman D.G. Systematic reviews of evaluations of prognostic variables. BMJ 323 7306 (2001) 224-228
59. Riley R.D., Abrams K.R., Sutton A.J., Lambert P.C., Jones D.R., Heney D., et al. Reporting of prognostic markers: current problems and development of guidelines for evidence-based practice in the future. Br J Cancer 88 (2003) 1191-1198
60. Khan K.S., Bachmann L.M., and ter Riet G. Systematic reviews with individual patient data meta-analysis to evaluate diagnostic tests. Eur J Obstet Gynecol Reprod Biol 108 (2003) 121-125