Early warning scores for detecting deterioration in adult hospital patients: Systematic review and critical appraisal of methodology

The BMJ

Volumn 369, Issue 3, 2020, Pages

  Gerry, Stephen ^a   Bonnici, Timothy ^b   Birks, Jacqueline ^a,c   Kirtley, Shona ^a   Virdee, Pradeep S ^a   Watkinson, Peter J ^a   Collins, Gary S ^a,c  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; BOOTSTRAPPING; BREATHING RATE; CALIBRATION; CINAHL; CLINICAL PRACTICE; CROSS VALIDATION; DETERIORATION; EARLY WARNING SCORE; ELIGIBILITY CRITERIA; EMBASE; FEMALE; HEART RATE; HOSPITAL PATIENT; HUMAN; MALE; MEDLINE; OXYGEN SATURATION; PATIENT CARE; PREDICTION; PSYCINFO; REVIEW; SAMPLE SIZE; SYSTEMATIC REVIEW; SYSTOLIC BLOOD PRESSURE; UNITED KINGDOM; UNITED STATES; VALIDATION PROCESS; VALIDATION STUDY; AGED; BLOOD PRESSURE; DEATH; HEART ARREST; HOSPITAL; INTENSIVE CARE; METABOLISM; MIDDLE AGED; PHYSIOLOGY; PROCEDURES; PROGNOSIS; TEMPERATURE;

OXYHEMOGLOBIN;

ADULT; AGED; BLOOD PRESSURE; CLINICAL DETERIORATION; CRITICAL CARE; DEATH; EARLY WARNING SCORE; FEMALE; HEART ARREST; HEART RATE; HOSPITALS; HUMANS; INPATIENTS; MALE; MIDDLE AGED; OXYHEMOGLOBINS; PROGNOSIS; RESPIRATORY RATE; TEMPERATURE; UNITED KINGDOM; UNITED STATES;

EID: 85085156752     PISSN: 09598146     EISSN: 17561833     Source Type: Journal    
DOI: 10.1136/bmj.m1501     Document Type: Review

References (155)

1. Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med 1991;324: 370-6. doi: 10.1056/NEJM199102073240604
2. Donaldson M, Corrigan J, Kohn L. To err is human: Building a safer health system. 2000. doi.org/10.17226/9728
3. Vincent C, Neale G, Woloshynowych M. Adverse events in British hospitals: Preliminary retrospective record review. BMJ 2001;322: 517-9. doi: 10.1136/bmj.322.7285.517
4. Hillman KM, Bristow PJ, Chey T, et al. Duration of life-threatening antecedents prior to intensive care admission. Intensive Care Med 2002;28: 1629-34. doi: 10.1007/s00134-002-1496-y
5. Morgan R, Lloyd-Williams F, Wright M, Morgan-Warren RJ. An early warning scoring system for detecting developing critical illness. 1997. https://www.scienceopen.com/document?vid=28251d22-8476-40a6-916d-1a34796816e4
6. Prince Charles Hospital. Modified Early Warning Score (MEWS), Escalation and ISBAR, The Prince Charles Hospital Procedure TPCHS10085.
7. Deteriorating Patient Programme and National Early Warning System (NEWS) -HSE.ie. https://www.hse.ie/eng/about/who/cspd/ncps/acute-medicine/national-early-warning-score/.
8. Churpek MM, Yuen TC, Park SY, Meltzer DO, Hall JB, Edelson DP. Derivation of a cardiac arrest prediction model using ward vital signs. Crit Care Med 2012;40: 2102-8. doi: 10.1097/CCM.0b013e318250aa5a
9. Douw G, Huisman-de Waal G, van Zanten AR, van der Hoeven JG, Schoonhoven L. Nurses' 'worry' as predictor of deteriorating surgical ward patients: A prospective cohort study of the Dutch-Early-Nurse-Worry-Indicator-Score. Int J Nurs Stud 2016;59: 134-40. doi: 10.1016/j.ijnurstu.2016.04.006
10. National Institute for Health and Care Excellence. Acutely ill adults in hospital: Recognising and responding to deterioration. https://www.nice.org.uk/guidance/cg50.
11. Gerry S, Birks J, Bonnici T, Watkinson PJ, Kirtley S, Collins GS. Early warning scores for detecting deterioration in adult hospital patients: A systematic review protocol. BMJ Open 2017;7: E019268. doi: 10.1136/bmjopen-2017-019268
12. Royal College of Physicians. Royal College of Physicians National Early Warning Score (NEWS) 2: Standardising the assessment of acuteillness severity in the NHS. Updated report of a working party. RCP, 2017.
13. Silcock DJ, Corfield AR, Gowens PA, Rooney KD. Validation of the National Early Warning Score in the prehospital setting. Resuscitation 2015;89: 31-5. doi: 10.1016/j.resuscitation.2014.12.029
14. Corfield AR, Lees F, Zealley I, et al, Scottish Trauma Audit Group Sepsis Steering Group. Utility of a single early warning score in patients with sepsis in the emergency department. Emerg Med J 2014;31: 482-7. doi: 10.1136/emermed-2012-202186
15. Brangan E, Banks J, Brant H, Pullyblank A, Le Roux H, Redwood S. Using the National Early Warning Score (NEWS) outside acute hospital settings: A qualitative study of staff experiences in the West of England. BMJ Open 2018;8: E022528. doi: 10.1136/bmjopen-2018-022528
16. Inada-Kim M, Nsutebu E. NEWS 2: An opportunity to standardise the management of deterioration and sepsis. BMJ 2018;360: K1260. doi: 10.1136/bmj.k1260
17. Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD statement. Ann Intern Med 2015;162: 55-63. doi: 10.7326/M14-0697
18. Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, Van Calster B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol 2019;110: 12-22. doi: 10.1016/j.jclinepi.2019.02.004
19. Damen JAAG, Hooft L, Schuit E, et al. Prediction models for cardiovascular disease risk in the general population: Systematic review. BMJ 2016;353: I2416. doi: 10.1136/bmj.i2416
20. Collins GS, Mallett S, Omar O, Yu LM. Developing risk prediction models for type 2 diabetes: A systematic review of methodology and reporting. BMC Med 2011;9: 103. doi: 10.1186/1741-7015-9-103
21. Bouwmeester W, Zuithoff NPA, Mallett S, et al. Reporting and methods in clinical prediction research: A systematic review. PLoS Med 2012;9: 1-12. doi: 10.1371/journal.pmed.1001221
22. Collins GS, Omar O, Shanyinde M, Yu LM. A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods. J Clin Epidemiol 2013;66: 268-77. doi: 10.1016/j.jclinepi.2012.06.020
23. Gao H, McDonnell A, Harrison DA, et al. Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med 2007;33: 667-79. doi: 10.1007/s00134-007-0532-3
24. Smith MEB, Chiovaro JC, O'Neil M, et al. Early warning system scores for clinical deterioration in hospitalized patients: A systematic review. Ann Am Thorac Soc 2014;11: 1454-65. doi: 10.1513/AnnalsATS.201403-102OC
25. Goldstein BA, Navar AM, Pencina MJ, Ioannidis JPA. Opportunities and challenges in developing risk prediction models with electronic health records data: A systematic review. J Am Med Inform Assoc 2017;24: 198-208. doi: 10.1093/jamia/ocw042
26. Goldstein BA, Pomann GM, Winkelmayer WC, Pencina MJ. A comparison of risk prediction methods using repeated observations: An application to electronic health records for hemodialysis. Stat Med 2017;36: 2750-63. doi: 10.1002/sim.7308
27. Churpek MM, Yuen TC, Edelson DP. Predicting clinical deterioration in the hospital: The impact of outcome selection. Resuscitation 2013;84: 564-8. doi: 10.1016/j.resuscitation.2012.09.024
28. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)-a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42: 377-81. doi: 10.1016/j.jbi.2008.08.010
29. Moons KGM, De Groot JAH, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: The CHARMS checklist. PLoS Med 2014;11: E1001744. doi: 10.1371/journal.pmed.1001744
30. Wolff RF, Moons KGM, Riley RD, et al, PROBAST Group. PROBAST: A tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med 2019;170: 51-8. doi: 10.7326/M18-1376
31. Albert BL, Huesman L. Development of a modified early warning score using the electronic medical record. Dimens Crit Care Nurs 2011;30: 283-92. doi: 10.1097/DCC.0b013e318227761d
32. Alvarez CA, Clark CA, Zhang S, et al. Predicting out of intensive care unit cardiopulmonary arrest or death using electronic medical record data. BMC Med Inform Decis Mak 2013;13: 28. doi: 10.1186/1472-6947-13-28.
33. Badriyah T, Briggs JS, Meredith P, et al. Decision-tree early warning score (DTEWS) validates the design of the National Early Warning Score (NEWS). Resuscitation 2014;85: 418-23. doi: 10.1016/j.resuscitation.2013.12.011
34. Bleyer AJ, Vidya S, Russell GB, et al. Longitudinal analysis of one million vital signs in patients in an academic medical center. Resuscitation 2011;82: 1387-92. doi: 10.1016/j.resuscitation.2011.06.033
35. Churpek MM, Yuen TC, Park SY, Gibbons R, Edelson DP. Using electronic health record data to develop and validate a prediction model for adverse outcomes in the wards. Crit Care Med 2014;42: 841-8. doi: 10.1097/CCM.0000000000000038
36. Churpek MM, Yuen TC, Winslow C, et al. Multicenter development and validation of a risk stratification tool for ward patients. Am J Respir Crit Care Med 2014;190: 649-55. doi: 10.1164/rccm.201406-1022OC
37. Churpek MM, Adhikari R, Edelson DP. The value of vital sign trends for detecting clinical deterioration on the wards. Resuscitation 2016;102: 1-5. doi: 10.1016/j.resuscitation.2016.02.005
38. Cuthbertson BH, Boroujerdi M, Prescott G. The use of combined physiological parameters in the early recognition of the deteriorating acute medical patient. J R Coll Physicians Edinb 2010;40: 19-25. doi: 10.4997/JRCPE.2010.105
39. Duckitt RW, Buxton-Thomas R, Walker J, et al. Worthing physiological scoring system: Derivation and validation of a physiological early-warning system for medical admissions. An observational, population-based single-centre study. Br J Anaesth 2007;98: 769-74. doi: 10.1093/bja/aem097
40. Dziadzko MA, Novotny PJ, Sloan J, et al. Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital. Crit Care 2018;22: 286. doi: 10.1186/s13054-018-2194-7
41. Escobar GJ, LaGuardia JC, Turk BJ, Ragins A, Kipnis P, Draper D. Early detection of impending physiologic deterioration among patients who are not in intensive care: Development of predictive models using data from an automated electronic medical record. J Hosp Med 2012;7: 388-95. doi: 10.1002/jhm.1929
42. Faisal M, Scally AJ, Jackson N, et al. Development and validation of a novel computer-aided score to predict the risk of in-hospital mortality for acutely ill medical admissions in two acute hospitals using their first electronically recorded blood test results and vital signs: A cross-sectional study. BMJ Open 2018;8: E022939. doi: 10.1136/bmjopen-2018-022939
43. Ghosh E, Eshelman L, Yang L, Carlson E, Lord B. Early Deterioration Indicator: Data-driven approach to detecting deterioration in general ward. Resuscitation 2018;122: 99-105. doi: 10.1016/j.resuscitation.2017.10.026
44. Goldhill DR, McNarry AF. Physiological abnormalities in early warning scores are related to mortality in adult inpatients. Br J Anaesth 2004;92: 882-4. doi: 10.1093/bja/aeh113
45. Harrison GA, Jacques T, McLaws ML, Kilborn G. Combinations of early signs of critical illness predict in-hospital death-the SOCCER study (signs of critical conditions and emergency responses). Resuscitation 2006;71: 327-34. doi: 10.1016/j.resuscitation.2006.05.008
46. Jones M. NEWSDIG: The National Early Warning Score Development and Implementation Group. Clin Med (Lond) 2012;12: 501-3. doi: 10.7861/clinmedicine.12-6-501
47. Kellett J, Deane B. The Simple Clinical Score predicts mortality for 30 days after admission to an acute medical unit. QJM 2006;99: 771-81. doi: 10.1093/qjmed/hcl112
48. Kellett J, Deane B, Gleeson M. Derivation and validation of a score based on Hypotension, Oxygen saturation, low Temperature, ECG changes and Loss of independence (HOTEL) that predicts early mortality between 15 min and 24 h after admission to an acute medical unit. Resuscitation 2008;78: 52-8. doi: 10.1016/j.resuscitation.2008.02.011
49. Kipnis P, Turk BJ, Wulf DA, et al. Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU. J Biomed Inform 2016;64: 10-9. doi: 10.1016/j.jbi.2016.09.013
50. Kirkland LL, Malinchoc M, O'Byrne M, et al. A clinical deterioration prediction tool for internal medicine patients. Am J Med Qual 2013;28: 135-42. doi: 10.1177/1062860612450459
51. Kwon JM, Lee Y, Lee Y, Lee S, Park J. An algorithm based on deep learning for predicting in-hospital cardiac arrest. J Am Heart Assoc 2018;7: 26. doi: 10.1161/JAHA.118.008678
52. Kyriacos U, Jelsma J, James M, Jordan S. Monitoring vital signs: Development of a modified early warning scoring (MEWS) system for general wards in a developing country. PLoS One 2014;9: E87073. doi: 10.1371/journal.pone.0087073
53. Luís L, Nunes C. Short National Early Warning Score-developing a modified early warning score. Aust Crit Care 2018;31: 376-81. doi: 10.1016/j.aucc.2017.11.004
54. Moore CC, Hazard R, Saulters KJ, et al. Derivation and validation of a universal vital assessment (UVA) score: A tool for predicting mortality in adult hospitalised patients in sub-Saharan Africa. BMJ Glob Health 2017;2: E000344. doi: 10.1136/bmjgh-2017-000344
55. Nickel CH, Kellett J, Cooksley T, Bingisser R, Henriksen DP, Brabrand M. Combined use of the National Early Warning Score and D-dimer levels to predict 30-day and 365-day mortality in medical patients. Resuscitation 2016;106: 49-52. doi: 10.1016/j.resuscitation.2016.06.012
56. Perera YS, Ranasinghe P, Adikari AM, et al. The value of the Modified Early Warning Score and biochemical parameters as predictors of patient outcome in acute medical admissions a prospective study. Acute Med 2011;10: 126-32.
57. Prytherch DR, Smith GB, Schmidt PE, Featherstone PI. ViEWS-Towards a national early warning score for detecting adult inpatient deterioration. Resuscitation 2010;81: 932-7. doi: 10.1016/j.resuscitation.2010.04.014
58. Redfern OC, Pimentel MAF, Prytherch D, et al. Predicting in-hospital mortality and unanticipated admissions to the intensive care unit using routinely collected blood tests and vital signs: Development and validation of a multivariable model. Resuscitation 2018;133: 75-81. doi: 10.1016/j.resuscitation.2018.09.021
59. Silke B, Kellett J, Rooney T, Bennett K, O'Riordan D. An improved medical admissions risk system using multivariable fractional polynomial logistic regression modelling. QJM 2010;103: 23-32. doi: 10.1093/qjmed/hcp149
60. Tarassenko L, Clifton DA, Pinsky MR, Hravnak MT, Woods JR, Watkinson PJ. Centile-based early warning scores derived from statistical distributions of vital signs. Resuscitation 2011;82: 1013-8. doi: 10.1016/j.resuscitation.2011.03.006
61. Watkinson PJ, Pimentel MAF, Clifton DA, Tarassenko L. Manual centilebased early warning scores derived from statistical distributions of observational vital-sign data. Resuscitation 2018;129: 55-60. doi: 10.1016/j.resuscitation.2018.06.003
62. Wheeler I, Price C, Sitch A, et al. Early warning scores generated in developed healthcare settings are not sufficient at predicting early mortality in Blantyre, Malawi: A prospective cohort study [correction: PLoS One 2014;9: E91623]. PLoS One 2013;8: E59830. doi: 10.1371/journal.pone.0059830
63. Abbott TEF, Torrance HDT, Cron N, Vaid N, Emmanuel J. A singlecentre cohort study of National Early Warning Score (NEWS) and near patient testing in acute medical admissions. Eur J Intern Med 2016;35: 78-82. doi: 10.1016/j.ejim.2016.06.014
64. Abbott TE, Vaid N, Ip D, et al. A single-centre observational cohort study of admission National Early Warning Score (NEWS). Resuscitation 2015;92: 89-93. doi: 10.1016/j.resuscitation.2015.04.020
65. Atmaca O, Turan C, Güven P, Arikan H, Eryüksel SE, Karakurt S. Usage of NEWS for prediction of mortality and in-hospital cardiac arrest rates in a Turkish university hospital. Turk J Med Sci 2018;48: 1087-91. doi: 10.3906/sag-1706-67
66. Bartkowiak B, Snyder AM, Benjamin A, et al. Validating the Electronic Cardiac Arrest Risk Triage (eCART) score for risk stratification of surgical inpatients in the postoperative setting: Retrospective cohort study. Ann Surg 2019;269: 1059-63. doi: 10.1097/SLA.0000000000002665
67. Beane A, De Silva AP, De Silva N, et al. Evaluation of the feasibility and performance of early warning scores to identify patients at risk of adverse outcomes in a low-middle income country setting. BMJ Open 2018;8: E019387. doi: 10.1136/bmjopen-2017-019387
68. Brabrand M, Hallas P, Hansen SN, Jensen KM, Madsen JLB, Posth S. Using scores to identify patients at risk of short term mortality at arrival to the acute medical unit: A validation study of six existing scores. Eur J Intern Med 2017;45: 32-6. doi: 10.1016/j.ejim.2017.09.042
69. Brabrand M, Henriksen DP. CURB-65 score is equal to NEWS for identifying mortality risk of pneumonia patients: An observational study. Lung 2018;196: 359-61. doi: 10.1007/s00408-018-0105-y
70. Cei M, Bartolomei C, Mumoli N. In-hospital mortality and morbidity of elderly medical patients can be predicted at admission by the Modified Early Warning Score: A prospective study. Int J Clin Pract 2009;63: 591-5. doi: 10.1111/j.1742-1241.2008.01986.x
71. Churpek MM, Snyder A, Sokol S, Pettit NN, Edelson DP. Investigating the impact of different suspicion of infection criteria on the accuracy of quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores. Crit Care Med 2017;45: 1805-12. doi: 10.1097/CCM.0000000000002648
72. Churpek MM, Snyder A, Han X, et al. Quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med 2017;195: 906-11. doi: 10.1164/rccm.201604-0854OC
73. Churpek MM, Yuen TC, Edelson DP. Risk stratification of hospitalized patients on the wards. Chest 2013;143: 1758-65. doi: 10.1378/chest.12-1605
74. Churpek MM, Yuen TC, Huber MT, Park SY, Hall JB, Edelson DP. Predicting cardiac arrest on the wards: A nested case-control study. Chest 2012;141: 1170-6. doi: 10.1378/chest.11-1301
75. Cooksley T, Kitlowski E, Haji-Michael P. Effectiveness of Modified Early Warning Score in predicting outcomes in oncology patients. QJM 2012;105: 1083-8. doi: 10.1093/qjmed/hcs138.
76. De Meester K, Das T, Hellemans K, et al. Impact of a standardized nurse observation protocol including MEWS after Intensive Care Unit discharge. Resuscitation 2013;84: 184-8. doi: 10.1016/j.resuscitation.2012.06.017
77. DeVoe B, Roth A, Maurer G, et al. Correlation of the predictive ability of early warning metrics and mortality for cardiac arrest patients receiving in-hospital Advanced Cardiovascular Life Support. Heart Lung 2016;45: 497-502. doi: 10.1016/j.hrtlng.2016.08.010
78. Douw G, Huisman-de Waal G, van Zanten ARH, van der Hoeven JG, Schoonhoven L. Capturing early signs of deterioration: The dutchearly-nurse-worry-indicator-score and its value in the Rapid Response System. J Clin Nurs 2017;26: 2605-13. doi: 10.1111/jocn.13648
79. Eccles SR, Subbe C, Hancock D, Thomson N. CREWS: Improving specificity whilst maintaining sensitivity of the National Early Warning Score in patients with chronic hypoxaemia. Resuscitation 2014;85: 109-11. doi: 10.1016/j.resuscitation.2013.08.277
80. Fairclough E, Cairns E, Hamilton J, Kelly C. Evaluation of a modified early warning system for acute medical admissions and comparison with C-reactive protein/albumin ratio as a predictor of patient outcome. Clin Med (Lond) 2009;9: 30-3. doi: 10.7861/clinmedicine.9-1-30
81. Finlay GD, Rothman MJ, Smith RA. Measuring the modified early warning score and the Rothman index: Advantages of utilizing the electronic medical record in an early warning system. J Hosp Med 2014;9: 116-9. doi: 10.1002/jhm.2132
82. Forster S, Housley G, McKeever TM, Shaw DE. Investigating the discriminative value of Early Warning Scores in patients with respiratory disease using a retrospective cohort analysis of admissions to Nottingham University Hospitals Trust over a 2-year period. BMJ Open 2018;8: E020269. doi: 10.1136/bmjopen-2017-020269
83. Garcea G, Jackson B, Pattenden CJ, et al. Early warning scores predict outcome in acute pancreatitis. J Gastrointest Surg 2006;10: 1008-15. doi: 10.1016/j.gassur.2006.03.008
84. Gardner-Thorpe J, Love N, Wrightson J, Walsh S, Keeling N. The value of Modified Early Warning Score (MEWS) in surgical in-patients: A prospective observational study. Ann R Coll Surg Engl 2006;88: 571-5. doi: 10.1308/003588406X130615
85. Ghanem-Zoubi NO, Vardi M, Laor A, Weber G, Bitterman H. Assessment of disease-severity scoring systems for patients with sepsis in general internal medicine departments. Crit Care 2011;15: R95. doi: 10.1186/cc10102
86. Green M, Lander H, Snyder A, Hudson P, Churpek M, Edelson D. Comparison of the Between the Flags calling criteria to the MEWS, NEWS and the electronic Cardiac Arrest Risk Triage (eCART) score for the identification of deteriorating ward patients. Resuscitation 2018;123: 86-91. doi: 10.1016/j.resuscitation.2017.10.028
87. Hodgson LE, Dimitrov BD, Congleton J, Venn R, Forni LG, Roderick PJ. A validation of the National Early Warning Score to predict outcome in patients with COPD exacerbation. Thorax 2017;72: 23-30. doi: 10.1136/thoraxjnl-2016-208436
88. Hydes TJ, Meredith P, Schmidt PE, Smith GB, Prytherch DR, Aspinall RJ. National Early Warning Score accurately discriminates the risk of serious adverse events in patients with liver disease. Clin Gastroenterol Hepatol 2018;16: 1657-1666.e10. doi: 10.1016/j.cgh.2017.12.035
89. Jo S, Jeong T, Lee JB, Jin Y, Yoon J, Park B. Validation of modified early warning score using serum lactate level in community-acquired pneumonia patients. The National Early Warning Score-Lactate score. Am J Emerg Med 2016;34: 536-41. doi: 10.1016/j.ajem.2015.12.067
90. Kellett J, Kim A. Validation of an abbreviated Vitalpac™ Early Warning Score (ViEWS) in 75, 419 consecutive admissions to a Canadian regional hospital. Resuscitation 2012;83: 297-302. doi: 10.1016/j.resuscitation.2011.08.022
91. Kellett J, Murray A. Should predictive scores based on vital signs be used in the same way as those based on laboratory data? A hypothesis generating retrospective evaluation of in-hospital mortality by four different scoring systems. Resuscitation 2016;102: 94-7. doi: 10.1016/j.resuscitation.2016.02.020
92. Kim D, Jo S, Lee JB, et al. Comparison of the National Early Warning Score+Lactate score with the pre-endoscopic Rockall, Glasgow-Blatchford, and AIMS65 scores in patients with upper gastrointestinal bleeding. Clin Exp Emerg Med 2018;5: 219-29. doi: 10.15441/ceem.17.268
93. Kim WY, Lee J, Lee JR, et al. A risk scoring model based on vital signs and laboratory data predicting transfer to the intensive care unit of patients admitted to gastroenterology wards. J Crit Care 2017;40: 213-7. doi: 10.1016/j.jcrc.2017.04.024
94. Kovacs C, Jarvis SW, Prytherch DR, et al. Comparison of the National Early Warning Score in non-elective medical and surgical patients. Br J Surg 2016;103: 1385-93. doi: 10.1002/bjs.10267
95. Kruisselbrink R, Kwizera A, Crowther M, et al. Modified Early Warning Score (MEWS) identifies critical illness among ward patients in a resource restricted setting in Kampala, Uganda: A prospective observational study. PLoS One 2016;11: E0151408. doi: 10.1371/journal.pone.0151408
96. Le Lagadec MD, Dwyer T, Browne M. The efficacy of twelve early warning systems for potential use in regional medical facilities in Queensland, Australia. Aust Crit Care 2020;33: 47-53. doi: 10.1016/j.aucc.2019.03.001.
97. Lee YS, Choi JW, Park YH, et al. Evaluation of the efficacy of the National Early Warning Score in predicting in-hospital mortality via the risk stratification. J Crit Care 2018;47: 222-6. doi: 10.1016/j.jcrc.2018.07.011
98. Liljehult J, Christensen T. Early warning score predicts acute mortality in stroke patients. Acta Neurol Scand 2016;133: 261-7. doi: 10.1111/ane.12452
99. Mulligan A. Validation of a physiological track and trigger score to identify developing critical illness in haematology patients. Intensive Crit Care Nurs 2010;26: 196-206. doi: 10.1016/j.iccn.2010.03.002
100. Ohman MC, Atkins TEH, Cooksley T, Brabrand M. Validation of the MARS: A combined physiological and laboratory risk prediction tool for 5-to 7-day in-hospital mortality. QJM 2018;111: 385-8. doi: 10.1093/qjmed/hcy053
101. Opio MO, Nansubuga G, Kellett J. Validation of the VitalPAC™ Early Warning Score (ViEWS) in acutely ill medical patients attending a resource-poor hospital in sub-Saharan Africa. Resuscitation 2013;84: 743-6. doi: 10.1016/j.resuscitation.2013.02.007
102. Opio MO, Nansubuga G, Kellett J, Clifford M, Murray A. Performance of TOTAL, in medical patients attending a resource-poor hospital in sub-Saharan Africa and a small Irish rural hospital. Acute Med 2013;12: 135-40.
103. Pedersen NE, Rasmussen LS, Petersen JA, Gerds TA, Ostergaard D, Lippert A. Modifications of the National Early Warning Score for patients with chronic respiratory disease. Acta Anaesthesiol Scand 2018;62: 242-52. doi: 10.1111/aas.13020
104. Pimentel MAF, Redfern OC, Gerry S, et al. A comparison of the ability of the National Early Warning Score and the National Early Warning Score 2 to identify patients at risk of in-hospital mortality: A multi-centre database study. Resuscitation 2019;134: 147-56. doi: 10.1016/j.resuscitation.2018.09.026
105. Plate JD, Peelen LM, Leenen LP, Hietbrink F. Validation of the VitalPAC early warning score at the intermediate care unit. World J Crit Care Med 2018;7: 39-45. doi: 10.5492/wjccm.v7.i3.39
106. Redfern OC, Smith GB, Prytherch DR, Meredith P, Inada-Kim M, Schmidt PE. A comparison of the Quick Sequential (sepsisrelated) Organ Failure Assessment score and the National Early Warning Score in non-ICU patients with/without infection. Crit Care Med 2018;46: 1923-33. doi: 10.1097/CCM.0000000000003359
107. Roberts D, Djärv T. Preceding national early warnings scores among in-hospital cardiac arrests and their impact on survival. Am J Emerg Med 2017;35: 1601-6. doi: 10.1016/j.ajem.2017.04.072
108. Romero-Brufau S, Morlan BW, Johnson M, et al. Evaluating automated rules for rapid response system alarm triggers in medical and surgical patients. J Hosp Med 2017;12: 217-23. doi: 10.12788/jhm.2712.
109. Romero-Brufau S, Huddleston JM, Naessens JM, et al. Widely used track and trigger scores: Are they ready for automation in practice?Resuscitation 2014;85: 549-52. doi: 10.1016/j.resuscitation.2013.12.017
110. Rylance J, Baker T, Mushi E, Mashaga D. Use of an early warning score and ability to walk predicts mortality in medical patients admitted to hospitals in Tanzania. Trans R Soc Trop Med Hyg 2009;103: 790-4. doi: 10.1016/j.trstmh.2009.05.004
111. Smith GB, Prytherch DR, Schmidt PE, Featherstone PI. Review and performance evaluation of aggregate weighted 'track and trigger' systems. Resuscitation 2008;77: 170-9. doi: 10.1016/j.resuscitation.2007.12.004
112. Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation 2013;84: 465-70. doi: 10.1016/j.resuscitation.2012.12.016
113. Smith GB, Prytherch DR, Jarvis S, et al. A comparison of the ability of the physiologic components of medical emergency team criteria and the U.K. National Early Warning Score to discriminate patients at risk of a range of adverse clinical outcomes. Crit Care Med 2016;44: 2171-81. doi: 10.1097/CCM.0000000000002000
114. Smith HJ, Pasko DN, Haygood CLW, Boone JD, Harper LM, Straughn JMJr. Early warning score: An indicator of adverse outcomes in postoperative patients on a gynecologic oncology service. Gynecol Oncol 2016;143: 105-8. doi: 10.1016/j.ygyno.2016.08.153
115. Spagnolli W, Rigoni M, Torri E, Cozzio S, Vettorato E, Nollo G. Application of the National Early Warning Score (NEWS) as a stratification tool on admission in an Italian acute medical ward: A perspective study. Int J Clin Pract 2017;71. doi: 10.1111/ijcp.12934
116. Stark AP, Maciel RC, Sheppard W, Sacks G, Hines OJ. An early warning score predicts risk of death after inhospital cardiopulmonary arrest in surgical patients. Am Surg 2015;81: 916-21.
117. Stræde M, Brabrand M. External validation of the simple clinical score and the HOTEL score, two scores for predicting shortterm mortality after admission to an acute medical unit. PLoS One 2014;9: E105695. doi: 10.1371/journal.pone.0105695
118. Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified Early Warning Score in medical admissions. QJM 2001;94: 521-6. doi: 10.1093/qjmed/94.10.521
119. Suppiah A, Malde D, Arab T, et al. The Modified Early Warning Score (MEWS): An instant physiological prognostic indicator of poor outcome in acute pancreatitis. JOP 2014;15: 569-76. doi: 10.6092/1590-8577/2829.
120. Tirkkonen J, Olkkola KT, Huhtala H, Tenhunen J, Hoppu S. Medical emergency team activation: Performance of conventional dichotomised criteria versus national early warning score. Acta Anaesthesiol Scand 2014;58: 411-9. doi: 10.1111/aas.12277
121. Tirotta D, Gambacorta M, La Regina M, et al. Evaluation of the threshold value for the modified early warning score (MEWS) in medical septic patients: A secondary analysis of an Italian multicentric prospective cohort (SNOOPII study). QJM 2017;110: 369-73. doi: 10.1093/qjmed/hcw229
122. Vaughn JL, Kline D, Denlinger NM, Andritsos LA, Exline MC, Walker AR. Predictive performance of early warning scores in acute leukemia patients receiving induction chemotherapy. Leuk Lymphoma 2018;59: 1498-500. doi: 10.1080/10428194.2017.1376744
123. von Lilienfeld-Toal M, Midgley K, Lieberbach S, et al. Observationbased early warning scores to detect impending critical illness predict in-hospital and overall survival in patients undergoing allogeneic stem cell transplantation. Biol Blood Marrow Transplant 2007;13: 568-76. doi: 10.1016/j.bbmt.2006.12.455
124. Van Calster B, Wynants L, Verbeek JFM, et al. Reporting and interpreting decision curve analysis: A guide for investigators. Eur Urol 2018;74: 796-804. doi: 10.1016/j.eururo.2018.08.038
125. McGaughey J, Alderdice F, Fowler R, Kapila A, Mayhew A, Moutray M. Outreach and Early Warning Systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev 2007;(3): CD005529. doi: 10.1002/14651858.CD005529.pub2.
126. Alam N, Hobbelink EL, van Tienhoven AJ, van De Ven PM, Jansma EP, Nanayakkara PWB. The impact of the use of the Early Warning Score (EWS) on patient outcomes: A systematic review. Resuscitation 2014;85: 587-94. doi: 10.1016/j.resuscitation.2014.01.013
127. Wong D, Bonnici T, Knight J, Morgan L, Coombes P, Watkinson P. SEND: A system for electronic notification and documentation of vital sign observations. BMC Med Inform Decis Mak 2015;15: 68. doi: 10.1186/s12911-015-0186-y
128. Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: A resampling study. Stat Med 2016;35: 214-26. doi: 10.1002/sim.6787
129. Riley RD, Snell KI, Ensor J, et al. Minimum sample size for developing a multivariable prediction model: PART II -binary and time-toevent outcomes [correction in: Stat Med 2019;38: 5672]. Stat Med 2019;38: 1276-96. doi: 10.1002/sim.7992
130. van Smeden M, Moons KGM, De Groot JAH, et al. Sample size for binary logistic prediction models: Beyond events per variable criteria. Stat Methods Med Res 2019;28: 2455-74. doi: 10.1177/0962280218784726
131. Little RJA. Regression with missing X's: A review. J Am Stat Assoc 1992;87: 1227. doi: 10.2307/2290664.
132. Janssen KJ, Donders ART, Harrell FEJr, et al. Missing covariate data in medical research: To impute is better than to ignore. J Clin Epidemiol 2010;63: 721-7. doi: 10.1016/j.jclinepi.2009.12.008
133. Ambler G, Omar RZ, Royston P. A comparison of imputation techniques for handling missing predictor values in a risk model with a binary outcome. Stat Methods Med Res 2007;16: 277-98. doi: 10.1177/0962280206074466
134. Little RJA, Rubin DB. Statistical analysis with missing data. John Wiley & Sons, Inc, 2002, doi: 10.1002/9781119013563
135. Vergouwe Y, Royston P, Moons KG, Altman DG. Development and validation of a prediction model with missing predictor data: A practical approach. J Clin Epidemiol 2010;63: 205-14. doi: 10.1016/j.jclinepi.2009.03.017
136. White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med 2011;30: 377-99. doi: 10.1002/sim.4067
137. Wolbers M, Koller MT, Witteman JCM, Steyerberg EW. Prognostic models with competing risks: Methods and application to coronary risk prediction. Epidemiology 2009;20: 555-61. doi: 10.1097/EDE.0b013e3181a39056
138. Gerds TA, Scheike TH, Andersen PK. Absolute risk regression for competing risks: Interpretation, link functions, and prediction. Stat Med 2012;31: 3921-30. doi: 10.1002/sim.5459
139. Wolbers M, Koller MT, Stel VS, et al. Competing risks analyses: Objectives and approaches. Eur Heart J 2014;35: 2936-41. doi: 10.1093/eurheartj/ehu131
140. Austin PC, Lee DS, Fine JP. Introduction to the analysis of survival data in the presence of competing risks. Circulation 2016;133: 601-9. doi: 10.1161/CIRCULATIONAHA.115.017719
141. Blanche P, Dartigues J-F, Jacqmin-Gadda H. Estimating and comparing time-dependent areas under receiver operating characteristic curves for censored event times with competing risks. Stat Med 2013;32: 5381-97. doi: 10.1002/sim.5958
142. Austin PC, Lee DS, D'Agostino RB, Fine JP. Developing points-based risk-scoring systems in the presence of competing risks [correction in: Stat Med 2018;37: 1405]. Stat Med 2016;35: 4056-72. doi: 10.1002/sim.6994
143. Gerds TA, Andersen PK, Kattan MW. Calibration plots for risk prediction models in the presence of competing risks. Stat Med 2014;33: 3191-203. doi: 10.1002/sim.6152
144. Steyerberg EW, Harrell FEJr, Borsboom GJ, Eijkemans MJC, Vergouwe Y, Habbema JDF. Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 2001;54: 774-81. doi: 10.1016/S0895-4356(01)00341-9
145. Moons KG, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): Explanation and elaboration. Ann Intern Med 2015;162: W1-73. doi: 10.7326/M14-0698
146. Harrell FEJr, Lee KL, Mark DB. Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996;15: 361-87. doi: 10.1002/(SICI)1097-0258(19960229)15: 4 3.0.CO;2-4
147. Steyerberg EW, Borsboom GJJM, van Houwelingen HC, Eijkemans MJC, Habbema JDF. Validation and updating of predictive logistic regression models: A study on sample size and shrinkage. Stat Med 2004;23: 2567-86. doi: 10.1002/sim.1844
148. Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: From utopia to empirical data. J Clin Epidemiol 2016;74: 167-76. doi: 10.1016/j.jclinepi.2015.12.005
149. Perel P, Edwards P, Wentz R, Roberts I. Systematic review of prognostic models in traumatic brain injury. BMC Med Inform Decis Mak 2006;6: 38. doi: 10.1186/1472-6947-6-38
150. Meads C, Ahmed I, Riley RD. A systematic review of breast cancer incidence risk prediction models with meta-analysis of their performance. Breast Cancer Res Treat 2012;132: 365-77. doi: 10.1007/s10549-011-1818-2
151. Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med 2014;33: 517-35. doi: 10.1002/sim.5941
152. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: A framework for traditional and novel measures. Epidemiology 2010;21: 128-38. doi: 10.1097/EDE.0b013e3181c30fb2
153. Wynants L, Kent DM, Timmerman D, Lundquist CM, Van Calster B. Untapped potential of multicenter studies: A review of cardiovascular risk prediction models revealed inappropriate analyses and wide variation in reporting. Diagn Progn Res 2019;3: 6. doi: 10.1186/s41512-019-0046-9
154. Linnen DT, Escobar GJ, Hu X, Scruth E, Liu V, Stephens C. Statistical modeling and aggregate-weighted scoring systems in prediction of mortality and ICU transfer: A systematic review. J Hosp Med 2019;14: 161-9. doi: 10.12788/jhm.3151
155. NHS England. National Early Warning Score (NEWS). https://www.england.nhs.uk/ourwork/clinical-policy/sepsis/nationalearlywarningscore/.