Adjusting weighted kappa for severity of mistriage decreases reported reliability of emergency department triage systems: a comparative study

Journal of Clinical Epidemiology

Volumn 62, Issue 11, 2009, Pages 1196-1201

  van der Wulp, Ineke ^a   van Stel, Henk F ^a  

^a UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

Author keywords

Emergency medicine; Emergency service hospital; Reproducibility of results; Statistics as topics; Triage; Validation studies as topic

Indexed keywords

ARTICLE; COMPARATIVE STUDY; EMERGENCY HEALTH SERVICE; EMERGENCY MEDICINE; HUMAN; MEDICAL LITERATURE; PRIORITY JOURNAL; RELIABILITY; STATISTICAL ANALYSIS; VALIDATION STUDY;

ALGORITHMS; DATA INTERPRETATION, STATISTICAL; EMERGENCY SERVICE, HOSPITAL; HUMANS; OBSERVER VARIATION; REPRODUCIBILITY OF RESULTS; TRAUMA SEVERITY INDICES; TRIAGE;

EID: 70349312904     PISSN: 08954356     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2009.01.007     Document Type: Article

References (55)

1. Tanabe P., Gimbel R., Yarnold P.R., Kyriacou D.N., and Adams J.G. Reliability and validity of scores on The Emergency Severity Index version 3. Acad Emerg Med 11 (2004) 59-65
2. Fernandes C.M., Tanabe P., Gilboy N., Johnson L.A., McNair R.S., Rosenau A.M., et al. Five-level triage: a report from the ACEP/ENA Five-level Triage Task Force. J Emerg Nurs 31 (2005) 39-50
3. Gilboy N., Tanabe P., Travers D., Rosenau A., and Eitel D.R. Emergency Severity Index, Version 4: implementation handbook (2005), Agency for Healthcare Research and Quality, Rockville
4. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 20 (1960) 37-46
5. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull 70 (1968) 213-220
6. Google Scholar Available at http://scholar.google.com. Accessed Oct. 6, 2008.
7. Fleiss J.L., Levin B., and Paik M.C. The measurement of inter rater agreement. Statistical methods for rates and proportions (2003), Wiley, New York
8. AGREE for Windows [computer program]. Version 7.002 1999. University of Groningen. Available at: URL: http://www.scienceplus.nl/component/page,shop.product_details/flypage,shop.flypage_splus/product_id,59/category_id,13/manufacturer_id,0/option,com_virtuemart/Itemid,26/
9. Dong S.L., Bullard M.J., Meurer D.P., Colman I., Blitz S., Holroyd B.R., et al. Emergency triage: comparing a novel computer triage program with standard triage. Acad Emerg Med 12 (2005) 502-507
10. Dong S.L., Bullard M.J., Meurer D.P., Blitz S., Ohinmaa A., Holroyd B.R., et al. Reliability of computerized emergency triage. Acad Emerg Med 13 (2006) 269-275
11. Dong S.L., Bullard M.J., Meurer D.P., Blitz S., Holroyd B.R., and Rowe B.H. The effect of training on nurse agreement using an electronic triage system. CJEM 9 (2007) 260-266
12. Durani Y., Brecher D., Walmsley D., Attia M.W., and Loiselle J.M. The Emergency Severity Index (Version 4): reliability in pediatric patients. Acad Emerg Med 14 Suppl 1 (2007) S95-S96
13. Göransson K., Ehrenberg A., Marklund B., and Ehnfors M. Accuracy and concordance of nurses in emergency department triage. Scand J Caring Sci 19 (2005) 432-438
14. Gravel J., Gouin S., Bailey B., Roy M., Bergeron S., and Amre D. Reliability of a computerized version of the Pediatric Canadian Triage and Acuity Scale. Acad Emerg Med 14 (2007) 864-869
15. van der Wulp I., Baar van M.E., and Schrijvers A.J.P. Reliability and validity of the Manchester Triage System in a general emergency department patient population in the Netherlands: results of a simulation study. Emerg Med J 25 (2008) 431-434
16. Baumann M.R., and Strout T.D. Evaluation of the Emergency Severity Index (version 3) triage algorithm in pediatric patients. Acad Emerg Med 12 (2005) 219-224
17. Bergeron S., Gouin S., Bailey B., and Patel H. Comparison of triage assessments among pediatric registered nurses and pediatric emergency physicians. Acad Emerg Med 9 (2002) 1397-1401
18. Bergeron S., Gouin S., Bailey B., Amre D.K., and Patel H. Agreement among pediatric health care professionals with the pediatric Canadian triage and acuity scale guidelines. Pediatr Emerg Care 20 (2004) 514-518
19. Beveridge R., Ducharme J., Janes L., Beaulieu S., and Walter S. Reliability of the Canadian emergency department triage and acuity scale: interrater agreement. Ann Emerg Med 34 (1999) 155-159
20. Eitel D.R., Travers D.A., Rosenau A.M., Gilboy N., and Wuerz R.C. The emergency severity index triage algorithm version 2 is reliable and valid. Acad Emerg Med 10 (2003) 1070-1080
21. Grafstein E., Innes G., Westman J., Christenson J., and Thorne A. Inter-rater reliability of a computerized presenting-complaint-linked triage system in an urban emergency department. CJEM 5 (2003) 323-329
22. Kim T.G., Cho J.K., Kim S.H., Lee H.S., Gu H.D., and Chung S.W. Reliability and validity of the modified Emergency Severity Index-2 as a triage tool. J Korean Soc Emerg Med 17 (2006) 154-164
23. Maningas P.A., Hime D.A., and Parker D.E. The use of the Soterion Rapid Triage System in children presenting to the Emergency Department. J Emerg Med 31 (2006) 353-359
24. Maningas P.A., Hime D.A., Parker D.E., and McMurry T.A. The Soterion Rapid Triage System: evaluation of inter-rater reliability and validity. J Emerg Med 30 (2006) 461-469
25. Manos D., Petrie D.A., Beveridge R.C., Walter S., and Ducharme J. Inter-observer agreement using the Canadian Emergency Department Triage and Acuity Scale. CJEM 4 (2002) 16-22
26. Rutschmann O.T., Kossovsky M., Geissbuhler A., Perneger T.V., Vermeulen B., Simon J., et al. Interactive triage simulator revealed important variability in both process and outcome of emergency triage. J Clin Epidemiol 59 (2006) 615-621
27. Travers D.A., Waller A.E., Bowling J.M., Flowers D., and Tintinalli J. Five-level triage system more effective than three-level in tertiary emergency department. J Emerg Nurs 28 (2002) 395-400
28. Worster A., Gilboy N., Fernandes C.M., Eitel D., Eva K., Geisler R., et al. Assessment of inter-observer reliability of two five-level triage and acuity scales: a randomized controlled trial. CJEM 6 (2004) 240-245
29. Wuerz R.C., Milne L.W., Eitel D.R., Travers D., and Gilboy N. Reliability and validity of a new five-level triage instrument. Acad Emerg Med 7 (2000) 236-242
30. Wuerz R.C., Travers D., Gilboy N., Eitel D.R., Rosenau A., and Yazhari R. Implementation and refinement of the emergency severity index. Acad Emerg Med 8 (2001) 170-176
31. Sim J., and Wright C.C. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 85 (2005) 257-268
32. Brillman J.C., Doezema D., Tandberg D., Sklar D.P., Davis K.D., Simms S., et al. Triage: limitations in predicting need for emergent care and hospital admission. Ann Emerg Med 27 (1996) 493-500
33. Considine J., LeVasseur S.A., and Villanueva E. The Australasian Triage Scale: examining emergency department nurses' performance using computer and paper scenarios. Ann Emerg Med 44 (2004) 516-523
34. Gerdtz M.F., and Bucknall T.K. Influence of task properties and subjectivity on consistency of triage: a simulation study. J Adv Nurs 58 (2007) 180-190
35. Gill J.M., Reese C.L., and Diamond J.J. Disagreement among health care professionals about the urgent care needs of emergency department patients. Ann Emerg Med 28 (1996) 474-479
36. Goodacre S.W., Gillett M., Harris R.D., and Houlihan K.P. Consistency of retrospective triage decisions as a standardised instrument for audit. J Accid Emerg Med 16 (1999) 322-324
37. Hay E., Bekerman L., Rosenberg G., and Peled R. Quality assurance of nurse triage: consistency of results over three years. Am J Emerg Med 19 (2001) 113-117
38. Maldonado T., and Avner J.R. Triage of the pediatric patient in the emergency department: are we all in agreement?. Pediatrics 114 (2004) 356-360
39. Nakagawa J., Ouk S., Schwartz B., and Schriger D.L. Interobserver agreement in emergency department triage. Ann Emerg Med 41 (2003) 191-195
40. Wollaston A., Fahey P., McKay M., Hegney D., Miller P., and Wollaston J. Reliability and validity of the Toowoomba adult trauma triage tool: a Queensland, Australia study. Accid Emerg Nurs 12 (2004) 230-237
41. Wuerz R., Fernandes C.M., and Alarcon J. Inconsistency of emergency department triage. Emergency Department Operations Research Working Group. Ann Emerg Med 32 (1998) 431-435
42. Albin S.L., Wassertheil-Smoller S., Jacobson S., and Bell B. Evaluation of emergency room triage performed by nurses. Am J Public Health 65 (1975) 1063-1068
43. Bazarian J.J., Stern R.A., and Wax P. Accuracy of ED triage of psychiatric patients. Am J Emerg Med 22 (2004) 249-253
44. Considine J., Ung L., and Thomas S. Triage nurses' decisions using the National Triage Scale for Australian emergency departments. Accid Emerg Nurs 8 (2000) 201-209
45. Cooke M.W., and Jinks S. Does the Manchester triage system detect the critically ill?. J Accid Emerg Med 16 (1999) 179-181
46. Durojaiye L., and O'Meara M. A study of triage of paediatric patients in Australia. Emerg Med (Fremantle) 14 (2002) 67-76
47. Toulson K., Laskowski-Jones L., and McConnell L.A. Implementation of the five-level emergency severity index in a level I trauma center emergency department with a three-tiered triage scheme. J Emerg Nurs 31 (2005) 259-264
48. Worster A., Sardo A., Eva K., Fernandes C.M., and Upadhye S. Triage tool inter-rater reliability: a comparison of live versus paper case scenarios. J Emerg Nurs 33 (2007) 319-323
49. Fernandes C.M., Wuerz R., Clark S., and Djurdjev O. How reliable is emergency department triage?. Ann Emerg Med 34 (1999) 141-147
50. Gerven van R., Delooz H., and Sermeus W. Systematic triage in the emergency department using the Australian National Triage Scale: a pilot project. Eur J Emerg Med 8 (2001) 3-7
51. Landis J.R., and Koch G.G. The measurement of observer agreement for categorical data. Biometrics 33 (1977) 159-174
52. Cicchetti D., Bronen R., Spencer S., Haut S., Berg A., Oliver P., et al. Rating scales, scales of measurement, issues of reliability. J Nerv Ment Dis 194 (2006) 557-564 Ref Type: Magazine Article
53. Feinstein A.R., and Cicchetti D.V. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 43 (1990) 543-549
54. Brenner H., and Kliebsch U. Dependence of weighted kappa coefficients on the number of categories. Epidemiology 7 (1996) 199-202
55. Cicchetti D.V., and Feinstein A.R. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol 43 (1990) 551-558