Consumer notes clinical indicators: Determining inter-rater reliability with multiple raters, nominal categories and several cases

Australian Journal of Advanced Nursing

Volumn 21, Issue 3, 2004, Pages 14-19

  Gaskin, Cadeyrn J ^a,b   O'Brien, Anthony P ^c   Hardy, Derrylea J ^a  

^a MASSEY UNIVERSITY   (New Zealand)

^b VICTORIA UNIVERSITY   (Australia)

^c SOUTHERN CROSS UNIVERSITY   (Australia)

Author keywords

Inter rater reliability; Mental health; Rater agreement

Indexed keywords

EID: 22244434123     PISSN: 08130531     EISSN: 14474328     Source Type: Journal    
DOI: None     Document Type: Article

References (24)

1. Australian and New Zealand College of Mental Health Nurses Inc. (ANZCMHN). 1995. Standards of practice for mental health nursing in New Zealand.Greenacres. South Australia: ANZCMHN.
2. Agresti, A. 1992. Modelling patterns of agreement and disagreement. Statistical Methods in Medical Research. 1:201-218.
3. Banerjee, M., Capozzoli, M., McSweeney, L. and Sinha, D. 1999. Beyond kappa: A review of interrater agreement measures. The Canadian Journal of Statistics. 27:3-23.
4. Bloch, D.A. and Kraemer, H.C. 1989. 2x2 kappa coefficients: Measures of agreement or association. Biometrics. 45:269-287.
5. Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 20:37-46.
6. Gaskin, C.J., O'Brien, A.P. and Hardy, D.J. 2003. The development of a professional practice audit questionnaire for mental health nursing in Aotearoa/New Zealand. International Journal of Mental Health Nursing. 12:259-270.
7. Hardy, D.J., O'Brien, A.P., Gaskin, C.J., O'Brien, A.J., Morrison-Ngatai, E., Skews, G., Ryan, T. and McNulty, N. In press. The Delphi prioritisation of New Zealand mental health nursing clinical indicators: A bi-cultural study. Journal of Advanced Nursing.
8. Judd, C.M., Smith, E.R. and Kidder, L.H. 1991. Research methods in social relations. 6th edn. Fort Worth, TX: Holt, Rinehart and Winston.
9. Ministry of Health. 2002. AIM: Advice to the Incoming Minister of Health. Retrieved October 11, 2002 from http://www.moh.govt.nz.
10. Nelson, J.C. and Pepe, M.S. 2000. Statistical description of interrater variability in ordinal ratings. Statistical Methods in Medical Research. 9:475-496.
11. O'Brien, A.P., O'Brien, A.J., McNulty, N.G., Morrison-Ngatai, E., Skews, G., Ryan, T., Hardy, D.J., Gaskin, C.J. and Boddy, J.M. 2002a. Clinical indicators for mental health standards of practice in Aotearoa/New Zealand: A report to the Health Research Council of New Zealand. Palmerston North. New Zealand: Massey University.
12. O'Brien, A.P., O'Brien, A.J., McNulty, N.G., Morrison-Ngatai, E., Skews, G., Ryan T., Hardy, D.J., Gaskin, C.J. and Boddy, J.M. 2002b. Clinical indicators for mental health nursing standards of practice in Aotearoa/New Zealand: Consumer Notes Clinical Indicators audit booklet. Palmerston North. New Zealand: Massey University.
13. O'Brien, A.P., O'Brien, A.J., McNulty, N.G., Morrison-Ngatai, E., Skews, G., Ryan T., Hardy, D.J., Gaskin, C.J. and Boddy, J.M. 2002c. Clinical indicator statements for mental health nursing standards of practice in Aotearoa/New Zealand: A guide to the Professional Practice Audit Questionnaire. Palmerston North. New Zealand: Massey University.
14. O'Brien, A.P., O'Brien, A.J., McNulty, N.G., Morrison-Ngatai, E., Skews, G., Ryan T., Hardy, D.J., Gaskin, C.J. and Boddy, J.M. 2002d. Clinical indicators for mental health nursing standards of practice in Aotearoa/New Zealand: Consumer Notes Clinical Indicators audit guide. Palmerston North. New Zealand: Massey University.
15. O'Brien, A.P., O'Brien, A.J., Hardy, D.J., Morrison-Ngatai, E., Gaskin, C.J., Boddy, J.M., McNulty, N., Ryan, T. and Skews, G. 2003. The New Zealand development and trial of mental health nursing clinical indicators - A bicultural study. International Journal of Nursing Studies. 40:835-861.
16. Pearson, K. 1900. Mathematical contribution to the theory of evolution VII: On the correlation of characters not quantitatively measurable. Philosophical Transactions of the Royal Society of London. Series A: Containing papers of a mathematical or physical character. 195:1-47.
17. Polit, D.F. and Hungler, B.P. 1999. Nursing research: Principles and methods. 6th edn. Philadelphia: Lippincott-Williams & Wilkins.
18. Shaughnessy, J.J. and Zechmeister, E.B. 1997. Research methods in psychology. 4th edn. New York: McGraw-Hill.
19. Shrout, P.E. and Fleiss, J.L. 1979. Interclass correlations: Uses in assessing rater reliability. Psychological Bulletin. 86:420-428.
20. Statistics New Zealand. 2001. Resident Maori population estimates as at 30 June 2001. Retrieved December 21, 2001 from http://www.stats.govt.nz/domino/external/web/prod_serv.nsf/8048d3976fd325824c2567e2000de0fc2/76502b12a6e113bcc25694200808060$/FILE/alltabls5.xls.
21. Szalai, J.P. 1993. The statistics of agreement on a single item or object by multiple raters. Perceptual and Motor Skills. 77:377-378.
22. Szalai, J.P. 1998. KappaSC. A measure of agreement on a single rating category for a single item or object rated by multiple raters. Psychological Reports. 82:1321-1322.
23. Tanner, M.A. and Young, M.A. 1985. Modeling agreement among raters. Journal of the American Statistical Association. 80:175-180.
24. Uebersax, J.S. and Grove, W.M. 1990. Latent class analysis of diagnostic agreement. Statistics in Medicine. 9:559-572.