Evaluating interrater agreement in SPICE-based assessments

Computer Standards and Interfaces

Volumn 25, Issue 5, 2003, Pages 477-499

  Jung, Ho Won ^a  

^a Korea University   (South Korea)

Author keywords

Interrater agreement; Kappa coefficient; Observed agreement; Software process assessments; SPICE

Indexed keywords

COMPUTER AIDED SOFTWARE ENGINEERING; DATA REDUCTION; STATISTICAL METHODS;

SOFTWARE PROCESS ASSESSMENTS (SPA);

COMPUTER SOFTWARE;

EID: 0041466364     PISSN: 09205489     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0920-5489(03)00018-7     Document Type: Article

References (42)

1. Agresti A. Categorical Data Analysis. 1990;Wiley, New York.
2. Agresti A. An Introduction to Categorical Data Analysis. 1996;Wiley, New York.
3. Armitage P. Test for linear trend in proportions and frequencies. Biometrics. 11:1955;375-386.
4. Byrt T., Bishop J., Carlin J.B. Bias, prevalence and Kappa. Journal of Clinical Epidemiology. 46(5):1993;423-424.
5. Casella G. Refining binomial confidence intervals. Canadian Journal of Statistics. 14:1987;113-129.
6. Cicchetti D.V., Feinstein A.R. High agreement but low Kappa: II. The problems of two paradoxes. Journal of Clinical Epidemiology. 43(6):1990;551-558.
7. Cochran W.G. Some methods for strengthening the common tests. Biometrics. 10:1954;417-454.
8. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. XX(1):1960;37-46.
9. Cohen J. Weighted Kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin. 70(4):1968;213-220.
10. El-Emam K., Goldenson D. SPICE: an empiricist's perspective. Proceedings of the Second IEEE International Software Engineering Standards Symposium. 1995;84-97 IEEE Computer Society Press, Los Alamitos, CA.
11. El-Emam K., Goldenson D., Briand L., Marshall P. Interrater agreement in SPICE based assessments: some preliminary results. Proceedings of the Fourth International Conference on the Software Process. 1996;149-156 IEEE Computer Society Press, Los Alamitos, CA.
12. El-Emam K., Briand L., Smith R. Assessor agreement in rating SPICE processes. Software Process Improvement and Practice. 2(4):1996;291-306.
13. El-Emam K., Smith R., Fusaro P. Modeling the reliability of SPICE based assessments. Proceedings of the Third International Symposium on Software Engineering Standards. 1997;69-82 IEEE Computer Society Press, Los Alamitos, CA.
14. El-Emam K. The internal consistency of the ISO/IEC 15504 software process capability scale. Proceedings of the 5th International Symposium on Software Metrics. 1998;72-81 IEEE Computer Society Press, Los Alamitos, CA.
15. El-Emam K., Marshall P. Interrater agreement in assessment ratings. El-Emam K., Drouin J.-N., Melo W. SPICE: The Theory and Practice of Software Process Improvement and Capability Determination. 1998;357-362 IEEE Computer Society Press, Los Alamitos, CA.
16. El-Emam K., Simon J.-M., Rousseau S., Jacquet E. Cost implications of interrater agreement for software process assessment. Proceedings of the 5th International Symposium on Software Metrics. 1998;38-51 IEEE Computer Society Press, Los Alamitos, CA.
17. El-Emam K. Benchmarking Kappa: interrater agreement in software process assessments. Empirical Software Engineering: An International Journal. 4(2):1999;113-133.
18. El-Emam K., Goldenson D. An empirical review of software process assessments. Advances in Computers. 53:2000;319-423.
19. El-Emam K., Jung H.-W. An evaluation of the ISO/IEC 15504 assessment model. Journal of Systems and Software. 59(1):2001;23-41.
20. Feinstein A.R., Cicchetti D.V. High agreement but low Kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology. 43(6):1990;543-549.
21. Fleiss J.L. Statistical Methods for Rates and Proportions. 1981;Wiley, New York.
22. Fleiss J.L., Cohen J. The equivalence of weighted kappa and the interclass correlation coefficient as measured of reliability. Educational and Psychological Measurement. 33:1973;613-619.
23. Freeman D.H. Jr. Applied Categorical Data Analysis. 1987;Marcel Dekker, New York.
24. Fusaro P., El-Emam K., Smith B. Evaluating the interrater agreement of process capability ratings. Proceedings of the Fourth International Software Metrics Symposium. 1997;2-11 IEEE Computer Society Press, Los Alamitos, CA.
25. Fusaro P., El-Emam K., Smith B. The internal consistencies of the 1987 SEI maturity questionnaire and the SPICE capability dimension. Empirical Software Engineering: An International Journal. 3(2):1998;179-201.
26. Guggenmoos-Holzmann I., Vonk R. Kappa-like indices of observer agreement viewed from a latent class perspective. Statistics in Medicine. 17:1998;797-812.
27. ISO/IEC JTC1 Directive, Procedures for the Technical Work of ISO/IEC JTC1 (ISO, Geneva, Switzerland, 1999), http://www.jtc1.org/directives/toc.htm .
28. ISO/IEC TR2 15504, Information Technology - Software Process Assessment: Part 1-9, ISO, Geneva, Switzerland, 1998 (Part 5 was published in 1999).
29. Jung H.-W., Hunter R., Goldenson R., El-Emam K. Findings from Phase 2 of the SPICE Trials. Software Process Improvement and Practice: International Journal. 6(5):2001;205-242.
30. Jung H.W., Hunter R. The Relationship between ISO/IEC 15504 process capability levels, ISO 9001 certification, and organizational size: an empirical study. Journal of Systems and Software. 59(1):2001;43-55.
31. Jung H.-W., Hunter R. An evaluation of SPICE rating scale with regard to the internal consistency of capability measurement. Proceedings of SPICE2002. 2002;105-115 Nuovo Studio Tecna, Rome, Italy.
32. H.-W. Jung, Evaluating the internal consistency of SPICE process capability indictors, submitted for publication, 2002.
33. A Guideline for KSPICE Assessment Procedure. 2002;Korea SPICE, Seoul, Korea.
34. Lantz C.A., Nebenzahl E. Behavior and interpretation of the statistic: resolution of the two paradoxes. Journal of Clinical Epidemiology. 49(4):1996;431-434.
35. Lee H.-Y., Jung H.-W., Chung C.-S., Lee J.-M., Lee K.-W., Jeong H.-J. Analysis of interrater agreement In ISO/IEC 15504-based software process assessment. Proceedings of the 2nd Asia-Pacific Conference on Quality Software (APAQS2001). 2001;341-348 IEEE Computer Society Press, Los Alamitos, CA.
36. Pickard L.M., Kitchenham B.A., Jones P.W. Combining empirical results in software engineering. Information and Software Technology. 40(12):1998;811-821.
37. Montgomery D.C., Runger G.C., Hubele N.F. Engineering Statistics. 1998;Wiley, New York.
38. Standard CMMI Appraisal Method for Process Improvement (SCAMPI), Version 1.1: Method Definition Document (CMU/SEI-HB-2001). 2000;Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA.
39. Assessment Requirements for CMMI (ARC), Version 1.0 (CMU/SEI-2000-TR-011). 2000;Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA.
40. J.-M. Simon, K. El-Emam, S.-R. Jacquet, F. Babet, The Reliability of ISO/IEC PDTR 15504 assessments, International Software Engineering Research Network, Technical Report ISERN-97-28, 1997.
41. StatXact-4 for Windows, Software for Exact Nonparametric Inference, Cytel Software, Cambridge, MA, 1999.
42. J.S. Uebersax, Statistical Methods for Rater Agreement, http://ourworld.compuserve.com/homepages/jsuebersax/agree.htm, 2001.