Assessing Interrater Agreement via the Average Deviation Index Given a Variety of Theoretical and Methodological Problems

Organizational Research Methods

Volumn 16, Issue 1, 2013, Pages 127-151

  Smith Crowe, Kristin ^a   Burke, Michael J ^b   Kouchaki, Maryam ^a   Signal, Sloane M ^c  

^a UNIVERSITY OF UTAH   (United States)

^b TULANE UNIVERSITY   (United States)

^c JACKSON STATE UNIVERSITY   (United States)

Author keywords

aggregation; average deviation (AD); interrater agreement; multilevel research; null distribution

Indexed keywords

EID: 84873695238     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428112465898     Document Type: Article

References (56)

1. Avery D. R.,McKay P. F.,Wilson D. C.What are the odds? How demographic similarity affects the prevalence of perceived employment discrimination.Journal of Applied Psychology. 2008;93:235-249.
2. Bledow R.,Frese M.A situational judgment test of personal initiative and its relationship to performance.Personnel Psychology. 2009;62:229-258.
3. Borucki C. C.,Burke M. J.An examination of service-related antecedents to retail store performance.Journal of Organizational Behavior. 1999;20:943-962.
4. Brief A. P.,Umphress E. E.,Dietz J.,Burrows J. W.,Butz R. M.,Scholten L.Community matters: Realistic group conflict theory and the impact of diversity.Academy of Management Journal. 2005;48:830-844.
5. Brown R. D.,Hauenstein N. M. A.Interrater agreement reconsidered: An alternative to the rwg indices.Organizational Research Methods. 2005;8:165-184.
6. Burke M. J.,Dunlap W. P.Estimating interrater agreement with the average deviation index: A user's guide.Organizational Research Methods. 2002;5:159-172.
7. Burke M. J.,Finkelstein L. M.,Dusig M. S.On average deviation indices for estimating interrater agreement.Organizational Research Methods. 1999;2:49-68.
8. Cascio W. F.Applied psychology in human resource management. Upper Saddle River, NJ: Prentice Hall; 1998:.
9. Cashen L. H.,Geiger S. W.Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies.Organizational Research Methods. 2004;7 (2): 151-167.
10. Chan D.Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models.Journal of Applied Psychology. 1998;83:234-246.
11. Cohen A.,Doveh E.,Nahum-Shani I.Testing agreement for multi-item scales with the indices rWG(J) and ADM(J).Organizational Research Methods. 2009;12:148-164.
12. Cohen J.Statistical power analysis for the behavioral sciences. New York, NY: Academic Press; 1977:.
13. Cortina J. M.,Folger R. G.When is it acceptable to accept a null hypothesis: No way, Jose?.Organizational Research Methods. 1998;1:334-350.
14. Dawson J. F.,Gonzalez-Roma V.,Davis A.,West M. A.Organizational climate and climate strength in UK hospitals.European Journal of Work and Organizational Psychology. 2008;17:89-111.
15. DeRue D. S.,Hollenbeck J. R.,Ilgen D. R.,Feltz D.Efficacy dispersion in teams: Moving beyond agreement and aggregation.Personnel Psychology. 2010;63:1-40.
16. Dunlap W. P.,Burke M. J.,Smith-Crowe K.Accurate tests of statistical significance for rWG and average deviation indexes.Journal of Applied Psychology. 2003;88:356-362.
17. Edwards J. R.,Berry J. W.The presence of something or the absence of nothing: Increasing theoretical precision in management research.Organizational Research Methods. 2010;13:668-689.
18. Grant A. M.,Mayer D. M.Good soldiers and good actors: Prosocial and impression management motives as interactive predictors of affiliate citizenship behaviors.Journal of Applied Psychology. 2009;94:900-912.
19. Greene W. H.Econometric analysis. Upper Saddle River, NJ: Prentice Hall; 1997:.
20. Greenwald A. G.Consequences of prejudice against the null hypothesis.Psychological Bulletin. 1975;82:1-20.
21. Guion R. M.Assessment, measurement, and prediction for personnel decisions. Mahwah, NJ: Lawrence Erlbaum; 1998:.
22. Harrison D. A.,Klein K. J.What's the difference? Diversity constructs as separation, variety, or disparity in organizations.Academy of Management Review. 2007;32:1199-1228.
23. Heidemeier H.,Moser K.Self-other agreement in job performance ratings: A meta-analytic test of a process model.Journal of Applied Psychology. 2009;94:353-370.
24. Hudson J. P.,Campion J. E.Hindsight bias in an application of the Angoff method for setting cutoff scores.Journal of Applied Psychology. 1994;79:860-865.
25. Hunter J. E.,Schmidt F. L.Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage; 2004:.
26. James L. R.,Demaree R. G.,Wolf G.Estimating within-group interrater reliability with and without response bias.Journal of Applied Psychology. 1984;69:85-98.
27. James L. R.,Demaree R. G.,Wolf G.rwg: An assessment of within-group interrater agreement.Journal of Applied Psychology. 1993;78:306-309.
28. Katz-Navon T.,Naveh E.,Stern Z.Active learning: When is more better? The case of resident physicians' medical errors.Journal of Applied Psychology. 2009;94:1200-1209.
29. Klein K. J.,Conn A. B.,Smith D. B.,Sorra J. S.Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment.Journal of Applied Psychology. 2001;86:3-16.
30. Klein K. J.,Dansereau F.,Hall R. J.Levels issues in theory development, data collection, and analysis.Academy of Management Review. 1994;19:195-229.
31. Kline T. J. B.,Hambley L. A.Four multi-item interrater agreement options: Comparisons and outcomes.Psychological Reports. 2007;101:1001-1010.
32. Kozlowski S. W. J.,Hattrup K.A disagreement about within-group agreement: Disentangling issues of consistency versus consensus.Journal of Applied Psychology. 1992;77:161-167.
33. Kozlowski S. W. J.,Klein K. J.Multilevel theory, research, and methods in organizations. Klein K. J.Kozlowski S. W. J., ed. San Francisco, CA: Jossey-Bass; 2000:3-90.
34. Kreiner G. E.,Hollensbe E. C.,Sheep M. L.Balancing borders and bridges: Negotiating the work-home interface via boundary work tactics.Academy of Management Journal. 2009;52:704-730.
35. Lawrence K. L.,Lenk P.,Quinn R. E.Behavioral complexity in leadership: The psychometric properties of a new instrument to measure behavioral repertoire.Leadership Quarterly. 2009;20:87-102.
36. LeBreton J. M.,Senter J. L.Answers to 20 questions about interrater reliability and interrater agreement.Organizational Research Methods. 2008;11:815-852.
37. Lindell M. K.,Brandt C. J.Measuring interrater agreement for ratings of a single target.Applied Psychological Measurement. 1997;21:271-278.
38. Lipsey M. W.,Wilson D. B.Practical meta-analysis. Thousand Oaks, CA: Sage; 2001:.
39. Lüdtke O.,Robitzsch A.Assessing within-group agreement: A critical examination of a random-group resampling approach.Organizational Research Methods. 2009;12:461-487.
40. McCall R. B.Fundamental statistics for psychology. New York, NY: Harcourt, Brace, & World, Inc; 1970:.
41. Messick D. M.Some cheap tricks for making inferences about distribution shapes from variances.Educational and Psychological Measurement. 1982;42:749-758.
42. Meyer R. D.,Mumford T. V.,Campion M. A.The practical consequences of null distribution choice on rwg. 2010:.
43. Newman D. A.,Sin H-P.How do missing data bias estimates of within-group agreement? Sensitivity of SDWG, CVWG, rWG(J), rWG(J)*, and ICC to systematic nonresponses.Organizational Research Methods. 2009;12:113-147.
44. Nicklin J. M.,Roch S. G.Letters of recommendation: Controversy and consensus from expert perspectives.International Journal of Selection and Assessment. 2009;17:76-91.
45. Parsons R.Statistical analysis. New York, NY: Harper & Row; 1978:.
46. Roberson Q. M.,Sturman M. C.,Simons T. L.Does the measure of dispersion matter in multilevel research? A comparison of the relative performance of dispersion indexes.Organizational Research Methods. 2007;10:564-588.
47. Rousseau D. M.Issues of level in organizational research: Multi-level and cross-level perspectives.Research in Organizational Behavior. 1985;7:1-37.
48. Schriesheim C. A.The effect of grouping or randomizing items on leniency response bias.Educational and Psychological Measurement. 1981;41:401-411.
49. Schwarz N.Self-reports: How questions shape the answers.American Psychologist. 1999;54:93-105.
50. Schwarz N.,Knauper B.,Hippler H. J.,Noelle-Neumann E.,Clark F.Rating scales: Numeric values may change the meaning of scale labels.Public Opinion Quarterly. 1991;55:570-582.
51. Smith-Crowe K.,Burke M.J.Interpreting the statistical significance of observed AD interrater agreement values: Corrections to Burke and Dunlap (2002).Organizational Research Methods. 2003;6:129-131.
52. Takeuchi R.,Chen G.,Lepak D. P.Through the looking glass of a social system: Cross-level effects of high performance work systems on employees' attitudes.Personnel Psychology. 2009;62:1-29.
53. Trougakos J. P.,Beal D. J.,Green S. G.,Weiss H. M.Making the break count: An episodic examination of recovery activities, emotional experiences, and positive affective displays.Academy of Management Journal. 2008;51:131-146.
54. Van Kleef G. A.,Homan A. C.,Beersma B.,Van Knippenberg D.,Knippenberg B. V.,Damen F.Searing sentiment or cold calculation? The effects of leader emotional displays on team performance depend on follower epistemic motivation.Academy of Management Journal. 2009;52:562-580.
55. Wagner S. M.,Rau C.,Lindermann E.Multiple informant methodology: A critical review and recommendations.Sociological Methods and Research. 2010;38:582-618.
56. Walumbwa F. O.,Schaubroeck J.Leader personality traits and employee voice behavior: Mediating roles of ethical leadership and work group psychological safety.Journal of Applied Psychology. 2009;94:1275-1286.