Statistical significance criteria for the rwg and average deviation interrater agreement indices

Journal of Applied Psychology

Volumn 99, Issue 2, 2014, Pages 239-261

  Smith Crowe, Kristin ^a   Burke, Michael J ^b   Cohen, Ayala ^c   Doveh, Etti ^c  

^a UNIVERSITY OF UTAH   (United States)

^b TULANE UNIVERSITY   (United States)

^c TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

Ad; Data aggregation; Interrater agreement; Multilevel research; Rwg

Indexed keywords

COMPUTER SIMULATION; HUMAN; PROCEDURES; PSYCHOLOGY; STATISTICAL ANALYSIS; STATISTICAL DISTRIBUTION;

COMPUTER SIMULATION; DATA INTERPRETATION, STATISTICAL; HUMANS; PSYCHOLOGY, APPLIED; STATISTICAL DISTRIBUTIONS;

EID: 84897638323     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0034556     Document Type: Article

References (64)

1. Bernardin, H. J., & Buckley, M. R. (1981). Strategies for rating training. Academy of Management Review, 6, 205-212.
2. WG(J) in leadership research and some best practice guidelines. Leadership Quarterly, 23, 66-80.
3. Bledow, R., & Frese, M. (2009). A situational judgment test of personal initiative and its relationship to performance. Personnel Psychology, 62, 229-258.
4. Bliese, P. (2012). Multilevel modeling in R (2.4): A brief introduction to R, the multilevel package, and the nlme package. Retrieved from http:// cran.rproject.org/doc/contrib/Bliese_Multilevel.pdf
5. Bliese, P. D., & Halverson, R. R. (1996). Individual and nomothetic models of job stress: An examination of work hours, cohesion, and well-being. Journal of Applied Social Psychology, 26, 1171-1189.
6. Bliese, P. D., & Halverson, R. R. (2002). Using random group resampling in multilevel research. Leadership Quarterly, 13, 53-68.
7. Bliese, P. D., Halverson, R. R., & Rothberg, J. (2000). Using random group resampling (RGR) to estimate within-group agreement with examples using the statistical language R. Unpublished manuscript.
8. Borucki, C. C., & Burke, M. J. (1999). An examination of service-related antecedents to retail store performance. Journal of Organizational Behavior, 20, 943-962.
9. Brown, R. D., & Hauenstein, N. M. A. (2005). Interrater agreement reconsidered: An alternative to the rwg indices. Organizational Research Methods, 8, 165-184.
10. Burke, M. J., Chan-Serafin, S., Salvador, R., Smith, A., & Sarpy, S. (2008). The role of national culture and organizational climate in safety training effectiveness. European Journal of Work and Organizational Psychology, 17, 133-152.
11. Burke, M. J., & Dunlap, W. P. (2002). Estimating interrater agreement with the average deviation index: A user's guide. Organizational Research Methods, 5, 159-172.
12. Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49-68.
13. Chen, Z., Lam, W., & Zhong, J. A. (2007). Leader-member exchange and member performance: A new look at individual-level negative feedbackseeking behavior and team-level empowerment climate. Journal of Applied Psychology, 92, 202-212.
14. Chuang, C., & Liao, H. (2010). Strategic human resource management in service context: Taking care of business by taking care of employees and customers. Personnel Psychology, 63, 153-196.
15. WG(J) index of agreement. Psychological Methods, 6, 297-310.
16. M(J). Organizational Research Methods, 12, 148-164.
17. Colquitt, J. A., Noe, R. A., & Jackson, C. L. (2002). Justice in teams: Antecendents and consequences of procedural justice climate. Personnel Psychology, 55, 83-109.
18. Dawson, J. F., González-Romá, V., Davis, A., & West, M. A. (2008). Organisational climate and climate strength in UK hospitals. European Journal of Work and Organizational Psychology, 17, 89-111.
19. WG and average deviation indexes. Journal of Applied Psychology, 88, 356-362.
20. Ehrhart, M. G. (2004). Leadership and procedural justice climate as antecedents of unit-level organizational citizenship behavior. Personnel Psychology, 57, 61-94.
21. Farh, J.-L., Lee, C., & Farh, C. I. C. (2010). Task conflict and team creativity: A question of how much and when. Journal of Applied Psychology, 95, 1173-1180.
22. French-Lazovik, G., & Gibson, C. L. (1984). Effects of verbally labeled anchor points on the distributional parameters of rating measures. Applied Psychological Measurement, 8, 49 -57.
23. George, J. M., & James, L. R. (1993). Personality, affect, and behavior in groups revisited: Comment on aggregation, levels of analysis, and a recent application of within and between analysis. Journal of Applied Psychology, 78, 798-804.
24. González-Romá, V., Fortes-Ferreira, L., & Peiró, J. (2009). Team climate, climate strength and team performance: A longitudinal study. Journal of Occupational and Organizational Psychology, 82, 511-536.
25. Harrison, D. A., & Klein, K. J. (2007). What's the difference? Diversity constructs as separation, variety, or disparity in organizations. Academy of Management Review, 32, 1199 -1228.
26. Hoffman, B. J., Gorman, C. A., Blair, C. A., Meriac, J. P., Overstreet, B., & Atchley, E. K. (2012). Evidence for the effectiveness of an alternative multisource performance rating methodology. Personnel Psychology, 65, 531-563.
27. Hofmann, D. A., & Mark, B. (2006). An investigation of the relationship between safety climate and medication errors as well as other nurse and patient outcomes. Personnel Psychology, 59, 847-869.
28. James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
29. Katz-Navon, T., Naveh, E., & Stern, Z. (2009). Active learning: When is more better? The case of resident physicians' medical errors. Journal of Applied Psychology, 94, 1200-1209.
30. Klein, K. J., Conn, A. B., Smith, D. B., & Sorra, J. S. (2001). Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86, 3-16.
31. LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815-852.
32. Leisch, F., Weingessel, A., Kaiser, S., & Hornik, K. (2011). Orddata: Generation of artificial ordinal and binary data (R package version 0.1/r4). [Computer software]. Available from http://ie.technion.ac.il/Labs/ Stat/
33. Lindell, M. K., & Brandt, C. J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21, 271-278.
34. Lüdtke, O., & Robitzsch, A. (2009). Assessing within-group agreement: A critical examination of a random-group resampling approach. Organizational Research Methods, 12, 461-487.
35. Messick, D. M. (1982). Some cheap tricks for making inferences about distribution shapes from variances. Educational and Psychological Measurement, 42, 749-758.
36. Meyer, R. D., Mumford, T. V., & Campion, M. A. (2010, August). The practical consequences of null distribution choice on rwg. Paper presented at the annual meeting of the Academy of Management, Montreal, Canada.
37. Murphy, K. R., Cronin, B. E., & Tam, A. P. (2003). Controversy and consensus regarding the use of cognitive ability testing in organizations. Journal of Applied Psychology, 88, 660-671.
38. Neal, A., & Griffin, M. A. (2006). A longitudinal study of the relationships among, safety climate, safety behavior, and accidents at the individual and group levels. Journal of Applied Psychology, 91, 946-953.
39. Ng, K.-Y., Koh, C., Ang, S., Kennedy, J. C., & Chang, K.-Y. (2011). Rating leniency and halo in multisource feedback ratings: Testing cultural assumptions of power distance and individualism-collectivism. Journal of Applied Psychology, 96, 1033-1044.
40. Nicklin, J. M., & Roch, S. G. (2009). Letters of recommendation: Controversy and consensus from expert perspectives. International Journal of Selection and Assessment, 17, 76-91.
41. Nunnally, J. C. (1967). Psychometric theory. New York, NY: McGraw Hill.
42. R Development Core Team. (2011). R: A language and environment for statistical computing. Available from http://www.R-project.org
43. Rego, A., Ribeiro, N., & Cunha, M. P. (2010). Perceptions of organizational virtuousness and happiness as predictors of organizational citizenship behaviors. Journal of Business Ethics, 93, 215-235.
44. Rego, A., Ribeiro, N., Cunha, M. P., & Jesuino, J. C. (2011). How happiness mediates the organizational virtuousness and affective commitment relationship. Journal of Business Research, 64, 524-532.
45. Rego, A., Sousa, F., Marques, S., & Cunha, M. P. (2012). Authentic leadership promoting employees' psychological capital and creativity. Journal of Business Research, 65, 429-437.
46. Rego, A., Vitória, A., Magalhães, A., Ribeiro, N., & Cunha, M. P. (2013). Are authentic leaders associated with more virtuous, committed and potent teams? Leadership Quarterly, 24, 61-79.
47. Roberson, Q. M., Sturman, M. C., & Simons, T. L. (2007). Does the measure of dispersion matter in multilevel research? A comparison of the relative performance of dispersion indexes. Organizational Research Methods, 10, 564-588.
48. Rynes, S. L., Colbert, A. E., & Brown, K. G. (2002). HR professionals' beliefs about effective human resource practices: Correspondence between research and practice. Human Resource Management, 41, 149- 174.
49. Salanova, M., Agut, S., & Peiró, J. M. (2005). Linking organizational resources and work engagement to employee performance and customer loyalty: The mediation of service climate. Journal of Applied Psychology, 90, 1217-1227.
50. Scandura, T. A., & Graen, G. B. (1984). Moderating effects of initial leader-member exchange status on the effects of a leadership intervention. Journal of Applied Psychology, 71, 579-584.
51. Schriesheim, C. A. (1981). The effect of grouping or randomizing items on leniency response bias. Educational and Psychological Measurement, 41, 401-411.
52. Schwarz, N. (1999). Self-reports: How questions shape the answers. American Psychologist, 54, 93-105.
53. Simons, T., & Roberson, Q. (2003). Why managers should care about fairness: The effects of aggregate justice perceptions on organizational outcomes. Journal of Applied Psychology, 88, 432-443.
54. Smith-Crowe, K., & Burke, M. J. (2003). Interpreting the statistical significance of observed AD interrater agreement values: Correction to Burke and Dunlap (2002). Organizational Research Methods, 6, 129- 131.
55. Smith-Crowe, K., Burke, M. J., Kouchaki, M., & Signal, S. M. (2013). Assessing interrater agreement via the average deviation index given a variety of theoretical and methodological problems. Organizational Research Methods, 16, 127-151.
56. Smith-Crowe, K., Burke, M. J., & Landis, R. (2003). Organizational climate as a moderator of safety knowledge-safety performance relationships. Journal of Organizational Behavior, 24, 861-876.
57. Sonnentag, S., & Grant, A. (2012). Doing good at work feels good a home, but not right away: When and why perceived prosocial impact predicts positive affect. Personnel Psychology, 65, 495-530.
58. Sowinski, D. R., Fortmann, K. A., & Lezotte, D. V. (2008). Climate for service and the moderating effects of climate strength on customer satisfaction, voluntary turnover, and profitability. European Journal of Work and Organizational Psychology, 17, 73-88.
59. Takeuchi, R., Chen, G., & Lepak, D. P. (2009). Through the looking glass of a social system: Cross-level effects of high-performance work systems on employees' attitudes. Personnel Psychology, 62, 1-29.
60. Tordera, N., González-Romá, V., & Peiró, J. M. (2008). The moderator effect of psychological climate on the relationship between leader- member exchange (LMS) quality and role overload. European Journal of Work and Organizational Psychology, 17, 55-72.
61. Towler, A., Lezotte, D. V., & Burke, M. J. (2011). An examination of the service climate-firm performance chain: The role of customer retention. Human Resource Management, 50, 391-406.
62. Venkataramani, V., & Tangirala, S. (2010). When and why do central employees speak up? An examination of mediating and moderating variables. Journal of Applied Psychology, 95, 582-591.
63. Wallace, J. C., Popp, E., & Mondore, S. (2006). Safety climate as a mediator between foundation climates and occupational accidents. A group level study. Journal of Applied Psychology, 91, 681-688.
64. Wang, X. M., Wong, K. F. E., & Kwong, J. Y. Y. (2010). The roles of rater goals and rate performance levels in the distortion of performance ratings. Journal of Applied Psychology, 95, 546-561.