Testing agreement for multi-item scales with the indicesrWG(J) and AD M(J)

Organizational Research Methods

Volumn 12, Issue 1, 2009, Pages 148-164

  Cohen, Ayala ^a   Doveh, Etti ^a   Nahum Shani, Inbal ^a  

^a TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

AD; Indices of agreement; Multi item scale; RWG; Significance tests

Indexed keywords

EID: 57849153967     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428107300365     Document Type: Article

References (32)

1. Bliese, P.D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analyses. In K. J. Klein & S. W.J. Kozlowski (Eds.), Multilevel theory, research, and methods in organization: Foundations, extensions, and new directions (pp 349-381). San Francisco: Jossey-Bass.
2. Bliese, P.D. (2006). Multilevel modeling in R (2.2): A Brief Introduction to R, the Multilevel Package and the nlme Package. Washington, DC: Walter Reed Army Institute of Research.
3. Bliese, P.D., & Halverson, R.R. (1996). Individual and nomothetic models of job stress: An examination of work hours, cohesion, and well-being. Journal of Applied Social Psychology, 26, 1171-1189.
4. Bliese, P.D., & Halverson, R.R. (2002). Using random group resampling in multilevel research: An example of the buffering effects of leadership climate. Leadership Quarterly, 13, 53-68.
5. Bliese, P.D., Halverson, R.R., & Rothberg, J.M. (1994). Within-group agreement scores: Using resampling techniques to estimate expected variance. In D. P. Moore (Ed), Best paper proceedings (pp. 303-307). Dallas, TX: Academy of Management.
6. Bliese, P.D., Halverson, R.R., & Rothberg, J.M. (2000). Using random group resampling (RGR) to estimate within-group agreement with examples using the statistical language R. Unpublished working paper, Walter Reed Army Institute.
7. Bliese, P.D., Halverson, R.R., & Schriesheim, C.A. (2002). Benchmarking multilevel methods in leadership: The articles, the model, and the data set. Leadership Quarterly, 13, 3-14.
8. Brown, R.D., & Hauenstein, N.M.A. (2005). Interrater agreement reconsidered: An alternative to the rwg indices. Organizational Research Methods, 8, 165-184.
9. Burke, M.J., & Dunlap, W.P. (2002). Estimating interrater agreement with the average deviation index: A user's guide. Organizational Research Methods, 5, 159-172.
10. Burke, M.J., Finkelstein, L.M., & Dusig, M.S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49-68.
11. Cario, M.C., & Nelson, B.L. (1997). Modeling and generating random vectors with arbitrary marginal distributions and correlation matrix (Tech. Rep.). Evanston, IL: Northwestern University, Department of Industrial Engineering and Management Sciences.
12. Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composite models. Journal of Applied Psychology, 83 (2). 234-246.
13. Charnes, J.M., & Schriesheim, C.A. (1995). Estimation of quantiles for the sampling distribution of the rWG(J) within-groups agreement index. Educational and Psychological Measurement, 55, 435-437.
14. Chen, G., Mathieu, J.E., & Bliese, P. (2004). A framework for conducting multilevel construct validation. In F. J. Dansereau & F. Yammarino (Eds.). Research in multi-level issues: Multi-level issues in organizational behavior and processes (Vol. 3, pp. 273-303). Elsevier: Oxford, UK
15. Cohen, A., Doveh, E., & Eick, U. (2001). Statistical properties of the rWG(J) index of agreement. Psychological Methods, 6, 297-310.
16. Derogatis, L.R., & Spencer, P.M. (1982). The Brief Symptom Inventory: Administration, scoring and procedures manual - I. Baltimore: Johns Hopkins School of Medicine, Clinical Psychometrics Research Unit.
17. Dunlap, W.P., Burke, M.J., & Smith-Crowe, K. (2003). Accurate tests of statistical significance for rWG and average deviation interrater agreement indexes. Journal of Applied Psychology, 88, 356-362.
18. Ehrhart, M.G. (2004). Leadership and procedural justice climate as antecedents of unit-level organizational citizenship behavior. Personnel Psychology, 57, 61-94.
19. George, J.M. (1990). Personality, affect and behavior in groups. Journal of Applied Psychology, 75, 107-116.
20. Ghosh, S., & Henderson, S.G. (2002). Chessboard distributions and random vectors with specified marginals and covariance matrix. Operations Research, 50, 820-834.
21. Gonzalez-Roma, V., Peiro, J.M., & Tordera, N. (2002). An examination of the antecedents and moderator influences of climate strength. Journal of Applied Psychology, 87, 465-473.
22. Harvey, R.J., & Hollander, E. (2004). Benchmarking rWG interrater agreement indices: Let's drop the.70 rule-of-thumb. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Chicago.
23. James, L.R., Demaree, R.G., & Wolf, G. (1984). Estimating within groups interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
24. Kozlowski, S.W.J., & Hattrup, K. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161-167.
25. Lance, C.E., Butts, M.M., & Michels, L.C. (2006). The sources of four commonly reported cutoff criteria: What did they really say ? Organizational Research Methods, 9, 202-220.
26. LeBreton, J.M., Burgess, J.R.D., Kaiser, R.B., Atchley, E.K., & James, L.R. (2003). The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar ? Organizational Research Methods, 6, 80-128.
27. LeBreton, J.M., James, L.R., & Lindell, M.K. (2005). Recent issues regarding rWG, r*WG, rWG(J), and r*WG(J). Organizational Research Methods, 8, 128-138.
28. Li, S.T., & Hammond, J.L. (1975). Generation of pseudorandom numbers with specified univariate distributions and correlation coefficients. IEEE Transactions on Systems, Man and Cybernetics, 5, 557-561.
29. Lindell, M.K., & Brandt, C.J. (1997). Assessing interrater agreement on the job relevance of a test: A comparison of the CVI, T, rWG(J), r*WG(J) and indexes. Journal of Applied Psychology, 84, 640-647.
30. Lindell, M.K., Brandt, C.J., & Whitney, D.J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23, 127-135.
31. Schriesheim, C.A., Cogliser, C.C., & Neider, L. (1995). Is it ''trustworthy''? A multi-level of analysis reexamination of an Ohio State leadership study with implications for future research. Leadership Quarterly, 6, 111-145.
32. Tinsley, H.E.A., & Weiss, D.J. (1975). Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology, 22, 358-376.