Assessing within-group agreement: A critical examination of a random-group resampling approach

Organizational Research Methods

Volumn 12, Issue 3, 2009, Pages 461-487

  Lüdtke, Oliver ^a   Robitzsch, Alexander ^b  

^a MAX PLANCK INSTITUTE FOR HUMAN DEVELOPMENT   (Germany)

^b Institute for Educational Progress   (Germany)

Author keywords

Intraclass correlation; Levels of analysis; Multilevel analysis; Within group agreement

Indexed keywords

EID: 67649566009     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428108317406     Document Type: Article

References (45)

1. Baumert, J., Lehmann, R.H., Lehrke, M., Schmitz, B., Clausen, M., Hosenfeld, I., et al. (1997). TIMSS: Mathematisch-Naturwissenschaftlicher Unterricht im internationalen Vergleich. [TIMSS: Mathematics and science instruction in international comparison.]. Opladen, Germany: Leske & Budrich.
2. Bliese, P.D. (2000). Within-group agreement, nonindependence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 349-381). San Francisco: Jossey-Bass.
3. Bliese, P.D. (2006). Multilevel modeling in R (2.2): A brief introduction to R, the multilevel package and the NLME package. Washington, DC: Walter Reed Army Institute of Research.
4. Bliese, P.D., & Halverson, R.R. (2002). Using random group resampling in multilevel research: An example of the buffering effects of leadership climate. Leadership Quarterly, 13, 53-68.
5. Bliese, P.D., Halverson, R.R., & Rothberg, J.M. (1994). Within-group agreement scores: Using resampling procedures to estimate expected variance. In D. P. Moore (Ed.), Academy of Management Best Paper Proceedings (pp. 303-307). Dallas, TX: Academy of Management.
6. Burke, M.J., & Dunlap, W.P. (2002). Estimating interrater agreement with the average deviation index: A user's guide. Organizational Research Methods, 5, 159-172.
7. Burke, M.J., Finkelstein, L.M., & Dusig, M.S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49-68.
8. Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246.
9. Cohen, A., & Doveh, E. (2005). More on the comparison of intra-class correlation coefficients (ICCs) as measures of homogeneity. In F. Dansereau & F. J. Yammarino (Eds.), Research in multi-level issues (Vol. 4, pp. 455-459). Oxford, UK: Elsevier.
10. Cohen, A., Doveh, E., & Eick, U. (2001). Statistical properties of the rWG(J) index of agreement. Psychological Methods, 6, 297-310. (Pubitemid 33600384)
11. Cohen, A., Doveh, E., & Nahum-Shani, I. (in press). Testing agreement for multi-item scales with the indices rWG(J)and ADM(J). Organizational Research Methods.
12. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
13. Dunlap, W.P., Burke, M.J., & Smith-Crowe, K. (2003). Accurate tests of statistical significance for rWG and average deviation interrater agreement indexes. Journal of Applied Psychology, 88, 356-362. (Pubitemid 36507880)
14. Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. New York: Chapman & Hall.
15. Finn, R.H. (1970). A note on estimating the reliability of categorical data. Educational and Psychological Measurement, 30, 71-76.
16. Fox, J. (2000). Nonparametric simple regression: Smoothing scatterplots. Thousand Oaks, CA: Sage.
17. George, J.M., & James, L.R. (1993). Personality, affect, and behavior in groups revisited: Comment on aggregation, levels of analysis, and a recent application of within and between analysis. Journal of Applied Psychology, 78, 798-804.
18. Gonzalez-Roma, V., Peiro, J.M., & Tordera, N. (2002). An examination of the antecedents and moderator influences of climate strength. Journal of Applied Psychology, 87, 465-473. (Pubitemid 36288922)
19. Good, P.I. (2005). Permutation, parametric, and bootstrap tests of hypotheses. Berlin, Germany: Springer.
20. Hanges, P.J., & Lyon, J.S. (2005). Interpreting changes in ICCs: To agree or not to agree, that is the question. In F. Dansereau & F. J. Yammarino (Eds.), Research in multi-level issues (Vol. 4, pp. 421-431). Oxford, UK: Elsevier.
21. Hoyt, W.T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it ? Psychological Methods, 5, 64-86.
22. James, L.R. (1982). Aggregation bias in estimates of perceptual agreement. Journal of Applied Psychology, 67, 219-229.
23. James, L.R., Demaree, R.G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
24. Klein, K.J., Conn, A.B., Smith, B.D., & Sorra, J.S. (2001). Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86, 3-16. (Pubitemid 33404724)
25. Lance, C.E., Butts, M.M., & Michels, L.C. (2006). The sources of four commonly reported cutoff criteria: What did they really say ? Organizational Research Methods, 9, 202-220. (Pubitemid 43907720)
26. LeBreton, J.M., Burgess, J.R.D., Kaiser, R.B., Atchley, E.K.P., & James, L.R. (2003). The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar ? Organizational Research Methods, 6, 80-128.
27. LeBreton, J.M., James, L.R., & Lindell, M.K. (2005). Recent issues regarding rWG, rWG, rWG(J), and rWG(J). Organizational Research Methods, 8, 128-138.
28. LeBreton, J.M., & Senter, J.L. (in press). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods.
29. Lindell, M.K. (2001). Assessing and testing interrater agreement on a single target using multi-rating scales. Applied Psychological Measurement, 25, 89-99.
30. Lindell, M.K., & Brandt, C.J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21, 271-278.
31. Lindell, M.K., Brandt, C.J., & Whitney, D.J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23, 127-135.
32. Li1/4dtke, O., Ki¶ller, O., Marsh, H.W., & Trautwein, U. (2005). Teacher frame of reference and the big-fish-littlepond effect. Contemporary Educational Psychology, 30, 263-285. (Pubitemid 40757530)
33. Lidtke, O., Trautwein, U., Kunter, M., & Baumert, J. (2006). Reliability and agreement of student ratings of the classroom environment: A reanalysis of TIMSS data. Learning Environments Research, 9, 215-230.
34. Manoukian, E.E. (1985). Modern concepts and theorems of mathematical statistics. New York: Springer.
35. Masterson, S.S. (2001). A trickle-down model of organizational justice: Relating employees' and customers' perceptions of and reactions to fairness. Journal of Applied Psychology, 86, 594-604. (Pubitemid 33570881)
36. McGraw, K.O., & Wong, S.P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46. (Pubitemid 126421252)
37. Moritz, S., & Watson, C.B. (1998). Levels of analysis issues in group psychology: Using efficacy as an example of a multilevel model. Group Dynamics: Theory, Research, and Practice, 2, 285-298. (Pubitemid 128682823)
38. R Development Core Team. (2006). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.
39. Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical linear models (2 nd ed.). Thousand Oaks, CA: Sage.
40. Ryan, A.M., Gheen, M.H., & Midgley, C. (1998). Why do some students avoid asking for help? An examination of the interplay among students' academic efficacy, teachers' social-emotional role, and the classroom goal structure. Journal of Educational Psychology, 90, 528-535. (Pubitemid 128467783)
41. Schneider, B., White, S.S., & Paul, M.C. (1998). Linking service climate and customer perceptions of service quality: Test of a causal model. Journal of Applied Psychology, 83, 150-163. (Pubitemid 128374310)
42. Snijders, T.A.B., & Bosker, R.J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.
43. Urdan, T., Midgley, C., & Anderman, E.M. (1998). The role of classroom goal structure in students' use of self-handicapping strategies. American Educational Research Journal, 35, 101-122. (Pubitemid 128358627)
44. Wong, A.F., Young, D.J., & Fraser, B.J. (1997). A multilevel analysis of learning environments and student attitudes. Educational Psychology, 17, 449-468. (Pubitemid 127155429)
45. Wooldridge, J.M. (2002). Econometric analysis of cross section and panel data. Cambridge: MIT Press.