Estimating within-group agreement in small groups: A proposed adjustment for the average deviation index

European Journal of Work and Organizational Psychology

Volumn 23, Issue 3, 2014, Pages 456-468

  Lohse Bossenz, Hendrik ^a   Kunina Habenicht, Olga ^a   Kunter, Mareike ^a  

^a GOETHE UNIVERSITY   (Germany)

Author keywords

Average Deviation Index; Group agreement; Multilevel

Indexed keywords

EID: 84894549495     PISSN: 1359432X     EISSN: 14640643     Source Type: Journal    
DOI: 10.1080/1359432X.2012.748189     Document Type: Article

References (46)

1. Bashshur, M. R., Hernandez, A. and Gonzalez-Roma, V. 2011. When managers and their teams disagree: A longitudinal look at the consequences of differences in perceptions of organizational support. Journal of Applied Psychology, 96: 558 - 573. doi:10.1037/a0022675
2. Biemann, T. and Kearney, E. 2010. Size does matter: How varying group sizes in a sample affect the most common measures of group diversity. Organizational Research Methods, 13: 582 - 599. doi:10.1177/1094428109338875
3. Bledow, R. and Frese, M. 2009. A situational judgement test of personal initiative and its relationship to perfomance. Personnel Psychology, 62: 229 - 258. doi:10.1111/j.1744-6570.2009.01137.x
4. Bliese, P. D. 2009. Multilevel modeling in R (2.3): A brief introduction to R, the multilevel package and the nlme package Unpublished manuscript
5. Brew, A. and Ginns, P. 2008. The relationship between engagement in the scholarship of teaching and learning and students' course experiences. Assessment & Evaluation in Higher Education, 33: 535 - 545. doi:10.1080/02602930701698959
6. WG indices. Organizational Research Methods, 8: 165 - 184. doi:10.1177/1094428105275376
7. Burke, M. J., Chan-Serafin, S., Salvador, R., Smith, A. and Sarpy, S. A. 2008. The role of national culture and organizational climate in safety training effectiveness. European Journal of Work and Organizational Psychology, 17: 133 - 152. doi:10.1080/13594320701307503
8. Burke, M. J. and Dunlap, W. P. 2002. Estimating interrater agreement with the Average Deviation Index: A user's guide. Organizational Research Methods, 5: 159 - 172. doi:10.1177/1094428102005002002
9. Burke, M. J., Finkelstein, L. M. and Dusig, M. S. 1999. On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2: 49 - 68. doi:10.1177/109442819921004
10. Chuang, C. H. and Liao, H. 2010. Strategic human resource management in service context: Taking care of business by taking care of employees and customers. Personnel Psychology, 63: 153 - 196. doi:10.1111/j.1744-6570.2009.01165.x
11. M(J). Organizational Research Methods, 12: 148 - 164. doi:10.1177/1094428107300365
12. Côté, S., Lopes, P. N., Salovey, P. and Miners, C. T. H. 2010. Emotional intelligence and leadership emergence in small groups. The Leadership Quarterly, 21: 496 - 508. doi:10.1016/j.leaqua.2010.03.012
13. Dawson, J. F., Gonzalez-Roma, V., Davis, A. and West, M. A. 2008. Organizational climate and climate strength in UK hospitals. European Journal of Work and Organizational Psychology, 17: 89 - 111. doi:10.1080/13594320601046664
14. Dettmers, S., Trautwein, U., Lüdtke, O., Kunter, M. and Baumert, J. 2010. Homework works if homework quality is high: Using multilevel modeling to predict the development of achievement in mathematics. Journal of Educational Psychology, 102: 467 - 482. doi:10.1037/a0018453
15. Diefendorff, J. M., Richard, E. M. and Croyle, M. H. 2006. Are emotional display rules formal job requirements? Examination of employee and supervisor perceptions. Journal of Occupational and Organizational Psychology, 79: 273 - 298. doi:10.1348/096317905x68484
16. WG and average deviation interrater agreement indexes. Journal of Applied Psychology, 88: 356 - 262. doi:10.1037/0021-9010.88.2.356
17. Efron, B. and Tibshirani, R. J. 1993. An introduction to the bootstrap, New York, NY: Chapman & Hill.
18. Gamero, N., Gonzalez-Roma, V. and Peiro, J. M. 2008. The influence of intra-team conflict on work teams' affective climate: A longitudinal study. Journal of Occupational and Organizational Psychology, 81: 47 - 69. doi:10.1348/096317907x180441
19. Gärtner, H. 2010. Wie schülerinnen und schüler ihre lernumwelt wahrnehmen: Ein vergleich verschiedener maße zur übereinstimmung von schülerwahrnehmungen [How students perceive their learning environment: A comparison of different measures for agreement of student perceptions]. Zeitschrift für Pädagogische Psychologie, 24: 111 - 122. doi:10.1024/1010-0652/a000009
20. Ginns, P. and Ellis, R. A. 2009. Evaluating the quality of e-learning at the degree level in the student experience of blended learning. British Journal of Educational Technology, 40: 652 - 663. doi:10.1111/j.1467-8535.2008.00861.x
21. Ginns, P., Prosser, M. and Barrie, S. 2007. Students' perceptions of teaching quality in higher education: The perspective of currently enrolled students, 603 - 615. Studies in Higher Education. doi:10.1080/03075070701573773
22. Gonzalez-Roma, V., Fortes-Ferreira, L. and Peiro, J. M. 2009. Team climate, climate strength and team performance: A longitudinal study. Journal of Occupational and Organizational Psychology, 82: 511 - 536. doi:10.1348/096317908x370025
23. Gorard, S. 2005. Revisiting a 90-year-old debate: The advantages of the mean deviation. British Journal of Educational Studies, 53: 417 - 430. doi:10.1111/j.1467-8527.2005.00304.x
24. Henri, J. F. 2006. Organizational culture and performance measurement systems. Accounting Organizations and Society, 31: 77 - 103. doi:10.1016/j.aos.2004.10.003
25. Homburg, C., Grozdanovic, M. and Klarmann, M. 2007. Responsiveness to customers and competitors: The role of affective and cognitive organizational systems. Journal of Marketing, 71 (3): 18 - 38. doi:10.1509/jmkg.71.3.18
26. James, L. R., Demaree, R. G. and Wolf, G. 1984. Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69: 85 - 98. doi:10.1037/0021-9010.69.1.85
27. WG: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78: 306 - 309. doi:10.1037/0021-9010.78.2.306
28. Kouvonen, A., Kivimäki, M., Elovainio, M., Pentti, J., Linna, A., Virtanen, M. and Vahtera, J. 2006. Effort/reward imbalance and sedentary lifestyle: An observational study in a large occupational cohort. Occupational and Environmental Medicine, 63: 422 - 427. doi:10.1136/oem.2005.020974
29. Kunina-Habenicht, O., Lohse-Bossenz, H., Kunter, M., Dicke, T., Förster, D., Gößling, J., & Terhart, E. (in press). Welche bildungswissenschaftlichen Inhalte sind wichtig in der Lehrerbildung? Ergebnisse einer Delphi-Studie. [Which general pedagogical content is necessary for teacher education? Results of a Delphi study]. Zeitschrift für Erziehungswissenschaft
30. Kunze, F., Boehm, S. A. and Bruch, H. 2011. Age diversity, age discrimination climate and performance consequences-a cross organizational study. Journal of Organizational Behavior, 32: 264 - 290. doi:10.1002/job.698
31. Lance, C. E., Butts, M. M. and Michels, L. C. 2006. The sources of four commonly reported cutoff criteria. Organizational Research Methods, 9: 202 - 220. doi:10.1177/1094428105284919
32. LeBreton, J. M. and Senter, J. L. 2008. Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11: 815 - 852. doi:10.1177/1094428106296642
33. WG(J) indexes. Journal of Applied Psychology, 84: 640 - 647. doi:10.1037/0021-9010.84.4.640
34. Lüdtke, O., Robitzsch, A., Trautwein, U. and Kunter, M. 2009. Assessing the impact of learning environments: How to use student ratings of classroom or school characteristics in multilevel modeling. Contemporary Educational Psychology, 34: 120 - 131. doi:10.1016/j.cedpsych.2008.12.001
35. Lüdtke, O., Trautwein, U., Kunter, M. and Baumert, J. 2006. Reliability and agreement of student ratings of the classroom environment: A reanalysis of TIMSS data. Learning Environments Research, 9: 215 - 230. doi:10.1007/s10984-006-9014-8
36. Maronna, R. A., Martin, R. D. and Yohai, V. J. 2006. Robust statistics: Theory and methods, Chichester: Wiley.
37. Meade, A. W. and Eby, L. T. 2007. Using indices of group agreement in multilevel construct validation. Organizational Research Methods, 10: 75 - 96. doi:10.1177/1094428106289390
38. R Development Core Team. 2011. R: A language and environment for statistical computing, Vienna: R Foundation for Statistical Computing.
39. Ramani, G. and Kumar, V. 2008. Interaction orientation and firm performance. Journal of Marketing, 72 (1): 27 - 45. doi:10.1509/jmkg.72.1.27
40. Roberson, Q. M., Sturman, M. C. and Simons, T. L. 2007. Does the measure of dispersion matter in multilevel research? A comparison of the relative performance of dispersion indexes. Organizational Research Methods, 10: 564 - 588. doi:10.1177/1094428106294746
41. Ruscher, J. B., Santuzzi, A. M. and Hammer, E. Y. 2003. Shared impression formation in the cognitively interdependent dyad. British Journal of Social Psychology, 42: 411 - 425. doi:10.1348/014466603322438233
42. Smith-Crowe, K. and Burke, M. J. 2003. Interpreting the statistical significance of observed AD interrater agreement values: Correction to Burke and Dunlap (2002). Organizational Research Methods, 6: 129 - 131. doi:10.1177/1094428102239428
43. Takeuchi, R., Chen, G. and Lepak, D. P. 2009. Through the looking glass of a social system: Cross-level effects of high-performance work systems on employees' attitudes. Personnel Psychology, 62: 1 - 29. doi:10.1111/j.1744-6570.2008.01127.x
44. Terhart, E., Schulze, F., Kunina-Habenicht, O., Dicke, T., Förster, D., Lohse-Bossenz, H., & Leutner, D. (2012). Bildungswissenschaftliches Wissen und der Erwerb professioneller Kompetenz in der Lehramtsausbildung - Eine Kurzdarstellung des BilWiss-Projekts [General pedagogical knowledge and the acquisition of professional competence in teacher education - a short presentation of the BilWiss-Project]. Lehrerbildung auf dem Prüfstand, 5, 96-106
45. Wagner, S. M., Rau, C. and Lindemann, E. 2010. Multiple informant methodology: A critical review and recommendations. Sociological Methods & Research, 38: 582 - 618. doi:10.1177/0049124110366231
46. Walker, A. G., Smither, J. W. and Waldman, D. A. 2008. A longitudinal examination of concomitant changes in team leadership and customer satisfaction. Personnel Psychology, 61: 547 - 577. doi:10.1111/j.1744-6570.2008.00122.x