On Average Deviation Indices for Estimating Interrater Agreement

Organizational Research Methods

Volumn 2, Issue 1, 1999, Pages 49-68

  Burke, Michael J ^a   Finkelstein, Lisa M ^b   Dusig, Michelle S ^b  

^a Tulane University   (United States)

^b NONE

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0002249088     PISSN: 10944281     EISSN: None     Source Type: Journal    
DOI: 10.1177/109442819921004     Document Type: Article

References (30)

1. Berk, R. A. (1979). Generalizability of behavioral observations: A clarification of interobserver agreement and interobserver reliability. American Journal of Mental Deficiency, 83, 460-472.
2. Bullock, R. J., & Svyantek, D. J. (1985). Analyzing meta-analysis: Potential problems, an unsuccessful replication, and evaluation criteria. Journal of Applied Psychology, 70, 108-115.
3. Burke, M. J., Borucki, C. C., & Hurley, A. E. (1992). Reconceptualizing psychological climate in a retail service environment: A multiple stakeholder perspective. Journal of Applied Psychology, 77, 717-729.
4. Burke, M. J., Rupinski, M. T., Dunlap, W. P., & Davison, H. K. (1996). Do situational variables act as substantive causes of relationships between individual difference variables? Two large-scale tests of "common cause" models. Personnel Psychology, 49, 573-598.
5. Campion, M. A., Medsker, G. J., & Higgs, A. C. (1993). Relations between work group characteristics and effectiveness: Implications for designing effective work groups. Personnel Psychology, 46, 823-850.
6. Dutton, J. E., & Webster, J. (1988). Patterns of interest around issues: The role of uncertainty and feasibility. Academy of Management Journal, 31, 663-675.
7. Finn, R. H. (1970). A note on estimating the reliability of categorical data. Educational and Psychological Measurement, 30, 71-76.
8. George, J. M. (1990). Personality, affect, and behavior in groups. Journal of Applied Psychology, 75, 107-116.
9. George, J. M., & Bettenhausen, K. (1990). Understanding prosocial behavior, sales performance, and turnover: A group-level analysis in a service context. Journal of Applied Psychology, 75, 698-709.
10. George, J. M., & James, L. R. (1993). Personality, affect, and behavior in groups revisited: Comment on aggregation, levels of analysis, and a recent application of within and between analysis. Journal of Applied Psychology, 78, 798-804.
11. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20.
12. Greller, M., & Parsons, C. K. (1988). Psychosomatic complaints scale of stress: Measure development and psychometric properties. Educational and Psychological Measurement, 48, 1051-1065.
13. Hater, J. J., & Bass, B. M. (1988). Superiors' evaluations and subordinates' perceptions of transformational and transactional leadership. Journal of Applied Psychology, 73, 695-702.
14. Huber, P. J. (1980). Robust statistics. New York: John Wiley.
15. James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
16. wg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306-309.
17. Jones, A. P., Johnson, L. A., Butler, M. C., & Main, D. S. (1983). Apples and oranges: An empirical comparison of commonly used indices of interrater agreement. Academy of Management Journal, 26, 507-519.
18. Kozlowski, S.W.J., & Hattrup, K. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161-167.
19. Kozlowski, S.W.J., & Hults, B. M. (1987). An exploration of climates for technical updating and performance. Personnel Psychology, 40, 539-563.
20. Lawlis, G. F., & Lu, E. (1972). Judgment of counseling process: Reliability, agreement, and error. Psychological Bulletin, 78, 17-20.
21. Lindell, M. K., & Brandt, C. J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21, 271-278.
22. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
23. Parsons, R. (1978). Statistical analysis: A decision-making approach. New York: Harper & Row.
24. Russell, C. J., Pinto, J. K., & Bobko, P. (1991). Appropriate moderated regression and inappropriate research strategy: A demonstration of information loss due to scale coarseness. Applied Psychological Measurement, 15, 257-266.
25. Schmidt, F. L., & Hunter, J. E. (1989). Interrater reliability coefficients cannot be computed when only one stimulus is rated. Journal of Applied Psychology, 74, 368-370.
26. Smith, P. C. (1976). Behaviors, results, and organizational effectiveness: The problem of criteria. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology (pp. 745-775). Chicago: Rand McNally.
27. Tesluk, P. E., Farr, J. L., Mathieu, J. E., & Vance, R. J. (1995). Generalization of employee involvement training: Individual and situational effects. Personnel Psychology, 48, 607-632.
28. Tinsley, H.E.A., & Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology, 22, 358-376.
29. Tukey, J. W. (1960). A survey of sampling from contaminated distributions. In I. Olkin, S. G. Ghurye, W. Hoeffding, W. G. Madow, & H. B. Mann (Eds.), Contributions to probability and statistics: Essays in honor of Harold Hotelling (pp. 448-485). Stanford, CA: Stanford University Press.
30. Watson, D. (1988). The vicissitudes of mood measurement: Effects of varying descriptors, time frames, and response formats on measures of positive and negative affect. Journal of Personality and Social Psychology, 55, 128-141.