Applying Item Response Theory to Multisource Performance Ratings: What Are the Consequences of Violating the Independent Observations Assumption?

Organizational Research Methods

Volumn 6, Issue 1, 2003, Pages 44-60

  Craig, S Bartholomew ^a   Kaiser, Robert B ^b  

^a NORTH CAROLINA STATE UNIVERSITY   (United States)

^b Kaplan DeVries Inc   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0037232302     PISSN: 10944281     EISSN: None     Source Type: Journal    
DOI: 10.1177/1094428102239425     Document Type: Article

References (40)

1. Atwater, L., & Yammarino, F. J. (1997). Antecedents and consequences of self-other rating agreement: A review and model. In G. Ferris (Ed.), Research in personnel and human resources management. Greenwich, CT: JAI.
2. Baker, F. (1995). EQUATE 2.1: Computer program for equating two metrics in item response theory. Madison: University of Wisconsin, Laboratory of Experimental Design.
3. Bass, B. M. (1990). Bass and Stogdill's handbook of leadership: Theory, research, and managerial applications (3rd ed.). New York: Free Press.
4. Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: An application of an EM algorithm. Psychometrika, 46, 443-459.
5. Campion, M. A., Pursell, E. D., & Brown, B. K. (1988). Structured interviewing: Raising the psychometric properties of the employment interview. Personnel Psychology, 41, 25-42.
6. Cattell, R. B. (1966). The meaning and strategic use of factor analysis. In R. B. Cattell (Ed.), Handbook of multivariate experimental psychology (pp. 174-243). Chicago: Rand McNally.
7. Church, A. H. (1997). Managerial self-awareness in high-performing individuals in organizations. Journal of Applied Psychology, 82, 281-292.
8. Conway, J. M., & Huffcutt, A. I. (1997). Psychometric properties of mulitsource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331-360.
9. Conway, J. M., Jako, R. A., & Goodman, D. F. (1995). A meta-analysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80, 565-579.
10. Facteau, J., & Craig, S. B. (2001). Are performance appraisal ratings obtained from different rating sources comparable? Journal of Applied Psychology, 86, 215-227.
11. Fiedler, F. E. (1967). A theory of leadership effectiveness. New York: McGraw-Hill.
12. Fleenor, J. W., McCauley, C. D., & Brutus, S. (1996). Self-other rating agreement and leader effectiveness. Leadership Quarterly, 7, 487-506.
13. Fleishman, E. A. (1973). Twenty years of consideration and structure. In E. A. Fleishman & J. G. Hunt (Eds.), Current developments in the study of leadership. Carbondale: Southern Illinois University Press.
14. Greguras, G. J., & Robie, C. (1998). A new look at within-source interrater reliability of 360-degree feedback ratings. Journal of Applied Psychology, 83, 960-968.
15. Harris, M. M., Becker, A. S., & Smith, D. E. (1993). Does the assessment center scoring method affect the cross-situational consistency of ratings? Journal of Applied Psychology, 78, 675-678.
16. Hazucha, J. F., Hezlett, S. A., & Schneider, R. J. (1993). The impact of 360-degree feedback on management skills development. Human Resource Management, 32, 325-351.
17. Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141-151.
18. Lombardo, M. M., & McCauley, C. D. (1994). Benchmarks: A manual and trainer's guide. Greensboro, NC: Center for Creative Leadership.
19. London, M., & Smither, J. W. (1995). Can multi-source feedback change perceptions of goal accomplishment, self-evaluations, and performance-related outcomes? Theory-based applications and directions for research. Personnel Psychology, 48, 803-839.
20. Maurer, T. J., Raju, N. S., & Collins, W. C. (1998). Peer and subordinate performance appraisal measurement equivalence. Journal of Applied Psychology, 83, 693-702.
21. McCauley, C. D., & Lombardo, M. M. (1990). Benchmarks: An instrument for diagnosing managerial strengths and weaknesses. In K. E. Clark & M. B. Clark (Eds.), Measures of leadership. West Orange, NJ: Leadership Library of America.
22. McCauley, C. D., Lombardo, M. M., & Usher, C. J. (1989). Diagnosing management development needs: An instrument based on how managers develop. Journal of Management, 15, 389-403.
23. McDaniel, M. A., Whetzel, D. L., Schmidt, F. L. & Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79, 599-616.
24. Mislevy, R. J., & Bock, R. D. (1990). Bilog 3 user manual. Mooresville, IN: Scientific Software.
25. Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., & Hezlett, S. A. (1998). Trait, rater and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557-576.
26. Raju, N. S. (1999a). DFIT4P: A computer program for analyzing differential item and test functioning. Chicago: Illinois Institute of Technology.
27. Raju, N. S. (Chair). (1999b, April). IRT-based evaluation of 360° feedback assessments: The BENCHMARKS story. Symposium presented at the annual conference of the Society for Industrial-Organizational Psychology, Atlanta, GA.
28. Raju, N. S., Leslie, J. B., McDonald-Mann, D., & Craig, S. B. (2001). Measurement equivalence of a 360-degree feedback assessment. Unpublished manuscript.
29. Raju, N. S., van der Linden, W., & Fleer, P. (1995). An IRT-based internal measure of test bias with applications for differential item functioning. Applied Psychological Measurement, 19, 353-368.
30. Reckase, M. D. (1979). Unifactor latent trait models applied to multi-factor tests: Results and implications. Journal of Educational Statistics, 4, 207-230.
31. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores (Psychometrika Monograph No. 17).
32. Schriesheim, C. A., House, R. J., & Kerr, S. (1976). Leader initiating structure: A reconciliation of discrepant research results and some empirical tests. Organizational Behavior and Human Performance, 15, 297-321.
33. Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
34. Stocking, M. L., & Lord, F. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7, 201-210.
35. Supovitz, J. A., MacGowan, A., Slattery, J. (1997). Assessing agreement: An examination of the interrater reliability of portfolio assessment in Rochester, New York. Educational Assessment, 4, 237-259.
36. Thissen, D. (1995). MULTILOG 6.3: A computer program for multiple, categorical item analysis and test scoring using item response theory. Chicago: Scientific Software.
37. Thornton, G. C. (1992). Assessment centers in human resource management. Reading, MA: Addison-Wesley.
38. Tornow, W. (1993). Perceptions or reality: Is multi-perspective measurement a means or an end? Human Resource Management, 32, 221-230.
39. Tornow, W. W., & London, M. (1998). Maximizing the value of 360-degree feedback. San Francisco: Jossey-Bass.
40. Zedeck, S. (1995). [Review of benchmarks]. In J. Conoley & J. Impara (Eds.), The twelfth mental measurements yearbook (Vol. 1, pp. 128-129). Lincoln, NE: Buros Institute of Mental Measurements.