An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales

Journal of Applied Psychology

Volumn 86, Issue 5, 2001, Pages 965-973

  Borman, Walter C ^b   Buck, Daren E ^b   Motowidlo, Stephan J ^c   Hanson, Mary Ann ^b   Stark, Stephen ^a   Drasgow, Fritz ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b Pers Decisions Research Institutes   (United States)

^c University of Illinois ^*

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; COMPUTER SIMULATION; FEMALE; HUMAN; INFORMATION PROCESSING; MALE; PERSONNEL MANAGEMENT; REPRODUCIBILITY; SENSITIVITY AND SPECIFICITY; TASK PERFORMANCE; VALIDATION STUDY; VIDEORECORDING;

ADULT; AUTOMATIC DATA PROCESSING; COMPUTER SIMULATION; EMPLOYEE PERFORMANCE APPRAISAL; FEMALE; HUMANS; MALE; PERSONNEL SELECTION; REPRODUCIBILITY OF RESULTS; SENSITIVITY AND SPECIFICITY; TASK PERFORMANCE AND ANALYSIS; VIDEO RECORDING;

EID: 85047681029     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/0021-9010.86.5.965     Document Type: Article

References (35)

1. Bendig, A. W. (1954). Reliability and number of rating scale categories. Journal of Applied Psychology, 38, 38-40.
2. Bernardin, H. J. (1977). Behavioral expectation scales versus summated scales: A fairer comparison. Journal of Applied Psychology, 62, 422-427.
3. Blanz, R., & Ghiselli, E. E. (1972). The mixed standard scale: A new rating system. Personnel Psychology, 25, 185-200.
4. Borman, W. C. (1977). Consistency of rating accuracy and rater errors in the judgment of human performance. Organizational Behavior and Human Performance, 20, 238-252.
5. Borman, W. C. (1979). Format and training effects on rating accuracy and rater errors. Journal of Applied Psychology, 64, 410-421.
6. Borman, W. C. (1991). Job behavior, performance, and effectiveness. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (pp. 271-326). Palo Alto, CA: Consulting Psychologists Press.
7. Borman, W. C., Hanson, M. A., Kubisiak, U. C., & Buck, D. E. (2000). Computerized adaptive rating scales (CARS): Development and evaluation of the concept (Institute Rep. No. 350). Tampa, FL: Personnel Decisions Research Institutes.
8. Borman, W. C., & Motowidlo, S. J. (1993). Expanding the criterion domain to include elements of contextual performance. In N. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations (pp. 71-98). San Francisco: Jossey-Bass.
9. Borman, W. C., Motowidlo, S. J., Rose, S. R., & Hanser, L. M. (1983). Development of a model of soldier effectiveness (Institute Rep. No. 95). Minneapolis, MN: Personnel Decisions Research Institutes.
10. Brief, A. P., & Motowidlo, S. J. (1986). Prosocial organizational behaviors. Academy of Management Review, 11, 710-725.
11. Campbell, J. P., Dunnette, M. D., Arvey, R. D., & Hellervik, L. W. (1973). The development and evaluation of behaviorally based rating scales. Journal of Applied Psychology, 57, 15-22.
12. Campbell, J. P., McCloy, R. A., Oppler, S. H., & Sager, C. E. (1993). A theory of job performance. In N. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations (pp. 35-70). San Francisco: Jossey-Bass.
13. Coleman, V. I., & Borman, W. C. (2000). Investigating the underlying structure of the citizenship performance domain. Human Resources Management Review, 10, 25-44.
14. Coombs, C. H. (1950). Psychological scaling without a unit of measurement. Psychological Review, 57, 145-158.
15. Cronbach, L. J. (1955). Processes affecting scores on understanding of others and assumed "similarity." Psychological Bulletin, 52, 177-193.
16. Dickinson, T. L., & Zellinger, P. M. (1980). A comparison of the behaviorally anchored rating and mixed standard scale formats. Journal of Applied Psychology, 65, 147-154.
17. Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51, 327-355.
18. Guion, R. M., & Gibson, W. M. (1988). Personnel selection and placement. In M. R. Rosenzweig & L. W. Porter (Eds.), Annual review of psychology (Vol. 39, pp. 349-374). Palo Alto, CA: Annual Reviews.
19. Kane, J. S. (1986). Performance distribution assessment. In R. Berk (Ed.), Performance assessment: Methods and applications (pp. 237-273). Baltimore: Johns Hopkins University Press.
20. Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87, 72-107.
21. Landy, F. J., & Farr, J. L. (1985). The measurement of work performance: Methods, theory and applicctions. New York: Academic Press.
22. Latham, G. P., & Wexley, K. V. (1981). Increasing productivity through performance appraisal. Reading, MA: Addison-Wesley.
23. Madden, J. M., & Bourdon, R. D. (1964). Effects of variations in rating scale format on judgment. Journal of Applied Psychology, 48, 147-151.
24. Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage.
25. Organ, D. W. (1988). Organizational citizenship behavior: The good soldier syndrome. Lexington, MA: Lexington Books.
26. Rosenthal, R. (1987). Judgment studies: Design, analysis, and meta-analysis. New York: Cambridge University Press.
27. Schwab, D. P., Heneman, H. G., III, & DeCotiis, T. (1975). Behaviorally anchored rating scales: A review of the literature. Personnel Psychology, 28, 549-562.
28. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
29. Smith, P. C., & Kendall, L. M. (1963). Retranslation of expectations: An approach to the constructior of unambiguous anchors for rating scales. Journal of Applied Psychology, 47, 149-155.
30. Stark, S., & Drasgow, F. (1998, April). Application of an IRT ideal point model to computer adaptive assessment of job performance. Paper presented at the annual conference of the Society for Industrial and Organizational Psychology, Dallas, TX.
31. Taylor, E. K., Parker, J. W., & Ford, G. L. (1959). Rating scale content: IV. Predictability of structured and unstructured scales. Personal Psychology, 12, 247-266.
32. Taylor, E. K., & Wherry, R. J. (1951). A study of leniency in two rating systems. Personnel Psychology, 4, 245-252.
33. Thurstone, L. L. (1927). Psyohophysical analysis. American Journal of Psychology, 38, 368-389.
34. Viswesvaran, C., Schmidt, F., & Ones, D. S. (2001). Intercorrelations of rated job performance dimensions in the absence of halo error: Implications for halo research and theories of job performance. Manuscript submitted for publication.
35. Zinnes, J. L., & Griggs, R. A. (1974). Probabilistic, multidimensional unfolding analysis. Psychometrika, 39, 327-350.