The Reliability of Instructor Evaluations of Crew Performance: Good News and Not So Good News

International Journal of Aviation Psychology

Volumn 12, Issue 3, 2002, Pages 241-261

  Brannick, Michael T ^a   Prince, Carolyn ^b   Salas, Eduardo ^b  

^a UNIVERSITY OF SOUTH FLORIDA   (United States)

^b UNIVERSITY OF CENTRAL FLORIDA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0041877400     PISSN: 10508414     EISSN: None     Source Type: Journal    
DOI: 10.1207/S15327108IJAP1203_4     Document Type: Article

References (39)

1. Arvey, R. D., & Murphy, K. R. (1998). Performance evaluation in work settings. Annual Review of Psychology, 49, 141-168.
2. Baker, D. P., & Salas, E. (1992). Principles for measuring teamwork skills. Human Factors, 34, 469-475.
3. Bernardin, H. J., & Beatty, R. W. (1984). Performance appraisal: Assessing human behavior at work. Boston: Kent.
4. Bernardin, H. J., & Pence, E. C. (1980). Effects of rater training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology, 65, 60-66.
5. Brannick, M. T., Prince, C., Prince, A., & Salas, E. (1995). The measurement of team process. Human Factors, 37, 641-651.
6. Brannick, M. T., Roach, R. M., & Salas, E. (1993). Understanding team performance: A multimethod study. Human Performance, 6, 287-308.
7. Brannick, M. T., Salas, E., & Prince, C. (Eds.). (1997). Team performance assessment and measurement. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
8. Bretz, R. D., Jr., Milkovich, G. T., & Read, W. (1992). The current state of performance appraisal research and practice: Concerns, directions, and implications. Journal of Management, 18, 321-352.
9. Cronbach, L. J. (1955). Processes affecting scores on "understanding of others" and "assumed similarity." Psychological Bulletin, 52, 177-193.
10. Cronbach, L. J. (1990). Essentials of psychological testing (5th ed.). New York: HarperCollins.
11. Dwyer, D. J., Fowlkes, J. E., Oser, R. L., Salas, E., & Lane, N. E. (1997). Team performance measurement in distributed environments: The TARGETS methodology. In M. T. Brannick, E. Salas, & C. Prince (Eds.), Team performance assessment and measurement: Theory, methods, and applications (pp. 137-153). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
12. Dyer, J. (1984). Team research and training: A state-of-the-art review. In F. A. Muckler (Ed.), Human factors review: 1984. Santa Monica, CA: Human Factors Society.
13. Fowlkes, J., Dwyer, D. J., Oser, R., & Salas, E. (1998). Event-based approach to training (EBAT). International Journal of Aviation Psychology, 8, 209-221.
14. George Mason University. (1996). Developing and evaluating CRM procedures for a regional air carrier: Phase I report. Washington, DC: Federal Aviation Administration.
15. Goldstein, W. M., & Hogarth, R. M. (Eds.). (1997). Research on judgment and decision making: Currents, connections and controversies. New York: Cambridge University Press.
16. Gregorich, S. E., & Wilhelm, J. A. (1993). Crew resource management training assessment. In E. L. Wiener, B. G. Kanki, & R. L. Helmreich (Eds.), Cockpit resource management (pp. 173-198). San Diego, CA: Academic.
17. Guion, R. M. (1998). Assessment, measurement and prediction for personnel decisions. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
18. Hammond, K. R. (1996). Human judgment and social policy: Irreducible uncertainty, inevitable error, unavoidable injustice. New York: Oxford University Press.
19. Hammond, K. R., McClelland, G. H., & Mumpower, J. (1980). Human judgment and decision making: Theories, methods, and procedures. New York: Praeger.
20. Hedge, J. W., & Kavanagh, M. J. (1988). Improving the accuracy of performance evaluations: Comparison of three methods of performance appraiser training. Journal of Applied Psychology, 73, 68-73.
21. Helmreich, R. L., Chidester, R. R., Foushee, H. C., Gregorich, S. E., & Wilhelm, J. A. (1990). How effective is cockpit resource management training? Issues in evaluating the impact of programs to enhance crew coordination. Flight Safety Digest, 9, 1-17.
22. Helmreich, R. L., & Foushee, H. C. (1993). Why crew resource management? Empirical and theoretical bases of human factors training in aviation. In E. L. Wiener, B. G Kanki, & R. L. Helmreich (Eds.), Cockpit resource management (pp. 3-45). San Diego, CA: Academic.
23. Helmreich, R. L., Merritt, A. C., & Wilhelm, J. A. (1999). The evolution of crew resource management training in commercial aviation. International Journal of Aviation Psychology, 9, 19-32.
24. Kanki, B. G., Lozito, S. C., & Foushee, H. C. (1987). Communication indexes of crew coordination. Aviation, Space, and Environmental Medicine, 60, 56-60.
25. Koonee, J. M. (1984). A brief history of aviation psychology. Human Factors, 26, 499-508.
26. Landy, F. J., & Farr, J. L. (1983). The measurement of workper formance. New York: Academic.
27. Lauber, J. K. (1987). Cockpit resource management: Background studies and rationale. In H. W. Orlay & H. C. Foushee (Eds.), Cockpit resource management training (Tech. Rep. No. NASA CP 2455). Moffett Field, CA: NASA Ames Research Center.
28. Maurino, D. E. (1999). Crew resource management: A time for reflection. In D. J. Garland, J. A. Wise, & V. D. Hokpin (Eds.), Handbook of aviation human factors (pp. 215-234). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
29. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
30. Prince, C., & Salas, E. (1993). Training and research for teamwork in the military aircrew. In E. L. Wiener, B. G. Kanki, & R. L. Helmreich (Eds.), Cockpit resource management (pp. 337-366). San Diego, CA: Academic.
31. Prince, C., & Salas, E. (1999). Team processes and their training in aviation. In D. J. Garland, J. A. Wise, & V. D. Hokpin (Eds.), Handbook of aviation human factors (pp. 193-213). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
32. Salas, E., Fowlkes, J. E., Stout, R. J., Milanovich, D. M., & Prince, C. (1999). Does CRM training improve teamwork skills in the cockpit?: Two evaluation studies. Human Factors, 41, 326-343.
33. Stamoulis, D. T., & Hauenstein, N. M. A. (1993). Rater training and rating accuracy: Training for dimensional accuracy versus training for ratee differentiation. Journal of Applied Psychology, 78, 994-1003.
34. Sulsky, L. M., & Balzer, W. K. (1988). Meaning and measurement of performance rating accuracy: Journal of Applied Psychology, 73, 497-506.
35. Sulsky, L. M., & Day, D. V. (1994). Effects of frame-of-reference training on rater accuracy under alternative time delays. Journal of Applied Psychology, 79, 535-543.
36. Wigdor, A. K., & Green, B. F., Jr. (Eds.). (1991a). Performance assessment for the workplace (Vol. 1). Washington, DC: National Academy Press.
37. Wigdor, A. K., & Green, B. F., Jr. (Eds.). (1991b). Performance assessment for the workplace (Vol. 2). Washington, DC: National Academy Press.
38. Williams, D. M., Holt, R. W., & Boehm-Davis, D. A. (1997). Training for inter-rater reliability: Baselines and benchmarks. In R. S. Jensen & L. Rakovan (Eds.), Proceedings of the Ninth International Symposium on Aviation Psychology (pp. 514-519). Columbus: The Ohio State University.
39. Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189-205.