Improving Rater Calibration in Aviation: A Case Study

International Journal of Aviation Psychology

Volumn 12, Issue 3, 2002, Pages 305-330

  Holt, Robert W ^a   Hansberger, Jeffrey T ^a   Boehm Davis, Deborah A ^a  

^a GEORGE MASON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0042378642     PISSN: 10508414     EISSN: None     Source Type: Journal    
DOI: 10.1207/S15327108IJAP1203_7     Document Type: Article

References (25)

1. AmericanTransAir. (1994, November). Line operational simulations: LOFT scenario design, conduct and validation. LOFT Design Focus Group, AQP Subcommittee Report, Washington, DC.
2. Baker, D. P., & Dismukes, R. K. (in press). Training raters to assess crew performance: Theory practice and future directions. International Journal of Aviation Psychology.
3. Baker, D. P., & Mulqueen, C. (1999). Pilot instructor/evaluator rater training: Guidelines for development. In Proceedings of the Tenth International Symposium on Aviation Psychology (pp. 291-300). Columbus: Ohio State University.
4. Bernardin, H. J., & Buckley, M. R. (1981). Strategies in rater training. Academy of Management Review, 6, 205-212.
5. Birnback, R. A., & Longridge, T. M. (1993). The regulatory perspective. In E. L. Wiener, B. G. Kanki, & R. L. Helmreich (Eds.), Cockpit resource management (pp. 263-281). New York: Academic.
6. Boehm-Davis, D. A., Holt, R. W., & Seamster, T. (2001). Airline resource management programs, In E. Salas, C. A. Bowers, & E. Edens (Eds.), Improving teamwork in organizations (pp. 191-215). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
7. Borman, W. C. (1975). Effects of instruction to avoid halo error on reliability and validity of performance evaluation ratings. Journal of Applied Psychology, 60, 556-560.
8. Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
9. George Mason University. (1996). Developing and evaluating CRM procedures for a regional air carrier phase I report. Federal Aviation Administration, Office of the Chief Scientific and Technical Advisor for Human Factors, Washington, DC.
10. Greenwood, D. M., Holt, R. W., & Boehm-Davis, D. A. (2000). Training instructor pilots to evaluate air crew performance in a workshop setting (Tech. Rep). Fairfax, VA: George Mason University.
11. Hamman, W. R., Seamster, T. L., Smith, K. M., & Lofaro, R. J. (1991). The future of LOFT scenario design and validation. Proceedings of the 6th International Symposium on Aviation Psychology, 589-594.
12. Holt, R. W., Meiman, E., & Seamster, T. L. (1996). Evaluation of aircraft pilot team performance. In Proceeding of the Human Factors and Ergonomics Society 40th Annual Meeting, 40, 44-48.
13. Jako, R. A., & Murphy, K. R. (1991). Distributional ratings, judgment decomposition, and their impact on interrater agreement and rating accuracy. Journal of Applied Psychology, 75, 500-505.
14. James, L. R., Demaree, R. G., & Wolfe, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
15. wg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306-309.
16. Law, J. R., & Sherman, P. J. (1995). Do raters agree? Assessing inter-rater agreement in the evaluation of air crew resource management skills. In Proceedings of the Eighth International Symposium on Aviation Psychology. Columbus: Ohio State University.
17. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlations coefficients: Correction. Psychological Methods, 1, 390.
18. Nunnally, J. C. (1967). Psychometric theory. New York: McGraw-Hill.
19. Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach (Student ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
20. Prince, C., Oser, R., Salas, E., & Woodruff, W. (1993). Increasing hits and reducing misses in CRM/LOS scenarios: Guidelines for simulator scenario development. International Journal of Aviation Psychology, 3, 69-82.
21. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.
22. Smith, P. C., & Kendall, L. M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47, 149-155.
23. Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1997). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.
24. Williams, D. M., Holt, R. W., & Boehm-Davis, D. A. (1997). Training for inter-rater reliability: Baselines and benchmarks. In R. S. Jensen & L. Rakovan (Eds.), Proceedings of the Ninth International Symposium on Aviation Psychology (pp. 514-519). Columbus: Ohio State University.
25. Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189-205.