Improving the Construct Validity of line Operational Simulation Ratings: Lessons Learned from the Assessment Center

International Journal of Aviation Psychology

Volumn 14, Issue 1, 2004, Pages 1-17

  Beaubien, J Matthew ^a   Baker, David P ^a   Salvaggio, Amy Nicole ^a  

^a AMERICAN INSTITUTES FOR RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 1042279205     PISSN: 10508414     EISSN: None     Source Type: Journal    
DOI: 10.1207/s15327108ijap1401_1     Document Type: Article

References (30)

1. Austin, J. T., Klimoski, R. J., & Hunt, S. T. (1996). Dilemmatics in public sector assessment: A framework for developing and evaluating selection systems. Human Performance, 93, 177-198.
2. Baker, D. P., Mulqueen, C., & Dismukes, R. K. (2001). Training raters to assess resource management skills. In E. Salas, C. Bowers, & E. Edens (Eds.), Improving teamwork in organizations: Applications of resource management training (pp. 131-145). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
3. Baker, D. P., Swezey, R. W., & Dismukes, R. K. (1998). A methodology for developing gold standards for rater training video tapes. Unpublished technical report, prepared under contract to the NASA-Ames Research Center (NASA Cooperative Agreement No. NCC 2-911), Moffet Field, CA.
4. Beaubien, J. M., Holt, R. W., & Hamman, W. R. (1999). An evaluation of the rating process used by instructor/evaluators in a line-operational simulation: Preliminary evidence of internal structure validity (Tech. Rep. No. 98-002). Fairfax, VA: George Mason University.
5. Bernardin, H. J., & Buckley, M. R. (1981). Strategies in rater training. Academy of Management Review, 6, 205-212.
6. Brannick, M. T., Prince, A., Prince, C., & Salas, D. (1995). The measurement of team process. Human Factors, 37, 641-651.
7. Burki-Cohen, J., Kendra, A. J., Kanki, B. G., & Lee, A. T. (2000). Realistic radio communication in pilot simulator training (FAA Tech. Rep. No. DOT-VNTSC-FAA-00-13). Cambridge, MA: Volpe National Transportation Systems Center.
8. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
9. Cascio, W. F. (1991). Applied psychology in personnel management (4th ed.). Englewood Cliffs, NJ: Prentice Hall.
10. Donahue, L. M., Truxillo, D. M., Cornwell, J. M., & Gerrity, M. J. (1997). Assessment center construct validity and behavioral checklists: Some additional findings. Journal of Social Behavior & Personality, 12(5), 85-108.
11. Federal Aviation Administration. (1990). Line operational simulations: Line oriented flight training, special purpose operational training, line operational evaluation (Advisory Circular No. 120-35B). Washington, DC: Author.
12. Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 41, 327-358.
13. Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., & Bentson, C. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology, 72, 493-511.
14. Howard, A. (1997). A reassessment of assessment centers: Challenges for the 21st century. Journal of Social Behavior and Personality, 12, 13-52.
15. Klimoski, R. J., & Brickner, M. (1987). Why do assessment centers work? The puzzle of assessment center validity. Personnel Psychology, 40, 243-260.
16. Lauber, J. K., & Foushee, H. C. (1981). Guidelines for line-oriented flight training (Vol. 1, National Aeronautics and Space Administration Conference Publication No. 2184). Moffett Field, CA: NASA-Ames Research Center.
17. Lievens, F., & Conway, J. M. (2001). Dimension and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies. Journal of Applied Psychology, 86, 1202-1222.
18. Lovler, B., Rose, M., & Wesley, S. (2002, April). Finding assessment center construct validity: Try behaviors instead of dimensions. Paper presented at the 17th annual meeting of the Society for Industrial and Organizational Psychology, Toronto, Ontario, Canada.
19. Moses, J. L., & Byham, W. C. (1977). Applying the assessment center method. New York: Pergamon.
20. Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage.
21. Orasanu, J. M., & Backer, P. (1996). Stress and military performance. In J. E. Driskell & E. Salas (Eds.). Stress and human performance (pp. 89-125). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
22. Prince, A., Brannick, M. T., Prince, C., & Salas, E. (1997). The measurement of team process behaviors in the cockpit: Lessons learned. In M. T. Brannick, E. Salas, & C. Prince (Eds.), Team performance assessment and measurement (pp. 289-310). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
23. Prince, C., Oser, R., Salas, E., & Woodruff, W. (1993). Increasing hits and reducing misses in CRM/LOS scenarios: Guidelines for simulator scenario development. International Journal of Aviation Psychology, 3, 69-82.
24. Reilly, R. R., Henry, S., & Smither, J. W. (1990). An examination of the effects of using behavior checklists on the construct validity of assessment center dimensions. Personnel Psychology, 43, 71-84.
25. Seamster, T. L., Boehm-Davis, D. A., Holt, R. W., & Schultz, K. (1998). Developing advanced crew resource management (ACRM) training: A training manual. Washington, DC: Federal Aviation Administration.
26. Smith-Jentsch, K. A., Johnson, J. H., & Payne, S. C. (1998). Measuring team-related expertise in complex environments. In J. A. Cannon-Bowers & E. Salas (Eds.), Making decisions under stress: Implications for individual and team training (pp. 61-87). Washington, DC: American Psychological Association.
27. Thornton, G. C., & Zorich, S. (1980). Training to improve observer accuracy. Journal of Applied Psychology, 65, 351-354.
28. Trumpower, D. L., Johnson, P. J., & Goldsmith, T. E. (1999). Structural analysis of line-oriented evaluation data. In R. S. Jensen (Ed.), Proceedings of the 10th International Symposium on Aviation Psychology (pp. 1220-1223). Columbus: The Ohio State University Press.
29. Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A meta-analytic review. Journal of Occupational and Organizational Psychology, 67, 189-205.
30. Wood, R. E. (1986). Task complexity: Definition of a construct. Organizational Behavior and Human Decision Processes, 37, 60-82.