Enhancing the design, delivery, and evaluation of scenario-based training: Can situational judgment tests contribute?

Situational Judgment Tests: Theory, Measurement, and Application

Volumn , Issue , 2013, Pages 301-318

  Fritzsche, Barbara A ^a   Stagl, Kevin C ^a   Salas, Eduardo ^a   Burke, C Shawn ^a  

^a UNIVERSITY OF CENTRAL FLORIDA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33750991169     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203774878     Document Type: Chapter

References (72)

1. Anderson, J. P. (1985). Cognitive psychology and its implications (2nd). New York: Freeman.
2. Bassi, L. J., & Van Buren, M. E. (1998). The 1998 ASTD state of the industry report [on-line]. Available http://www. Astd.org
3. Bowers, C., Jentsch, F., Baker, D., Prince, C., & Salas, E. (1997). Rapidly reconfigurable event-set based line operational evaluation scenarios. Proceedings of the Human Factors and Ergonomics Society 41st Annual Meeting (pp. 912–915). Santa Monica, CA: Human Factors and Ergonomics Society.
4. Breuer, K., & Streufert, S. (1995). Strategic management simulations: The German case. In M. Mulder & W. J. Nijhof, Corporate training for effective performance (pp. 195–208). Boston, MA: Kluwer
5. Brown, K., & Gerhardt, M. (2002). Formative evaluation: An integrated practice model and case study. Personnel Psychology, 55, 951–983.
6. Buch, G., & Diehl, A. (1984). An investigation of the effectiveness of pilot judgment training. Aviation Psychology, 2, 557–564.
7. Campbell, J. P. (1990). The role of theory in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough, Handbook of industrial and organizational psychology (Vol. 1, pp. 39–74). Palo Alto, CA: Consulting Psychologists Press.
8. Campbell, J. P., & Kuncel, N. R. (2002). Individual and team training. In N. Anderson, D. S. Ones, H. K. Sinangil, & C. Viswesvaran, Handbook of industrial, work and organizational psychology (pp. 272–312). London: Sage.
9. Campbell, J. P., McCloy, R. A., Oppler, S. H., & Sager, C. E. (1993). A theory of performance. In N. Schmitt & W. C. Borman, Personnel selection in organizations (pp. 35–70). San Francisco: Jossey-Bass.
10. Cannon-Bowers, J. A., Bums, J. J., Salas, E., & Pruitt, J. S. (1998). Advanced technology in scenario-based training. In J. A. Cannon-Bowers & E. Salas, Making decisions under stress: Implications for individual and team training (pp. 365–374). Washington, DC: American Psychological Association.
11. Cascio W. F. (1998). Applied psychology in human resource management (5th) Upper Saddle River, NJ: Prentice-Hall.
12. Chan, D., & Schmitt, N. (1997). Video based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82, 143–159.
13. Chan, D., & Schmitt, N. (2002). Video-based versus paper-and-pencil method of assessment in situational judgment tests. Journal of Applied Psychology, 82, 143–159.
14. Cianciolo, A. T., Antonakis, J., & Sternberg, R. J. (2004). Practical intelligence and leadership: Using experience as a mentor. In D. Day, S. J. Zaccaro, & S. M. Halpin, Leader development for transforming organizations (pp. 211–236). Mahwah, NJ: Lawrence Erlbaum Associates.
15. Clevenger, J., Pereira, G. M., Wiechmann, D., Schmitt, N., & Harvey, V. S. (2001). Incremental validity of situational judgment tests. Journal of Applied Psychology, 86, 410–417.
16. Cronbach, L. J. (1957). The two disciplines of scientific study. American Psychologists, 12, 671–684.
17. Day, D., Zaccaro, S. J., & Halpin, S. M. (2004), Leader development for transforming organizations. Mahwah, NJ: Lawrence Erlbaum Associates.
18. Day, D. V. (2000). Leadership development: A review in context. Leadership Quarterly, 11, 581–613.
19. Dwyer, D. J., Oser, R. L., & Fowlkes, J. E. (1995). A case study of distributed training and training performance. Proceedings of the Human Factors and Ergonomics Society 39th Annual Meeting (pp. 1316–1320). Santa Monica, CA: Human Factors and Ergonomics Society.
20. Edmondson, A., Bohmer, R., & Pisano, G. (2001). Disrupted routines: Team learning and technology implementation in hospitals. Administrative Sciences Quarterly, 46, 685–716.
21. Ellis, A. P. J., Ilgen, D. R., Hollenbeck, J. R., & Porter, C. O. L. H. (2001). Capacity collaboration, and commonality: A framework for understanding team learning. Paper presented at the annual conference of the Academy of Management, Washington, DC.
22. Endsley, M. R. (1988). Design and evaluation for situation awareness enhancement. In Proceedings of the Human Factors Society 32nd Annual Meeting (pp. 97–101). Santa Monica, CA: Human Factors Society.
23. Endsley, M. R. (1995). Toward a theory of situation awareness. Human Factors, 1, 32–64.
24. Fowlkes, J. E., & Burke, C. S. (2004). Event-Based Approach to Training (EBAT). In N. Stanton, Hendrick, S. Konz, K. Parsons, & E. Salas, Handbook of human factors and ergonomics methods (pp. 47–1-47–5). London: Taylor & Francis.
25. Fowlkes, J. E., Lane, N. E., Salas, E., Franz, T., & Oser, R. (1994). Improving the measurement of team performance: The TARGETS methodology. Military Psychology, 6, 47–61.
26. Funke, J. (1992). The validity of a personnel assessment simulation on complex problem solving. Kongre der Deutschen Gesellschaft fur Psychologie, Trier.
27. Funke, J. (1998). Computer-based testing and training with scenarios from complex problemsolving research: Advantages and disadvantages. International Journal of Selection and Assessment, 2, 90–96.
28. Goldstein, I. L. (1991). Training in work organizations. In M. D. Dunnette & L. M. Hough, Handbook of industrial and organizational psychology (Vol. 2, pp. 507–619). Palo Alto, CA: Consulting Psychologist Press.
29. Graf, J. (1992). Scenarios-simulated realities for tomorrows' manager. Speyer: Gabal-Verlag.
30. Hanson, M. A., & Borman, W. C. (1995, April). Construct validation of a measure of supervisory job knowledge. Poster presented at the 10th annual conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
31. Hanson, M. A., Horgen, K. E., & Borman, W. C. (1996, November). Situational judgment: An alternative approach to selection test development. Paper presented at the 38th annual conference of the International Military Testing Association.
32. Hasselmann, D., & Straub, B. (1995, November). Challenge complexity, Element 2: Textile factory. Hamburg, Germany: Windmuhle.
33. Hedge, J. W., Borman, W. C., & Hanson, M. A. (1996). Videotaped crew resource management scenarios for selection and training applications. Paper presented at the 38th annual conference of the International Military Testing Association.
34. Hunter, D. R. (2003). Measuring general aviation pilot judgment using a situational judgment technique. The International Journal of Aviation Psychology, 13, 373–386.
35. Hunter, D. R., Martinussen, M., & Wiggins, M. (2003). Understanding how pilots make weather-related decisions. The International Journal of Aviation Psychology, 13, 73–87.
36. Jackson, S. E., & Schuler, R. S. (1990). Human resource planning: Challenges for industrial/organizational psychologists. American Psychologist, 45, 223–239.
37. Johnston, J., Cannon-Bowers, J. A., & Jentsch, K. A. S. (1995). Event based performance measurement system for shipboard command teams. Proceedings of the first International Symposium on Command and Control Research and Technology (pp. 274–276). Washington, DC: The Center for Advanced Command and Technology.
38. Johnston, J., Smith-Jentsch, K. A., & Cannon-Bowers, J. A. (1997). Performance measurement tools for enhancing team decision-making. In M. T. Brannick, E. Salas, & C. Prince, Team performance assessment and measurement: Theory, methods and applications (pp. 311–327). Hillsdale, NJ: Lawrence Erlbaum Associates.
39. Joiner, D. A. (2002). Assessment centers: What's new? Public Personnel Management, 31, 179–185.
40. Kirkpatrick, D. L. (1959). Techniques for evaluating training programs, fournal of ASTD, 13, 3–9.
41. Kleinmann, M., & Straub, B. (1998). Validity and application of computer-simulated scenarios in personnel assessment. Validity and Implementation of Computer-Simulated Scenarios, 2, 97106.
42. Kozlowski, S. W. J., Brown, K. G., Weissbein, D. A., Cannon-Bowers, J. A., & Salas, E. (2000). A multilevel approach to training effectiveness: Enhancing horizontal and vertical transfer. In K. Klein & S. W. J. Kozlowski, Multilevel theory, research and methods in Organization (pp. 157–210). San Francisco, CA: Jossey-Bass.
43. Kraiger K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311–328.
44. Kreuzig, H. W. (1995). The computer simulation Manage!. In T. Geilhardt & T. Muhlbradt, Planspiele im Personal-und Organizations Management (pp. 387–400). Gottingen, Germany: Verlag fur Angewandte Psychologie.
45. Marks, M. A. (2000). A critical analysis of computer simulations for conducting team research. Small Group Research, 31, 653–675.
46. McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., Campion, M. A., & Braverman, E. P. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 730–740.
47. McDaniel, M. A., & Nguyen, N. T. (2001). Situational judgment tests: A review of practice and constructs assess International Journal of Selection and Assessment, 9, 103–113.
48. Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640–647.
49. Motowidlo, S. J., Hanson, M. A., & Crafts, J. L. (1997). Low-fidelity simulations. In D. L. Whetzel, & G. R. Wheaton, Applied measurement methods in industrial psychology (pp. 241–260). Palo Alto, CA: Consulting Psychologists Press.
50. Mullins, M. E. (2000). The effects of practice variability and velocity feedback on the development of basic and strategic training skills. Unpublished doctoral dissertation, Michigan State University, Ann Arbor.
51. Obermann, C. (1988). Development of a personnel assessment instrument based on the complex problem solving paradigm. Mannheim: Institut fur Psychologie.
52. Olson-Buchanan, J. B., Drasgow, F., Moberg, P. J., Mead, A. D., Keenan, P. A., & Donovan, M. A. (1998). Interactive video assessment of conflict resolution skills. Personnel Psychology, 51, 1–24.
53. Oser, R. L., Cannon-Bowers, J. A., Salas, E., & Dwyer, D. J. (1999). Enhancing human performance in technology-rich environments: Guidelines for scenario based training. Human Technology Interaction in Complex Systems, 9, 175–202.
54. Paley, M., Baker, K., Entin, E., Serfaty, D., & Salas, E. (2002). Scenario-based training for stability and support operations (SASO) with the distributed dynamic decision-making (DDD) simulation (AP-R-1200). Woburn, MA: Aptima, Inc.
55. Ployhart, R. E., & Ehrhart, M. G. (2003). Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International journal of Selection & Assessment, 11, 1–16.
56. Ployhart, R. E., Porr, W. B., & Ryan, A. M. (2004). A construct oriented approach for developing situational judgment tests in a service context. In P. R. Sackett (Chair), New developments in SJTs: Scoring, coaching, and incremental validity. Paper presented at the 19th annual conference of the Society for Industrial and Organizational Psychology, Chicago, IL.
57. Rosen, N. A. (1961). How to supervise? Personnel Psychology, 14, 87–99.
58. Rutkowski, K. A., Steelman, L. A., & Griffith, R. L. (2004, April). An empirical examination of accountability for performance development. Paper presented at the 19th annual meeting for the Society of Industrial Organizational Psychology
59. Salas, E., & Cannon-Bowers, J. A. (2001). The science of training: A decade of progress. Annual Review of Psychology, 52, 471–499.
60. Salas, E., Fowlkes, J. E., Stout, R. J., Milanovich, D. M., & Prince, C. (1999). Does CRM training improve teamwork skills in the cockpit?: Two evaluation studies. Human Factors, 41(2), 326–343.
61. Salas, E., Stagl, K. C., & Burke, C. S. (2004). 25 years of team effectiveness in organizations: Research themes and emerging ne In C. L. Cooper & I. T. Robertson, International review of industrial and organizational psychology (pp. 47–91). New York: Wiley.
62. Schneider, B. (1987). The people make the place. Personnel Psychology, 40, 450–467.
63. Schmidt, F. L., & Hunter, J. E. (1993). Tacit knowledge, practical intelligence, general mental ability and job knowledge. Current Directions in Psychological Science, 2, 8–9.
64. Schmitt, N., & Mills, A. E. (2001). Traditional tests and job simulations: Minority and majority performance and test validities. Journal of Applied Psychology, 3, 451–458.
65. Scolaro, J., & Santarelli, T. (2002). Cognitive modeling teamwork, taskwork, and instructional behavior in synthetic teammates. Proceedings of the 11th Conference on Computer Generated Forces and Behavioral Representation. Orlando, FL: Institute for Simulation and Training.
66. Smith-Jentsch, K. A., Zeisig, R. L., Acton, B., & McPherson, J. A. (1998). Team dimensional training: A strategy for guided team self-correction. In J. A. Cannon-Bowers & E. Salas (). Making decisions under stress (pp. 271–297). Washington, DC: American Psychological Association.
67. Sternberg, R. J., Forsythe, G. B., Hedlund, J., Horvath, J. A., Wagner, R. K., Williams, W. M., Snook, S. A., & Grigorenko, E. L. (2000). Practical intelligence: An example from the military workplace. In Practical intelligence in everyday life. Cambridge: Cambridge University Press.
68. Stout, R. J., Salas, E., & Fowlkes, J. E. (1997). Enhancing teamwork in complex environments through team training. Group Dynamics: Theory, Research and Practice, 1, 169–182.
69. Streufert, S., Pogash, R., & Piasecki, M. (1988). Simulation-based assessment of managerial competence: Reliability and validity. Personnel Psychology, 41, 537–557.
70. Swezey, R. W., Hutcheson, T. D., & Swezey, L. L. (2000). Development of a second-generation computer-based team performance assessment technology. International Journal of Cognitive Ergonomics, 4, 163–170.
71. Weekley, J. A. & Jones, C. (1997). Video-based situational testing. Personnel Psychology, 50, 25–49.
72. Wernimont, P., & Campbell, J. P. (1968). Sign, sample and criteria. Journal of Applied Psychology, 52, 372–376.