How "situational" is judgment in situational judgment tests?

Journal of Applied Psychology

Volumn 100, Issue 2, 2015, Pages 399-416

  Krumm, Stefan ^a   Lievens, Filip ^b   Hüffmeier, Joachim ^c   Lipnevich, Anastasiya A ^d   Bendels, Hanna ^e   Hertel, Guido ^e  

^a FREIE UNIVERSITÄT BERLIN   (Germany)

^b GHENT UNIVERSITY   (Belgium)

^c FEDERAL INSTITUTE FOR OCCUPATIONAL SAFETY AND HEALTH   (Germany)

^d CITY UNIVERSITY OF NEW YORK   (United States)

^e UNIVERSITY OF MÜNSTER   (Germany)

Author keywords

Contextualization; Knowledge; Simulation; Situational judgment test; Validity

Indexed keywords

ADOLESCENT; ADULT; AGED; DECISION MAKING; DEVICES; EMPLOYMENT; FEMALE; HUMAN; MALE; MIDDLE AGED; PSYCHOLOGIC TEST; PSYCHOMETRY; REPRODUCIBILITY; STANDARDS; YOUNG ADULT;

ADOLESCENT; ADULT; AGED; EMPLOYMENT; FEMALE; HUMANS; JUDGMENT; MALE; MIDDLE AGED; PSYCHOLOGICAL TESTS; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS; YOUNG ADULT;

EID: 84929841347     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0037674     Document Type: Article

References (61)

1. Arvey, R. D., Strickland, W., Drauden, G., & Martin, C. (1990). Motivational components of test taking. Personnel Psychology, 43, 695-716. doi:10.1111/j.1744-6570.1990.tb00679.x.
2. Beauducel, A., & Wittmann, W. W. (2005). Simulation study on fit indexes in CFA based on data with slightly distorted simple structure. Structural Equation Modeling, 12, 41-75. doi:10.1207/s15328007sem1201_3.
3. Becker, T. E. (2005). Development and validation of a situational judgment test of employee integrity. International Journal of Selection and Assessment, 13, 225-232. doi:10.1111/j.1468-2389.2005.00319.x.
4. Byrne, B. M. (1989). A primer of LISREL: Basic applications and programming for confirmatory factor analytic models. New York, NY: Springer.
5. Byrne, B. M. (2004). Testing for multigroup invariance using AMOS graphics: A road less traveled. Structural Equation Modeling, 11, 272-300. doi:10.1207/s15328007sem1102_8.
6. Cabin, R. J., & Mitchell, R. J. (2000). To Bonferroni or not to Bonferroni: When and how are the questions. Bulletin of the Ecological Society of America, 81, 246-248.
7. Campion, M. C., & Ployhart, R. E. (2013). Assessing personality with situational judgment measures: Interactionist psychology operationalized. In N. D. Christiansen & R. P. Tett (Eds.), Handbook of personality at work (pp. 439-456). New York, NY: Routledge.
8. Catano, V. M., Brochu, A., & Lamerson, C. D. (2012). Assessing the reliability of situational judgment tests used in high-stakes situations. International Journal of Selection and Assessment, 20, 333-346. doi: 10.1111/j.1468-2389.2012.00604.x.
9. Christian, M. S., Edwards, B. D., & Bradley, J. C. (2010). Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validity. Personnel Psychology, 63, 83-117. doi: 10.1111/j.1744-6570.2009.01163.x.
10. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46. doi:10.1177/001316446002000104.
11. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analyses for the behavioral sciences (3rd ed.). Hillsdale, NJ: Erlbaum.
12. Crook, A. E., Beier, M. E., Cox, C. B., Kell, H. J., Hanks, A. R., & Motowidlo, S. J. (2011). Measuring relationships between personality, knowledge, and performance using single-response situational judgment tests. International Journal of Selection and Assessment, 19, 363-373. doi:10.1111/j.1468-2389.2011.00565.x.
13. Dragoni, L., Oh, I.-S., Vankatwyk, P., & Tesluk, P. E. (2011). Developing executive leaders: The relative contribution of cognitive ability, personality, and the accumulation of work experience in predicting strategic thinking competency. Personnel Psychology, 64, 829-864. doi:10.1111/j.1744-6570.2011.01229.x.
14. Endler, N. S., & Magnusson, D. (1976). Toward an interactional psychology of personality. Psychological Bulletin, 83, 956-974. doi:10.1037/0033-2909.83.5.956.
15. Fleeson, W., & Gallagher, P. (2009). The implications of Big-Five standing for the distribution of trait manifestation in behavior: Fifteen experiencesampling studies and a meta-analysis. Journal of Personality and Social Psychology, 97, 1097-1114. doi:10.1037/a0016786.
16. Gigerenzer, G., & Gaissmaier, W. (2011). Heuristic decision making. Annual Review of Psychology, 62, 451-482. doi:10.1146/annurevpsych-120709-145346.
17. Glass, G. V., McGraw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
18. Gudykunst, W. B., Matsumoto, Y., Ting-Toomey, S., Nishida, T., Kim, K. S., & Heyman, S. (1996). The influence of cultural individualism-collectivism, self construals, and individual values on communication styles across cultures. Human Communication Research, 22, 510-543. doi:10.1111/j.1468-2958.1996.tb00377.x.
19. Hall, E. T. (1976). Beyond culture. New York, NY: Doubleday.
20. Holtz, B. C., Ployhart, R. E., & Dominguez, A. (2005). Testing the rules of justice. The effects of frame-of-reference and pre-test information on personality test responses and test perceptions. International Journal of Selection and Assessment, 13, 75-86. doi:10.1111/j.0965-075X.2005.00301.x.
21. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi:10.1080/10705519909540118.
22. Huffcutt, A. I., Conway, J. M., Roth, P. L., & Stone, N. J. (2001). Identification and meta-analytic assessment of psychological constructs measured in employment interviews. Journal of Applied Psychology, 86, 897-913. doi:10.1037/0021-9010.86.5.897.
23. Hunter, D. R. (2003). Measuring general aviation pilot judgment using a situational judgment technique. International Journal of Aviation Psychology, 13, 373-386. doi:10.1207/S15327108IJAP1304_03.
24. Irvine, S. H., & Kyllonen, P. C. (2002). Item generation for test development. Mahwah, NJ: Erlbaum.
25. Jansen, A., Melchers, K. G., Lievens, F., Kleinmann, M., Brändli, M., Fraefel, L., & König, C. J. (2013). Situation assessment as an ignored factor in the behavioral consistency paradigm underlying the validity of personnel selection procedures. Journal of Applied Psychology, 98, 326-341. doi:10.1037/a0031257.
26. Kanning, U. P., Grewe, K., Hollenberg, S., & Hadouche, M. (2006). From the subjects' point of view: Reactions to different types of situational judgment items. European Journal of Psychological Assessment, 22, 168-176. doi:10.1027/1015-5759.22.3.168.
27. Kasten, N., & Freund, P. A. (2013, September). Eine metaanalytische Reliabilitätsgeneralisierung von Situational Judgment Tests (SJTs) [A meta-analytic reliability generalization of situational judgment tests]. Paper presented at the 12th meeting of the section Differential Psychology, Personality Psychology, and Psychological Assessment of the German Psychological Society, Greifswald, Germany.
28. Kittler, M. G., Rygl, D., & Mackinnon, A. (2011). Special review article: Beyond culture or beyond control? Reviewing the use of Hall's high-/low-context concept. International Journal of Cross-Cultural Management, 11, 63-82. doi:10.1177/1470595811398797.
29. Kline, R. B. (2004). Principles and practices of structural equation modeling (2nd ed.). New York, NY: Guilford Press.
30. Leeds, J. P. (2012). The theory of cognitive acuity: Extending psychophysics to the measurement of situational judgment. Journal of Neuroscience, Psychology, and Economics, 5, 166-181. doi:10.1037/a0027294.
31. Lievens, F., & De Soete, B. (2012). Simulations. In N. Schmitt (Ed.), Oxford handbook of assessment and selection (pp. 383-410). New York, NY: Oxford University Press.
32. Lievens, F., & Sackett, P. R. (2006). Video-based versus written situational judgment tests: A comparison in terms of predictive validity. Journal of Applied Psychology, 91, 1181-1188. doi:10.1037/0021-9010.91.5.1181.
33. Lievens, F., & Sackett, P. R. (2007). Situational judgment tests in highstakes settings: Issues and strategies with generating alternate forms. Journal of Applied Psychology, 92, 1043-1055. doi:10.1037/0021-9010.92.4.1043.
34. McCarthy, J. M., & Goffin, R. D. (2003). Is the test attitude survey psychometrically sound? Educational and Psychological Measurement, 63, 446-464. doi:10.1177/0013164403063003007.
35. McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & Grubb, W. L. (2007). Situational judgment tests, response instructions, and validity: A metaanalysis. Personnel Psychology, 60, 63-91. doi:10.1111/j.1744-6570.2007.00065.x.
36. Mischel, W., & Shoda, Y. (1995). A cognitive-affective system theory of personality: Reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. Psychological Review, 102, 246-268. doi:10.1037/0033-295X.102.2.246.
37. Mittring, G., & Rost, D. H. (2008). Die verflixten Distraktoren: Über den Nutzen einer theoretischen Distraktorenanalyse bei Matrizentests [The darn distractors: On the usefulness of theoretical distractor analyses in matrix tests]. Diagnostica, 54, 193-201. doi:10.1026/0012-1924.54.4.193.
38. Morgeson, F. P., DeRue, D. S., & Karam, E. P. (2010). Leadership in teams: A functional approach to understanding leadership structures and processes. Journal of Management, 36, 5-39. doi:10.1177/0149206309347376.
39. Motowidlo, S. J., & Beier, M. E. (2010). Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test. Journal of Applied Psychology, 95, 321-333. doi:10.1037/a0017975.
40. Motowidlo, S. J., Crook, A. E., Kell, H. J., & Naemi, B. (2009). Measuring procedural knowledge more simply with a single-response situational judgment test. Journal of Business and Psychology, 24, 281-288. doi: 10.1007/s10869-009-9106-4.
41. Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An alternative selection procedures: The low fidelity simulation. Journal of Applied Psychology, 75, 640-647. doi:10.1037/0021-9010.75.6.640.
42. Motowidlo, S. J., Hooper, A. C., & Jackson, H. L. (2006). Implicit policies about relations between personality traits and behavioral effectiveness in situational judgment items. Journal of Applied Psychology, 91, 749-761. doi:10.1037/0021-9010.91.4.749.
43. Motowidlo, S. J., Martin, M. P., & Crook, A. E. (2013). Relations between personality, knowledge, and behavior in professional service encounters. Journal of Applied Social Psychology, 49, 1851-1861.
44. Murtha, T., Kanfer, R., & Ackerman, P. L. (1996). Toward an interactionist taxonomy of personality and situations: An integrative situationaldispositional representation of personality traits. Journal of Personality and Social Psychology, 71, 193-207. doi:10.1037/0022-3514.71.1.193.
45. Neisser, U. (1976). General, academic, and artificial intelligence. In L. Resnick (Ed.), The nature of intelligence (pp. 135-144). Hillsdale, NJ: Erlbaum.
46. O'Neill, T., Goffin, R. D., & Gellatly, I. R. (2012). The knowledge, skill, and ability requirements for teamwork: Revisiting the Teamwork-KSA Test's validity. International Journal of Selection and Assessment, 20, 36-52. doi:10.1111/j.1468-2389.2012.00578.x.
47. Ployhart, R. E., Porr, W., & Ryan, A. M. (2004, April). New development in SJTs: Scoring, coaching and incremental validity. Paper presented at the 19th annual convention of the Society for Industrial and Organizational Psychology, Chicago, IL.
48. Potosky, D., & Bobko, P. (2004). Selection testing via the Internet: Practical considerations and exploratory empirical findings. Personnel Psychology, 57, 1003-1034. doi:10.1111/j.1744-6570.2004.00013.x.
49. Richman-Hirsch, W. L., Olson-Buchanan, J. B., & Drasgow, F. (2000). Examining the impact of administration medium on examinee perceptions and attitudes. Journal of Applied Psychology, 85, 880-887. doi: 10.1037/0021-9010.85.6.880.
50. Rockstuhl, R., Ang, S., Lievens, F., & Van Dyne, L. (in press). Putting judging situations into situational judgment tests: Verbal protocols and incremental validity. Journal of Applied Psychology.
51. Stevens, M. J., & Campion, M. A. (1994). The knowledge, skills, and ability requirements for teamwork: Implications for human resource management. Journal of Management, 20, 503-530. doi:10.1016/0149-2063(94)90025-6.
52. Stevens, M. J., & Campion, M. A. (1996). Teamwork-KSA information guide. Arlington, VA: Vangent.
53. Stevens, M. J., & Campion, M. A. (1999). Staffing work teams: Development and validation of a selection test for teamwork settings. Journal of Management, 25, 207-228. doi:10.1016/S0149-2063(99)80010-5.
54. Thorndike, R. L., & Stein, S. (1937). An evaluation of the attempts to measure social intelligence. Psychological Bulletin, 34, 275-285. doi: 10.1037/h0053850.
55. Thornton, G., & Cleveland, J. (1990). Developing managerial talent through simulation. American Psychologist, 45, 190-199. doi:10.1037/0003-066X.45.2.190.
56. Tippins, N. T., & Adler, S. (2011). Technology-enhanced assessment of talent. San Francisco, CA: Jossey-Bass. doi:10.1002/9781118256022.
57. Topping, K., Bremner, W., & Holmes, E. A. (2000). Social competence. The handbook of emotional intelligence. San Francisco, CA: Jossey-Bass.
58. van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994). The think aloud method: A practical guide to modeling cognitive processes. London, England: Academic Press.
59. Wagner, R. K., & Sternberg, R. J. (1985). Practical intelligence in real-world pursuits: The role of tacit knowledge. Journal of Personality and Social Psychology, 49, 436-458. doi:10.1037/0022-3514.49.2.436.
60. Weekley, J. A., Ployhart, R. E., & Holtz, B. C. (2006). On the development of situational judgment tests: Issues in item development, scaling, and scoring. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement, and application (pp. 157-182). Mahwah, NJ: Erlbaum.
61. White, A. P., & Zammarelli, J. E. (1981). Convergence principles: Information in the answer sets of some multiple-choice intelligence tests. Applied Psychological Measurement, 5, 21-27. doi:10.1177/014662168100500103.