Putting judging situations into situational judgment tests: Evidence from intercultural multimedia SJTs

Journal of Applied Psychology

Volumn 100, Issue 2, 2015, Pages 464-480

  Rockstuhl, Thomas ^a   Ang, Soon ^a   Ng, Kok Yee ^a   Lievens, Filip ^b   Van Dyne, Linn ^c  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^b GHENT UNIVERSITY   (Belgium)

^c MICHIGAN STATE UNIVERSITY   (United States)

Author keywords

Intercultural skills; Performance; Situational judgment test; Verbal protocol analysis

Indexed keywords

ADULT; CULTURAL COMPETENCE; DECISION MAKING; DEVICES; FEMALE; HUMAN; MALE; MULTIMEDIA; PSYCHOLOGIC TEST; PSYCHOMETRY; STANDARDS; YOUNG ADULT;

ADULT; CULTURAL COMPETENCY; FEMALE; HUMANS; JUDGMENT; MALE; MULTIMEDIA; PSYCHOLOGICAL TESTS; PSYCHOMETRICS; YOUNG ADULT;

EID: 84929836145     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0038098     Document Type: Article

References (74)

1. Ang, S., Rockstuhl, T., & Ng, K. Y. (2014). Performance-based cultural intelligence (CQ): Development and validation of an intercultural situational judgment test (iSJT). Singapore: Nanyang Technological University, Center for Leadership and Cultural Intelligence.
2. Ang, S., & Van Dyne, L. (2008). Conceptualization of cultural intelligence: Definition, distinctiveness, and nomological network. In S. Ang & L. Van Dyne (Eds.), Handbook of cultural intelligence: Theory, measurement, and applications (pp. 3-15). Armonk, NY: Sharpe.
3. Ang, S., Van Dyne, L., Koh, C., Ng, K. Y., Templer, K. J., Tay, C., & Chandrasekar, N. A. (2007). Cultural intelligence: Its measurement and effects on cultural judgment and decision making, cultural adaptation, and task performance. Management and Organization Review, 3, 335-371. doi:10.1111/j.1740-8784.2007.00082.x.
4. Bandura, A. (1997). Self-efficacy: The exercise of control. New York, NY: Freeman.
5. Blau, P. M. (1977). Inequality and heterogeneity. New York, NY: Free Press.
6. Byrne, B. M., Shavelson, R. J., & Muthén, B. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of practical measurement invariance. Psychological Bulletin, 105, 456-466. doi:10.1037/0033-2909.105.3.456.
7. Byham, W. C. (1977). Assessor selection and training. In J. L. Moses & W. C. Byham (Eds.), Applying the assessment center method (pp. 89-126). New York, NY: Pergamon Press. doi:10.1016/B978-0-08-019581-0.50011-6.
8. Campion, M. C., & Ployhart, R. E. (2013). Assessing personality with situational judgment measures: Interactionist psychology operationalized. In N. D. Christiansen & R. P. Tett (Eds.), Handbook of personality at work (pp. 439-456). New York, NY: Routledge.
9. Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82, 143-159. doi:10.1037/0021-9010.82.1.143.
10. Christian, M. S., Edwards, B. D., & Bradley, J. C. (2010). Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validities. Personnel Psychology, 63, 83-117. doi: 10.1111/j.1744-6570.2009.01163.x.
11. Clevenger, J., Pereira, G. M., Wiechmann, D., Schmitt, N., & Schmidt-Harvey, V. (2001). Incremental validity of situational judgment tests. Journal of Applied Psychology, 86, 410-417. doi:10.1037/0021-9010.86.3.410.
12. Davis, M. H. (1983). Measuring individual differences in empathy: Evidence for a multidimensional approach. Journal of Personality and Social Psychology, 44, 113-126. doi:10.1037/0022-3514.44.1.113.
13. Deardorff, D. K. (2009). Implementing intercultural competence assessment. In D. K. Deardorff (Ed.), The Sage handbook of intercultural competence (pp. 477-491). Thousand Oaks, CA: Sage.
14. Earley, P. C., & Ang, S. (2003). Cultural intelligence: Individual interactions across cultures. Palo Alto, CA: Stanford University Press.
15. Elfenbein, H. A., & Ambady, N. (2002). On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin, 128, 203-235. doi:10.1037/0033-2909.128.2.203.
16. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
17. Farh, J. L., Zhong, C. B., & Organ, D. W. (2004). Organizational citizenship behavior in the People's Republic of China. Organization Science, 15, 241-253. doi:10.1287/orsc.1030.0051.
18. Feldman, J. M., & Lynch, J. G. Jr. (1988). Self-generated validity and other effects of measurement on belief, attitude, intention, and behavior. Journal of Applied Psychology, 73, 421-435. doi:10.1037/0021-9010.73.3.421.
19. Gelfand, M. J., Erez, M., & Aycan, Z. (2007). Cross-cultural organizational behavior. Annual Review of Psychology, 58, 479-514. doi:10.1146/annurev.psych.58.110405.085559.
20. Gibson, C. B., & Zellmer-Bruhn, M. E. (2001). Metaphors and meaning: An intercultural analysis of the concept of teamwork. Administrative Science Quarterly, 46, 274-303. doi:10.2307/2667088.
21. Goldberg, L. R. (1999). A broad-bandwidth, public-domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. J. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe (Vol. 7, pp. 7-28). Tilburg, the Netherlands: Tilburg University Press.
22. Heider, F. (1958). The psychology of interpersonal relations. New York, NY: Wiley. doi:10.1037/10628-000.
23. House, R. J., Hanges, P. J., Javidan, M., Dorfman, P. W., & Gupta, V. (2004). Culture, leadership, and organizations: The GLOBE study of 62 societies. Palo Alto, CA: Sage.
24. Jansen, A., Melchers, K. G., Lievens, F., Kleinmann, M., Brändli, M., Fraefel, L., & König, C. J. (2013). Situation assessment as an ignored factor in the behavioral consistency paradigm underlying the validity of personnel selection procedures. Journal of Applied Psychology, 98, 326-341. doi:10.1037/a0031257.
25. Johnson, J. W., & LeBreton, J. M. (2004). History and use of relative importance indices in organizational research. Organizational Research Methods, 7, 238-257. doi:10.1177/1094428104266510.
26. Jöreskog, K. G., & Sörbom, D. (1996). LISREL 8: User's reference guide. Chicago, IL: Scientific Software International.
27. Kenny, D. A. (1995). SOREMO Version 2: A FORTRAN program for the analysis of round-robin data structures. Unpublished manuscript, University of Connecticut.
28. Kenny, D. A., & La Voie, L. (1984). The social relations model. Advances in Experimental Social Psychology, 18, 141-182. doi:10.1016/S0065-2601(08)60144-6.
29. Kruglanski, A. W., & Webster, D. M. (1996). Motivated closing of the mind: "Seizing" and "freezing". Psychological Review, 103, 263-283. doi:10.1037/0033-295X.103.2.263.
30. Lance, C. E., Woehr, D. J., & Meade, A. W. (2007). A Monte Carlo investigation of assessment center construct validity models. Organizational Research Methods, 10, 430-448. doi:10.1177/10944 28106289395.
31. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174. doi:10.2307/2529310.
32. Latham, G. P., Saari, L. M., Pursell, E. D., & Campion, M. A. (1980). The situational interview. Journal of Applied Psychology, 65, 422-427. doi:10.1037/0021-9010.65.4.422.
33. LeBreton, J. M., & Senter, J. L. (2007). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815-852. doi:10.1177/1094428106296642.
34. Leung, K., Ang, S., & Tan, M. L. (2014). Intercultural competence. Annual Review of Organizational Psychology and Organizational Behavior, 1, 489-519. doi:10.1146/annurev-orgpsych-031413-091229.
35. Lievens, F. (2006). International situational judgment tests. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement and application (pp. 279-300). Mahwah, NJ: Erlbaum.
36. Lievens, F., De Corte, W., & Westerveld, L. (2012). Understanding the building blocks of selection procedures: Effects of response fidelity on performance and validity. Journal of Management. Advance online publication. doi:10.1177/0149206312463941.
37. Lievens, F., Peeters, H., & Schollaert, E. (2008). Situational judgment tests: A review of recent research. Personnel Review, 37, 426-441. doi:10.1108/00483480810877598.
38. Magnusson, D., & Ekehammar, B. (1975). Perceptions of and reactions to stressful situations. Journal of Personality and Social Psychology, 31, 1147-1154. doi:10.1037/h0077032.
39. McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & Grubb, W. L. (2007). Situational judgment tests, response instructions, and validity: A metaanalysis. Personnel Psychology, 60, 63-91. doi:10.1111/j.1744-6570.2007.00065.x.
40. McDaniel, M. A., Psotka, J., Legree, P. J., Yost, A. P., & Weekley, J. A. (2011). Toward an understanding of situational judgment item validity and group differences. Journal of Applied Psychology, 96, 327-336. doi:10.1037/a0021983.
41. Molinsky, A. L. (2013). The psychological processes of cultural retooling. Academy of Management Journal, 56, 683-710. doi:10.5465/amj.2010.0492.
42. Motowidlo, S. J., & Beier, M. E. (2010). Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test. Journal of Applied Psychology, 95, 321-333. doi:10.1037/a0017975.
43. Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640-647. doi:10.1037/0021-9010.75.6.640.
44. Motowidlo, S. J., Hooper, A. C., & Jackson, H. L. (2006). A theoretical basis for situational judgment tests. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement, and practice (pp. 57-81). Mahwah, NJ: Erlbaum.
45. Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.
46. Parker, S. M., & Axtell, C. M. (2001). Seeing another viewpoint: Antecedents and outcomes of employee perspective taking. Academy of Management Journal, 44, 1085-1100. doi:10.2307/3069390.
47. Ployhart, R. E. (2006). The predictor response process model. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement and application (pp. 83-105). Mahwah, NJ: Erlbaum.
48. Ployhart, R. E., & MacKenzie, W. I. Jr. (2011). Situational judgment tests: A critical review and agenda for future research. In S. Zedeck (Ed.), APA handbook of industrial and organizational psychology (Vol. 2, pp. 237-252.). Washington, DC: American Psychological Association.
49. Pulakos, E. D. (1984). A comparison of rater training programs: Error training and accuracy training. Journal of Applied Psychology, 69, 581-588. doi:10.1037/0021-9010.69.4.581.
50. Rockstuhl, T., Seiler, S., Ang, S., Van Dyne, L., & Annen, H. (2011). Beyond EQ and IQ: The role of cultural intelligence in cross-border leadership effectiveness in a globalized world. Journal of Social Issues, 67, 825-840. doi:10.1111/j.1540-4560.2011.01730.x.
51. Ryan, A. M., & Greguras, G. J. (1998). Life is not multiple choice: Reactions to the alternatives. In M. D. Hakel (Ed.), Beyond multiple choice: Evaluating alternatives to traditional testing for selection (pp. 183-202). Mahwah, NJ: Erlbaum.
52. Sackett, P. R. (1987). Assessment centers and content validity: Some neglected issues. Personnel Psychology, 40, 13-25. doi:10.1111/j.1744-6570.1987.tb02374.x.
53. Schmitt, N., & Chan, D. (2006). Situational judgment tests: Method or construct? In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement and application (pp. 135-155). Mahwah, NJ: Erlbaum.
54. Shore, L. M., Randel, A. E., Chung, B. G., Dean, M. A., Ehrhart, K. H., & Singh, G. (2011). Inclusion and diversity in work groups: A review and model for future research. Journal of Management, 37, 1262-1289. doi:10.1177/0149206310385943.
55. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428. doi: 10.1037/0033-2909.86.2.420.
56. Silvester, J., Patterson, F., Koczwara, A., & Ferguson, E. (2007). "Trust me.": Psychological and behavioral predictors of perceived physician empathy. Journal of Applied Psychology, 92, 519-527. doi:10.1037/0021-9010.92.2.519.
57. Smith, P. C., & Kendall, L. M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors to rating scales. Journal of Applied Psychology, 47, 149-155. doi:10.1037/h0047060.
58. Spencer-Oatey, H. (2008). Face, (im)politeness, and rapport. In H. Spencer-Oatey (Ed.), Culturally speaking: Culture, communication, and politeness theory (2nd ed., pp. 11-47). London, England: Continuum.
59. Stajkovic, A. D., & Luthans, F. (1998). Self-efficacy and work-related performance: A meta-analysis. Psychological Bulletin, 124, 240-261. doi:10.1037/0033-2909.124.2.240.
60. Stone-Romero, E., Stone, D. L., & Salas, E. (2003). The influence of culture on role conceptions and role behavior in organizations. Applied Psychology: An International Review, 52, 328-362. doi:10.1111/1464-0597.00139.
61. Strauss, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park, CA: Sage.
62. Thomas, K. W. (1976). Conflict and conflict management. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology (pp. 889-935). Chicago, IL: Rand McNally.
63. Thornton, G. C., III, & Rupp, D. E. (2006). Assessment centers in human resources management. Mahwah, NJ: Erlbaum.
64. Triandis, H. C. (2006). Cultural intelligence in organizations. Group & Organization Management, 31, 20-26. doi:10.1177/1059601105275253.
65. Van Dyne, L., Ang, S., Ng, K. Y., Rockstuhl, T., Tan, M. L., & Koh, C. (2012). Sub-dimensions of the four factor model of cultural intelligence: Expanding the conceptualization and measurement of cultural intelligence. Social and Personality Psychology Compass, 6, 295-313. doi: 10.1111/j.1751-9004.2012.00429.x.
66. Van Dyne, L., & LePine, J. A. (1998). Helping and voice extra-role behaviors: Evidence of construct and predictive validity. Academy of Management Journal, 41, 108-119. doi:10.2307/256902.
67. Weber, E. U., & Johnson, E. J. (2009). Mindful judgment and decision making. Annual Review of Psychology, 60, 53-85. doi:10.1146/annurev.psych.60.110707.163633.
68. Weekley, J. A., & Jones, C. (1997). Video-based situational testing. Personnel Psychology, 50, 25-49. doi:10.1111/j.1744-6570.1997.tb00899.x.
69. Weekley, J. A., & Ployhart, R. E. (2006). An introduction to situational judgment testing. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement, and practice (pp. 1-10). Mahwah, NJ: Erlbaum.
70. Weekley, J. A., Ployhart, R. E., & Holtz, B. C. (2006). On the development of situational judgment tests: Issues in item development, scaling, and scoring. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement, and practice (pp. 157-182). Mahwah, NJ: Erlbaum.
71. Weiner, B. (1995). Judgments of responsibility: A foundation for a theory of social conduct. New York, NY: Guilford Press.
72. Whetzel, D. L., & McDaniel, M. A. (2009). Situational judgment tests: An overview of current research. Human Resource Management Review, 19, 188-202. doi:10.1016/j.hrmr.2009.03.007.
73. Williams, L. J., & Anderson, S. E. (1991). Job satisfaction and organizational commitment as predictors of organizational citizenship and in-role behavior. Journal of Management, 17, 601-618. doi:10.1177/014920639101700305.
74. Wonderlic, E. F. (1999). Wonderlic Personnel Test user's manual. Libertyville, IL: Wonderlic.