On the development of situational judgment tests: Issues in item development, scaling, and scoring

Situational Judgment Tests: Theory, Measurement, and Application

Volumn , Issue , 2013, Pages 157-182

  Weekley, Jeff A ^a   Ployhart, Robert E ^b   Holtz, Brian C ^c  

^a Kenexa ^*  (United States)

^b UNIVERSITY OF SOUTH CAROLINA   (United States)

^c UNIVERSITY OF CALGARY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84920817389     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203774878     Document Type: Chapter

References (61)

1. Arad, S., Borman, W. C., & Pulakos, E. D. (1999, April). Construct validation of a situational judgment test of adaptive behavior. Paper presented at the 14th annual convention of the Society for Industrial and Organizational Psychology, Atlanta, GA.
2. Beauregard, R. S. (2000). Construct explication of a situational judgment test: Addressing multidimensionality through item development, content analysis, and scoring procedures. Unpublished doctoral dissertation, Wright State University, Dayton OH.
3. Becker, T. E. (2004, April). Development and validation of a scenarios-based measure of employee integrity. Paper presented at the 19th annual conference of the Society for Industrial and Organizational Psychology, Chicago, IL.
4. Bergman, M. E., Drasgow, F., Donovan, M. A., & Juraska, S. E. (2003, April). Scoring situational judgment tests. Paper presented at the 18th annual conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
5. Birkeland, S., Manson, T., Kisamore, J., Brannick, M., & Liu, Y. (2003, April). A metaanalytic investigation of job applicant faking on personality measures. Paper presented at the 18th annual conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
6. Blanz, F., & Ghiselli (1972). The mixed standard scale: A new rating system. Personnel Psychology, 25, 185–199.
7. Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82, 143–159.
8. Chan, D., & Schmitt, N. (2002). Situational judgment and job performance. Human Performance, 15, 233–254.
9. Cucina, J. M., Vasilopoulos, N. L., & Leaman, J. A. (2003, April). The bandwidth-fidelity dilemma and situational judgment test validity. Paper presented at the 18th annual conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
10. Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 41, 237–358.
11. Frederickson, N. (1972). Toward a taxonomy of situations. American Psychologist, 27, 114–123.
12. Gillespie, M. A., Oswald, F. L., Schmitt, N., Manheim, L., & Kim, B. H. (2002). Construct validation of a situational judgment test of college student success. Paper presented at the 17th annual conference of the Society for Industrial and Organizational Psychology. Toronto, CA.
13. Goldstein, I. L., Zedeck, S., & Schneider, B. (1993). An exploration of the job analysis-content validation process. In N. Schmitt & W. C. Borman, Personnel selection in organizations, San Francisco, CA: Jossey-Bass.
14. Haas, A. C., Smith, K. C., & McDaniel, M. A. (1999, April). Faking strategies: Effects on a situational judgment test. Paper presented at the 14th annual convention of the Society for Industrial and Organizational Psychology, Atlanta, GA.
15. Hanson, M. A., Borman, W. C., Mogilka, H. J., & Manning, C. (1999). Computerized assessment of skill for a highly technical job. In F. Drasgow & J. B. Olson-Buchanan, Innovations in computerized assessment. Mahwah, NJ: Lawrence Earlbaum Associates.
16. Hooper, A. C., Jackson, H. L., & Motowidlo, S. J. (2004, April). Faking on personality-based measures: Sjts compared to a traditional format. Paper presented at the 19th annual conference of the Society for Industrial and Organizational Psychology, Chicago, IL.
17. Hunter, D. R. (2003). Measuring general aviation pilot judgment using a situational judgment technique. International Journal of Aviation Psychology, 13, 373–386.
18. Juraska, S., & Drasgow, F. (2001, April). Faking situational judgment: A test of the conflict resolution skills assessment. Paper presented at the 16th annual conference of the Society for Industrial and Organizational Psychology, San Diego, CA.
19. Kim, B. H., Schmitt, N., Oswald, F. L., Gillespie, M., & Ramsay, L. J. (2003, April), lob knowledge tests on the path to successful performance. Paper presented at the 18th annual conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
20. Krokos, K. J., Meade, A. W., Cantwell, A. R., Pond, S. B., & Wilson, M. A. (2004, April). Empirical keying of situational judgment tests: Rationale and some examples. Paper presented at the 19th annual conference of the Society for Industrial and Organizational Psychology, Chicago, IL.
21. Legree, P. (1995). Evidence for an oblique social intelligence factor established with a likert-based testing procedure. Intelligence, 21, 247–266.
22. Lievens, F., Buyse, T., & Sackett, P. R. (In press). The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology.
23. MacLane, C. N., Barton, M. G., Holloway-Lundy, A. E., Nickles, B. J. (2001, April). Keeping score: Expert weights on situational judgment responses. Paper presented at the 16th annual conference of the Society for Industrial and Organizational Psychology, San Diego, CA.
24. McDaniel, M. A., Hartman, N. S., & Grubb, W. L. (2003, April). Response instructions as moderators of the validity of situational judgment tests. Paper presented at the 18th annual conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
25. McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., Campion, M. A., & Braverman, E. P. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 80, 730–740.
26. McDaniel, M. A., & Nguyen, N. T. (2001). Situational judgment tests: A review of practice and constructs assess International Journal of Selection and Assessment, 9, 103–113.
27. McElreath, J., & Vasilopoulos, N. L. (2002, April). Situational judgment: Are most and least likely responses the same? Paper presented at the 17th annual conference of the Society for Industrial and Organizational Psychology, Toronto, Canada.
28. Motowidlo, S. J., Diesch, A. C., & Jackson, H. L. (2003, April). Using the situational judgment format to measure personality characteristics. Paper presented at the 18th annual conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
29. Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640–647.
30. Motowidlo, S. J. Hanson, & Crafts, J. L. (1997). Low-fidelity simulations. In D. L. Whetzel & R. Wheaton, Applied measurement methods in industrial psychology (pp. 241–260). Palo Alto, CA: Davies-Black.
31. Nguyen, N. T., Biderman, M., & McDaniel, M. A. (2005). Effects of response instructions on faking in a situational judgment test. Manuscript submitted for publication.
32. O'Connell, M. S., Doverspike, D., Norris-Watts, C., & Hattrup, K. (2001). Predictors of organizational citizenship behavior among Mexican retail salespeople. International Journal of Organizational Analysis, 9, 272–280.
33. Olson-Buchanan, J. B., Drasgow, F., Moberg, P. J., Mead, A. D., Keenan, P. A., & Donovan, M. A. (1998). Interactive video assessment of conflict resolution skills. Personnel Psychology, 51, 1–24.
34. Paullin, C., & Hanson, M. A. (2001, April). Comparing the validity of rationally-derived and empirically-derived scoring keys for a situational judgment inventory. Paper presented at the 16th annual conference of the Society for Industrial and Organizational Psychology, San Diego, CA.
35. Peeters, H., & Lievens, F. (2005). Situational judgment tests and their predictiveness of college students' success: The influence of faking. Educational and Psychological Measurement, 65, 70–89.
36. Ployhart, R. E., & Ehrhart, M. G. (2003). Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection and Assessment, 71, 1–16.
37. Ployhart, R. E., & Ryan, A. M. (2000, April). Integrating personality tests with situational judgment tests for the prediction of customer service performance. Paper presented at the 15th annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
38. Ployhart, R. E., Weekley, J. A., Holtz, B. C., & Kemp, C. F. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: Are personality, biodata, and situational judgment tests comparable? Personnel Psychology, 56, 733–752.
39. Porr, W. B., & Ployhart, R. E. (2004, April). The validity of empirically and construct-oriented situational judgment tests. Paper presented at the 19th annual conference of the Society for Industrial and Organizational Psychology, Chicago, IL.
40. Pulakos, E. D., & Schmitt, N. (1996). An evaluation of two strategies for reducing adverse impact and their effects on criterion-related validity. Human Performance, 9, 241–258.
41. Reynolds, D. H., Sydell, E. J., Scott D. R., & Winter J. L. (2000, April). Factors Affecting Situational judgment Test Characteristics. Paper presented at the 15th annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
42. Reynolds, D. H., Winter, J. L., & Scott, D. R. (1999, April). Development, validation, and translation of a professional-level situational judgment inventory. Paper presented at the 14th annual conference of the Society for Industrial and Organizational Psychology, Atlanta, GA.
43. Sacco, J. M., Scheu, C., Ryan, A. M., & Schmitt, N. W. (2000, April). Understanding race differences on situational judgment tests using readability statistics. Paper presented at the 15th Annual Convention of the Society for Industrial and Organizational Psychology, New Orleans, LA.
44. Sacco, J. M., Schmidt, D. B., & Rogg, K. L. (2000, April). Using readability statistics and reading comprehension scores to predict situational judgment test performance, black-white differences, and validity. Paper presented at the 15th annual convention of the Society for Industrial and Organizational Psychology, New Orleans, LA.
45. Schmidt, D. B., & Wolf, P. P. (2003). Susceptibility of SJTs to applicant faking: An examination of applicant and incumbent samples. Paper presented at the 18th annual conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
46. Sinar, E. F., Paquet, S. L., & Scott D. R. (2002, April). Internet versus paper selection tests: Exploring comparability issues. Paper presented at the 17th annual conference of the Society for Industrial and Organizational Psychology, Toronto, Canada.
47. Stevens, M. A., & Campion, M. ]. (1994). The knowledge, skill, and ability requirements for teamwork: Implications for human resource management. Journal of Management, 20, 503–530.
48. Stevens, M. A., & Campion, M. J. (1999). Staffing work teams: Development and validation of a selection test for teamwork settings. Journal of Management, 25, 207–208.
49. Such, M. J., & Schmidt, D. B. (2004, April). Examining the effectiveness of empirical keying: A cross-cultural perspective. Paper presented at the 19th annual conference of the Society for Industrial and Organizational Psychology, Chicago, IL.
50. Trippe, M. D., & Foti, R. J. (2003, April). An evaluation of the construct validity of situational judgment tests. Paper presented at the 18th annual conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
51. Viswesvaran, C., & Ones, D. S. (1999). Meta-analysis of fakability estimates: Implications for personality measurement. Educational and Psychological Measurement, 59, 197–210.
52. Vroom, V. H. (2000). Leadership and the decision-making process. Organizational Dynamics, 28, 82–94.
53. Vroom, V. H., & Yetton, P. W. (1973). Leadership and decision making. Pittsburgh, PA: University of Pittsburgh Press.
54. Wagner, R. K. (1987). Tacit knowledge in everyday intelligent behavior. Journal of Personality and Social Psychology, 52, 1236–1247.
55. Wagner, R. K., & Sternberg, R. J. (1985). Practical intelligence in real-world pursuits: The role of tacit knowledge. Journal of Personality and Social Psychology, 48, 436–458.
56. Weekley, J. A., Harding, R., Creglow, A., & Ployhart, R. E. (2004, April). Scoring situational judgment tests: Does the middle matter? Paper presented at the 19th annual conference of the Society for Industrial and Organizational Psychology, Chicago, IL.
57. Weekley, J. A., & Jones, C. (1997). Video-based situational testing. Personnel Psychology, 50, 25–49.
58. Weekley, J. A., & Jones, C. (1999). Further studies of situational tests. Personnel Psychology, 52, 679–700.
59. Weekley, J. A., & Ployhart, R. E. (2005). Situational judgment: Antecedents and relationships with performance. Human Performance, 18, 81–104.
60. Weekley, J. A., & Ployhart, R. E., & Harold, C. (2004). Personality and situational judgment tests across applicant and incumbent contexts: An examination of validity, measurement, and subgroup differences. Human Performance, 17, 433–461.
61. Wemimont, P., & Campbell, J. P. (1968). Signs, samples, and criteria. Journal of Applied Psychology, 52, 372–376.