Assessing the Reliability of Situational Judgment Tests Used in High-Stakes Situations

International Journal of Selection and Assessment

Volumn 20, Issue 3, 2012, Pages 333-346

  Catano, Victor M ^a   Brochu, Anne ^a   Lamerson, Cheryl D ^b  

^a SAINT MARY S UNIVERSITY   (Canada)

^b PsychWell Associates of Canada Ltd   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84864821733     PISSN: 0965075X     EISSN: 14682389     Source Type: Journal    
DOI: 10.1111/j.1468-2389.2012.00604.x     Document Type: Article

References (70)

1. American Educational Research Association (AERA), American Psychological Association (APA), and National Council on Measurement in Education (NCME). (1999). Standards for Educational and Psychological Testing. Washington, DC: AERA.
2. *Banki, S., & Latham, G. P. (2010). The criterion-related validities and perceived fairness of the situational interview and the situational judgment test in an Iranian Organisation. Applied Psychology: An International Review, 59, 124-142.
3. Becker, T. E. (2004). Development and validation of a scenario-based measure of employee integrity. Paper presented at the 19th annual convention of the Society for Industrial and Organizational Psychology, Chicago, IL, April.
4. Becker, T. E. (2005). Development and validation of a situational judgment test of employee integrity. International Journal of Selection and Assessment, 13, 225-232.
5. *Bledow, R., & Frese, M. (2009). A situational judgment test of personal initiative and its relationship to performance. Personnel Psychology, 62, 229-258.
6. Canadian Council of Human Resources Associations. (2008) Sample questions from the National Professional Practice Exam . Available at http://www.chrp.ca/become-a-chrp/exam-centre/ (accessed 27 June 2012).
7. Catano, V. M., Darr, W., & Campbell, C. A. (2007). Performance appraisal of behavior-based competencies: A reliable and valid procedure. Personnel Psychology, 60, 199-228.
8. Catano, V. M., & Lamerson, C. D. (2008). Examining the validity of the National Professional Practice Assessment: History and development. Ottawa, ON: Canadian Council of Human Resources Associations.
9. *Chan, D. (2006). Interactive effects of situational judgment effectiveness and proactive personality on work perceptions and work outcomes. Journal of Applied Psychology, 91, 475-481.
10. *Chan, D., & Schmitt, N. (1997). 'Video-based versus paper-and-pencil method of assessment is situational judgment tests: Subgroup differences in test performance and face validity perceptions'. Journal of Applied Psychology, 82, 143-159.
11. *Chan, D., & Schmitt, N. (2002). Situational judgment and job performance. Human Performance, 15, 233-254.
12. *Clause, C. C., Mullins, M. E., Nee, M. T., Pulakos, E. D., & Schmitt, N. (1998). Parallel test form development: A procedure for alternative predictors and an example. Personnel Psychology, 51, 193-208.
13. *Clevenger, J., Pereira, G. M., Wiechmann, D., Schmitt, N., & Schmidt-Harvey, V. S. (2001). Incremental validity of situational judgment tests. Journal of Applied Psychology, 86, 410-417.
14. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
15. Cronbach, L. J., Schönemann, P., & McKie, D. (1965). Alpha coefficient for stratified-parallel tests. Educational and Psychological Measurement, 25, 291-312.
16. *Cullen, M. J., Sackett, P. R., & Lievens, F. (2006). Threats to the operational use of situational judgment tests in the college admission process. International Journal of Selection and Assessment, 14, 142-155.
17. *de Meijer, L. A. L., Born, M. P., van Zeilst, J., & van der Molen, H. T. (2010). Construct-driven development of a video-based situational judgment test for integrity: A study in a multi-ethnic police setting. European Psychologist, 15, 229-236.
18. Downing, S. M., & Haladyna, T. M. (1997). Test item development: Validity evidence from quality assurance procedures. Applied Measurement in Education, 10, 61-82.
19. Goldberg, L. R. (1999). A broad-bandwidth, public-domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe (Vol. 7, pp. 7-28). Tilburg: Tilburg University Press.
20. Guion, R. M. (1965). Personnel testing. New York: McGraw-Hill.
21. *Hunter, D. (2003). Measuring general aviation pilot judgment using a situational judgment technique. The International Journal of Aviation Psychology, 13, 373-386.
22. Johnson, J. W., & Oswald, F. L. (2010). Use of test scores. In J. L. Farr & N. T. Tippins (Eds.), Handbook of employee selection (pp. 151-170). New York: Routledge.
23. *Lievens, F., Buyse, T., & Sackett, P. R. (2005a). Retest effects in operational selection settings: Development and test of a framework. Personnel Psychology, 58, 981-1007.
24. *Lievens, F., Buyse, T., & Sackett, P. R. (2005b). The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology, 90, 442-452.
25. *Lievens, F., & Coetsier, P. (2002). Situational tests in student selection: An examination of predictive validity, adverse impact, and construct validity. International Journal of Selection and Assessment, 10, 245-257.
26. *Lievens, F., & Patterson, F. (2011). The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. Journal of Applied Psychology, 96, 927-940.
27. *Lievens, F., & Peeters, H. (2008). Impact of elaboration on responding to situational judgment test items. International Journal of Selection and Assessment, 16, 345-355.
28. Lievens, F., Peeters, H., & Schollaert, E. (2008). Situational judgment tests: A review of recent research. Personnel Review, 37, 426-441.
29. *Lievens, F., & Sackett, P. R. (2006). Video-based versus written situational judgment tests: A comparison in terms of predictive validity. Journal of Applied Psychology, 91, 1181-1188.
30. *Lievens, F., & Sackett, P. R. (2007). Situational judgment tests in high-stakes settings: Issues and strategies with generating alternate forms. Journal of Applied Psychology, 92, 1043-1055.
31. Lievens, F., Sackett, P. R., & Buyse, T. (2009). The effects of response instructions on situational judgment test performance and validity in a high-stakes context. Journal of Applied Psychology, 94, 1095-1101.
32. McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & Grubb, W. L. III. (2007). Situational judgment tests, response instructions, and validity: A meta-analysis. Personnel Psychology, 60, 63-91.
33. McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., & Campion, M. A. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 730-740.
34. McDaniel, M. A., & Nguyen, N. T. (2001). Situational judgment tests: A review of practice and constructs assessed. International Journal of Selection and Assessment, 9, 103-113.
35. McDaniel, M. A., Whetzel, D. L., Hartman, N. S., & Grubb, W. L. III. (2007). Situational judgment tests, response instructions, and validity: A meta-analysis. Personnel Psychology, 60, 63-91.
36. McDaniel, M. A., Whetzel, D. L., Hartman, N. S., Nguyen, N. T., & Grubb, W. L. III. (2006). Situational judgment tests: Validity and an integrative model. In J. Weekley & R. E. Ployhart (Eds.), Situational judgment tests (pp. 183-204). Mahwah, NJ: Lawrence Erlbaum and Associates.
37. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46.
38. *Morgeson, F. P., Reider, M. H., & Campion, M. A. (2005). Selecting individuals in team settings: The importance of social skills, personality characteristics, and teamwork knowledge. Personnel Psychology, 58, 583-611.
39. *Motowidlo, S. J., & Beier, M. E. (2010). Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test. Journal of Applied Psychology, 95, 321-333.
40. *Motowidlo, S. J., Crook, A. E., Kell, H. J., & Naemi, B. (2009). Measuring procedural knowledge more simply with a single-response situational judgment test. Journal of Business Psychology, 24, 281-288.
41. *Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An alternate selection procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640-647.
42. *Motowidlo, S. J., & Tippins, N. (1993). Further studies of the low-fidelity simulation in the form of a situational inventory. Journal of Occupational and Organizational Psychology, 66, 337-344.
43. *Mumford, T. V., Van Iddekinge, C. H., Morgeson, F. P., & Campion, M. A. (2008). The team role test: Development and validation of a team role knowledge situational judgment test. Journal of Applied Psychology, 93, 250-267.
44. National Commission for Certifying Agencies (NCCA). (2004). Standards for accreditation of certifying programs. Washington, DC: Institute for Credentialing Excellence.
45. *Nguyen, N. T., Biderman, M. D., & McDaniel, M. A. (2005). Effects of response instructions on faking a situational judgment tests. International Journal of Selection and Assessment, 13, 250-260.
46. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.) New York: McGraw-Hill.
47. *O'Connell, M. S., Hartman, N. S., McDaniel, M. A., Grubb, W. L., & Lawrence, A. (2007). Incremental validity of situational judgment tests for task and contextual job performance. International Journal of Selection and Assessment, 15, 19-29.
48. *Oswald, F. L., Friede, A. J., Schmitt, N., Kim, B. K., & Ramsay, L. J. (2005). Extending a practical method for developing alternate test forms using independent sets of items. Organizational Research Methods, 8, 149-164.
49. *Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. J., & Gillespie, M. A. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89, 187-208.
50. Patterson, F., Baron, H., Carr, V., Plint, S., & Lane, P. (2009). Evaluation of three short-listing methodologies for selection into postgraduate training in general practice. Medical Education., 43, 50-57.
51. *Peeters, H., & Lievens, F. (2005). Situational judgment tests and their predictiveness of college student success: The influence of faking. Educational and Psychological Measurement, 65, 70-89.
52. *Ployhart, R. E., & Ehrhart, M. G. (2003). Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection & Assessment., 11, 1-16.
53. Ployhart, R. E., & MacKenzie, J. I., Jr. (2011). Situational judgment tests: A critical review and agenda for the future. In S. Zedek (Ed.), APA handbook of industrial and organizational psychology (Vol. 2, pp. 237-252). Washington, DC: American Psychological Association.
54. Ployhart, R. E., Porr, W., & Ryan, A. M. (2004). New development in SJTs: scoring, coaching and incremental validity. Paper presented at the 19th annual convention of the Society for Industrial and Organizational Psychology, Chicago, IL, April.
55. *Ployhart, R. E., Weekley, J. A., Hotz, B. C., & Kemp, C. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: Are personality, biodata, and situational judgment tests comparable? Personnel Psychology, 56, 733-752.
56. *Pulakos, E. D., & Schmitt, N. (1996). An evaluation of two strategies for reducing adverse impact and their effects on criterion-related validity. Human Performance, 9, 241-258.
57. Schmidt, F. L., & Hunter, J. E. (1996). Measurement errors in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199-223.
58. Schmitt, N., & Chan, D. (2006). Situational judgment tests: Method or construct? J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests (pp. 135-156). Mahwah, NJ: Lawrence Erlbaum.
59. *Schmitt, N., Oswald, F. L., Kim, B. H., Gillespie, M. A., & Ramsay, L. J. (2004). The impact of justice and self-serving bias explanations of the perceived fairness of different types of selection tests. International Journal of Selection and Assessment, 12, 160-171.
60. *Schmitt, N., Oswald, F. L., Kim, B. H., Imus, A., Merritt, S., Friede, A., & Ramsay, L. J. (2007). The use of background and ability profiles to predict college student outcomes. Journal of Applied Psychology, 92, 165-179.
61. Stemler, S. E., & Sternberg, R. J. (2006). Using situational judgment tests to measure practical intelligence. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement and application (pp. 107-131). Mahwah, NJ: Lawrence Erlbaum Associates.
62. *Stevens, M. J., & Campion, M. A. (1999). Staffing work teams: Development and validation of a selection test for teamwork. Journal of Management, 25, 207-228.
63. *Weekley, J. A., & Jones, C. (1999). Further studies of situational tests. Personnel Psychology, 52, 679-700.
64. *Weekley, J. A., & Ployhart, R. E. (2005). Situational judgment: Antecedents and relationships with performance. Human Performance, 18, 81-104.
65. Weekley, J. A., & Ployhart, R. E. (2006). Situational judgment tests: Theory, measurement and application. Mahwah, NJ: Lawrence Erlbaum & Associates.
66. Weekley, J. A., Ployhart, R. E., & Holtz, B. C. (2006). On the development of situational judgment tests: Issues in item development, scaling, and scoring. In J. Weekley & R. E. Ployhart (Eds.), Situational judgment tests (pp. 157-182). Mahwah, NJ: Lawrence Erlbaum and Associates.
67. *Weekly, J. A., & Jones, C. (1997). Video-based situational testing. Personnel Psychology, 50, 25-49.
68. *Weekly, J. A., Ployhart, R. E., & Harold, C. M. (2004). Personality and situational judgment tests across applicant and incumbent settings: An examination of validity, measurement, and subgroup differences. Human Performance, 17, 433-461.
69. *Westring, A. J. F., Oswald, F. L., Schmitt, N., Drzakpwski, S., Imus, A., Kim, B., & Shivpuri, S. (2009). Estimating trait and situational variance in a situational judgment test. Human Performance, 22, 44-63.
70. Whetzel, D. L., & McDaniel, M. A. (2009). Situational judgment tests: An overview of current research. Human Resource Management Review, 19, 188-202.