Scoring situational judgment tests: Once you get the data, your troubles begin

International Journal of Selection and Assessment

Volumn 14, Issue 3, 2006, Pages 223-235

  Bergman, Mindy E ^a,e   Drasgow, Fritz ^b   Donovan, Michelle A ^c   Henning, Jaime B ^a   Juraska, Suzanne E ^d  

^a TEXAS A AND M UNIVERSITY   (United States)

^b UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^c INTEL CORPORATION   (United States)

^d Personnel Decisions Research Institutes   (United States)

^e Department of Psychology ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33845419000     PISSN: 0965075X     EISSN: 14682389     Source Type: Journal    
DOI: 10.1111/j.1468-2389.2006.00345.x     Document Type: Review

References (59)

1. Arnold, J.A., Arad, S., Rhoades, J.A. and Drasgow, F. (2000) The empowering leadership questionnaire: The construction of a new scale for measuring leader behaviors. Journal of Organizational Behavior, 21, 249-269.
2. Ashworth, S.D. and Joyce, T.M. (1994) Developing score protocols for a computerized multimedia in-basket exercise. Paper presented at the Ninth Annual Conference of the Society for Industrial and Organizational Psychology, Nashville, TN, April.
3. Borman, W.C., White, L.A., Pulakos, E.D. and Oppler, S.H. (1991) Models of supervisory job performance ratings. Journal of Applied Psychology, 76, 863-872.
4. Breiman, L., Friedman, J.H., Olshen, R.A. and Stone, C.J. (1984) Classification and regression trees. Belmont, CA: Wadsworth.
5. Campbell, D.T. and Fiske, D.W. (1959) Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
6. Campbell, J.P. (1990a) Modeling the performance prediction problem in industrial and organizational psychology. In M.D. Dunnette and L.M. Hough (Eds), Handbook of industrial and organizational psychology, Vol. 1 (pp. 687-732). Palo Alto: Consulting Psychologists Press.
7. Campbell, J.P. (1990b) The role of theory in industrial and organizational psychology. In M.D. Dunnette and L.M. Hough (Eds), Handbook of industrial and organizational psychology, Vol. 1 (pp. 39-73). Palo Alto: Consulting Psychologists Press.
8. Campbell, J.P, Dunnette, M.D., Lawler, E.E. III. and Weick, K.E. (1970) Managerial behavior, performance, and effectiveness. New York: McGraw-Hill.
9. Cattell, R.B., Cattell, A.K. and Cattell, H.E. (1993) Sixteen personality factor questionnaire, 5th Edn). Champaign, IL: Institute for Personality and Ability Testing Inc.
10. Chan, D. and Schmitt, N. (1997) Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82, 143-159.
11. Chan, D. and Schmitt, N. (2002) Situational judgment and job performance. Human Performance, 15, 233-254.
12. Chan, D. and Schmitt, N. (2005) Situational judgment tests. In A. Evers, N. Anderson and O. Voskuijil (Eds), Handbook of personnel selection (pp. 219-246). Oxford: Blackwell.
13. Cleary, T.A. (1968) Test bias: Prediction of grades of Negro and White students in integrated colleges. Journal of Educational Measurement, 5, 115-124.
14. Clevenger, J., Pereira, G.M., Wiechmann, D., Schmitt, N. and Harvey, VS. (2001) Incremental validity of situational judgment tests. Journal of Applied Psychology, 86, 410-417.
15. Cureton, E.E. (1950) Validity, reliability, and baloney. Educational and Psychological Measurement, 10, 94-96.
16. Dalessio, A.T. (1994) Predicting insurance agent turnover using a video-based situational judgment test. Journal of Business and Psychology, 9, 23-32.
17. Desmarais, L.B., Masi, D.L., Olson, M.J., Barbara, K.M. and Dyer, P.J. (1994) Scoring a multimedia situational judgment test: IBM's experience. Paper presented at the Ninth Annual Conference of the Society for Industrial and Organizational Psychology, Nashville, TN, April.
18. Devlin, S.E., Abrahams, N.M. and Edwards, J.E. (1992) Empirical keying of biographical data: Cross-validity as a function of scaling procedure and sample size. Military Psychology, 4, 119-136.
19. Dodrill, C.B. (1983) Long term reliability of the Wonderlic Personnel Test. Journal of Consulting and Clinical Psychology, 51, 316-317.
20. Dodrill, C.B. and Warner, M.H. (1988) Further studies of the Wonderlic Personnel Test as a brief measure of intelligence. Journal of Consulting and Clinical Psychology, 56, 145-147.
21. England, G.W. (1961) Development and use of weighted application blanks. Dubuque: Brown.
22. Flanagan, J.C. (1954) The critical incident technique. Psychological Bulletin, 51, 327-358.
23. Hein, M. and Wesley, S. (1994) Scaling biodata through subgrouping. In G.S. Stokes, M.D. Mumford and W.A. Owens (Eds), Biodata handbook: Theory, research, and use of biographical information in selection and performance prediction (pp. 171-196). Palo Alto: Consulting Psychologists Press.
24. Hesketh, B. (1999) Introduction to the International Journal of Selection and Assessment special issue on biodata. International Journal of Selection and Assessment, 7, 55-56.
25. Hogan, J.B. (1994) Empirical keying of background data measures. In O.S. Stokes, M.D. Mumford and W.A. Owens (Eds), Biodata handbook: Theory, research, and use of biographical information in selection and performance prediction (pp. 69-107). Palo Alto: Consulting Psychologists Press.
26. Hough, L. and Paullin, C. (1994) Construct-oriented scale construction: The rational approach. In G.S. Stokes, M.D. Mumford and W.A. Owens (Eds), Biodata handbook: Theory, research, and use of biographical information in selection and performance prediction (pp. 109-145). Palo Alto: Consulting Psychologists Press.
27. Judge, T.A., Bono, J.E., Hies, R. and Gerhart, M.W. (2002) Personality and leadership: A qualitative and quantitative review. Journal of Applied Psychology, 87, 765-780.
28. Karas, M. and West, J. (1999) Construct-oriented biodata development for selection to a differentiated performance domain. International Journal of Selection and Assessment, 7, 86-96.
29. Krukos, K., Meade, A.W., Cantwell, A., Pond, S.B. and Wilson, M.A. (2004). Empirical keying of situational judgment tests: Rationale and some examples. Paper presented at the 19th annual meeting of the Society for Industrial/Organizational Psychology, Chicago, IL.
30. Legree, P.J., Psotka, J., Tremble, T. and Bourne, D.R. (2005) Using consensus based measurement to assess emotional intelligence. In R. Schulze and R.D. Roberts (Eds), Emotional intelligence: An international handbook (pp. 155-180). Cambridge, MA: Hogrefe and Huber.
31. Liden, R.C. and Arad, S. (1996) A power perspective of empowerment and work groups: Implications for human resources management research. In G.R. Ferris (Ed.), Research in personnel and human resources management (pp. 205-252). Greenwich, CT: JAI Press.
32. MacLane, C.N., Barton, M.G., Holloway-Lundy, A.E. and Nickels, B.J. (2001). Keeping score: Expert weights on situational judgment responses. Paper presented at the 16th Annual Conference of the Society for Industrial and Organizational Psychology, San Diego, CA.
33. Mael, F.A. (1991) A conceptual rationale for the domain and attributes of biodata items. Personnel Psychology, 44, 763-792.
34. McDaniel, M.A., Morgeson, F.P., Finnegan, E.B., Campion, M.A. and Braverman, E.P. (2001) Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 60-79.
35. McDaniel, M.A. and Nguyen, N.T. (2001) Situational judgment tests: A review of practice and constructs assessed. International Journal of Selection and Assessment, 9, 103-113.
36. McHenry, J.J. and Schmitt, N. (1994) Multimedia testing. In M.J. Rumsey, C.D. Walker and J. Harris (Eds), Personnel selection and classification research (pp. 193-232). Mahwah, NJ: Lawrence Erlbaum Publishers.
37. Mead, A.D. (2000) Properties of a resampling validation technique for empirically scoring psychological assessments. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign.
38. Mitchell, T.W. and Klimoski, R.J. (1982) Is it rational to be empirical? A test of methods for scoring biographical data. Journal of Applied Psychology, 67, 411-418.
39. Motowidlo, S.J., Dunnette, M.D. and Carter, G.W. (1990) An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640-647.
40. Mumford, M.D. (1999) Construct validity and background data: Issues, abuses, and future directions. Human Resource Management Review, 9, 117-145.
41. Mumford, M.D. and Owens, W.A. (1987) Methodology review: Principles, procedures, and findings in the application of background data measures. Applied Psychological Measurement, 11, 1-31.
42. Mumford, M.D. and Stokes, G.S. (1992) Developmental determinants of individual action: Theory and practice in applying background measures. In M.D. Dunnette and L.M. Hough (Eds), Handbook of industrial and organizational psychology, 2nd Edn (pp. 61-138). Palo Alto: Consulting Psychologists Press.
43. Mumford, M.D. and Whetzel, D.L. (1997) Background data. In D. Whetzel and G. Wheaton (Eds), Applied measurement methods in industrial psychology (pp. 207-239). Palo Alto: Davies-Black Publishing.
44. Nickels, B.J. (1994) The nature of biodata. In G.S. Stokes, M.D. Mumford and W.A. Owens (Eds), Biodata handbook: Theory, research, and use of biographical information in selection and performance prediction (pp. 1-16). Palo Alto: Consulting Psychologists Press.
45. Olson-Buchanan, J.B., Drasgow, F., Moberg, P.J., Mead, A.D., Keenan, P.A. and Donovan, M.A. (1998) An interactive video assessment of conflict resolution skills. Personnel Psychology, 51, 1-24.
46. Paullin, C. and Hanson, M.A. (2001) Comparing the validity of rationally-derived and empirically-derived scoring keys for a situational judgment inventory. Paper presented at the 16th Annual Conference of the Society for Industrial and Organizational Psychology, San Diego, CA.
47. Ployhart, R.E. and Ehrhart, M.G. (2003) Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection and Assessment, 11, 1-16.
48. Schoenfeldt, L.F. (1999) From dust bowl empiricism to rational constructs in biographical data. Human Resource Management Review, 9, 147-167.
49. Schoenfeldt, L.F. and Mendoza, J.L. (1994) Developing and using factorially derived biographical scales. In G.S. Stokes, M.D. Mumford and W.A. Owens (Eds), Biodata handbook: Theory, research, and use of biographical information in selection and performance prediction (pp. 147-169). Palo Alto: Consulting Psychologists Press.
50. Smith, K.C. and McDaniel, M.A. (1998). Criterion and construct validity evidence for a situational judgment measure. Poster presented at the 13th Annual Meeting of the Society for Industrial and Organizational Psychology, Dallas, TX, April.
51. Stokes, G.S. and Searcy, C.A. (1999) Specification of scales in biodata form development: Rational vs. empirical and global vs. specific. International Journal of Selection and Assessment, 7, 72-85.
52. Such, M.J. and Hemingway, M.A. (2003) Examining the usefulness of empirical keying in the cross-cultural implementation of a biodata inventory. Paper presented in F. Drasgow (Chair), Resampling and Other Advances in Empirical Keying. Symposium conducted at the 18th annual conference of the Society for Industrial and Organizational Psychology.
53. Such, M.J. and Schmidt, D.B. (2004) Examining the effectiveness of empirical keying: A cross-cultural perspective. Paper presented at the 19th Annual Conference of the Society for Industrial and Organizational Psychology, Chicago, IL.
54. Vroom, V.H. (2000) Leadership and the decision-making process. Organizational Dynamics, 28, 82-94.
55. Vroom, V.H. and Jago, A.G. (1978) On the validity of the Vroom-Yetton model. Journal of Applied Psychology, 63, 151-162.
56. Vroom, V.H. and Yetton, P.W. (1973) Leadership and decision making. Pittsburgh: University of Pittsburgh Press.
57. Weekley, J.A. and Jones, C. (1997) Video-based situational testing. Personnel Psychology, 50, 25-49.
58. Weekley, J.A. and Jones, C. (1999) Further studies of situational tests. Personnel Psychology, 52, 679-700.
59. Wonderlic Personnel Test Inc. (1992) User's manual for the Wonderlic Personnel Test and the Scholastic Level Exam. Libertyville, IL: Wonderlic Personnel Test Inc.