Situational Judgment Tests in High-Stakes Settings: Issues and Strategies With Generating Alternate Forms

Journal of Applied Psychology

Volumn 92, Issue 4, 2007, Pages 1043-1055

  Lievens, Filip ^a   Sackett, Paul R ^b  

^a GHENT UNIVERSITY   (Belgium)

^b UNIVERSITY OF MINNESOTA   (United States)

Author keywords

alternate forms; high stakes testing; item generation; situational judgment tests

Indexed keywords

ADOLESCENT; ADULT; ARTICLE; DECISION MAKING; FEMALE; HUMAN; MALE; PSYCHOLOGIC TEST; SOCIAL ENVIRONMENT;

ADOLESCENT; ADULT; FEMALE; HUMANS; JUDGMENT; MALE; PSYCHOLOGICAL TESTS; SOCIAL ENVIRONMENT;

EID: 34548827973     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/0021-9010.92.4.1043     Document Type: Article

References (38)

1. Barrick M.R., and Mount M.K. Autonomy as a moderator of the relationships between the Big Five personality dimensions and job performance. Journal of Applied Psychology 78 (1993) 111-118
2. Bejar I.I. The psychometrics of mental rotation (1986), Educational Testing Service, Princeton, NJ
3. Bejar I.I. Generative testing: From conception to implementation. In: Irvine S.H., and Kyllonen P.C. (Eds). Item generation and test development (2002), Lawrence Erlbaum, Mahwah, NJ 199-217
4. Brannick M.T., Michaels C.E., and Baker D.P. Construct validity of in-basket scores. Journal of Applied Psychology 74 (1989) 957-963
5. Chan D., and Schmitt N. Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology 82 (1997) 143-159
6. Clause C.C., Mullins M.E., Nee M.T., Pulakos E.D., and Schmitt N. Parallel test form development: A procedure for alternative predictors and an example. Personnel Psychology 51 (1998) 193-208
7. Collis J.M., Tapsfield P.G.C., Irvine S.H., Dann P.L., and Wright D.E. The British Army recruit battery goes operational: From theory to practice in computer based testing using item-generation techniques. International Journal of Selection and Assessment 3 (1995) 251-259
8. Cullen M.J., Sackett P.R., and Lievens F. Coachability as a threat to the operational use of situational judgment tests in the college admission process. International Journal of Selection and Assessment 14 (2006) 142-155
9. Dennis I., Handley S.J., Bradon P., Evans J., and Newstead S.E. Towards a predictive model of the difficulty of analytical reasoning items. In: Irvine S.H., and Kyllonen P.C. (Eds). Item generation and test development (2002), Erlbaum, Mahwah, NJ 53-71
10. Embretson S. A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods 3 (1998) 380-396
11. Hunter J.E., and Schmidt F.L. Methods of meta-analysis. 2nd ed. (2004), Sage, Thousand Oaks, CA
12. Irvine S.H. The foundation of item generation. In: Irvine S.H., and Kyllonen P.C. (Eds). Item generation and test development (2002), Erlbaum, Mahwah, NJ 3-34
13. Irvine S.H., Dann P.L., and Anderson J.D. Towards a theory of algorithm-determined cognitive test construction. British Journal of Psychology 81 (1990) 173-195
14. Irvine S.H., and Kyllonen P.C. Item generation and test development (2002), Erlbaum, Mahwah, NJ
15. Kulik J.A., Kulik C.C., and Bangert R.L. Effects of practice on aptitude and achievement test scores. American Educational Research Journal 21 (1984) 435-447
16. Kyllonen P.C. Item generation for repeated testing of human performance. In: Irvine S.H., and Kyllonen P.C. (Eds). Item generation and test development (2002), Erlbaum, Mahwah, NJ 251-275
17. Kyllonen P.C., and Christal R.E. Reasoning ability is (little more than) working-memory capacity?. Intelligence 14 (1990) 389-433
18. Lance C.E., Lambert T.A., Gewin A.G., Lievens F., and Conway J.M. Revised estimates of dimension and exercise variance components in assessment center post-exercise dimension ratings. Journal of Applied Psychology 89 (2004) 377-385
19. Lievens F., Buyse T., and Sackett P.R. The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology 90 (2005) 442-452
20. Lievens F., Buyse T., and Sackett P.R. Retest effects in operational selection settings: Development and test of a framework. Personnel Psychology 58 (2005) 981-1007
21. Lievens F., and Sackett P.R. Video-based versus written situational judgment tests: A comparison in terms of predictive validity. Journal of Applied Psychology 91 (2006) 1181-1188
22. McDaniel M.A., Morgeson F.P., Finnegan E.B., Campion M.A., and Braverman E.P. Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology 86 (2001) 730-740
23. McDaniel M.A., and Nguyen N.T. Situational judgment tests: A review of practice and constructs assessed. International Journal of Selection and Assessment 9 (2001) 103-113
24. Meng X.L., Rosenthal R., and Rubin D.B. Comparing correlated correlation coefficients. Psychological Bulletin 111 (1992) 172-175
25. Messick S., and Jungeblut A. Time and method in coaching for the SAT. Psychological Bulletin 89 (1981) 191-216
26. Motowidlo S.J., Dunnette M.D., and Carter G.W. An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology 75 (1990) 640-647
27. Motowidlo S.J., Hooper A.C., and Jackson H.L. A theoretical basis for situational judgment tests. In: Weekley J., and Ployhart R. (Eds). Situational judgment tests (2006), Erlbaum, Mahwah, NJ 57-81
28. Nunnally J.C., and Bernstein I.H. Psychometric theory. 3rd ed. (1994), McGraw-Hill, New York
29. Oswald F.L., Friede A.J., Schmitt N., Kim B.K., and Ramsay L.J. Extending a practical method for developing alternate test forms using independent sets of items. Organizational Research Methods 8 (2005) 149-164
30. Oswald F.L., Schmitt N., Kim B.H., Ramsay L.J., and Gillespie M.A. Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology 89 (2004) 187-207
31. Ryan A.M., and Greguras G. Life is not multiple choice: Reactions to the alternatives. In: Hakel M.D. (Ed). Beyond multiple choice: Evaluating alternatives to traditional testing for selection (1998), Erlbaum, Mahwah, NJ 183-202
32. Sackett P.R. Performance assessment in education and professional certification: Lessons for personnel selection?. In: Hakel M.D. (Ed). Beyond multiple choice: Evaluating alternatives to traditional testing for selection (1998), Erlbaum, Mahwah, NJ 113-129
33. Sackett P.R., Burris L.R., and Ryan A.M. Coaching and practice effects in personnel selection. In: Cooper C.L., and Robertson I.T. (Eds). International review of industrial and organizational psychology (1989), Wiley, New York 145-183
34. Schmitt N., and Chan D. Situational judgment tests: Method or construct?. In: Weekley J., and Ployhart R.E. (Eds). Situational judgment tests (2006), Erlbaum, Mahwah, NJ 135-156
35. Shavelson R.J., Baxter G.P., Pine J., Yure J., Goldman S.R., and Smith B. Alternative technologies for large-scale science assessment: Instrument of education reform. School Effectiveness and School Improvement 2 (1991) 1-18
36. Shavelson R.J., Mayberry P.W., Li W., and Webb N.M. Generalizability of job performance measurements: Marine Corps rifleman. Military Psychology 2 (1990) 129-144
37. Swanson D.B., Norman G.R., and Linn R.L. Performance-based assessment: Lessons from the health professions. Educational Researcher 24 (1995) 5-11
38. Vu N.V., and Barrows H.S. Use of standardized patients in clinical assessments: Recent developments and measurement findings. Educational Researcher 23 (1994) 23-30