The Validity and Incremental Validity of Knowledge Tests, Low-Fidelity Simulations, and High-Fidelity Simulations for Predicting Job Performance in Advanced-Level High-Stakes Selection

Journal of Applied Psychology

Volumn 96, Issue 5, 2011, Pages 927-940

  Lievens, Filip ^a   Patterson, Fiona ^b  

^a GHENT UNIVERSITY   (Belgium)

^b CITY UNIVERSITY   (United Kingdom)

Author keywords

Assessment centers; High stakes selection; Medical education; Situational judgment tests

Indexed keywords

ADULT; APTITUDE; APTITUDE TEST; ARTICLE; DECISION MAKING; EDUCATION; FEMALE; HUMAN; KNOWLEDGE; MALE; MEDICAL SCHOOL; PHYSIOLOGY; PREDICTIVE VALUE; PROBLEM SOLVING; PSYCHOMETRY; REPRODUCIBILITY; STANDARD;

ADULT; APTITUDE; APTITUDE TESTS; FEMALE; HUMANS; JUDGMENT; KNOWLEDGE; MALE; PREDICTIVE VALUE OF TESTS; PROBLEM SOLVING; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS; SCHOOL ADMISSION CRITERIA; SCHOOLS, MEDICAL;

EID: 80052828201     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0023496     Document Type: Article

References (62)

1. Standards for educational and psychological testing 1999, American Educational Research Association, American Psychological AssociationAmerican Psychological Association, National Council on Measurement in EducationNational Council on Measurement in Education, American Educational Research Association, Washington, DC.
2. Arthur W., Day E.A., McNelly T.L., Edens P.S. A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology 2003, 56:125-153. 10.1111/j.1744-6570.2003.tb00146.x.
3. Arthur W., Villado A.J. The importance of distinguishing between constructs and methods when comparing predictors in personnel selection research and practice. Journal of Applied Psychology 2008, 93:435-442. 10.1037/0021-9010.93.2.435.
4. Bentler P.M. EQS: Structural equations program manual 1995, Multivariate Software, Encino, CA.
5. Bledow R., Frese M. A situational judgment test of personal initiative and its relationship to performance. Personnel Psychology 2009, 62:229-258. 10.1111/j.1744-6570.2009.01137.x.
6. Brannick M.T. Back to basics of test construction and scoring. Industrial and Organizational Psychology: Perspectives on Science and Practice 2008, 1:131-133. 10.1111/j.1754-9434.2007.00025.x.
7. Callinan M., Robertson I.T. Work sample testing. International Journal of Selection and Assessment 2000, 8:248-260. 10.1111/1468-2389.00154.
8. Christian M.S., Edwards B.D., Bradley J.C. Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validities. Personnel Psychology 2010, 63:83-117. 10.1111/j.1744-6570.2009.01163.x.
9. Clogg C.C., Petkova E., Haritou A. Statistical methods for comparing regression coefficients between models. American Journal of Sociology 1995, 100:1261-1293. 10.1086/230638.
10. Cullen M.J., Sackett P.R., Lievens F. Threats to the operational use of situational judgment tests in the college admission process. International Journal of Selection and Assessment 2006, 14:142-155. 10.1111/j.1468-2389.2006.00340.x.
11. Dayan K., Kasten R., Fox S. Entry-level police candidate assessment center: An efficient tool or a hammer to kill a fly?. Personnel Psychology 2002, 55:827-849. 10.1111/j.1744-6570.2002.tb00131.x.
12. Goldstein H.W., Zedeck S., Goldstein I.L. g: Is this your final answer?. Human Performance 2002, 15:123-142. 10.1207/S15327043HUP1501&02_08.
13. Goldstein I.L., Zedeck S., Schneider B. An exploration of the job analysis-content validity process (pp. 2-34). Personnel selection in organizations 1993, 3-34. Jossey-Bass, San Francisco, CA. N. Schmitt, W.C. Borman (Eds.).
14. Howard A. Making assessment centers work the way they are supposed to. Industrial and Organizational Psychology: Perspectives on Science and Practice 2008, 1:98-104. 10.1111/j.1754-9434.2007.00018.x.
15. Hunter J.E., Hunter R.F. Validity and utility of alternative predictors of job performance. Psychological Bulletin 1984, 96:72-98. 10.1037/0033-2909.96.1.72.
16. Guidelines and ethical considerations for assessment center operations. International Journal of Selection and Assessment 2009, 17:243-253. International Task Force on Assessment Center Guidelines. 10.1111/j.1468-2389.2009.00467.x.
17. Jansen P.G.W., Stoop B.A.M. The dynamics of assessment center validity: Results of a 7-year study. Journal of Applied Psychology 2001, 86:741-753. 10.1037/0021-9010.86.4.741.
18. Jefferis T. Selection for specialist training: What can we learn from other countries?. BMJ 2007, 334:1302-1304. 10.1136/bmj.39238.447338.AD.
19. Kanfer R., Ackerman P.L. Motivation and cognitive abilities: An integrative/aptitude-treatment interaction approach to skill acquisition. Journal of Applied Psychology 1989, 74:657-690. 10.1037/0021-9010.74.4.657.
20. Lance C.E. Why assessment centers do not work the way they are supposed to. Industrial and Organizational Psychology: Perspectives on Science and Practice 2008, 1:84-97. 10.1111/j.1754-9434.2007.00017.x.
21. Lane S., Stone C.A. Performance assessments. Educational measurement 2006, 387-431. American Council on Education, Westport, CT. B. Brennan (Ed.).
22. Lievens F., Buyse T., Sackett P.R. The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology 2005, 90:442-452. 10.1037/0021-9010.90.3.442.
23. Lievens F., Sackett P.R. Video-based versus written situational judgment tests: A comparison in terms of predictive validity. Journal of Applied Psychology 2006, 91:1181-1188. 10.1037/0021-9010.91.5.1181.
24. Lievens F., Sackett P.R., Buyse T. The effects of response instructions on situational judgment test performance and validity in a high-stakes context. Journal of Applied Psychology 2009, 94:1095-1101. 10.1037/a0014628.
25. Lievens F., Tett R.P., Schleicher D.J. Assessment centers at the crossroads: Toward a reconceptualization of assessment center exercises. Research in personnel and human resources management 2009, 99-152. JAI Press, Bingley, England. J.J. Martocchio, H. Liao (Eds.).
26. McCloy R.A., Campbell J.P., Cudeck R. A confirmatory test of a model of performance determinants. Journal of Applied Psychology 1994, 79:493-505. 10.1037/0021-9010.79.4.493.
27. McDaniel M.A., Hartman N.S., Whetzel D.L., Grubb W.L. Situational judgment tests, response instructions, and validity: A meta-analysis. Personnel Psychology 2007, 60:63-91. 10.1111/j.1744-6570.2007.00065.x.
28. McDaniel M.A., Morgeson F.P., Finnegan E.B., Campion M.A., Braverman E.P. Predicting job performance using situational judgment tests: A clarification of the literature. Journal of Applied Psychology 2001, 86:730-740. 10.1037/0021-9010.86.4.730.
29. Motowidlo S.J., Beier M.E. Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test. Journal of Applied Psychology 2010, 95:321-333. 10.1037/a0017975.
30. Motowidlo S.J., Crook A.E., Kell H.J., Naemi B. Measuring procedural knowledge more simply with a single-response situational judgment test. Journal of Business and Psychology 2009, 24:281-288. 10.1007/s10869-009-9106-4.
31. Motowidlo S.J., Dunnette M.D., Carter G.W. An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology 1990, 75:640-647. 10.1037/0021-9010.75.6.640.
32. Motowidlo S.J., Hooper A.C., Jackson H.L. Implicit policies about relations between personality traits and behavioral effectiveness in situational judgment items. Journal of Applied Psychology 2006, 91:749-761. 10.1037/0021-9010.91.4.749.
33. Olson-Buchanan J.B., Drasgow F., Moberg P.J., Mead A.D., Keenan P.A., Donovan M.A. Interactive video assessment of conflict resolution skills. Personnel Psychology 1998, 51:1-24. 10.1111/j.1744-6570.1998.tb00714.x.
34. Oswald F.L., Schmitt N., Kim B.H., Ramsay L.J., Gillespie M.A. Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology 2004, 89:187-207. 10.1037/0021-9010.89.2.187.
35. Patterson F., Baron H., Carr V., Plint S., Lane P. Evaluation of three short-listing methodologies for selection into postgraduate training in general practice. Medical Education 2009, 43:50-57. 10.1111/j.1365-2923.2008.03238.x.
36. Patterson F., Ferguson E., Lane P., Farrell K., Martlew J., Wells A. A competency model for general practice: Implications for selection and development. British Journal of General Practice 2000, 50:188-193.
37. Patterson F., Ferguson E., Norfolk T., Lane P. A new selection system to recruit GP registrars: Preliminary findings from a validation study. BMJ 2005, 330:711-714. 10.1136/bmj.330.7493.711.
38. Ployhart R.E., Holtz B.C. The diversity-validity dilemma: Strategies for reducing racio-ethnic and sex subgroup differences and adverse impact in selection. Personnel Psychology 2008, 61:153-172. 10.1111/j.1744-6570.2008.00109.x.
39. Raymond M.R. Job analysis and the specification of content for licensure and certification examinations. Applied Measurement in Education 2001, 14:369-415. 10.1207/S15324818AME1404_4.
40. Raymond M.R., Neustel S., Anderson D. Retest effects on identical and parallel forms in certification and licensure testing. Personnel Psychology 2007, 60:367-396. 10.1111/j.1744-6570.2007.00077.x.
41. Ree M.J., Carretta T.R., Earles J.A., Albert W. Sign changes when correcting for restriction of range: A note on Pearson's and Lawley's selection formulas. Journal of Applied Psychology 1994, 79:298-301. 10.1037/0021-9010.79.2.298.
42. Ree M.J., Carretta T.R., Teachout M.S. Role of ability and prior knowledge in complex training performance. Journal of Applied Psychology 1995, 80:721-730. 10.1037/0021-9010.80.6.721.
43. Rolfhus E.L., Ackerman P.L. Assessing individual differences in knowledge: Knowledge structures and traits. Journal of Educational Psychology 1999, 91:511-526. 10.1037/0022-0663.91.3.511.
44. Roth P., Bobko P., McFarland L.A. A meta-analysis of work sample test validity: Updating and integrating some classic literature. Personnel Psychology 2005, 58:1009-1037. 10.1111/j.1744-6570.2005.00714.x.
45. Ryan A.M., Greguras G.J. Life is not multiple choice: Reactions to the alternatives. Beyond multiple-choice: Alternatives to traditional testing 1998, 183-202. Erlbaum, Mahwah, NJ. M. Hakel (Ed.).
46. Sackett P.R. Performance assessment in education and professional certification: Lessons for personnel selection?. Beyond multiple choice tests: Evaluating alternatives to traditional testing for selection 1998, 113-129. Erlbaum, Mahwah, NJ. M.D. Hakel (Ed.).
47. Sackett P.R., Borneman M.J., Connelly B.S. High-stakes testing in higher education and employment: Appraising the evidence for validity and fairness. American Psychologist 2008, 63:215-227. 10.1037/0003-066X.63.4.215.
48. Sackett P.R., Lievens F. Personnel selection. Annual Review of Psychology 2008, 59:419-450. 10.1146/annurev.psych.59.103006.093716.
49. Sackett P.R., Schmitt N., Ellingson J.E., Kabin M.B. High-stakes testing in employment, credentialing, and higher education: Prospects in a post-affirmative action world. American Psychologist 2001, 56:302-318. 10.1037/0003-066X.56.4.302.
50. Sackett P.R., Yang H. Correction for range restriction: An expanded typology. Journal of Applied Psychology 2000, 85:112-118. 10.1037/0021-9010.85.1.112.
51. Salas E., Cannon-Bowers J.A. The science of training: A decade of progress. Annual Review of Psychology 2001, 52:471-499. 10.1146/annurev.psych.52.1.471.
52. Schmidt F.L., Hunter J.E. The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin 1998, 124:262-274. 10.1037/0033-2909.124.2.262.
53. Schmitt N., Chan D. Situational judgment tests: Method or construct?. Situational judgment tests: Theory, measurement, and practice 2006, 135-156. Erlbaum, Mahwah, NJ. J. Weekley, R.E. Ployhart (Eds.).
54. Schmitt N., Keeney J., Oswald F.L., Pleskac T., Quinn A., Sinha R., Zorzie M. Prediction of 4-year college student performance using cognitive and noncognitive predictors and the impact of demographic status on admitted students. Journal of Applied Psychology 2009, 94:1479-1497. 10.1037/a0016810.
55. Schmitt N., Mills A.E. Traditional tests and job simulations: Minority and majority performance and test validities. Journal of Applied Psychology 2001, 86:451-458. 10.1037/0021-9010.86.3.451.
56. Thorndike R.L. Personnel selection: Test and measurement techniques 1949, Wiley, New York, NY.
57. Thornton G.C., Cleveland J.N. Developing managerial talent through simulation. American Psychologist 1990, 45:190-199. 10.1037/0003-066X.45.2.190.
58. Thornton G.C., Rupp D. Assessment centers in human resource management: Strategies for prediction, diagnosis, and development 2006, Erlbaum, Mahwah, NJ.
59. Van Iddekinge C.H., Ployhart R.E. Developments in the criterion-related validation of selection procedures: A critical review and recommendations for practice. Personnel Psychology 2008, 61:871-925. 10.1111/j.1744-6570.2008.00133.x.
60. Weekley J.A., Ployhart R.E., Holtz B.C. On the development of situational judgment tests: Issues in item development, scaling, and scoring. Situational judgment tests 2006, 157-182. Jossey-Bass, San Francisco, CA. J.A. Weekley, R.E. Ployhart (Eds.).
61. Wernimont P.F., Campbell J.P. Signs, samples, and criteria. Journal of Applied Psychology 1968, 52:372-376. 10.1037/h0026244.
62. Whetzel D.L., McDaniel M.A. Situational judgment tests: An overview of current research. Human Resource Management Review 2009, 19:188-202. 10.1016/j.hrmr.2009.03.007.