Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100

Medical Teacher

Volumn 38, Issue 1, 2016, Pages 3-17

  Patterson, Fiona ^a   Zibarras, Lara ^b   Ashworth, Vicki ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b CITY UNIVERSITY   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

BEHAVIOR; DECISION MAKING; EDUCATION; HEALTH CARE PERSONNEL; HUMAN; PERSONALITY; PROCEDURES; PSYCHOMETRY; REPRODUCIBILITY; STANDARDS;

BEHAVIOR; EDUCATIONAL MEASUREMENT; HEALTH PERSONNEL; HUMANS; JUDGMENT; PERSONALITY; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS; SCHOOL ADMISSION CRITERIA;

EID: 84941348986     PISSN: 0142159X     EISSN: 1466187X     Source Type: Journal    
DOI: 10.3109/0142159X.2015.1072619     Document Type: Article

References (117)

1. Albanese MA, Snow MH, Skochelak SE, Huggett KN, Farrell PM. 2003. Assessing personal qualities in medical school admissions. Acad Med 78(3):313-321.
2. Anderson N, Lievens F, van Dam K, Born M. 2006. A construct-driven investigation of gender differences in a leadership-role assessment center. J Appl Psychol 91(3):555-566.
3. Arnold J, Randall R, Patterson F, Silvester J, Robertson I, Cooper C, Burnes B, Swailes S, Harris D, Axtell C, Den Hartog D. 2010. Work psychology: Understanding human behaviour in the workplace. 5th ed. England: Pearson Education Limited.
4. Ashworth V, Fung K, Shaw R. 2014. National University of Singapore Situational Judgment Test and Focused Skills Assessment: Analysis & Evaluation.
5. Bauer TN, Truxillo DM, Sanchez RJ, Craig JM, Ferrara P, Campion MA. 2001. Applicant reactions to selection: Development of the selection procedural justice scale (SPJS). Pers Psychol 54:387-419.
6. Birkeland SA, Manson TM, Kisamore JL, Brannick MT, Smith MA. 2006. A meta-analytic investigation of job applicant faking on personality measures. Int J Sel Assess 14(4):317-335.
7. BMA. 2009. Equality and diversity in UK medical schools. London: BMA.
8. Campbell J, Gasser M, Oswald F. 1996. The substantive nature of job performance variability. In: Murphy KR, editor. Individual differences and behavior in organizations. San Francisco: Jossey-Boss. pp 258-299.
9. Cascio WF, Aguinis H. 2008. Staffing twenty-first-century organizations. Acad Manag Ann 2(1):133-165.
10. Catano VM, Brochu A, Lamerson CD. 2012. Assessing the reliability of situational judgment tests used in high-stakes situations. Int J Sel Assess 20(3):333-346.
11. Cavendish C. 2013. The Cavendish Review: An Independent Review into Healthcare Assistants and Support Workers in the NHS and social care settings. England: Department of Health.
12. Chambers BA. 2002. Applicant reactions and their consequences: Review, advice, and recommendations for future research. Int J Manag Rev 4(4):317-333.
13. Chan D, Schmitt N. 1997. Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions. J Appl Psychol 82(1): 143-159.
14. Chan D, Schmitt N. 2002. Situational judgment and job performance. Hum Perform 15(3):233-254.
15. Chan D, Schmitt N, Sacco JM, DeShon RP. 1998. Understanding pretest and posttest reactions to cognitive ability and personality tests. J Appl Psychol 83(3):471-485.
16. Christian MS, Edwards BD, Bradley JC. 2010. Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validities. Pers Psychol 63(1):83-117.
17. Clevenger J, Pereira GM, Wiechmann D, Schmitt N, Harvey VS. 2001. Incremental validity of situational judgment tests. J Appl Psychol 86(3):410-417.
18. Crook AE, Beier ME, Cox CB, Kell HJ, Hanks AR, Motowidlo SJ. 2011. Measuring relationships between personality, knowledge, and performance using single-response situational judgment tests. Int J Sel Assess 19(4):363-373.
19. Cullen MJ, Sackett PR, Lievens F. 2006. Threats to the operational use of situational judgment tests in the college admission process. Int J Sel Assess 14(2):142-155.
20. Dore KL, Reiter HI, Eva KW, Krueger S, Scriven E, Siu E, Hilsden S, Thomas J, Norman GR. 2009. Extending the interview to all medical school candidates-Computer-based multiple sample evaluation of noncognitive skills (CMSENS). Acad Med 84(10):S9-S12.
21. Escudier M, Flaxman C, Cousans F, Woolford M, Patterson F. 2015. Pilot results from a new assessment of professionalism. Proceedings of the AMEE Conference 2015, Glasgow, UK.
22. Eva KW, Reiter HI, Trinh K, Wasi P, Rosenfeld J, Norman GR. 2009. Predictive validity of the multiple mini-interview for selecting medical trainees. Med Educ 43(8):767-775.
23. Ferguson E, James D, Madeley L. 2002. Factors associated with success in medical school: Systematic review of the literature. BMJ 324(April): 952-957.
24. Ferguson E, James D, O'Hehir F, Sanders A. 2003. Pilot study of the roles of personality, references, and personal statements in relation to performance over the five years of a medical degree. BMJ. 326:429-432.
25. Ferguson E, Sanders A, O'Hehir F, James D. 2000. Predictive validity of personal statements and the role of the five-factor model of personality in relation to medical training. J Occup Organ Psychol 73(3):321-344.
26. Ferguson E, Semper H, Yates J, Fitzgerald JE, Skatova A, James D. 2014. The ''dark side'' and ''bright side'' of personality: When too much conscientiousness and too little anxiety are detrimental with respect to the acquisition of medical knowledge and skill. PLoS One 9(2):1-11.
27. Francis R. 2013. Report of the Mid Staffordshire NHS Foundation Trust Public Inquiry: Executive Summary. England: NHS Foundation Trust.
28. Gaugler BB, Rosenthal DB, Thornton GC, Bentson C. 1987. Meta-analysis of assessment center validity. J Appl Psychol 72(3):493-511.
29. Gilliland SW. 1993. The perceived fairness of selection systems: An organizational justice perspective. Acad Manag Rev 18(4):694-734.
30. Gray MJA. 1997. Evidence-based healthcare. London: Churchill Livingstone.
31. Hausknecht JP, Day DV, Thomas SC. 2004. Applicant reactions to selection procedures: An updated model and meta-analysis. Pers Psychol 57(3):639-683.
32. Hülsheger UR, Anderson N. 2009. Applicant perspectives in selection: Going beyond preference reactions. Int J Sel Assess 17(4):335-345.
33. Husbands A, Mathieson A, Dowell J, Cleland JA, MacKenzie R. 2014. Predictive validity of the UK clinical aptitude test in the final years of medical school: A prospective cohort study. BMC Med Educ 14(1):88.
34. James D, Yates J, Nicholson S. 2010. Comparison of A level and UKCAT performance in students applying to UK medical and dental schools in 2006: Cohort study. BMJ 340:c478.
35. Kerrin M, Aitkenhead A, Shaw R. 2014. Public Health Situational Judgement Test 2014: Final Report.
36. Kerrin M, Rowett E, Lopes S. 2015. The University of Nottingham: School of Veterinary Medicine & Science (SVMS): Situational Judgement Test Pilot & Operational Delivery 2014.
37. Kline P. 2000. Handbook of Psychological Testing, Vol. 12, 2nd ed. London, UK: Routledge.
38. Koczwara A, Ashworth V. 2013. Selection and assessment. In: Lewis R, Zibarras L, editors. Work and occupational psychology: Integrating theory and practice. London: SAGE. pp 295-342.
39. Koczwara A, Patterson F, Zibarras L, Kerrin M, Irish B, Wilkinson M. 2012. Evaluating cognitive ability, knowledge tests and situational judgement tests for postgraduate selection. Med Educ 46(4):399-408.
40. Kreiter C, Axelson R. 2013. A perspective on medical school admission research and practice over the last 25 years. Teach Learn Med 25(S1): S50-S56.
41. Libbrecht N, Lievens F, Carette B, Cote S. 2014. Emotional intelligence predicts success in medical school. Emotion 14(1):64-73.
42. Lievens F. 2013. Adjusting medical school admission: Assessing interpersonal skills using situational judgement tests. Med Educ 47:182-189.
43. Lievens F. 2014. Diversity in medical school admission: Insights from personnel recruitment and selection. Med Educ 49(1):7-20.
44. Lievens F, Buyse T, Sackett PR. 2005a. The operational validity of a videobased situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. J Appl Psychol 90(3):442-452.
45. Lievens F, Buyse T, Sackett PR. 2005b. Retest effects in operational selection settings: Development and test of a framework. Pers Psychol 58(4):981-1007.
46. Lievens F, Buyse T, Sackett PR, Connelly BS. 2012. The effects of coaching on situational judgment tests in high-stakes selection. Int J Sel Assess 20(3):272-282.
47. Lievens F, Patterson F. 2011. The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. J Appl Psychol 96(5):927-940.
48. Lievens F, Peeters H, Schollaert E. 2008. Situational judgment tests: A review of recent research. Pers Rev 37(4):426-441.
49. Lievens F, Sackett PR. 2006. Video-based versus written situational judgment tests: A comparison in terms of predictive validity. J Appl Psychol 91(5):1181-1188.
50. Lievens F, Sackett PR. 2007. Situational judgment tests in high-stakes settings: Issues and strategies with generating alternate forms. J Appl Psychol 92(4):1043-1055.
51. Lievens F, Sackett PR. 2012. The validity of interpersonal skills assessment via situational judgment tests for predicting academic success and job performance. J Appl Psychol 97(2):460-468.
52. Lopes S, Baron H, Patterson F. 2015a. Specialty recruitment assessment results & scoring: Psychiatry CT1 entry. Report to HEE.
53. Lopes S, Baron H, Patterson F. 2015b. Specialty recruitment assessment results & scoring: Ophthalmology ST1 entry. Report to HEE.
54. Luschin-Ebengreuth M, Dimai HP, Ithaler D, Neges HM, Reibnegger G. 2015. Situational judgment test as an additional tool in a medical admission test: An observational investigation. BMC Res Notes 8(1):1-7.
55. Martin M, Motowidlo SJ. 2010. A single-response SJT for measuring procedural knowledge for human factors professionals. Poster Session Presented at 25th Annual Meeting of the Society for Industrial Organizational Psychology, Atlanta, GA.
56. McCrae RR, Costa PT. 2008. Empirical and Theoretical status of the fivefactor model of personality traits. In: Boyle GJ, Matthews G, Saklofske DH, editors. SAGE handbook of personality theory and assessment. London: SAGE Publications. pp 273-294.
57. McDaniel MA, Hartman NS, Grubb, III WL. 2003. Situational judgment tests, knowledge, behavioral tendency and validity: A meta-analysis. 18th Annual Conference of the Society for Industrial Organizational Psychology Orlando.
58. McDaniel MA, Hartman NS, Whetzel D, Grubb III WL. 2007. Situational judgment tests, response instructions, and validity: A meta-analysis. Pers Psychol 60(1):63-91.
59. McDaniel MA, Morgeson FP, Finnegan EB, Campion MA, Braverman EP. 2001. Use of situational judgment tests to predict job performance: A clarification of the literature. J Appl Psychol 86(4):730-740.
60. McDaniel MA, Nguyen NT. 2001. Situational judgment tests: A review of practice and constructs assessed. Int J Sel Assess 9(1/2):103-113.
61. McDaniel MA, Whetzel D. 2007. Situational judgment tests. In: Whetzel DL, Wheaton GR, editors. Applied measurement: Industrial psychology in human resources management. Mahwah, NJ: Lawrence Erlbaum and Associates. pp 235-257.
62. McDaniel MA, Whetzel D, Schmidt FL, Maurer SD. 1994. The validity of employment interviews: A comprehensive review and meta-analysis. J Appl Psychol 79(4):599-616.
63. McManus IC, Dewberry C, Nicholson S, Dowell J. 2013. The UKCAT-12 study: Educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of 12 UK medical schools. BMC Med 11:244.
64. McManus IC, Woolf K, Dacre J. 2008. Even one star at A level could be ''too little, too late'' for medical student selection. BMC Med Educ 8: 1-4.
65. Motowidlo SJ. 2003. Handbook of psychology. Hoboken, NJ: John Wiley & Sons, Inc.
66. Motowidlo SJ, Beier ME. 2010. Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test. J Appl Psychol 95(2):321-333.
67. Motowidlo SJ, Crook AE, Kell HJ, Naemi B. 2009. Measuring procedural knowledge more simply with a single-response situational judgment test. J Bus Psychol 24(3):281-288.
68. Motowidlo SJ, Dunnette MD, Carter GW. 1990. An alternative selection procedure: The low-fidelity simulation. J Appl Psychol 75(6): 640-647.
69. Motowidlo SJ, Hooper AC, Jackson HL. 2006. Implicit policies about relations between personality traits and behavioral effectiveness in situational judgment items. J Appl Psychol 91(4):749-761.
70. Mullins M, Schmitt N. 1998. Situational judgment testing: Will the real constructs please present themselves? 13th Annual Conference of the Society for Industrial and Organizational Psychology, Dallas, TX.
71. Nguyen NT, SBiderman MD, SMcDaniel MA. 2005. Effects of response instructions on faking a situational judgment test. Int J Sel Assess 13(4):250-260.
72. NHS. 2013. NHS Constitution. Englan: NHS.
73. O'Connell MS, Hartman NS, McDaniel MA, Grubb III WL, Lawrence A. 2007. Incremental validity of situational judgment tests for task and contextual job performance. Int J Sel Assess 15(1):19-29.
74. O'Neill L, Vonsild MC, Wallstedt B, Dornan T. 2013. Admission criteria and diversity in medical school. Med Educ 47(6):557-561.
75. Oswald F, Schmitt N, Kim B, Ramsay LJ, Gillespie MA. 2004. Developing a biodata measure and situational judgment inventory as predictors of college student performance. J Appl Psychol 89(2):187-207.
76. Parks L, Guay RP. 2009. Personality, values, and motivation. Pers Individ Dif 47(7):675-684.
77. Patterson F, Aitkenhead A, Edwards H, Flaxman C, Shaw R, Rosselli A. 2015a. Analysis of the situational judgement test for selection to the foundation programme 2015: Technical Report.
78. Patterson F, Ashworth V, Kerrin M, O'Neill P. 2013a. Situational judgement tests represent a measurement method and can be designed to minimise coaching effects. Med Educ 47(2):220-221.
79. Patterson F, Ashworth V, Mehra S, Falcon H. 2012a. Could situational judgement tests be used for selection into dental foundation training? Br Dent J 213(1):23-26.
80. Patterson F, Ashworth V, Zibarras L, Coan P, Kerrin M, O'Neill P. 2012b. Evaluations of situational judgement tests to assess non-academic attributes in selection. Med Educ 46:850-868.
81. Patterson F, Baron H, Carr V, Plint S, Lane P. 2009a. Evaluation of three short-listing methodologies for selection into postgraduate training in general practice. Med Educ 43(1):50-57.
82. Patterson F, Carr V, Zibarras L, Burr B, Berkin L, Plint S, Irish B, Gregory S. 2009b. New machine-marked tests for selection into core medical training: Evidence from two validation studies. Clin Med (Northfield, IL) 9(5):417-420.
83. Patterson F, Ferguson E, Lane P, Farrell K, Martlew J, Wells A. 2000. A competency model for general practice: Implications for selection, training, and development. Br J Gen Pract 50(452):188-193.
84. Patterson F, Ferguson E, Thomas S. 2008. Using job analysis to identify core and specific competencies: Implications for selection and recruitment. Med Educ 42(12):1195-1204.
85. Patterson F, Knight A, Dowell J, Nicholson S, Cousans F, Cleland JA. 2016. How effective are selection methods in medical education and training? Evidence from a systematic review. Med Educ [in press].
86. Patterson F, Lievens F, Kerrin M, Munro N, Irish B. 2013b. The predictive validity of selection for entry into postgraduate training in general practice: Evidence from three longitudinal studies. Br J Gen Pract 63(616):734-741.
87. Patterson F, Lievens F, Kerrin M, Zibarras L, Carette B. 2012c. Designing selection systems for medicine: The importance of balancing predictive and political validity in high-stakes selection contexts. Int J Sel Assess 20(4):486-496.
88. Patterson F, Martin S. 2014. UKCAT SJT: A study to explore validation methodology and early findings.
89. Patterson F, Martin S, Baron H, Fung K, Flaxman C. 2014. UKCAT situational judgement test: Technical report.
90. Patterson F, Prescott-Clements L, Zibarras L, Edwards H, Kerrin M, Cousans F. 2015b. Recruiting for values in healthcare: A preliminary review of the evidence. Adv Heal Sci Educ 1-23.
91. Patterson F, Zibarras L, Carr V, Irish B, Gregory S. 2011. Evaluating candidate reactions to selection practices using organisational justice theory. Med Educ 45(3):289-297.
92. Plint S, Patterson F. 2010. Identifying critical success factors for designing selection processes into postgraduate specialty training: The case of UK general practice. Postgrad Med J 86(1016):323-327.
93. Ployhart RE, Ehrhart MG. 2003. Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. Int J Sel Assess 11(1):1-16.
94. Ployhart RE, Weekley J, Holtz BC, Kemp C. 2003. Web-based and paperand-pencil testing of applicants in a proctored setting: Are personality, biodata, and situational judgment tests comparable? Pers Psychol 56(3):733-752.
95. Poole P, Moriarty HJ, Wearn AM, Wilkinson T, Weller JM. 2009. Medical student selection in New Zealand: Looking to the future. N Z Med J 122(1306):88-100.
96. Prideaux D, Roberts C, Eva KW, Centeno A, McCrorie P, McManus IC, Patterson F, Powis D, Tekian A, Wilkinson D. 2011. Assessment for selection for the health care professions and specialty training: Consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach 33(3):215-223.
97. Rankin B. 2013. Emotional intelligence: Enhancing values-based practice and compassionate care in nursing. J Adv Nurs 69(12):2717-2725.
98. Roberts C, Clark T, Burgess A, Frommer M, Grant M, Mossman K. 2014. The validity of a behavioural multiple-mini-interview within an assessment centre for selection into specialty training. BMC Med Educ 14:1-11.
99. Roberts C, Togno JM. 2011. Selection into specialist training programs: An approach from general practice. Med J Aust 194(2):93-95.
100. Roth PL, Bobko P, Buster MA. 2013. Situational judgment tests: The influence and importance of applicant status and targeted constructs on estimates of Black-White subgroup differences. J Occup Organ Psychol 86(3):394-409.
101. Rust J, Golombok S. 1999. Modern psychometrics: The science of psychological assessment, Vol. 2. London: Routledge.
102. Sartania N, McClure JD, Sweeting H, Browitt A. 2014. Predictive power of UKCAT and other pre-admission measures for performance in a medical school in Glasgow: A cohort study. BMC Med Educ 14(1):116-125.
103. Schmidt FL. 1994. The future of personnel selection in the U.S. Army. In Weekley JA, Ployhart RE, editors. Personnel selection and classification. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. pp 333-350.
104. Schmidt FL, Hunter JE. 1998. The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychol Bull 124(2):262-274.
105. Schmitt N, Chan D. 1999. The status of research on applicant reactions to selection tests and its implications for managers. Int J Manag Rev 1(1):45-62.
106. Sharma N, Nagle YK. 2015. Development of pictorial situational judgement test of affect. Psychology 6(March):400-408.
107. Simon E, Walsh K, Paterson-Brown F, Cahill D. 2015. Does a high ranking mean success in the situational judgement test? Clin Teach 12(1):42-45.
108. Stemig MS, Sackett PR, Lievens F. 2015. Effects of organizationally endorsed coaching on performance and validity of situational judgment tests. Int J Sel Assess 23(2):174-181.
109. St-Sauveur C, Girouard S, Goyette V. 2014. Use of situational judgment tests in personnel selection: Are the different methods for scoring the response options equivalent? Int J Sel Assess 22(3):225-239.
110. Wakeford R, Denney M-L, Ludka-Stempien K, Dacre J, McManus IC. 2015. Cross-comparison of MRCGP & MRCP(UK) in a database linkage study of 2,284 candidates taking both examinations: Assessment of validity and differential performance by ethnicity. BMC Med Educ 15(1):1-12.
111. Weekley J, Ployhart RE. 2005. Situational judgment: Antecedents and relationships with performance. Hum Perform 18(1):81-104.
112. Weekley J, Ployhart RE, Holtz BC. 2006. On the development of situational judgment tests: Issues in item development, scaling, and scoring. In: Weekley JA, Ployhart RE, editors. Situational judgment tests: Theory, measurement, and application. Mahwah, NJ: Lawrence Erlbaum Associates, Inc 157-182.
113. Wernimont P, Campbell J. 1968. Signs, samples, and criteria. J Appl Psychol 52(5):372-376.
114. Whetzel D, McDaniel MA. 2009. Situational judgment tests: An overview of current research. Hum Resour Manag Rev 19(3):188-202.
115. Whetzel D, McDaniel MA, Nguyen NT. 2008. Subgroup differences in situational judgment test performance: A meta-analysis. Hum Perform 21:291-309.
116. Woolf K, Potts HWW, McManus IC. 2011. Ethnicity and academic performance in UK trained doctors and medical students: Systematic review and meta-analysis. BMJ 342:d901.
117. Wyatt MRR, Pathak SB, Zibarras L. 2010. Advancing selection in an SME: Is best practice methodology applicable? Int Small Bus J 28(3):258-273.