The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains

Journal of Applied Psychology

Volumn 90, Issue 3, 2005, Pages 442-452

  Lievens, Filip ^a   Buyse, Tine ^a   Sackett, Paul R ^b  

^a GHENT UNIVERSITY   (Belgium)

^b UNIVERSITY OF MINNESOTA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADOLESCENT; ADULT; APTITUDE; ARTICLE; BELGIUM; CURRICULUM; DECISION MAKING; DENTAL STUDENT; EDUCATION; EDUCATIONAL STATUS; FEMALE; HUMAN; HUMAN RELATION; MALE; MEDICAL STUDENT; PROBABILITY; PSYCHOLOGIC TEST; PSYCHOLOGICAL ASPECT; PSYCHOMETRY; REPRODUCIBILITY; STATISTICS; VIDEORECORDING;

ADOLESCENT; ADULT; APTITUDE; BELGIUM; CURRICULUM; EDUCATIONAL STATUS; FEMALE; HUMANS; INTERPERSONAL RELATIONS; JUDGMENT; MALE; PROBABILITY; PSYCHOLOGICAL TESTS; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS; SCHOOL ADMISSION CRITERIA; STATISTICS; STUDENTS, DENTAL; STUDENTS, MEDICAL; VIDEO RECORDING;

EID: 20444436883     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/0021-9010.90.3.442     Document Type: Article

References (54)

1. Aguinis, H., Pierce, C. A., & Stone-Romero, E. F. (1994). Estimating the power to detect dichotomous moderators with moderated multiple regression. Educational and Psychological Measurement, 54, 690-692.
2. Blumberg, P. (2003). Multidimensional outcome considerations in assessing the efficacy of medical educational programs. Teaching and Learning in Medicine, 15, 210-214.
3. Borman, W. C., & Motowidlo, S. J. (1993). Expanding the criterion domain to include elements of contextual performance. In N. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations (pp. 71-98). San Francisco: Jossey-Bass.
4. Campbell, J. P., McCloy, R. A., Oppler, S. H., & Sager, C. E. (1993). A theory of performance. In N. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations (pp. 35-70). San Francisco: Jossey-Bass.
5. Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82, 143-159.
6. Chan, D., & Schmitt, N. (2002). Situational judgment and job performance. Human Performance, 15, 233-254.
7. Clause, C. C., Mullins, M. E., Nee, M. T., Pulakos, E. D., & Schmitt, N. (1998). Parallel test form development: A procedure for alternative predictors and an example. Personnel Psychology, 51, 193-208.
8. Clevenger, J., Pereira, G. M., Wiechmann, D., Schmitt, N., & Schmidt-Harvey, V. S. (2001). Incremental validity of situational judgment tests. Journal of Applied Psychology, 86, 410-417.
9. De Corte, W. (1999). Weighing job performance predictors to both maximize the quality of the selected workforce and control the level of adverse impact. Journal of Applied Psychology, 84, 695-702.
10. Edens, P. S., & Arthur, W. (2000, April). A meta-analysis investigating the susceptibility of self-report inventories to distortion. Paper presented at the 15th Annual Conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
11. Goldstein, H. W., Zedeck, S., & Goldstein, I. L. (2002). g: Is this your final answer. Human Performance, 15, 123-142.
12. Green, A., Peters, T. J., & Webster, D. J. (1991). An assessment of academic performance and personality. Medical Education, 25, 343-348.
13. Green, A., Peters, T. J., & Webster, D. J. (1993). Preclinical progress in relation to personality and academic profiles. Medical Education, 27, 137-142.
14. Haas, A., & McDaniel, M. A. (1999, April). Faking strategies: Effects on a situational judgment test. Paper presented at the 14th Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta, GA.
15. Hattrup, K., Rock, J., & Scalia, C. (1997). The effects of varying conceptualizations of job performance on adverse impact, minority hiring, and predictor performance. Journal of Applied Psychology, 82, 656-664.
16. Hedlund, J., Plamondon, K., Wilt, J., Nebel, K., Ashford, S., & Sternberg, R. J. (2001, April). Practical intelligence for business: Going beyond the GMAT. In J. Cortina (Chair), Out with the old, in with the new: Looking above and beyond what we know about cognitive predictors. Symposium conducted at the 16th Annual Convention of the Society for Industrial and Organizational Psychology, San Diego, CA.
17. Jansen, P. G. W., & Stoop, B. A. M. (2001). The dynamics of assessment center validity, results of a 7-year study. Journal of Applied Psychology, 86, 741-753.
18. Johnson, J. W. (2001). The relative importance of task and contextual performance dimensions to supervisor judgments of overall performance. Journal of Applied Psychology, 86, 984-996.
19. Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127, 162-181.
20. Lievens, F., & Coetsier, P. (2002). Situational tests in student selection: An examination of predictive validity, adverse impact, and construct validity. International Journal of Selection and Assessment, 10, 245-257.
21. McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., Campion, M. A., & Braverman, E. P. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 730-740.
22. McDaniel, M. A., & Nguyen, N. T. (2001). Situational judgment tests: A review of practice and constructs assessed. International Journal of Selection and Assessment, 9, 103-113.
23. McHenry, J. J., Hough, L. M., Toquam, J. L., Hanson, M. A., & Ashworth, S. (1990). Project A validity results: The relationship between predictor and criterion domains. Personnel Psychology, 43, 335-354.
24. McManus, I. C. (1982). A-level grades and medical school admission. British Medical Journal, 284, 1654.
25. Minnaert, A. (1996). Academic performance, cognition, metacognition and motivation. Assessing freshmen characteristics on task: A validation and replication study in higher education. Unpublished doctoral dissertation, University of Louvain, Belgium.
26. Mitchell, K., Haynes, R., & Koenig, J. (1994). Assessing the validity of the updated medical college admission test. Academic Medicine, 69, 394-401.
27. Montague, W., & Odds, F. C. (1990). Academic selection criteria and subsequent performance. Medical Education, 24, 44-47.
28. Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640-647.
29. Motowidlo, S. J., & Tippins, N. (1993). Further studies of the low-fidelity simulation in the form of a situational inventory. Journal of Occupational and Organizational Psychology, 66, 337-344.
30. Murphy, K. R., & Shiarella, A. H. (1997). Implications of the multidimensional nature of job performance for the validity of selection tests: Multivariate framework for studying test validity. Personnel Psychology, 50, 823-854.
31. Nguyen, N. T., McDaniel, M. A., & Biderman, M. D. (2002, April). Response instructions in situational judgment tests: Effects on faking and construct validity. Paper presented at the 17th Annual Conference of the Society for Industrial and Organizational Psychology, Toronto, Ontario.
32. Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. J., & Gillespie, M. A. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89, 187-207.
33. Peeters, H., & Lievens, F. (2005). Situational judgment tests and their predictiveness of college students' success: The influence of faking. Educational and Psychological Measurement, 65, 70-89.
34. Powis, D. A. (1994). Selecting medical students. Medical Education, 28, 443-469.
35. Ree, M. J., Carretta, T. R., Earles, J. A., & Albert, W. (1994). Sign changes when correcting for restriction of range: A note on Pearson's and Lawley's selection formulas. Journal of Applied Psychology, 79, 298-301.
36. Reeve, C. L., & Hakel, M. D. (2002). Asking the right questions about g. Human Performance, 15, 47-74.
37. Reiter-Palmon, R., & Connelly, M. S. (2000). Item selection counts: A comparison of empirical key and rational scale validities in theory-based and non-theory-based item pools. Journal of Applied Psychology, 85, 143-151.
38. Roessler, R., Lester, J. W., Butler, W. T., Rankin, B., & Collins, F. (1978). Cognitive and non-cognitive variables in the prediction of preclinical performance. Journal of Medical Education, 53, 678-681.
39. Rosse, J. G., Stechner, M. D., Levin, R. A., & Miller, J. L. (1998). The impact of response distortion on preemployment personality testing and hiring decisions. Journal of Applied Psychology, 83, 634-644.
40. Rothstein, M. G., Paunonen, S. V., Rush, J. C., & King, G. A. (1994). Personality and cognitive ability predictors of performance in graduate business school. Journal of Educational Psychology, 86, 516-530.
41. Rotundo, M., & Sackett, P. R. (2002). The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy capturing approach. Journal of Applied Psychology, 87, 66-80.
42. Sackett, P. R. (2005). The performance-diversity tradeoff in admissions testing. In W. Camara & E. Kimmel (Eds.), Choosing students: Higher education admission tools for the 21st century (pp. 109-125). Mahwah, NJ: Erlbaum.
43. Sackett, P. R., Schmitt, N., Ellingson, J. E., & Kabin, M. B. (2001). High-stakes testing in employment, credentialing, and higher education: Prospects in a post-affirmative action world. American Psychologist, 56, 302-318.
44. Sackett, P. R., & Yang, H. (2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85, 112-118.
45. Schmitt, N., & Chan, D. (1998). Personnel selection: A theoretical approach. Thousands Oaks, CA: Sage.
46. Schmitt, N., Clause, C. S., & Pulakos, E. D. (1996). Subgroup differences associated with different measures of some common job relevant constructs. International Review of Industrial and Organizational Psychology, 11, 115-139.
47. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
48. Stauffer, J. M., & Mendoza, J. L. (2001). The proper sequence for correcting correlation coefficients for range restriction and unreliability. Psychometrika, 66, 1-6.
49. Teutsch, C. (2003). Patient-doctor communication. Medical Clinics of North America, 87, 1115-1145.
50. Tomlinson, R. W. S., Clack, G. B., Pettingale, K. W., Anderson, J., & Ryan, K. C. (1977). The relative role of "A" level chemistry, physics and biology in the medical course. Medical Education, 11, 103-108.
51. Vey, M. A., Ones, D. S., Hezlett, S. A., Kuncel, N. R., Vannelli, J. R., Briggs, K. H., & Campbell, J. P. (2003, April). Relationships among college grade indices: A meta-analysis examining temporal influences. Paper presented at the Annual Conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
52. Vu, N. V., Dawson-Saunders, B., & Barrows, H. S. (1987). Use of medical reasoning aptitude test to help predict performance in medical school. Journal of Medical Education, 62, 325-335.
53. Weekley, J. A., & Jones, C. (1997). Video-based situational testing. Personnel Psychology, 50, 25-49.
54. Weekley, J. A., & Jones, C. (1999). Further studies of situational tests. Personnel Psychology, 52, 679-699.