The Effects of Coaching on Situational Judgment Tests in High-stakes Selection

International Journal of Selection and Assessment

Volumn 20, Issue 3, 2012, Pages 272-282

  Lievens, Filip ^a   Buyse, Tine ^a   Sackett, Paul R ^b   Connelly, Brian S ^c  

^a GHENT UNIVERSITY   (Belgium)

^b UNIVERSITY OF MINNESOTA   (United States)

^c UNIVERSITY OF TORONTO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84864815331     PISSN: 0965075X     EISSN: 14682389     Source Type: Journal    
DOI: 10.1111/j.1468-2389.2012.00599.x     Document Type: Article

References (41)

1. Austin, P. C. (2009). Some methods of propensity-score matching had superior performance to others: Results of an empirical investigation and Monte Carlo simulations. Biometrical Journal, 51, 171-184.
2. Austin, P. C., Grootendorst, P., Normand, S. L. T., & Anderson, G. M. (2007). Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: A Monte Carlo study. Statistics in Medicine, 26, 754-768.
3. Bangert-Drowns, R. L., Kulik, J. A., & Kulik, C. C. (1983). Effects of coaching programs on achievement test performance. Review of Educational Research, 53, 571-585.
4. Becker, B. J. (1990). Coaching for the Scholastic Aptitude Test: Further synthesis and appraisal. Review of Educational Research, 60, 373-417.
5. Ceci, S. J., & Papierno, P. B. (2005). The rhetoric and reality of gap closing: When the 'have-nots' gain but the 'haves' gain even more. American Psychologist, 60, 149-160.
6. Christian, M. S., Edwards, B. D., & Bradley, J. C. (2010). Situational judgment tests: Construct assessed and a meta-analysis of their criterion-related validities. Personnel Psychology, 63, 83-117.
7. Cullen, M. J., Sackett, P. R., & Lievens, F. (2006). Threats to the operational use of situational judgment tests in the college admission process. International Journal of Selection and Assessment, 14, 142-155.
8. D'Agostino, R. B. Jr. (1998). Tutorial in biostatistics: Propensity score method for bias reduction in the comparison of a treatment to a non-randomized control group. Statistics in Medicine, 17, 2265-2281.
9. D'Agostino, R. B. Jr, & Rubin, D. B. (2000). Estimating and using propensity scores with partially missing data. Journal of the American Statistical Association, 95, 749-759.
10. DerSimonian, R., & Laird, N. M. (1983). Evaluating the effect of coaching on SAT scores: A meta-analysis. Harvard Educational Review, 53, 1-15.
11. Harder, V. S., Stuart, E. A., & Anthony, J. (2010). Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychological Methods, 15, 234-249.
12. Hausknecht, J. P., Halpert, J. A., Di Paolo, N. T., & Moriarty Gerrard, M. O. (2007). Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology, 92, 373-385.
13. Klein, C., DeRouin, R. E., & Salas, E. (2006). Uncovering workplace interpersonal skills: A review, framework, and research agenda. In G. P. Hodgkinson & J. K. Ford (Eds.), International review of industrial and organizational psychology, Vol. 21 (pp. 80-126). New York: Wiley & Sons, Ltd.
14. Kulik, J. A., Bangert-Drowns, R. L., & Kulik, C. C. (1984). Effectiveness of coaching for aptitude tests. Psychological Bulletin, 95, 179-188.
15. Lievens, F., Buyse, T., & Sackett, P. R. (2005a). The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology, 90, 442-452.
16. Lievens, F., Buyse, T., & Sackett, P. R. (2005b). Retest effects in operational selection settings: Development and test of a framework. Personnel Psychology, 58, 981- 1007.
17. Lievens, F., Buyse, T., & Sackett, P. R. (2009). The effects of response instructions on situational judgment test performance and validity in a high-stakes context. Journal of Applied Psychology, 94, 1095-1101.
18. Lievens, F., & Sackett, P. R. (2006). Video-based versus written situational judgment tests: A comparison in terms of predictive validity. Journal of Applied Psychology, 91, 1181-1188.
19. Lievens, F., & Sackett, P. R. (2007). Situational judgment tests in high stakes settings: Issues and strategies with generating alternate forms. Journal of Applied Psychology, 92, 1043- 1055.
20. McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & Grubb, W. L. (2007). Situational judgment tests, response instructions, and validity: A meta-analysis. Personnel Psychology, 60, 63- 91.
21. McDaniel, M. A., Psotka, J., Legree, P. J., Yost, A. P., & Weekley, J. A. (2011). Toward an understanding of situational judgment item validity and group differences. Journal of Applied Psychology, 96, 327-336.
22. Messick, S., & Jungeblut, A. (1981). Time and method in coaching for the SAT. Psychological Bulletin, 89, 191-216.
23. Motowidlo, S. J., & Beier, M. E. (2010). Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test. Journal of Applied Psychology, 95, 321-333.
24. Motowidlo, S. J., Hooper, A. C., & Jackson, H. L. (2006). Implicit policies about relations between personality traits and behavioral effectiveness in situational judgment items. Journal of Applied Psychology, 91, 749-761.
25. Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. J., & Gillespie, M. A. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89, 187-207.
26. Painter, J. (2004). Propensity Matching via SPSS . Available at http://www.unc.edu/~painter/SPSSsyntax/propen.txt (accessed 10 February 2009).
27. Patterson, F., Ferguson, E., Norfolk, T., & Lane, P. (2005). A new selection system to recruit GP registrars: Preliminary findings from a validation study. BMJ (Clinical Research Ed.), 330, 711-714.
28. Picard, O. (2007). SJTs for GPST/GPVTS shortlisting: Situational judgement tests, professional dilemmas (2nd ed.) (GP recruitment stage 2). London: ISC Medical.
29. Powers, D. E., & Rock, D. A. (1999). Effects of coaching on SAT I: Reasoning test scores. Journal of Educational Measurement, 36, 93-118.
30. Ramsay, L. J., Gillespie, M. A., Kim, B. H., Schmitt, N., Oswald, F. L., Drzakowski, S. M., & Friede, A. J. (2003). Identifying and preventing score inflation on biodata and situational judgment inventory items . Invited Presentation to the College Board, New York, NY.
31. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41-55.
32. Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79, 516-524.
33. Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688-701.
34. Ryan, A. M., Ployhart, R. E., Greguras, G. J., & Schmit, M. J. (2006). Test preparation programs in selection contexts: Self-selection and program effectiveness. Personnel Psychology, 51, 599-621.
35. Sackett, P. R., Burris, L. R., & Ryan, A. M. (1989). Coaching and practice effects in personnel selection. In C. L. Cooper & I. T. Robertson (Eds.), International review of industrial and organizational psychology (pp. 145-183). Chichester: John Wiley.
36. Schleicher, D. J., Van Iddekinge, C. H., Morgeson, F. P., & Campion, M. A. (2010). If at first you don't succeed, try, try again: Understanding race, gender, and age differences in retesting score improvement. Journal of Applied Psychology, 95, 603-617.
37. Schmitt, N., Keeney, J., Oswald, F. L., Pleskac, T. J., Billington, A. Q., Sinha, R., & Zorzie, M. (2009). Prediction of 4-year college student performance using cognitive and noncognitive predictors and the impact on demographic status of admitted students. Journal of Applied Psychology, 94, 1479-1497.
38. Schmitt, N., Oswald, F. L., Kim, B. H., Gillespie, M. A., Ramsay, L. J., & Yoo, T. Y. (2003). Impact of elaboration on socially desirable responding and the validity of biodata measures. Journal of Applied Psychology, 88, 979-988.
39. Slack, W. V., & Porter, D. (1980). The scholastic aptitude test: A critical appraisal. Harvard Educational Review, 50, 154-175.
40. Whetzel, D. L., & McDaniel, M. A. (2009). Situational judgment tests: An overview of current research. Human Resource Management Review, 19, 188-202.
41. Whetzel, D. L., McDaniel, M. A., & Nguyen, N. T. (2008). Subgroup differences in situational judgment test performance: A meta-analysis. Human Performance, 21, 291-309.