Threats to the operational use of situational judgment tests in the college admission process

International Journal of Selection and Assessment

Volumn 14, Issue 2, 2006, Pages 142-155

  Cullen, Michael J ^a   Sackett, Paul R ^b   Lievens, Filip ^c  

^a Personnel Decisions Research Institutes Inc   (United States)

^b UNIVERSITY OF MINNESOTA   (United States)

^c GHENT UNIVERSITY   (Belgium)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33845391974     PISSN: 0965075X     EISSN: 14682389     Source Type: Journal    
DOI: 10.1111/j.1468-2389.2006.00340.x     Document Type: Article

References (51)

1. Aiken, L. and West, S. (1991) Multiple regression: Testing and interpreting interactions, Newbury Park, CA: Sage.
2. Allalouf, A. and Ben-Shakhar, G. (1998) The effect of coaching on the predictive validity of scholastic aptitude tests. Journal of Educational Measurement, 35, 31-47.
3. Alliger, G. and Dwight, S. (2000) A meta-analytic investigation of the susceptibility of integrity tests to faking and coaching. Educational and Psychological Measurement, 60, 59-72.
4. Alliger, G., Lilienfeld, S. and Mitchell, K. (1996) The susceptibility of overt and covert integrity tests to coaching and faking. Psychological Science, 7, 32-39.
5. Bangert-Drowns, R.L., Kulik, J.A. and Kulik, C.C. (1983) Effects of coaching programs on achievement test performance. Review of Educational Research, 53, 571-585.
6. Bashi, Y. (1976) Verbal and non-verbal abilities of 4th, 6th and 8th grade students in the Arab educational system in Israel. Jerusalem: School of Education, Hebrew University.
7. Becker, B.J. (1990) Coaching for the scholastic aptitude test: Further synthesis and appraisal. Review of Educational Research, 60, 373-417.
8. Briggs, D.C. (2001) The effect of admissions test preparation: Evidence from NELS: 88. Chance, 14, 10-18.
9. Chan, D. and Schmitt, N. (2002) Situational judgment and job performance. Human Performance, 15, 233-254.
10. Clause, C.S., Mullins, M.E., Nee, M.T., Pulakos, E. and Schmitt, N. (1988) Parallel test form development: A procedure for alternative predictors and an example. Personnel Psychology, 51, 193-208.
11. Clevenger, J., Pereira, G.M., Wiechmann, D., Schmitt, N. and Harvey, V.S. (2001) Incremental validity of situational judgment tests. Journal of Applied Psychology, 86, 410-417.
12. Cohen, J. and Cohen, P. (1983) Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
13. Dansereau, D.F., McDonald, B.A., Collins, D.W., Garland, J., Holley, C.D., Diekhoff, G.M. and Evans, S.H. (1979) Evaluation of a teaching strategy system. In H.F. O'Neill and C.D. Spielberger (Eds), Cognitive and affective learning strategies (pp. 3-43). New York: Academic Press.
14. DerSimonian, R. and Laird, N.M. (1983) Evaluating the effect of coaching on SAT scores: A meta-analysis. Harvard Educational Review, 53, 1-15.
15. Feingold, A. (1994) Gender differences in personality: A meta-analysis. Psychological Bulletin, 116, 429-456.
16. Fisher, S.L. and Ford, J.K. (1998) Differential effects of learner effort and goal orientation on two learning outcomes. Personnel Psychology, 51, 397-420.
17. Goldberg, L.R. (1999) A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Miervielde, I. Deary, F. De Fryyt and F. Ostendorf (Eds), Personality Psychology in Europe, Vol. 7 (pp. 7-28). Tilburg, The Netherlands: Tilburg University Press.
18. Gray, L.E. (1983). Aptitude constructs, learning processes and achievement. Unpublished doctoral dissertation. Stanford University.
19. Hausknecht, J.P., Trevor, C.O. and Farr, J.L. (2002) Retaking ability tests in a selection setting: Implications for practice effects, training performance, and turnover. Journal of Applied Psychology, 87, 243-254.
20. Hedlund, J., Plamondon, K., Wilt, J., Nebel, K., Ashford, S. and Sternberg, R.J. (2001, April) Practical intelligence for business: Going beyond the GMAT. Paper presented in J. Cortina (Chair), Out with the old, in with the new. Looking above and beyond what we know about cognitive predictors. Symposium presented at the 16th Annual Convention of the Society for Industrial and Organizational Psychology, San Diego, CA.
21. Holden, R.R. and Jackson, D.N. (1979) Item subtletly, and face validity in personality assessment. Journal of Consulting and Clinical Psychology, 47, 459-468.
22. Hooper, A.C., Cullen, M.J. and Sackett, P.R. (2006) Operational threats to the use of SJTs: Faking, coaching and retesting issues. In R. Ployhart and J. Weekley (Eds), Situational judgment tests. San Francisco, CA: Jossey-Bass.
23. Hurtz, G.M. and Alliger, G.M. (2002) Influence of coaching on integrity test performance and unlikely virtues scale scores. Human Performance, 15, 255-273.
24. Jensen, A.R. (1998) The g factor. Westport, CT: Praeger.
25. Kanfer, R., Ackerman, P.L., Murtha, T.C., Dugdale, B. and Nelson, L. (1994) Goal setting, conditions of practice, and task performance: A resource allocation perspective. Journal of Applied Psychology, 79, 826-835.
26. Kelley, P.L., Jacobs, R.R. and Farr, J.L. (1994) Effects of multiple administrations of the MMPI for employee screening. Personnel Psychology, 47, 575-591.
27. Kulik, J.A., Bangert-Drowns, R.L. and Kulik, C.C. (1984) Effectiveness of coaching for aptitude tests. Psychological Bulletin, 95, 179-188.
28. Kulik, J.A., Kulik, C.C. and Bangert, R.L. (1984) Effects of practice on aptitude and achievement test scores. American Educational Research Journal, 21, 435-447.
29. Lievens, F., Buyse, T. and Sackett, P.R. (2005) The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology, 90, 442-452.
30. Marron, J.E. (1965) Preparatory school test preparation: Special test preparation, its effect on College Board scores and the relationship of affected scores to subsequent college performance. West Point, NY: United States Military Academy.
31. Maurer, T., Solamon, J., Andrews, K. and Troxtel, D. (2001) Interviewee coaching, preparation strategies, and response strategies in relation to performance in situational employment interviews: An extension of Maurer, Solamon, and Troxtel (1998). Journal of Applied Psychology, 86, 709-717.
32. Maurer, T., Solamon, J. and Troxtel, D. (1998) Relationship of coaching with performance in situational employment interviews. Journal of Applied Psychology, 83, 128-136.
33. McDaniel, M. and Nguyen, N. (2001) Situational judgment tests: A review of practice and constructs assessed. International Journal of Selection and Assessment, 9, 103-113.
34. McDaniel, M., Morgeson, F.P., Finnegan, E.B., Campion, M.A. and Braverman, E.P. (2001) Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 730-740.
35. Messick, S. and Jungeblut, A. (1981) Time and method in coaching for the SAT. Psychological Bulletin, 89, 191-216.
36. Motowidlo, S.J., Dunnette, M.D. and Carter, G.W. (1990) An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640-647.
37. Ortar, G.R. (1960) Improving test validity by coaching. Educational Research, 2, 137-142.
38. Oswald, F.L., Schmitt, N., Kim, B.H., Ramsay, L.J. and Gillespie, M.A. (2004) Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89, 187-208.
39. Powers, D.E. (1981) Students' use of and reactions to alternative methods of preparing for the SAT. Measurement and Evaluation in Guidance, 14, 118-126.
40. Powers, D.E. (1985) Effects of test preparation on the validity of Graduate Admission Test. Applied Psychological Measurement, 9, 179-190.
41. Powers, D.E. (1993) Coaching for the SAT: Summary of the summaries and an update. Educational Measurement: Issues and Practice, 12, 24-30.
42. Powers, D.E. and Rock, D.A. (1999) Effects of coaching on SAT 1: Reasoning test scores. Journal of Educational Measurement, 36, 93-118.
43. Rigney, J.W., Munro, A. and Crook, D.E. (1979) Teaching taskoriented selective reading. A learning strategy. In H.F. O'Neill and C.D. Spielberger (Eds), Cognitive and affective learning strategies (pp. 177-206). New York; Academic Press.
44. Roznowski, M. and Bassett, J. (1992) Training test-wiseness and flawed item types. Applied Measurement in Education, 5, 35-48.
45. Ryan, A.M., Ployhart, R.E., Greguras, G.J. and Schmit, M.J. (1998) Test preparation programs in selection contexts: Self-selection and program effectiveness. Personnel Psychology, 51, 599-621.
46. Sackett, P.R., Burris, L.R. and Ryan, A.M. (1989) Coaching and practice effects in personnel selection. In C.L. Cooper and I.T. Robertson (Eds), International review of industrial and organizational psychology 1989. Chichester: John Wiley.
47. Slack, N.V. and Porter, D. (1980) The Scholastic Aptitude Test: A critical appraisal. Harvard Educational Review, 50, 154-175.
48. Sternberg, R.J. and the Rainbow Project Collaborators (2002). Enhancing the SAT through assessments of analytical, practical, and creative skills. Unpublished manuscript.
49. Sternberg, R.J., Wagner, R.K., Williams, W.M. and Horvath, J.A. (1995) Testing common sense. American Psychologist, 50, 912-927.
50. Te Nijenhuis, J., Voskuijl, O.F. and Schijve, N.B. (2001) Practice and coaching on IQ tests: Quite a lot of g. International Journal of Selection and Assessment, 9, 302-308.
51. Van der Molen, H.T., Te Nijenhuis, J. and Keen, G. (1995) The effects of intelligence test preparation. European Journal of Personality, 9, 43-56.