Contextual effects on SJT responses: An examination of construct validity and mean differences across applicant and incumbent contexts

Human Performance

Volumn 23, Issue 1, 2009, Pages 1-21

  MacKenzie, William I ^a   Ployhart, Robert E ^a   Weekley, Jeff A ^b   Ehlers, Clay ^b  

^a UNIVERSITY OF SOUTH CAROLINA   (United States)

^b Kenexa Corporation   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 74949123089     PISSN: 08959285     EISSN: None     Source Type: Journal    
DOI: 10.1080/08959280903400143     Document Type: Article

References (47)

1. Arthur, W., Woehr, D. J., & Maldegen, R. (2000). Convergrent and discriminant validity of assessment center dimensions: A conceptual and empirical reexamination of the assessment confer construct-related validity paradox. Journal of Management, 26, 813-835.
2. Barrett, G. V., Phillips, J. S., & Alexander, R. A. (1981). Concurrent and predictive-validity designs: A critical reanalysis. Journal of Applied Psychology, 66(1), 1-6.
3. Borman, W. C., McKee, A. S., & Schneider, R. J. (1995). Work styles. In N. G. Peterson, M. D. Mumford, W. C. Borman, P. R. Jeanneret, & E. A. Fleishman (Eds.), Development of a prototype Occupational Information Network (O*NET) content model (Vols. 1-2, pp. 1-26). Salt Lake City: Utah Department of Employment Security.
4. Bott, J. P., O'Connell, M. S., Ramakrishnan, M., & Doverspike, D. (2007). Practical limitations in making decisions re-garding the distribution of applicant personality test scores based on incumbent data. Journal of Business and Psychology, 22, 123-134.
5. Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82(1), 143-159.
6. Chan, D., & Schmitt, N. (2002). Situational judgment and job performance. Human Performance, 15, 233-254.
7. Chan, D., & Schmitt, N. (2004). An agenda for future research on applicant reactions to selection procedures: A con-struct-oriented approach. International Journal of Selection and Assessment, 12(1-2), 9-23.
8. Clevenger, J., Pereira, G. M., Wiechmann, D., Schmitt, N., & Harvey, V. S. (2001). Incremental validity of situational judgment tests. Journal of Applied Psychology, 86, 410-417.
9. Harold, C. M., McFarland, L. A., & Weekley, J. A. (2006). The validity of verifiable and non-verifiable biodata items: An examination across applicants and incumbents. International Journal of Selection and Assessment, 14, 336-346.
10. Hooper, A. C., Cullen, M. J., & Sackett, P. R. (2006). Operational threats to the use of SJTs: Faking, coaching, and retesting issues. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement and application (pp. 205-232). Mahwah, NJ: Erlbaum.
11. Hough, L. M. (1998). Effects of intentional distortion in personality measurement and evaluation of suggested palliatives. Human Performance, 11, 209-244.
12. Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72-98.
13. Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537-567.
14. McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & Grubb, W. L. (2007). Situational judgment tests, response instructions, and validity: A meta-analysis. Personnel Psychology, 60(1), 63-91.
15. McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., Campion, M. A., & Braverman, E. P. (2001). Use of situational judg-ment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 730-740.
16. McDaniel, M. A., & Nguyen, N. T. (2001). Situational judgment tests: A review of practice and constructs assessed. International Journal of Selection and Assessment, 9(1-2), 103-113.
17. McDaniel, M. A., Whetzel, D. L., Hartman, N. S., Nguyen, N. T., & Grubb, W. L., III. (2006). Situational judgment tests: Validity and an integrative model. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement and application (pp. 183-204). Mahwah, NJ: Erlbaum.
18. Messick, S. (1995). Validity of psychological-assessment: Validation of inferences from persons responses and perfor-mances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
19. Mischel, W., & Shoda, Y. (1995). A cognitive-affective system theory of personality: Reconceptualizing situations, dispo-sitions, dynamics, and invariance in personality structure. Psychological Review, 102, 246-268.
20. Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640-647.
21. Motowidlo, S. J., Hanson, M. A., & Crafts, J. L. (1997). Low-fidelity simulation. In D. L. Whetzel & G. R. Wheaton (Eds.), Applied measurement methods in industrial psychology (pp. 241-260). Palo Alto, CA: Davies-Black.
22. Motowidlo, S. J., Hooper, A. C., & Jackson, H. L. (2006). A theoretical basis for situational judgment tests. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement and application (pp. 57-82). Mahwah, NJ: Erlbaum.
23. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. New York: McGraw-Hill.
24. O'Connell, M. S., Hartman, N. S., McDaniel, M. A., Grubb, W. L., & Lawrence, A. (2007). Incremental validity of situational judgment tests for task and contextual job performance. International Journa lof Selection and Assessment, 15 (1), 19-29.
25. Peeters, H., & Lievens, F. (2005). Situational judgment tests and their predictiveness of college students'success: The influence of faking. Educational and Psychological Measurement, 64(1), 70-89.
26. Ployhart, R. E. (2006). The predictor response process model. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement and application (pp. 83-106). Mahwah, NJ: Erlbaum.
27. Ployhart, R. E., & Weekley, J. A. (2006). Situational judgment: Some suggestions for future science and practice. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement and application (pp. 345-350). Mahwah, NJ: Erlbaum.
28. Ployhart, R. E., Weekley, J. A., Holtz, B. C., & Kemp, C. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: Are personality, biodata, and situational judgment tests comparable? Personnel Psychology, 56, 733-752.
29. Principles for the validation and use of personnel selection procedures. (4th ed.). (2003). Bowling Green, OH: Society for Industrial and Organizational Psychology.
30. Rosse, J. G., Stecher, M. D., Miller, J. L., & Levin, R. A. (1998). The impact of response distortion on preemployment personality testing and hiring decisions. Journal of Applied Psychology, 83, 634-644.
31. Roth, P. L., Bevier, C. A., Bobko, P., Switzer, F. S., & Tyler, P. (2001). Ethnic group differences in cognitive ability in employment and educational settings: A meta-analysis. Personnel Psychology, 54, 297-330.
32. Russell, C. J., & Domm, D. R. (1995). Two field-tests of an explanation of assessment-center validity. Journal of Occupational and Organizational Psychology, 68, 25-47.
33. Schmidt, F. L., Hunter, J. E., & Outerbridge, A. N. (1986). Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of job performance. Journal of Applied Psychology, 71, 432-439.
34. Schmit, M. J., & Ryan, A. M. (1993). The Big-f in personnel-selection-factor structure in applicant and nonapplicant populations. Journal of Applied Psychology, 78, 966-974.
35. Schmitt, N., & Chan, D. (2006). Situational judgment tests: Method or construct? In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement and application (pp. 135-155). Mahwah, NJ: Erlbaum.
36. Standards for educational and psychological testing. (1999). Washington, DC: American Educational Research Association.
37. Stark, S., Chernyshenko, O. S., Chan, K. Y., Lee, W. C., & Drasgow, F. (2001). Effects of the testing situation on item responding: Cause for concern. Journal of Applied Psychology, 86, 943-953.
38. Stokes, G. S., Hogan, J. B., & Snell, A. F. (1993). Comparability of incumbent and applicant samples for the development of biodata keys: The influence of social desirability. Personnel Psychology, 46, 739-762.
39. Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response.Cambridge, UK: Cambridge University Press.
40. United States Department of Labor. (2001). Occupational information network. Retrieved from http://online.onetcenter.org/
41. Weekley, J. A., & Jones, C. (1997). Video-based situational testing. Personnel Psychology, 50(1), 25-49.
42. Weekley, J. A., & Jones, C. (1999). Further studies of situational tests. Personnel Psychology, 52, 679-700.
43. Weekley, J. A., & Ployhart, R. E. (2005). Situational judgment: Antecedents and relationships with performance. Human Performance, 18(1), 81-104.
44. Weekley, J. A., & Ployhart, R. E. (2006). Situational judgment tests: Theory, measurement and application. Mahwah, NJ: Erlbaum.
45. Weekley, J. A., Ployhart, R. E., & Cooper-Hakim, A. (2005). On the development of measures of the O*NET work styles. Paper presented at the annual conference for the Society for Industrial and Organizational Psychology, April, Los Angeles, CA.
46. Weekley, J. A., Ployhart, R. E., & Harold, C. M. (2004). Personality and situational judgment tests across applicant and in-cumbent settings: An examination of validity, measurement, and subgroup differences. Human Performance, 17, 433-461.
47. Weekley, J. A., Ployhart, R. E., & Holtz, B. C. (2006). On the development of situational judgment tests: Issues in item development, scaling, and scoring. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement and application (pp. 157-182). Mahwah, NJ: Erlbaum.