Situational judgment tests: The influence and importance of applicant status and targeted constructs on estimates of Black-White subgroup differences

Journal of Occupational and Organizational Psychology

Volumn 86, Issue 3, 2013, Pages 394-409

  Roth, Philip L ^a   Bobko, Philip ^b   Buster, Maury A ^c  

^a CLEMSON UNIVERSITY   (United States)

^b GETTYSBURG COLLEGE   (United States)

^c Alabama State Personnel Department   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84881525864     PISSN: 09631798     EISSN: 20448325     Source Type: Journal    
DOI: 10.1111/joop.12013     Document Type: Article

References (51)

1. Ackerman, P. L. (1992). Predicting individual differences in complex skill acquisition: Dynamics of ability determinants. Journal of Applied Psychology, 77, 598-614. doi:10.1037/0021-9010.77.5.598
2. Arthur, W., Jr, & Villado, A. J. (2008). The importance of distinguishing between constructs and methods when comparing predictors in personnel selection and research in practice. Journal of Applied Psychology, 93, 435-442. doi:10.1037/0021-9010.93.2.435
3. Bauer, T., & Truxillo, D. (2006). Applicant reactions to situational judgment tests: Research and related practical issues. In J. Weekley & R. Ployhart (Eds.), Situational judgment tests: Theory, measurement, & applications (pp. 233-249). Mahwah, NJ: Erlbaum.
4. Berry, C. M., Sackett, P. R., & Landers, R. N. (2007). Revisiting interview-cognitive ability relationships: Attending to specific range restriction mechanisms in meta-analysis. Journal of Applied Psychology, 60, 837-874. doi:10.1111/j.1744-6570.2007.00093.x
5. Bobko, P., & Roth, P. L. (2008). Psychometric accuracy and (the continuing need for) quality thinking in meta-analysis. Organizational Research Methods, 11, 114-126. doi:10.1177/1094428107303155
6. Bobko, P., & Roth, P. (2013). Reviewing, categorizing, and analyzing the literature on Black-White mean differences for predictors of job performance: Verifying some perceptions and updating/correcting others. Personnel Psychology, 66, 91-126. doi:10.1111/peps.12007
7. Bobko, P., Roth, P., & Bobko, C. (2001). Correcting the effect size of d for range restriction and unreliability. Organizational Research Methods, 4, 46-61. doi:10.1177/109442810141003
8. Breaugh, J. A. (2003). Effect size estimation: Factors to consider and mistakes to avoid. Journal of Management, 29, 79-97. doi:10.1177/014920630302900106
9. Brooks, M. E., & Highhouse, S. (2006). Can good judgment be measured? In J. Weekley & R. Ployhart (Eds.), Situational judgment tests: Theory, measurement, & applications (pp. 39-55). Mahwah, NJ: Erlbaum.
10. Campbell, J. P., McCloy, R. A., & Oppler, S. H. (1993). A theory of performance. In N. Schmitt & W. Borman (Eds.), Personnel selection in organizations. San Francisco, CA: Jossey Bass.
11. Christian, M. S., Edwards, B. D., & Bradley, J. C. (2010). Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validities. Personnel Psychology, 63, 83-117. doi:10.1111/j.1744-6570.2009.01163.x
12. Doverspike, D., Winter, J. L., Healy, M. C., & Barrett, G. V. (1996). Simulations as a method of illustrating the impact of differential weights on personnel selection outcomes. Human Performance, 9, 259-273. doi:10.1207/s15327043hup0903_5
13. Foldes, H. J., Duehr, E. E., & Ones, D. S. (2008). Group differences in personality: Meta-analysis comparing five US racial groups. Personnel Psychology, 61, 579-616. doi:10.1111/j.1744-6570.2008.00123.x
14. Goldstein, H. W., Yusko, K. P., & Nicolopoulos, V. (2001). Exploring black-white subgroup differences of managerial competencies. Personnel Psychology, 54, 783-807. doi:10.1111/j.1744-6570.2001.tb00232.x
15. Hanson, M. A., & Ramos, R. A. (1996). Situational judgment tests. In R. S. Barrett (Ed.), Fair employment strategies in human resource management (pp. 119-124). Westport, CT: Quorum Books/Greenwood.
16. Hough, L. M., Oswald, F. L., & Ployhart, R. E. (2001). Determinants, detection and amelioration of adverse impact in personnel selection procedures: Issues, evidence and lessons learned. International Journal of Selection and Assessment, 9, 152-194. doi:10.1111/1468-2389.00171
17. Huffcutt, A. I., Conway, J., Roth, P. L., & Stone, N. (2001). Identification and meta-analysis of constructs measured in employment interviews. Journal of Applied Psychology, 86, 897-913. doi:10.1037/0021-9010.86.5.897
18. Hunter, J. E. (1983a). A causal analysis of cognitive ability, job knowledge, and job performance, and supervisory ratings. In F. Landy, S. Zedeck & J. Cleveland (Eds.), Performance measurement and theory (pp. 257-266). Hillsdale, NJ: Erlbaum.
19. Hunter, J. E. (1983b). Test validation for 13, 000 jobs: An application of job classification and validity generalization for the General Aptitude Test Battery (GATB). (USES Test Research Report No. 450). Washington, DC: US Department of Labor.
20. Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72-98. doi:10.1037/0033-2909.96.1.72
21. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting for error and bias in research findings (2nd ed.). Newbury Park, CA: Sage.
22. Law, K. S., Schmidt, F. L., & Hunter, J. E. (1994). A test of two refinements in procedures for meta-analysis. Journal of Applied Psychology, 79, 978-986. doi:10.1037/0021-9010.79.6.978
23. McCloy, R. A., Campbell, J. P., & Cudeck, R. (1994). A confirmatory test of a model of performance determinants. Journal of Applied Psychology, 79, 493-505. doi:10.1037/0021-9010.79.4.493
24. McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & Grubb, L. (2007). Situational judgment tests, response instructions, and validity: A meta-analysis. Personnel Psychology, 60, 63-91. doi:10.1111/j.1744-6570.2007.00065.x
25. McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., Campion, M. A., & Braverman, E. P. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 730-740. doi:10.1037/0021-9010.86.4.730
26. McKay, P., & McDaniel, M. A. (2006). A re-examination of black-white mean differences in work performance: More data, more moderators. Journal of Applied Psychology, 91, 538-544. doi:10.1037/0021-9010.91.3.538
27. Motowidlo, S. J., Dunnette, M. J., & Carter, G. W. (1990). An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640-647. doi:10.1037/0021-9010.75.6.640
28. Motowidlo, S. J., Hooper, A. C., & Jackson, H. L. (2006). A theoretical basis for situational judgment tests. In J. Weekley & R. Ployhart (Eds.), Situational judgment tests: Theory, measurement, & applications (pp. 57-79). Mahwah, NJ: Erlbaum.
29. Mumford, T. V., Van Iddekinge, C. H., Morgeson, F. P., & Campion, M. A. (2008). The team role test: Development and validation of a team role knowledge situational judgment test. Journal of Applied Psychology, 93, 250-267. doi:10.1037/0021-9010.93.2.250
30. Murphy, K. R., & Myors, B. (2004). Statistical power analysis: A simple and general model for traditional and modern hypothesis tests (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
31. O'Connell, M. S., Hartman, N. S., McDaniel, M. A., Grubb, L., & Lawrence, A. (2007). Incremental validity of situational judgment tests for task and contextual job performance. International Journal of Selection and Assessment, 15, 19-29. doi:10.1111/j.1468-2389.2007.00364.x
32. Phillips, J. F. (1992). Predicting sales skills. Journal of Business and Psychology, 7, 151-160. doi:10.1007/BF01013925
33. Ployhart, R. E. (2006). Staffing in the 21st century: New challenges and strategic opportunities. Journal of Management, 32, 868-897. doi:10.1177/0149206306293625
34. Pulakos, E. D., & Schmitt, N. (1996). An evaluation of two strategies for reducing adverse impact and their effects on criterion-related validity. Human Performance, 9, 241-258. doi:10.1207/s15327043hup0903_4
35. Riggio, R. E. (1986). Assessment of basic social skills. Journal of Personality and Social Psychology, 5, 649-660. doi:10.1037/0022-3514.51.3.649
36. Riggio, R. E., & Riggio, H. R. (2001). Self-report measurement of interpersonal sensitivity. In J. Hall & F. Bernieri (Eds.), Interpersonal sensitivity theory and measurement (pp. 127-142). Mahwah, NJ: Erlbaum.
37. Roth, P. L., BeVier, C. A., Bobko, P., Switzer, F. S., III, & Tyler, P. (2001). Ethnic group differences in cognitive ability in employment and educational settings: A meta-analysis. Personnel Psychology, 54, 297-330. doi:10.1111/j.1744-6570.2001.tb00094.x
38. Roth, P. L., Bobko, P., & Buster, M. A. (2008). Applicant Black-White differences on Situational Judgment Tests: They may be surprising. In E. D. Scott (Chair), Legal issues for human resource management. Presented at the Academy of Management Meetings in Anaheim, CA.
39. Roth, P. L., Bobko, P., McFarland, L. A., & Buster, M. A. (2008). Work sample test ethnic group differences in personnel selection: A meta-analysis of overall and exercise scores. Personnel Psychology, 61, 635-659. doi:10.1111/j.1744-6570.2008.00125.x
40. Roth, P. L., Huffcutt, A. I., & Bobko, P. (2003). Ethnic group differences in measures of job performance: A new meta-analysis. Journal of Applied Psychology, 88, 694-706. doi:10.1111/j.1744-6570.2001.tb00094.x
41. Sackett, P. R., Borneman, M. J., & Connelly, B. S. (2008). High stakes testing in higher education and employment: Appraising the evidence for validity and fairness. American Psychologist, 63, 215-227. doi:10.1037/0003-066X.63.4.215
42. Sackett, P. R., & Ellingson, J. (1997). The effects of forming multi-predictor composites on group differences and adverse impact. Personnel Psychology, 50, 707-721. doi:10.1111/j.1744-6570.1997.tb00711.x
43. Sackett, P. R., Schmitt, N., Ellingson, J. E., & Kabin, M. E. (2001). High stakes testing in employment, credentialing, and higher education: Prospects in a post affirmative action world. American Psychologist, 56, 302-318. doi:10.1037/0003-066X.56.4.302
44. Schmidt, F. L., Hunter, J. E., & Outerbridge, A. N. (1986). Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of job performance. Journal of Applied Psychology, 71, 432-439. doi:10.1037/0021-9010.71.3.432
45. Stempler, S. E., & Sternberg, R. J. (2006). Using situational judgment tests to measure practical intelligence. In J. Weekley & R. Ployhart (Eds.), Situational judgment tests: Theory, measurement, & applications (pp. 107-131). Mahwah, NJ: Erlbaum.
46. Sternberg, R. J., & Wagner, R. K. (1993). The g-centric view of intelligence and job performance is wrong. Current Directions in Psychological Science, 2, 1-5. doi:10.1111/1467-8721.ep10770441
47. U.S. Equal Opportunity Employment Commission, U. S. Civil Service Commission, U.S. Department of Labor, U.S. Department of Justice. (1978). Uniform Guidelines on employee selection procedures. Federal Register, 43, 38295-38309.
48. Weekley, J. A., & Jones, C. (1997). Video-based situational testing. Personnel Psychology, 50, 25-49. doi:10.1111/j.1744-6570.1997.tb00899.x
49. Whetzel, D. L., & McDaniel, M. A. (2009). Situational judgment tests: An overview of current research. Human Resource Management Review, 19, 188-202. doi:10.1016/j.hrmr.2009.03.007
50. Whetzel, D. L., McDaniel, M. A., & Nguyen, N. T. (2008). Subgroup differences in situational judgment test performance. A meta-analysis. Human Performance, 21, 291-309. doi:10.1080/08959280802137820
51. Witt, L. A., & Ferris, G. A. (2003). Social skill as moderator of the conscientiousness-performance relationship: Convergent results across four studies. Journal of Applied Psychology, 88, 809-820. doi:10.1037/0021-9010.88.5.809