Toward Better Meta-Analytic Matrices: How Input Values Can Affect Research Conclusions In Human Resource Management Simulations

Personnel Psychology

Volumn 64, Issue 4, 2011, Pages 899-935

  Roth, Philip L ^a   Switzer Iii, Fred S ^a   Van Iddekinge, Chad H ^b   Oh, In Sue ^c  

^a CLEMSON UNIVERSITY   (United States)

^b FLORIDA STATE UNIVERSITY   (United States)

^c VIRGINIA COMMONWEALTH UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 81855194752     PISSN: 00315826     EISSN: 17446570     Source Type: Journal    
DOI: 10.1111/j.1744-6570.2011.01231.x     Document Type: Article

References (98)

1. Aguinis H, Culpepper SA, Pierce CA. (2010). Revival of test bias research in pre-employment testing. Journal of Applied Psychology, 95, 648-680.
2. Aguinis H, Dalton DR, Bosco FA, Pierce CA, Dalton CM. (2010). Meta-analytic choices and judgment calls: Implications for theory building and testing, obtained effect sizes, and scholarly impact. Journal of Management, doi:.
3. Aguinis H, Pierce CA, Bosco FA, Dalton DR, Dalton CM. (2011). Debunking myths and urban legends about meta-analysis. Organizational Research Methods, 14, 306-331.
4. Aguinis H, Smith MA. (2007). Understanding the impact of test validity and bias on selection errors and adverse impact. Personnel Psychology, 60, 165-199.
5. Aguinis H, Werner S, Abbot JL, Angert C, Park JH, Kohlhausen D. (2010). Customer-centric science: Reporting significant research results with rigor, relevance, and practical impact in mind. Organizational Research Methods, 13, 515-539.
6. Aguinis H, Whitehead R. (1997). Sampling variance in the correlations coefficient under indirect range restriction: Implication for validity generalization. Journal of Applied Psychology, 82, 528-538.
7. Arthur W, Jr., Villado AJ. (2008). The importance of distinguishing between constructs and methods when comparing predictors in personnel selection and research in practice. Journal of Applied Psychology, 93, 435-442.
8. nd Ed). Reading, MA: Addison Wesley.
9. Barrick MR, Mount MK. (1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1-26.
10. Berry CM, Sackett PR. (2009). Faking in personnel selection: Tradeoffs in performance versus fairness resulting from two cut-score strategies. Personnel Psychology, 62, 835-863.
11. Berry CM, Sackett PR, Landers RN. (2007). Revisiting interview-cognitive ability relationships: Attending to specific range restriction mechanisms in meta-analysis. Personnel Psychology, 60, 837-874.
12. Bobko P, Roth PL, Potosky D. (1999). Derivation and implications of a meta-analytic matrix incorporating cognitive ability, alternative predictors, and job performance. Personnel Psychology, 62, 561-589.
13. Bobko P, Stone-Romero E. (1998). Meta-analysis is another useful research tool, but it is not a panacea. In Ferris GR (Ed.), Research in personnel and human resources management, (Vol. 16, pp. 359-397). Greenwich, CT: JAI Press.
14. Boudreau JW, Berger CJ. (1985). Decision-theoretic utility analysis applied to employee separations and acquisitions. Journal of Applied Psychology, 70, 581-612.
15. Bullock RJ, Svyantek DJ. (1985). Analyzing meta-analysis: Potential problems, an unsuccessful replication, and evaluation criteria, Journal of Applied Psychology, 70, 108-115.
16. Carson KP, Becker JS, Henderson JA. (1998). Is utility really futile? A failure to replicate and an extension. Journal of Applied Psychology, 83, 84-96.
17. Colquitt JA, LePine JA, Noe RA. (2000). Toward an integrative theory of training motivation: A meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 85, 678-707.
18. Converse PD, Peterson MH, Griffith RL. (2009). Faking on personality measures: Implications for selection involving multiple predictors. International Journal of Selection and Assessment, 17, 47-60.
19. Cortina JM, Landis RS. (2009). When small effect sizes tell a big story, and when large effect sizes don't. In Lance CE, Vandenberg RJ (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 287-308). New York, NY: Routledge/Taylor & Francis Group.
20. Dean M. (1999). On biodata construct validity, criterion validity and adverse impact. Unpublished doctoral dissertation, Louisiana State University.
21. DeCorte W. (1999). Weighing job performance predictors to both maximize the quality of the selected workforce and control the level of adverse impact. Journal of Applied Psychology, 84, 695-702.
22. De Corte W, Lievens F. (2003). A practical procedure to estimate the quality and the adverse impact of single-stage selection decisions. International Journal of Selection and Assessment, 11, 89-97.
23. DeCorte W, Lievens F, Sackett PR. (2006). Predicting adverse impact and mean criterion performance in multi-stage selection. Journal of Applied Psychology, 91, 523-537.
24. DeCorte W, Lievens F, Sackett PR. (2007). Combining predictors to achieve optimal trade-offs between selection quality and adverse impact. Journal of Applied Psychology, 92, 1380-1393.
25. Doverspike D, Winter JL, Healy MC, Barrett GV. (1996). Simulations as a method of illustrating the impact of differential weights on personnel selection outcomes. Human Performance, 9, 259-273.
26. Dunleavy EM, Stuebing KK, Campion JE, Glenn DM. (2008). Using the 4/5ths rule as an outcome in regression analysis: A demonstrative simulation. Journal of Business and Psychology, 23, 103-114.
27. Finch DM, Edwards BD, Wallace CJ. (2009). Multistage selection strategies: Simulating the effects on adverse impact and expected performance for various predictor combinations. Journal of Applied Psychology, 94, 318-340.
28. Foldes H, Duehr E, Ones D. (2008). Group differences in personality: Meta-analyses comparing five U.S. racial groups. Personnel Psychology, 61, 579-616.
29. Geyskens I, Krishnan R, Steenkamp JEM, Cunha PV. (2009). A review and evaluation of meta-analysis practices in management research. Journal of Management, 35, 393-419.
30. Hartigan JA, Wigdor AK. (1989). Fairness in employment testing: Validity generalization, minority issues, and the General Aptitude Test Battery. Washington, D.C.: National Academies Press.
31. Harrison JR, Carroll GR, Carley KM. (2007). Simulation modeling in organizational and management research. Academy of Management Review, 32, 1229-1245.
32. Hattrup K, Rock J. (2002). A comparison of predictor-based and criterion-based methods of weighting predictors to reduce adverse impact. Applied H.R.M. Research, 7, 22-38.
33. Hattrup K, Rock J, Scalia C. (1997). The effects of varying conceptualizations of job performance on adverse impact, minority hiring, and predicted performance. Journal of Applied Psychology, 82, 656-664.
34. Huffcutt AI, Arthur W Jr. (1994). Hunter and Hunter (1984) revisited: Interview validity for entry-level jobs. Journal of Applied Psychology, 79, 184-190.
35. Huffcutt AI, Roth PL, McDaniel MA. (1996). A meta-analytic investigation of cognitive ability in employment interview evaluations: Moderating characteristics and implications for incremental validity. Journal of Applied Psychology, 81, 459-473.
36. Hunter JE. (1986). Cognitive ability, cognitive aptitudes, job knowledge, and job performance. Journal of Vocational Behavior, 29, 340-362.
37. Hunter JE, Hunter RF. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72-98.
38. Hunter JE, Schmidt FL. (2004). Methods of meta-analysis: Correcting for error and bias in research findings (2nd ed). Newbury Park, CA: Sage.
39. Hurtz GM, Donovan JJ. (2000). Personality and job performance: The Big Five revisited. Journal of Applied Psychology, 85, 869-879.
40. Johnson JW. (2001). The relative importance of task and contextual performance dimensions to supervisor judgments of overall performance. Journal of Applied Psychology, 86, 984-996.
41. Judge TA, Jackson CL, Shaw JC, Scott BA, Rich BL. (2007). Self-efficacy and work-related performance: The integral role of individual differences. Journal of Applied Psychology, 92, 107-127.
42. Komar S, Brown DJ, Komar JA, Robie C. (2008). The impact of faking on the validity of conscientiousness: A Monte Carlo investigation. Journal of Applied Psychology, 93, 140-154.
43. Kriska SD. (2001, April). The validity-adverse impact trade-off: Real data and mathematical model estimates. Paper presented at the 16th Annual Conference of the Society for Industrial and Organizational Psychology, San Diego, CA.
44. Law KS, Schmidt FL, Hunter JE. (1994). A test of two refinements in procedures for meta-analysis. Journal of Applied Psychology, 79, 978-986.
45. Lee Y, Kozar KA, Larsen KRT. (2003). The technology acceptance model: Past, present, and future. Communications of AIS, 12, 752-780.
46. McDaniel MA, Hartman NS, Whetzel DL, Grubb L. (2007). Situational judgment tests, response instructions, and validity: A meta-analysis. Personnel Psychology, 60, 63-91.
47. McDaniel MA, Morgeson FP, Finnegan EB, Campion MA, Braverman EP. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 730-740.
48. McDaniel MA, Rothstein HR, Whetzel DL. (2006). Publication bias: A case study of four test vendors. Personnel Psychology, 59, 927-953.
49. McDaniel MA, Whetzel DL, Schmidt FL, Maurer S. (1994). The validity of the employment interview: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79, 599-616.
50. McFarland LA, Ryan AM. (2000). Variance in faking across noncognitive measures. Journal of Applied Psychology, 85, 812-821.
51. McKay PF. (2010). Perspectives on adverse impact in work performance: What we know and what we could learn more about. In Outtz J. (Ed.), Adverse impact: Implications for organizational staffing and high stakes selection (pp. 249-270). New York, NY: Routledge.
52. McKay PF, McDaniel MA. (2006). A reexamination of Black-White mean differences in work performance: More data, more moderators. Journal of Applied Psychology, 91, 538-554.
53. Murphy KR, Shiarella AH. (1997). Implications of the multidimensional nature of job performance for the validity of selection tests: Multivariate frameworks for studying test validity. Personnel Psychology, 50, 823-850.
54. Oh I-S, Wang G, Mount MK. (2011). Validity of observer ratings of the five-factor model of personality: A meta-analysis. Journal of Applied Psychology, 96, 762-773.
55. Ones DS.(1993). The construct validity of integrity? tests. Unpublished doctoral dissertation, University of Iowa.
56. Ones DS, Viswesvaran C. (1998). Gender, age, and race differences on overt integrity tests: Results across four large-scale applicant data sets. Journal of Applied Psychology, 83, 35-42.
57. Ones DS, Viswesvaran C, Schmidt FL. (1993). Comprehensive meta-analysis of integrity test validities: Findings and implications for personnel selection and theories of job performance. Journal of Applied Psychology, 78, 679-703.
58. Payne SC, Youngcourt SS, Beaubien JM. (2007). A meta-analytic examination of the goal orientation nomological net. Journal of Applied Psychology, 92, 128-150.
59. Ployhart RE, Holtz BC. (2008). The diversity-validity dilemma: Strategies for reducing racioethnic and sex subgroup differences and adverse impact in selection. Personnel Psychology, 61, 153-172.
60. Potosky DP, Bobko P, Roth PL. (2005). Forming composites of cognitive ability and alternative measures to predict job performance and reduce adverse impact: Corrected estimates and realistic expectations. International Journal of Selection and Assessment, 13, 304-315.
61. Reilly RR, Warech MA. (1993). The validity and fairness of alternatives to cognitive ability tests. In Wing L, Gifford B (Eds.), Policy issues in employment testing. Boston, MA: Kluwer.
62. Rich JR, Boudreau JW. (1987). The effects of variability and risk in selection utility analysis: An empirical comparison. Personnel Psychology, 40, 55-84.
63. Roth PL, BeVier CA, Bobko P, Switzer FS III, Tyler P. (2001). Ethnic group differences in cognitive ability in employment and educational settings: A meta-analysis. Personnel Psychology, 54, 297-330.
64. Roth PL, Bobko P, McFarland LA. (2005). A meta-analysis of work sample test validity: Updating some classic literature. Personnel Psychology, 58, 1009-1037.
65. Roth P, Bobko P, McFarland L, Buster M. (2008). Work sample tests in personnel selection: A meta-analysis of Black-White differences in overall and exercise scores. Personnel Psychology, 61, 637-662.
66. Roth PL, Bobko P, Switzer FS III, Dean MA. (2001). Prior selection causes biased estimates of standardized ethnic group differences: Simulation and analysis. Personnel Psychology, 54, 591-617.
67. Roth PL, Van Iddekinge CH, Huffcutt AI, Eidson CE, Jr., Bobko P. (2002). Correcting for range restriction in structured interview ethnic group differences: The values may be larger than we thought. Journal of Applied Psychology, 87, 369-376.
68. Rothstein H, Schmidt FL, Erwin F, Owens W, Sparks CP. (1990). Biographical data in employment selection: Can validities be made generalizable? Journal of Applied Psychology, 75, 174-184.
69. Sackett PR, Ellingson J. (1997). The effects of forming multi-predictor composites on group differences and adverse impact. Personnel Psychology, 50, 707-721.
70. Sackett PR, Lievens F. (2008). Personnel selection. Annual Review of Psychology, 59, 1-32
71. Sackett PR, Lievens F, Berry CM, Landers RN. (2007). A cautionary note on the effects of range restriction on predictor intercorrelations. Journal of Applied Psychology, 92, 538-544.
72. Sackett PR, Roth L. (1991). A Monte Carlo examination of banding and rank order methods of test score use in personnel selection. Human Performance, 4, 279-295.
73. Sackett PR, Roth L. (1996). Multistage selection strategies: A Monte Carlo investigation of effects on performance and minority hiring. Personnel Psychology, 49, 1-18.
74. Sackett PR, Schmitt N, Ellingson JE, Kabin ME. (2001). High stakes testing in employment, credentialing, and higher education: Prospects in a post affirmative action world. American Psychologist, 56, 302-318.
75. Sackett PR, Shen W. (2010). Subgroup differences on cognitive tests in contexts other than personnel selection. In Outtz J (Ed.), Adverse impact: Implications for organizational staffing and high stakes selection (pp. 323-346). New York, NY: Routledge.
76. Sackett PR, Wanek JE. (1996). New developments in the use of measures of honest, integrity, conscientiousness, dependability, trustworthiness, and reliability. Personnel Psychology, 49, 797-829.
77. Salgado JF, Anderson N, Moscoso S, Bertuna C, Fruyt F, Rolland JP. (2003). A meta-analytic study of general mental ability validity for different occupations in the European Community. Journal of Applied Psychology, 88, 1068-1081.
78. Salgado JF, Moscoso S. (2002). Comprehensive meta-analysis of the construct validity of the employment interview. European Journal of Work and Organizational Psychology, 11, 299-324.
79. Schmidt FL, Hunter JE. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-274.
80. Schmidt FL, Le H, Oh I-S, Shaffer JA. (2007). General mental ability, job performance, and red herrings - Responses to Osterman, Hauser, and Schmitt. Academy of Management Perspectives, 21, 64-76.
81. Schmidt FL, Mack MJ, Hunter JE. (1984). Selection utility in the occupation of US Park Ranger for three modes of test use. Journal of Applied Psychology, 69, 490-497.
82. Schmidt, FL, Oh I-S, Hayes TL. (2009). Fixed- versus random-effects models in meta-analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62, 97-128.
83. Schmidt FL, Oh, I-S, Le H. (2006). Increasing the accuracy of corrections for range restriction: Implications for selection procedure validities and other research results. Personnel Psychology, 59, 281-305.
84. Schmidt FL, Shaffer JA, Oh I-S. (2008). Increased accuracy for range restriction corrections: Implications for the role of personality and general mental ability in job and training performance. Personnel Psychology, 61, 827-868.
85. Schmitt N. (2007). The value of personnel selection: Reflections on some remarkable claims. The Academy of Management Perspectives, 21, 19-23.
86. Schmitt N, Clause CS, Pulakos ED. (1996). Subgroup differences associated with different measures of some common job relevant constructs. In Cooper CL, Robertson IT (Eds.), International review of industrial and organizational psychology (pp. 115-139). New York, NY: Wiley.
87. Schmitt N, Keeney J, Oswald FL, Pleskac TJ, Billington AQ, Sinha R, Zorzie M. (2009). Prediction of 4-year college student performance using cognitive and noncognitive predictors and the impact on demographic status of admitted students. Journal of Applied Psychology, 94, 1479-1497.
88. Schmitt N, Oswald FL. (2006). The impact of corrections for faking on the validity of noncognitive measures in selection settings. Journal of Applied Psychology, 91, 613-621.
89. Schmitt N, Quinn A. (2010). Reductions in measured subgroup mean differences: What is possible? In Outtz J (Ed.), Adverse impact: Implications for organizational staffing and high stakes selection (pp. 425-452). New York, NY: Routledge.
90. Schmitt N, Rogers W, Chan D, Sheppard L, Jennings D. (1997). Adverse impact and predictive efficiency of various predictor combinations. Journal of Applied Psychology, 82, 719-730.
91. Schmitt N, Sackett PR, Ellingson JE. (2002). No easy solution to subgroup differences in high stakes testing. American Psychologist, 57, 305-306.
92. Scullen SE, Bergey PK, Aiman-Smith L. (2005). Forced choice distribution rating systems and the improvement of workforce potential: A baseline simulation. Personnel Psychology, 58, 1-32.
93. Stillwell WG, Seaver DA, Edwards W. (1981). A comparison of weight approximation techniques in multiattribute utility decision making. Organizational Behavior & Human Performance, 28, 62-77.
94. Tam AP, Murphy KR, Lyall JT. (2004). Can changes in differential dropout rates reduce adverse impact? A computer simulation study of a multi-wave selection system. Personnel Psychology, 57, 905-934.
95. Van Iddekinge CH, Raymark PH, Eidson CE, Jr., Attenweiler B. (2004). What do structured interviews really measure? The construct validity of behavior description interviews. Human Performance, 17, 71-93
96. Viswesvaran C, Ones DS. (1995). Theory testing: Combining psychometric meta-analysis and structural equations modeling. Personnel Psychology, 48, 865-885.
97. Whetzel D, McDaniel M, Nguyen N. (2008). Subgroup differences in situational judgment test performance: A meta-analysis. Human Performance, 21, 291-308.
98. Wanous JP, Sullivan SE, Malinak J. (1989). The role of judgment call in meta-analysis. Journal of Applied Psychology, 74, 259-264.