Validating synthetic validation: Comparing traditional and synthetic validity coefficients

Personnel Psychology

Volumn 63, Issue 3, 2010, Pages 755-795

  Johnson, Jeff W ^a   Carter, Gary W ^a  

^a Personnel Decisions Research Institutes   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77955532362     PISSN: 00315826     EISSN: 17446570     Source Type: Journal    
DOI: 10.1111/j.1744-6570.2010.01186.x     Document Type: Article

References (75)

1. Aguinis H, Beaty JC, Boik RJ, Pierce CA. Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology 2005, 90:94-107.
2. Aguinis H, Stone-Romero EF. Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology 1997, 82:192-206.
3. Alexander RA, DeShon RP. Effect of error variance heterogeneity on the power of tests for regression slope differences. Psychological Bulletin 1994, 115:308-314.
4. Standards for educational and psychological testing 1999, American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, Washington, DC, American Educational Research Association
5. Bartlett CJ, Bobko P, Mosier SB, Hannan R. Testing for fairness with a moderated multiple regression strategy: An alternative to differential analysis. Personnel Psychology 1978, 31:233-241.
6. Bobko P, Roth PL, Buster MA. The usefulness of unit weights in creating composite scores: A literature review, application to content validity, and meta-analysis. Organizational Research Methods 2007, 10:689-709.
7. Borman WC, Buck DE, Hanson MA, Motowidlo SJ, Stark S, Drasgow F. An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales. Journal of Applied Psychology 2001, 86:965-973.
8. Borman WC, Motowidlo SJ. Expanding the criterion domain to include elements of contextual performance. Personnel selection in organizations 1993, 71-98. Schmitt N, Borman WC. San Francisco, Jossey-Bass
9. Campbell JP. The definition and measurement of performance in the new age. The changing nature of performance: Implications for staffing, motivation, and development 1999, 399-429. Ilgen DRPulakos ED. San Francisco, Jossey-Bass
10. Cascio WF. Applied psychology in personnel management 1987, 3rd ed., Englewood Cliffs, NJ, Prentice-Hall
11. Cascio WF, Valenzi ER, Silbey V. Validation and statistical power: Implications for applied research. Journal of Applied Psychology 1978, 63:589-595.
12. Colihan J, Burger GK. Constructing job families: An analysis of quantitative techniques used for grouping jobs. Personnel Psychology 1995, 48:563-586.
13. Cranny CJ, Doherty ME. Importance ratings in job analysis: Note on the misinterpretation of factor analyses. Journal of Applied Psychology 1988, 73:320-322.
14. Dansereau F, Graen G, Haga W. A vertical dyad approach to leadership in formal organizations. Organizational Behavior and Human Performance 1975, 13:46-78.
15. Darlington RB. Multiple regression in psychological research and practice. Psychological Bulletin 1968, 69:161-182.
16. DeShon RP, Alexander RA. Alternative procedures for testing regression slope homogeneity when group error variances are unequal. Psychological Methods 1996, 1:261-277.
17. Ghiselli EE, Campbell JP, Zedeck S. Measurement theory for the behavioral sciences 1981, Rev. ed., San Francisco, Freeman
18. Graen G. Role making processes within complex organizations. Handbook of industrial and organizational psychology 1976, Dunnette MD., Chicago, Rand McNally
19. Green SB, Stutzman T. An evaluation of methods to select respondents to structured job-analysis questionnaires. Personnel Psychology 1986, 39:543-564.
20. Hamilton JW, Dickinson TL. Comparison of several procedures for generating J-coefficients. Journal of Applied Psychology 1987, 72:49-54.
21. Harvey RJ. Quantitative approaches to job classification: A review and critique. Personnel Psychology 1986, 39:267-289.
22. Harvey RJ. Job analysis. Handbook of industrial & organizational psychology 1991, 71-163. Dunnette MDHough LM. In, Eds., 2nd ed., pp, Palo Alto, CA, Consulting Psychologists Press
23. Hesketh B, Neal A. Technology and performance. The changing nature of performance: Implications for staffing, motivation, and development 1999, 21-55. Ilgen DR, Pulakos ED. San Francisco, Jossey-Bass
24. Hoffman CC, Holden LM, Gale K. So many jobs, so little " N": Applying expanded validation models to support generalization of cognitive test validity. Personnel Psychology 2000, 53:955-991.
25. Hoffman CC, McPhail SM. Exploring options for supporting test use in situations precluding local validation. Personnel Psychology 1998, 51:987-1003.
26. Hollenbeck JR, Whitener EM. Criterion-related validation for small sample contexts: An integrated approach to synthetic validity. Journal of Applied Psychology 1988, 73:536-544.
27. Hough LM. I/Owes its advances to personality. Applied personality psychology: The intersection of personality and industrial/organizational psychology 2001, 19-44. Roberts BWHogan RT. Washington, DC, APA
28. Hough LM, Ones DS. The structure, measurement, validity, and use of personality variables in industrial, work, and organizational psychology. Handbook of industrial, work & organizational psychology 2001, 1:233-277. Anderson NR, Ones DSSinangil HKViswesvaran C. In, Eds., Vol, pp, New York, Sage
29. Howell DC. Statistical methods for psychology 1987, 2nd ed., Boston, Duxbury Press
30. James LR, Demaree RG, Wolf G. Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology 1984, 69:85-98.
31. Jeanneret PR. Applications of job component/synthetic validity to construct validity. Human Performance 1992, 5:81-96.
32. Johnson JW. A heuristic method for estimating the relative weight of predictor variables in multiple regression. Multivariate Behavioral Research 2000a, 35:1-19.
33. Johnson JW. Factor analysis of importance ratings in job analysis: Note on the misinterpretation of Cranny and Doherty (1988). Organizational Research Methods 2000b, 3:267-284.
34. Johnson JW. The relative importance of task and contextual performance dimensions to supervisor judgments of overall performance. Journal of Applied Psychology 2001, 86:984-996.
35. Johnson JW. A procedure for conducting factor analysis with large numbers of variables 2003, April, Orlando, FL
36. Johnson JW. Synthetic validity: A technique of use (finally). Alternative validation strategies: Developing new and leveraging existing validity evidence 2007, 122-158. McPhail SM., pp, San Francisco, Jossey-Bass
37. Johnson JW, Carter GW, Davison HK, Oliver D. A synthetic validity approach to testing differential prediction hypotheses. Journal of Applied Psychology 2001, 86:774-780.
38. Johnson JW, Cochran CC. Studying the influence of stereotypes on personnel decisions in the real world. Industrial and Organizational Psychology: Perspectives on Science and Practice 2008, 1:423-425.
39. Johnson JW, LeBreton JM. History and use of relative importance indices in organizational research. Organizational Research Methods 2004, 7:238-257.
40. Johnson JW, Paullin C, Hennen M. Validation and development of an operational version of Exam 473. Development and validation of Exam 473 for the United States Postal Service 2005, Paullin C, Johnson JW, In, Eds., Institute Report #496, pp. 39-131, Minneapolis, MN, Personnel Decisions Research Institutes
41. Johnson JW, Steel P, Scherbaum CA, Hoffman CC, Jeanneret PR, Foster J. Validation is like motor oil: Synthetic is better. Industrial and Organizational Psychology: Perspectives on Science and Practice, 3 2010, 305-328.
42. Katz D, Kahn RL. The social psychology of organizations 1978, New York, Wiley
43. Lawshe CH. Employee selection. Personnel Psychology 1952, 6:31-34.
44. Lievens F, Conway JM, De Corte W. The relative importance of task, citizenship, and counterproductive performance to job performance ratings: Do rater source and team-based culture matter. Journal of Occupational and Organizational Psychology 2008, 81:11-27.
45. London M, Mone EM. Continuous learning. The changing nature of performance: Implications for staffing, motivation, and development 1999, 119-153. Ilgen DR, Pulakos ED. San Francisco, Jossey-Bass
46. McCormick EJ. The development of processes for indirect or synthetic validity: III. Application of job analysis to indirect validity. A symposium. Personnel Psychology 1959, 12:402-413.
47. Morris DC, Hoffman CC, Shultz KS. A comparison of job components validity estimates to meta-analytic validity estimates 2003, April, Orlando, FL
48. Mossholder KW, Arvey RD. Synthetic validity: A conceptual and comparative review. Journal of Applied Psychology 1984, 69:322-333.
49. Murphy KR, DeShon R. Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology 2000, 53:873-900.
50. Murphy KR, Schiarella AH. Implications of the multidimensional nature of job performance for the validity of selection tests: Multivariate frameworks for studying test validity. Personnel Psychology 1997, 50:823-854.
51. Nunnally JC, Bernstein IH. Psychometric theory 1994, 3rd ed., New York, McGraw-Hill
52. Pearlman K, Schmidt FL, Hunter JE. Validity generalization results for tests used to predict job proficiency and training success in clerical occupations. Journal of Applied Psychology 1980, 65:373-406.
53. Peterson NG, Wise LL, Arabian J, Hoffman RG. Synthetic validation and validity generalization: When empirical validation is not possible. Exploring the limits of personnel selection and classification 2001, 411-451. Campbell JP, Knapp DJ, Mahwah, NJ, Erlbaum
54. Primoff ES. The J-coefficient approach to jobs and tests. Personnel Administrator 1957, 20:31-40.
55. Primoff ES. Empirical validation of the J-coefficient. Personnel Psychology 1959, 12:413-418.
56. Pulakos ED, Arad S, Donovan MA, Plamondon KE. Adaptability in the workplace: Development of a taxonomy of adaptive performance. Journal of Applied Psychology 2000, 85:612-624.
57. Robie C, Johnson KM, Nilsen D, Hazucha J. The right stuff: Understanding cultural differences in leadership performance. Journal of Management Development 2001, 20:639-650.
58. Rotundo M, Sackett PR. The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy-capturing approach. Journal of Applied Psychology 2002, 87:66-80.
59. Russell CJ, Settoon RP, McGrath RN, Blanton AE, Kidwell RE, Lohrke FT. Investigator characteristics as moderators of personnel selection research: A meta-analysis. Journal of Applied Psychology 1994, 79:163-170.
60. Sackett PR, Arvey RD. Selection in small N settings. Personnel selection in organizations 1993, 418-447. Schmitt N, Borman WC. San Francisco, Jossey-Bass
61. Salgado JF. Sample size in validity studies of personnel selection. Journal of Occupational and Organizational Psychology 1998, 71:161-164.
62. Sanchez JI, Prager I, Wilson A, Viswesvaran C. Understanding within-job title variance in job-analytic ratings. Journal of Business and Psychology 1998, 12:407-419.
63. Scherbaum CA. Synthetic validity: Past, present, and future. Personnel Psychology 2005, 58:481-515.
64. Schmidt FL. The relative efficiency of regression and simple unit predictor weights in applied differential psychology. Educational & Psychological Measurement 1971, 31:699-714.
65. Schmidt FL, Hunter JE, Croll PR, McKenzie RC. Estimation of employment test validities by expert judgment. Journal of Applied Psychology 1983, 68:590-601.
66. Schmidt FL, Hunter JE, Pearlman K. Progress in validity generalization: Comments on Callender and Osburn and further developments. Journal of Applied Psychology 1982, 67:835-845.
67. Schmitt N, Cohen SA. Internal analysis of task ratings by job incumbents. Journal of Applied Psychology 1989, 74:96-104.
68. Schneider RJ, Hough LM, Dunnette MD. Broadsided by broad traits: How to sink science in five dimensions or less. Journal of Organizational Behavior 1996, 17:639-655.
69. Principles for the validation and use of personnel selection procedures 2003, Society for Industrial and Organizational Psychology, Bowling Green, OH, SIOP
70. Steel PDG, Huffcutt AI, Kammeyer-Mueller J. From the work one knows the worker: A systematic review of the challenges, solutions, and steps to creating synthetic validity. International Journal of Selection and Assessment 2006, 14:16-36.
71. Steel P, Kammeyer-Mueller J. Using a meta-analytic perspective to enhance job component validation. Personnel Psychology 2009, 62:533-552.
72. Stone-Romero EF, Anderson LE. Relative power of moderated multiple regression and the comparison of subgroup correlation coefficients for detecting moderating effects. Journal of Applied Psychology 1994, 79:354-359.
73. Stutzman TM. Within classification job differences. Personnel Psychology 1983, 36:503-516.
74. Trattner MH. Synthetic validity and its application to the Uniform Guidelines validation requirements. Personnel Psychology 1982, 35:383-397.
75. Viswesvaran C, Ones DS. Theory testing: Combining psychometric meta-analysis and structural equations modeling. Personnel Psychology 1995, 48:865-885.