Corrections for Criterion Reliability in Validity Generalization: A False Prophet in a Land of Suspended Judgment

Industrial and Organizational Psychology

Volumn 7, Issue 4, 2014, Pages 478-500

  Lebreton, James M ^a   Scherer, Kelly T ^b   James, Lawrence R ^c  

^a PENNSYLVANIA STATE UNIVERSITY   (United States)

^b PURDUE UNIVERSITY   (United States)

^c GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84921517327     PISSN: 17549426     EISSN: 17549434     Source Type: Journal    
DOI: 10.1111/iops.12184     Document Type: Article

References (105)

1. Aguinis, H. A. (2013). Performance management (3rd ed.). Upper Saddle River, NJ: Prentice Hall.
2. Aiken, L. R. (1988). Psychological testing and assessment (6th ed.). Boston, MA: Allyn and Bacon.
3. Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Prospect Heights, IL: Waveland Press.
4. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: Author.
5. Anastasi, A. (1968). Psychological testing (3rd ed.). New York, NY: The MacMillan Company.
6. Austin, J. T., & Villanova, P. (1992). The criterion problem: 1917-1992. Journal of Applied Psychology, 77, 836-874.
7. Barrick, M. R., & Mount, M. K. (1991). The big five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1-25.
8. Bhaskar-Shrinivas, P., Harrison, D. A., Shaffer, M., & Luk, D. M. (2005). Input-based and time-based models of international adjustment: Meta-analytic evidence and theoretical extensions. Academy of Management Journal, 48, 257-281.
9. Binning, J. F., & Barrett, G. V. (1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74, 478-494.
10. Binning, J. F., & LeBreton, J. M. (2009). Coherent conceptualization is useful for many things, and understanding validity is one of them. Industrial and Organizational Psychology: Perspectives on Science and Practice, 2, 486-492.
11. Borman, W. C. (1991). Job behavior, performance, and effectiveness. In M. D. Dunnette, & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol 2, pp. 271-326). Palo Alto, CA: Consulting Psychologists Press.
12. Bowling, N. A., Eschleman, K. J., Wang, Q., Kirkendall, C., & Alarcon, G. (2010). A meta-analysis of the predictors and consequences of organization-based self-esteem. Journal of Occupational and Organizational Psychology, 83, 601-626.
13. Bradshaw, F. F. (1930). The American Council on Education rating scale. Archives of Psychology, 199, 80.
14. Campion, M. A., Campion, J. E., & Hudson, J. P., Jr. (1994). Structured interviewing: A note on incremental validity and alternative question types. Journal of Applied Psychology, 79, 998-1002.
15. Cardy, R. L., & Dobbins, G. H. (1994). Performance appraisal: Alternative perspectives. Cincinnati, OH: South-Western.
16. Cascio, W. F., & Aguinis, H. (2005). Applied psychology in human resource management (6th ed.). Upper Saddle River, NJ: Prentice Hall.
17. Chiaburu, D. S., & Harrison, D. A. (2008). Do peers make the place? Conceptual synthesis and meta-analysis of coworker effects on perceptions, attitudes, OCBs, and performance. Journal of Applied Psychology, 93, 1082-1103.
18. Colquitt, J. A., Conlon, D. E., Wesson, M. J., Porter, C. O. L. H., & Ng, K. Y. (2001). Justice at the millennium: A meta-analytic review of 25 years of organizational justice research. Journal of Applied Psychology, 86, 425-445.
19. Colquitt, J. A., Scott, B. A., & LePine, J. A. (2007). Trust, trustworthiness, and trust propensity: A meta-analytic test of their unique relationships with risk taking and job performance. Journal of Applied Psychology, 92, 909-927.
20. Cooper, W. H. (1981). Ubiquitous halo. Psychological Bulletin, 90, 218-244.
21. Cooper-Hakim, A., & Viswesvaran, C. (2005). The construct of work commitment: Testing an integrative framework. Psychological Bulletin, 131, 241-259.
22. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: John Wiley & Sons.
23. DeShon, R. P. (1998). A cautionary note on measurement error corrections in structural equation models. Psychological Measurement, 3, 412-423.
24. DeShon, R. P. (2003). A generalizability theory perspective on measurement error corrections in validity generalization. In K. R. Murphy (Ed.), Validity generalization: A critical review (pp. 365-402). Mahwah, NJ: Lawrence Erlbaum.
25. Erdogan, B., Bauer, T. N., Truxillo, D. M., & Mansfield, L. R. (2012). Whistle while you work: A review of the life satisfaction literature. Journal of Management, 38, 1038-1083.
26. Feldman, J. M. (1981). Beyond attribution theory: Cognitive processes in performance appraisal. Journal of Applied Psychology, 66, 127-148.
27. Ford, M. T., Cerasoli, C. P., Higgins, J. A., & Decesare, A. L. (2011). Relationships between psychological, physical, and behavioural health and work performance: A review and meta-analysis. Work & Stress, 25, 185-204.
28. Fried, Y. (1991). Meta-analytic comparison of the job diagnostic survey and job characteristics inventory as correlates of work satisfaction and performance. Journal of Applied Psychology, 76, 690-697.
29. Fried, Y., Shirom, A., Gilboa, S., & Cooper, C. L. (2008). The mediating effects of job satisfaction and propensity to leave on role stress-job performance relationships: Combining meta-analysis and structural equation modeling. International Journal of Stress Management, 15, 305-328.
30. Ghiselli, E. E., & Brown, C. W. (1955). Personnel and industrial psychology (2nd ed.). New York, NY: McGraw-Hill.
31. Ghiselli, E. E., Campbell, J. P., & Zedeck, S. (1981). Measurement theory for the behavioral sciences. San Francisco, CA: W. H. Freeman & Company.
32. Guilford, J. P. (1936). Psychometric methods. New York, NY: McGraw-Hill.
33. Guilford, J. P. (1954). Psychometric methods (2nd ed.). New York, NY: McGraw-Hill.
34. Guilford, J. P., & Fruchter, B. (1973). Fundamental statistics in education and psychology (5th ed.). New York, NY: McGraw-Hill.
35. Gulliksen, H. (1950). Theory of mental tests. New York, NY: John Wiley & Sons.
36. Hermelin, E., & Robertson, I. T. (2001). A critique and standardization of meta-analytic validity coefficients in personnel selection. Journal of Occupational and Organizational Psychology, 74, 253-277.
37. Hoobler, J. M., Hu, J., & Wilson, M. (2010). Do workers who experience conflict between the work and family domains hit a "glass ceiling"?: A meta-analytic investigation. Journal of Vocational Behavior, 77, 481-494.
38. Huffcutt, A. I., & Arthur, W., Jr. (1994). Hunter and Hunter (1984) revisited: Interview validity for entry-level jobs. Journal of Applied Psychology, 79, 184-190.
39. Huffcutt, A. I., Roth, P. L., & McDaniel, M. A. (1996). A meta-analytic investigation of cognitive ability in employment interview evaluations: Moderating characteristics and implications for incremental validity. Journal of Applied Psychology, 81, 459-473.
40. Hunter, J. E. (1983). Test validation for 12, 000 jobs: An application of job classification and validity generalization analysis to the general aptitude test battery. USES test research report no. 45. Division of Counseling and Test Development Employment and Training Administration. Washington, DC: U.S. Department of Labor.
41. Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72-98.
42. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.
43. James, L. R. (1973). Criterion models and construct validity for criteria. Psychological Bulletin, 80, 75-83.
44. James, L. R., & LeBreton, J. M. (2012). Assessing the implicit personality through conditional reasoning. Washington, DC: American Psychological Association.
45. James, L. R., Mulaik, S. A., & Brett, J. M. (1982). Causal analysis: Assumptions, models, and data. Beverly Hills, CA: Sage.
46. Joseph, D. L., & Newman, D. A. (2010). Emotional intelligence: An integrative meta-analysis and cascading model. Journal of Applied Psychology, 95, 54-78.
47. Lance, C. E., Butts, M. M., & Michels, L. C. (2006). The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods, 9, 202-220.
48. Landy, F. J. (1985). Psychology of work behavior (3rd ed.). Homewood, IL: Dorsey Press.
49. Landy, F. J. (1986). Stamp collecting versus science: Validation as hypothesis testing. American Psychologist, 41, 1183-1192.
50. Landy, F. J., & Farr, J. L. (1980). Performance ratings. Psychological Bulletin, 87, 72-107.
51. Lanier, L. H. (1927). Prediction of the reliability of mental tests and tests of special abilities. Journal of Experimental Psychology, 10, 69-113.
52. Latham, G. P., & Wexley, K. N. (1981). Increasing productivity through performance appraisal. Reading, MA: Addison-Wesley.
53. LeBreton, J. M., Burgess, J. R. D., Kaiser, R. B., Atchley, E. K. P., & James, L. R. (2003). The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar? Organizational Research Methods, 6, 80-128.
54. LeBreton, J. M., & Senter, J. L. (2008). Answers to twenty questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815-852.
55. LeBreton, J. M., & Tonidandel, S. (2008). Multivariate relative importance: Extending relative weight analysis to multivariate criterion spaces. Journal of Applied Psychology, 93, 329-345.
56. Lilienfeld, S. O., Wood, J. M., & Garb, H. N. (2000). The scientific status of projective techniques. Psychological Science in the Public Interest, 1, 27-66.
57. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
58. Margenau, H. (1950). The nature of physical reality. New York, NY: McGraw Hill.
59. McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
60. McNemar, Q. (1962). Psychological statistics (3rd ed.). New York, NY: John Wiley & Sons.
61. Meriac, J. P., Hoffman, B. J., Woehr, D. J., & Fleisher, M. S. (2008). Further evidence for the validity of assessment center dimensions: A meta-analysis of the incremental criterion-related validity of dimension ratings. Journal of Applied Psychology, 93, 1042-1052.
62. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
63. Michel, J. S., Mitchelson, J. K., Kotrba, L. M., LeBreton, J. M., & Baltes, B. B. (2009). A comparative test of work-family conflict models and critical examination of work-family linkages. Journal of Vocational Behavior, 74, 199-218.
64. Morgeson, F. P., Campion, M. A., Dipboye, R. L., Hollenbeck, J. R., Murphy, K., & Schmitt, N. (2007). Reconsidering the use of personality tests in personnel selection contexts. Personnel Psychology, 60, 683-729.
65. Muchinsky, P. M. (1996). The correction for attenuation. Educational and Psychological Measurement, 56, 63-75.
66. Murphy, K. R., & Balzer, W. K. (1989). Rater errors and rating accuracy. Journal of Applied Psychology, 74, 619-624.
67. Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage.
68. Murphy, K. R., & Davidshofer, C. O. (2005). Psychological testing: Principles and applications (6th ed.). Upper Saddle River, NJ: Prentice Hall.
69. Murphy, K. R., & DeShon, R. (2000a). Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53, 873-900.
70. Murphy, K. R., & DeShon, R. (2000b). Progress in psychometrics: Can industrial and organizational psychology catch up? Personnel Psychology, 53, 913-924.
71. Murphy, K. R., & Shiarella, A. H. (1997). Implications of the multidimensional nature of job performance for the validity of selection tests: Multivariate frameworks for studying test validity. Personnel Psychology, 50, 823-854.
72. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.
73. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.
74. Oh, I., Wang, G., & Mount, M. K. (2011). Validity of observer ratings of the five-factor model of personality traits: A meta-analysis. Journal of Applied Psychology, 96, 762-773.
75. Ones, D. S., Viswesvaran, C., & Schmidt, F. L. (1993). Comprehensive meta-analysis of integrity test validities: Findings and implications for personnel selection and theories of job performance. Journal of Applied Psychology, 78, 679-703.
76. Oswald, F. L., & McCloy, R. A. (2003). Meta-analysis and the art of the average. In K. R. Murphy (Ed.), Validity generalization: A critical review (pp. 311-338). Mahwah, NJ: Lawrence Erlbaum.
77. Ployhart, R. E., Schneider, B., & Schmitt, N. (2006). Staffing organizations: Contemporary practice and theory (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
78. Putka, D. J., Le, H., McCloy, R. A., & Diaz, T. (2008). Ill-structured measurement designs in organizational research: Implications for estimating interrater reliability. Journal of Applied Psychology, 93, 959-981.
79. Riggle, R. J., Edmondson, D. R., & Hansen, J. D. (2009). A meta-analysis of the relationship between perceived organizational support and job outcomes: 20 years of research. Journal of Business Research, 62, 1027-1030.
80. Rockstuhl, T., Dulebohn, J. H., Ang, S., & Shore, L. M. (2012). Leader-member exchange (LMX) and culture: A meta-analysis of correlates of LMX across 23 countries. Journal of Applied Psychology, 97, 1097-1130.
81. Rosenthal, R. (1984). Meta-analysis procedures for social research. Beverly Hills, CA: Sage.
82. Rothstein, H. R., Schmidt, F. L., Erwin, F. W., Owens, W. A., & Sparks, C. P. (1990). Biographical data in employment selection: Can validities be made generalizable? Journal of Applied Psychology, 75, 175-184.
83. Rotundo, M., & Sackett, P. R. (2002). The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance. Journal of Applied Psychology, 87, 66-80.
84. Rugg, H. O. (1921). Is the rating of human character practicable? Journal of Educational Psychology, 12, 425-438.
85. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.
86. Salas, E., Rozell, D., Mullen, B., & Driskell, J. E. (1999). The effect of team building on performance: An integration. Small Group Research, 30, 309-329.
87. Schmidt, F. L. (1992). What do data really mean?: Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173-1181.
88. Schmidt, F. L., & Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540.
89. Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-274.
90. Schmidt, F. L., Hunter, J. E., & Outerbridge, A. N. (1986). Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of job performance. Journal of Applied Psychology, 71, 432-439.
91. Schmidt, F. L., Viswesvaran, C., & Ones, D. S. (2000). Reliability is not validity and validity is not reliability. Personnel Psychology, 53, 901-912.
92. Schmitt, N., & Chan, D. (1998). Personnel selection: A theoretical approach. Thousand Oaks, CA: Sage.
93. Seymour, R. T. (1988). Why plaintiffs' counsel challenge tests, and how they can successfully challenge the theory of "validity generalization." Journal of Vocational Behavior, 33, 331-364.
94. Smith, P. C. (1976). Behavior, results, and organizational effectiveness: The problem of the criteria. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology (pp. 745-775). Chicago, IL: Rand-McNally College.
95. Society for Industrial and Organizational Psychology (2003). Principles for the validation and use of personnel selection procedures (4th ed.). Bowling Green, OH: Author.
96. Spearman, C. (1910). Correlation calculated with faulty data. British Journal of Psychology, 3, 271-295.
97. Symonds, P. M. (1931). Diagnosing personality and conduct. New York, NY: D. Appleton-Century Company, Inc.
98. Thomas, J. P., Whitman, D. S., & Viswesvaran, C. (2010). Employee proactivity in organizations: A comparative meta-analysis of emergent proactive constructs. Journal of Occupational and Organizational Psychology, 83, 275-300.
99. Thorndike, E. L. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4, 25-29.
100. Thouless, R. H. (1939). The effects of errors of measurement on correlation coefficients. British Journal of Psychology: General Section, 29, 383-403.
101. Tornau, K., & Frese, M. (2013). Construct clean-up in proactivity research: A meta-analysis on the nomological net of work-related proactivity concepts and their incremental validities. Applied Psychology: An International Review, 62, 44-96.
102. Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.
103. Weiner, E. A., & Stewart, B. J. (1984). Assessing individuals: Psychological and educational tests and measurements. Boston, MA: Little, Brown, & Company.
104. Winne, P. H., & Belfry, M. J. (1982). Interpretive problems when correcting for attenuation. Journal of Educational Measurement, 19, 125-134.
105. Womer, F. B. (1968). Basic concepts in testing. Boston, MA: Houghton Mifflin Company.