Measurement Error Obfuscates Scientific Knowledge: Path to Cumulative Knowledge Requires Corrections for Unreliability and Psychometric Meta-Analyses

Industrial and Organizational Psychology

Volumn 7, Issue 4, 2014, Pages 507-518

  Viswesvaran, Chockalingam ^a   Ones, Deniz S ^b   Schmidt, Frank L ^c   Le, Huy ^d   Oh, In Sue ^e  

^a FLORIDA INTERNATIONAL UNIVERSITY   (United States)

^b UNIVERSITY OF MINNESOTA   (United States)

^c UNIVERSITY OF IOWA   (United States)

^d UNIVERSITY OF TEXAS AT SAN ANTONIO   (United States)

^e TEMPLE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84921441624     PISSN: 17549426     EISSN: 17549434     Source Type: Journal    
DOI: 10.1111/iops.12186     Document Type: Article

References (57)

1. Austin, J. T., & Villanova, P. (1992). The criterion problem: 1917-1992. Journal of Applied Psychology, 77(6), 836-874.
2. Binning, J. F., & Barrett, G. V. (1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74, 478-494.
3. Borman, W. C., & Brush, D. H. (1993). More progress towards a taxonomy of managerial performance requirements. Human Performance, 6, 1-21.
4. Campbell, J. P., Gasser, M. B., & Oswald, F. L. (1996). The substantive nature of job performance variability. In K. R. Murphy (Ed.), Individual differences and behavior in organizations (pp. 258-299). San Francisco, CA: Jossey-Bass.
5. Cattin, P. (1980). Estimation of the squared cross-validated multiple correlation of a regression-model. Psychological Bulletin, 87(1), 63-65.
6. Charles, E. P. (2005). The correction for attenuation due to measurement error: Clarifying concepts and creating confidence sets. Psychological Methods, 10(2), 206-226.
7. Connelly, B. S., & Ones, D. S. (2010). An other perspective on personality: Meta-analytic integration of observers' accuracy and predictive validity. Psychological Bulletin, 136, 1092-1122.
8. Conway, J. M., Jako, R. A., & Goodman, D. F. (1995). A metaanalysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80(5), 565-579.
9. Dierdorff, E. C., & Wilson, M. A. (2003). A meta-analysis of job analysis reliability. Journal of Applied Psychology, 88(4), 635-646.
10. Dilchert, S., Ones, D. S., & Krueger, R. F. (2014). Maladaptive personality constructs, measures, and work behaviors: Scientific background and employment practice recommendations. Industrial and Organizational Psychology: Perspectives on Science and Practice, 7, 98-110.
11. Facteau, J. D., & Craig, S. B. (2001). Are performance appraisal ratings from different rating sources comparable? Journal of Applied Psychology, 86, 215-227.
12. Harter, J. K., Schmidt, F. L., & Hayes, T. L. (2002). Business-unit-level relationships between employee satisfaction, employee engagement, and business outcomes: A meta-analysis. Journal of Applied Psychology, 87, 268-279.
13. Hauser, R. M. (2007). Will practitioners benefit from meta analysis? Academy of Management Perspectives, 21(3), 24-28.
14. Huffcutt, A. I., Culbertson, S., & Weyhrauch, W. S. (2013). Employment interview reliability: New meta-analytic estimates by structure and format. International Journal of Selection and Assessment, 21(3), 264-276.
15. Hunter, J. E., & Schmidt, F. L. (1982). Quantifying the effects of psychological interventions on employee job performance and workforce productivity. American Psychologist, 38, 473-478.
16. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
17. Hunter, J. E., Schmidt, F. L., & Le, H. (2006a). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91, 594-612.
18. Hunter, J. E., Schmidt, F. L., & Le, H. (2006b). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91(3), 594-612.
19. King, L. M., Hunter, J. E., & Schmidt, F. L. (1980). Halo in a multidimensional forced-choice performance evaluation scale. Journal of Applied Psychology, 65(5), 507-516.
20. LeBreton, J. M., Scherer, K. T., & James, L. R. (2014). Corrections for criterion reliability in validity generalization: A false prophet in a land of suspended judgment. Industrial and Organizational Psychology: Perspectives on Science and Practice, 7(4), 478-500.
21. Lievens, F. (2001). Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity. Journal of Applied Psychology, 86(2), 255-264.
22. Maurer, T. J., Raju, N. S., & Collins, W. C. (1998). Peer and subordinate performance appraisal measurement equivalence. Journal of Applied Psychology, 83(5), 693-702. doi: 10.1037/0021-9010.83.5.693
23. Mount, M. K., Judge, T. A., Scullen, S. E., et al. (1998). Trait, rater and level effects in 360-degree performance ratings. Personnel Psychology, 51(3), 557-576.
24. Ones, D. S., & Dilchert, S. (2012). Employee green behaviors. In S. E. Jackson, D. S. Ones, & S. Dilchert (Eds.), Managing HR for environmental sustainability (pp. 85-116). San Francisco, CA: Jossey-Bass/Wiley.
25. Ones, D. S., & Dilchert, S. (2013). Counterproductive work behaviors: Concepts, measurement, and nomological network. In K. F. Geisinger, B. A. Bracken, J. F. Carlson, J.-I. C. Hansen, N. R. Kuncel, S. P. Reise, & M. C. Rodriguez (Eds.), APA handbook of testing and assessment in psychology (pp. 643-659). Washington, DC: American Psychological Association.
26. Ones, D. S., Dilchert, S., & Viswesvaran, C. (2012). Cognitive abilities. In N. Schmitt (Ed.), Oxford handbook of personnel assessment and selection (pp. 179-224). New York, NY: Oxford University Press.
27. Ones, D. S., Viswesvaran, C., & Dilchert, S. (2005). Cognitive ability in personnel selection decisions. In A. Evers, O. Voskuijl, & N. Anderson (Eds.), Handbook of selection (pp. 143-173). Oxford, England: Blackwell.
28. Ones, D. S., Viswesvaran, C., & Schmidt, F. L. (2008). No new terrain: Reliability and construct validity of job performance ratings. Industrial and Organizational Psychology, 1, 174-179.
29. Osterman, P. (2007). Comment on Le, Oh, Shaffer, and Schmidt. Academy of Management Perspectives, 21(3), 16-18.
30. Ostroff, C., & Harrison, D. A. (1999). Meta-analysis, level of analysis, and best estimates of population correlations: Cautions for interpreting meta-analytic results in organizational behavior. Journal of Applied Psychology, 84(2), 260-270.
31. Putka, D. J., & Hoffman, B. J. (in press). "The" reliability of performance ratings equals 0.52. In C. E. Lance, & R. J. Vandenberg (Eds.), More statistical and methodological myths and urban legends. New York, NY: Taylor & Francis.
32. Putka, D. J., Le, H., McCloy, R. A., & Diaz, T. (2008). Ill-structured measurement designs in organizational research: Implications for estimating interrater reliability. Journal of Applied Psychology, 93, 959-981.
33. Rothstein, H. R. (1990). Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology, 75, 322-327.
34. Sackett, P. R., Laczo, R. M., & Arvey, R. D. (2002). The effects of range restriction on estimates of criterion interrater reliability: Implications for validation research. Personnel Psychology, 55(4), 807-825.
35. Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., De Fruyt, F., & Rolland, J. P. (2003). A meta-analytic study of general mental ability validity for different occupations in the European Community. Journal of Applied Psychology, 88, 1068-1081.
36. Salgado, J. F., & Moscoso, S. (1996). Meta-analysis of interrater reliability of job performance ratings. Perceptual and Motor Skills, 83, 1195-1201.
37. Salgado, J. F., Moscoso, S., & Lado, M. (2003). Test-retest reliability of job performance dimensions in managers. International Journal of Selection and Assessment, 11, 98-101.
38. Salgado, J. F., & Tauriz, G. (2014). The five-factor model, forced-choice personality inventories and performance: A comprehensive meta-analysis of academic and occupational validity studies. European Journal of Work and Organizational Psychology, 23, 3-30.
39. Schmidt, F. L., & Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62(5), 529-540.
40. Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1(2), 199-223.
41. Schmidt, F. L., & Hunter, J. E. (2014). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage.
42. Schmidt, F. L., Hunter, J. E., & Outerbridge, A. N. (1986). Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of performance. Journal of Applied Psychology, 71, 432-439.
43. Schmidt, F. L., & Kaplan, L. B. (1971). Composite versus multiple criteria: A review and resolution of the controversy. Personnel Psychology, 24, 419-434.
44. Schmidt, F. L., Le, H., Oh, I., & Schafer, J. (2007). General mental ability, job performance and red herrings: Responses to Osterman, Huaser and Schmitt. Academy of Management Perspectives, 21, 64-76.
45. Schmidt, F. L., Viswesvaran, C., & Ones, D. S. (2000). Reliability is not validity and validity is not reliability. Personnel Psychology, 53, 901-912.
46. Schmitt, N. (2007). The value of personnel selection: Reflections on some remarkable claims. Academy of Management Perspectives, 21(3), 19-23.
47. Shen, W., Beatty, A. S., & Sackett, P. R. (2009, September). Further moderators of the interrater reliability of supervisory ratings of job performance. Washington, DC: Personnel Testing Council of Metropolitan Washington Newsletter.
48. Society for Industrial and Organizational Psychology (SIOP). (2003). Principles for the validation and use of personnel selection procedures (4th ed.). Bowling Green, OH: Society for Industrial and Organizational Psychology.
49. Thompson, B. (1992). Two and one-half decades of leadership in measurement and evaluation. Journal of Counseling and Development, 70, 434-438.
50. Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20.
51. Viswesvaran, C., & Ones, D. S. (2000a). Perspectives on models of job performance. International Journal of Selection and Assessment, 8, 216-227.
52. Viswesvaran, C., & Ones, D. S. (2000b). Measurement error in "Big Five Factors" of personality assessment: Reliability generalization across studies and measures. Educational and Psychological Measurement, 60, 224-235.
53. Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.
54. Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2002). The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty. Journal of Applied Psychology, 87, 345-354.
55. Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108-131.
56. Voskuijl, O. F., & van Sliedregt, T. (2002). Determinants of interrater reliability of job analysis: Ameta-analysis. European Journal of Psychological Assessment, 18(1), 52-62.
57. Whitman, D. S., Van Rooy, D. L., & Viswesvaran, C. (2010). Satisfaction, citizenship behaviors and performance in work units: A meta-analysis of collective construct relations. Personnel Psychology, 63, 41-81.