The reliability of job performance ratings equals 0.52

More Statistical and Methodological Myths and Urban Legends

Volumn , Issue , 2014, Pages 248-275

  Putka, Dan J ^a   Hoffman, Brian J ^b  

^a HUMAN RESOURCES RESEARCH ORGANIZATION   (United States)

^b UNIVERSITY OF GEORGIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85086550869     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203775851-21     Document Type: Chapter

References (54)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing (4th ed.). Washington, DC: American Educational Research Association.
2. Bertua, C., Anderson, N., & Salgado, J. F. (2005). The predictive validity of cognitive ability tests: A UK meta-analysis. Journal of Occupational and Organizational Psychology, 78, 387–409.
3. Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer-Verlag.
4. Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models. Newbury Park, CA: Sage.
5. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: John Wiley.
6. Cronbach, L. J., Rajaratnam, N., & Gleser, G. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, Part 2, 137–163.
7. Cronbach, L. J., & Shavelson, R. J. (2004). My current thoughts on coeffi cient alpha and its successor procedures. Educational and Psychological Measurement, 64, 391–418.
8. Dawis, R. V., & Lofquist, L. H. (1984). A psychological theory of work adjustment: An individualdifferences model and its applications. Minneapolis, MN: University of Minnesota Press.
9. Deming, W. E. (1986). Out of the crisis. Cambridge: MIT Institute for Advanced Engineering Study.
10. DeShon, R. P. (2002). Generalizability theory. In F. Drasgow & N. Schmitt (Eds.), Measuring and analyzing behavior in organizations: Advances in measurement and data analysis (pp.189–220). San Francisco, CA: Jossey-Bass.
11. Ebel, R. L. (1951). Estimation of the reliability of ratings. Psychometrika, 16, 407–424.
12. Eysenck, H. J. (1978). An exercise in megasilliness. American Psychologist, 33, 517.
13. Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), Educational measurement (3rd ed.; pp. 105–146). Washington, DC: American Council on Education/Macmillan.
14. Gilboa, S., Shirom, A., Fried, Y., & Cooper, C. (2008). A meta-analysis of work demand stressors and job performance: Examining main and moderating effects. Personnel Psychology, 61, 227–271.
15. Greguras, G. J., & Robie, C. (1998). A new look at within-source interrater reliability of 360 degree feedback ratings. Journal of Applied Psychology, 83, 960–968.
16. Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed.; pp. 65–110). Westport, CT: American Council on Education and Praeger Publishers.
17. Harrison, D. A., Newman, D. A., & Roth, P. L. (2006). How important are job attitudes? Meta-analytic comparisons of integrative behavioral outcomes and time sequences. The Academy of Management Journal, 49, 305–325.
18. Hermelin, E., Lievens, F., & Robertson, I. T. (2007). The validity of assessment centres for the prediction of supervisory performance ratings: A meta-analysis. International Journal of Selection and Assessment, 15, 405–411.
19. Hoffman, B. J. & Woehr, D. J. (2009). Disentangling the meaning of multisource feedback: An examination of the nomological network surrounding source and dimension factors. Personnel Psychology, 62, 735–765.
20. Hoyt, C. (1941). Test reliability obtained by analysis of variance. Psychometrika, 6, 153–160.
21. Hoyt, W. T., & Kerns, M. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4, 403–424.
22. Huffcutt, A. I., Conway, J. M., Roth, P. L., & Stone, N. J. (2001). Identification and metaanalytic assessment of psychological constructs measured in employment interviews. Journal of Applied Psychology, 86, 897.
23. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage Publications.
24. Jackson, S., & Brashers, D. E. (1994). Random factors in ANOVA. Thousand Oaks, CA: Sage Publications.
25. Judge, T. A., Thoresen, C. J., Bono, J. E., & Patton, G. K. (2001). The job satisfaction–job performance relationship: A qualitative and quantitative review. Psychological Bulletin, 127, 376.
26. LaHuis, D. M., & Avis, J. M. (2007). Using multilevel random coeffi cient modeling to investigate rater effects in performance ratings. Organizational Research Methods, 10, 97–107.
27. MacKenzie, S. B., Podsakoff, P. M., & Jarvis, C. B. (2005). The problem of measurement model misspecifi cation in behavioral and organizational research and some recommended solutions. Journal of Applied Psychology, 90, 710–730.
28. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coeffi cients. Psychological Methods, 1, 30–46.
29. Murphy, K. R. (2008). Explaining the weak relationship between job performance and ratings of job performance. Industrial and Organizational Psychology: Perspectives on Research and Practice, 1, 148–160.
30. Murphy, K. R., Cleveland, J. N., & Mohler, C. J. (2001). Reliability, validity, and meaningfulness of multisource ratings. In D. W. Bracken, C. W. Timmreck, & A. H. Church (Eds.), Handbook of multisource feedback (pp. 130–148). San Francisco, CA: Jossey-Bass.
31. Murphy, K. R., & DeShon, R. (2000a). Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53, 873–900.
32. Murphy, K. R., & DeShon, R. (2000b). Progress in psychometrics: Can industrial and organizational psychology catch up? Personnel Psychology, 53, 873–900.
33. O’Neill, T. A., Goffin, R. D., & Gellatly, I. R. (2012). The use of random coefficient modeling for understanding and predicting job performance ratings: An application with field data. Organizational Research Methods, 15, 436–462.
34. Oswald, F., & McCloy, R. A. (2003). Meta-analysis and the art of the average. In K. R. Murphy (Ed.), Validity generalization: A critical review (pp. 311–338). Mahwah, NJ: Lawrence Erlbaum Associates.
35. Putka, D. J., Ingerick, M., & McCloy, R. A. (2008, April). Integrating reliability- and validity-based perspectives on error in performance ratings. In J. M. Cortina (Chair), Write, for these words are true: Uncovering complexity in I-O. Symposium conducted at the 23rd Annual Society for Industrial and Organizational Psychology Conference, San Francisco, CA.
36. Putka, D. J., Lance, C. L., Le, H., & McCloy, R. A. (2011). A cautionary note on modeling multitrait-multirater data arising from ill-structured measurement designs. Organizational Research Methods, 14, 503–529.
37. Putka, D. J., Le, H., McCloy, R. A., & Diaz, T. (2008). III-structured measurement designs in organizational research: Implications for estimating interrater reliability. Journal of Applied Psychology, 93, 959–981.
38. Putka, D. J., & McCloy, R. A. (2008). Estimating variance components in SPSS and SAS: An annotated reference guide. Alexandria, VA: Human Resources Research Organization. Available online at: Www.humrro.org/corpsite/sites/default/files/dputkaFiles/Estimating%20Variance%20Components%20in%20SPSS%20and%20SAS.pdf
39. Putka, D. J., & Sackett, P. R. (2010). Reliability and validity. In J. L. Farr & N. T. Tippins (Eds.), Handbook of employee selection (pp. 9–49). New York: Routledge.
40. Riketta, M. (2005). Organizational identification: A meta-analysis. Journal of Vocational Behavior, 66, 358–384.
41. Rothstein, H. R. (1990). Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology, 75, 322–327.
42. Sackett, P. R., Laczo, R. M., & Arvey, R. D. (2002). The effects of range restriction on estimates of criterion interrater reliability: Implications for validation research. Personnel Psychology, 55, 807–825.
43. Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199–223.
44. Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research fi ndings. Psychological Bulletin, 124, 262–274.
45. Schmidt, F. L., Viswesvaran, C., & Ones, D. S. (2000). Reliability is not validity and validity is not reliability. Personnel Psychology, 53, 901–912.
46. Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956–970.
47. Searle, S. R., Casella, G., & McCulloch, C. E. (1992). Variance components. New York, NY: Wiley.
48. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
49. Taylor, P. J., & Small, B. (2002). Asking applicants what they would do versus what they did do: A meta-analytic comparison of situational and past behaviour employment interview questions. Journal of Occupational and Organizational Psychology, 75, 277–294.
50. Thorndike, E. L. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4, 25–29.
51. Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557–574.
52. Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108–131.
53. Wong, S. P., & McGraw, K. O. (1999). Confidence intervals and F-tests for intraclass correlations based on three-way random effects models. Educational and Psychological Measurement, 59, 270–288.
54. Zhou, H., Muellerleile, P., Ingram, D., & Wong, S. P. (2011). Confidence intervals and F-tests for intraclass correlations based on three-way mixed effects models. Journal of Educational and Behavioral Statistics, 36, 638–671.