Reliability and validity

Handbook of Employee Selection

Volumn , Issue , 2010, Pages 9-49

  Putka, Dan J ^a   Sackett, Paul R ^b  

^a HUMAN RESOURCES RESEARCH ORGANIZATION   (United States)

^b UNIVERSITY OF MINNESOTA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84960287054     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203809808     Document Type: Chapter

References (133)

1. Aguinis, H., Henle, C. A., & Ostroff, C. (2001). Measurement in work and organizational psychology. In N. Anderson, D. S. Ones, H. K. Sinangil, & C. Viswesvaran (Eds.), Handbook of industrial, work, and organizational psychology: Volume 1: Personnel psychology (pp. 27-50). London, England: Sage.
2. Aiken, L. S., West, S. G., & Millsap, R. E. (2008). Graduate training in statistics, measurement, and methodology in psychology: Replication and extension of Aiken, West, Sechrest, and Reno’s (1990) survey of PhD programs in North America. American Psychologist, 63, 32-50.
3. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
4. American Psychological Association (joint committee). (1954). Technical recommendations for psychological tests and diagnostic techniques. Psychological Bulletin, 51, 201-238.
5. Bell, J. F. (1985). Generalizability theory: The software problem. Journal of Educational and Behavioral Statistics, 10, 19-29.
6. Binning, J. F., & Barrett, G.V. (1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74, 478-494.
7. Bliese, P. D. (2002). Multilevel random coefficient modeling in organizational research. In F. Drasgow & N. Schmitt (Eds.), Measuring and analyzing behavior in organizations (pp. 401-445). San Francisco, CA: Jossey-Bass.
8. Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71, 425-440.
9. Borsboom, D., & Mellenbergh, G. J. (2002). True scores, latent variables, and constructs: A comment on Schmidt and Hunter. Intelligence, 30, 505-514.
10. Bost, J. E. (1995). The effects of correlated errors on generalizability and dependability coefficients. Applied Psychological Measurement, 19, 191-203.
11. Brennan, R. L. (1998). Raw-score conditional standard errors of measurement in generalizability theory. Applied Psychological Measurement, 22, 307-331.
12. Brennan, R. L. (2001a). An essay on the history and future of reliability from the perspective of replications. Journal of Educational Measurement, 38, 295-317.
13. Brennan, R. L. (2001b). Generalizability theory. New York, NY: Springer-Verlag.
14. Brennan, R. L. (2006). Perspectives on the evolution and future of educational measurement. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 1-16). Westport, CT: American Council on Education and Praeger.
15. Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, NY: Guilford Press.
16. Brown, W. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296-322.
17. Bryk, A. S., & Raudenbush, S.W. (1992). Hierarchical linear models. Newbury Park, CA: Sage.
18. Campbell, J. P. (1976). Psychometric theory. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology (pp. 185-222). New York: John Wiley & Sons.
19. Cattell, R. B. (1966). The data box: Its ordering of total resources in terms of possible relational systems. In R. B. Cattell (Ed.), Handbook of multivariate experimental psychology (pp. 67-128). Chicago, IL: Rand McNally.
20. Cronbach, L. J. (1947). Test “reliability”: Its meaning and determination. Psychometrika, 12, 1-16.
21. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 292-334.
22. Cronbach, L. J. (1957). The two disciplines of scientific psychology. American Psychologist, 12, 671-684.
23. Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (pp. 221-237). Washington, DC: American Council on Education.
24. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: John Wiley.
25. Cronbach, L. J., Linn, R. L., Brennan, R. L., & Haertel, E. H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57, 373-399.
26. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-300.
27. Cronbach, L. J., Rajaratnam, N., & Gleser, G. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16(2), 137-163.
28. Cronbach, L. J., Schoneman, P., & McKie, D. (1965). Alpha coefficient for stratified-parallel tests. Educational & Psychological Measurement, 25, 291-312.
29. Cronbach, L. J., & Shavelson, R. J. (2004). My current thoughts on coefficient alpha and its successor procedures. Educational and Psychological Measurement, 64, 391-418.
30. Cureton, E. E. (1951). Validity. In E. F. Lindquist (Ed.), Educational measurement (pp. 621-694). Washington DC: American Council on Education.
31. DeShon, R. P. (1995, May). Restricted maximum likelihood estimation of variance components in generalizability theory: Overcoming balanced design requirements. Paper presented at the 10th annual conference of the Society of Industrial and Organizational Psychology, Orlando, FL.
32. DeShon, R. P. (1998). A cautionary note on measurement error corrections in structural equation models. Psychological Methods, 3, 412-423.
33. DeShon, R. P. (2002). Generalizability theory. In F. Drasgow & N. Schmitt (Eds.), Measuring and analyzing behavior in organizations (pp. 189-220). San Francisco, CA: Jossey-Bass.
34. Dunnette, M. D. (1966). Personnel selection and placement. Oxford, England: Wadsworth.
35. Ebel, R. L. (1951). Estimation of the reliability of ratings. Psychometrika, 16, 407-424.
36. Eid, M., Lischetzke, T., Nussbeck, F. W., & Trierweiler, L. I. (2003). Separating trait effects from trait-specific method effects in multitrait-multimethod models: A multiple-indicator CT-C(M-1) model. Psychological Methods, 8, 38-60.
37. Englehard, G. (1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93-112.
38. Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 105-146). New York: American Council on Education and Macmillan.
39. Fisher, R.A. (1925). Statistical methods for research workers (1st ed.). Edinburgh, Scotland: Oliver and Boyd.
40. Ghiselli, E. E. (1964). Theory of psychological measurement. New York: McGraw-Hill.
41. Green, S. B. (2003). A coefficient alpha for test-retest data. Psychological Methods, 8, 88-101.
42. Green, S. B., & Hershberger, S. L. (2000). Correlated errors in true score models and their effect on coefficient alpha. Structural Equation Modeling, 7, 251-270.
43. Greguras, G. J., & Robie, C. (1998). A new look at within-source interrater reliability of 360 degree feedback ratings. Journal of Applied Psychology, 83, 960-968.
44. Guion, R. M. (1974). Open a new window: Validities and values in psychological measurement. American Psychologist, 29, 287-296.
45. Guion, R. M. (1998). Assessment, measurement, and prediction for personnel decisions. Mahwah, NJ: Lawrence Erlbaum.
46. Gulliksen, H. (1950). Theory of mental tests. New York, NY: Wiley.
47. Guttman, L. A. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10, 255-282.
48. Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 65-110). Westport, CT: American Council on Education and Praeger.
49. Hocking, R. R. (1995). Variance component estimation in mixed linear models. In L. K. Edwards (Ed.), Applied analysis of variance in behavioral science (pp. 541-571). New York, NY: Marcel Dekker.
50. Hox, J. J. (1994). Hierarchical regression models for interviewer and respondent effects. Sociological Methods and Research, 22, 300-318.
51. Hoyt, C. (1941). Test reliability obtained by analysis of variance. Psychometrika, 6, 153-160.
52. Jackson, S., & Brashers, D. E. (1994). Random factors in ANOVA. Thousand Oaks, CA: Sage.
53. Joreskog, K. G. (1971). Statistical analysis of sets of congeneric tests. Psychometrika, 36, 109-133.
54. Joreskog, K. G., and Sorbom, D. (2001). LISREL 8: User’s reference guide. Lincolnwood, IL: Scientific Software International.
55. Kamata, A., Turhan, A., & Darandari, E. (2003, April). Estimating reliability for multidimensional composite scale scores. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
56. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17-64). Westport, CT: American Council on Education and Praeger.
57. Kenny, D. A., & Judd, C. M. (1986). Consequences of violating the independence assumption in analysis of variance. Psychological Bulletin, 99, 422-431.
58. Kenny, D. A., & Kashy, D. A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172.
59. Komaroff, E. (1997). Effect of simultaneous violations of essential tau-equivalence and uncorrelated errors on coefficient alpha. Applied Psychological Measurement, 21, 337-348.
60. Kuder, G. F., & Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2, 151-160.
61. Kuhn, T. S. (1962). The structure of scientific revolutions. Chicago: The University of Chicago Press.
62. Laenen, A., Vangeneugden, T., Geys, H., & Molenberghs, G. (2006). Generalized reliability estimation using repeated measurements. British Journal of Mathematical and Statistical Psychology, 59, 113-131.
63. LaHuis, D. M., & Avis, J. M. (2007). Using multilevel random coefficient modeling to investigate rater effects in performance ratings. Organizational Research Methods, 10, 97-107.
64. Lance, C. L. (2008). Why assessment centers don’t work the way they’re supposed to. Industrial and Organizational Psychology, 1, 84-97.
65. Lance, C. L., Lambert, T. A., Gewin, A. G., Lievens, F., & Conway, J. M. (2004). Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings. Journal of Applied Psychology, 89, 377-385.
66. Landy, F. J. (1986). Stamp collecting versus science: Validation as hypothesis testing. American Psychologist, 41, 1183-1192.
67. Le, H., & Putka, D. J. (2007). Reliability. In S. G. Rogelberg (Ed.), Encyclopedia of industrial and organizational psychology (Vol. 2, pp. 675-678). Thousand Oaks, CA: Sage.
68. Le, H., Schmidt, F. L., & Putka, D. J. (2009). The multifaceted nature of measurement artifacts and its implications for estimating construct-level relationships. Organizational Research Methods, 12, 165-200.
69. Lee, G., Dunbar, S. B., & Frisbie, D. A. (2001). The relative appropriateness of eight measurement models for analyzing scores from tests comprised of testlets. Educational and Psychological Measurement, 61, 958-975.
70. Lee, G., & Frisbie, D. A. (1999). Estimating reliability under a generalizability theory model for test scores composed of testlets. Applied Measurement in Education, 12, 237-255.
71. Littell, R. C., Milliken, G. A., Stroup, W. W., & Wolfinger, R. D. (1996). SAS system for mixed models. Cary, NC: SAS Institute.
72. Loevinger, J. (1957). Objective tests as instruments of psychological theory [Monograph No. 9]. Psychological Reports, 3, 635-694.
73. Lord, F. M. (1955). Estimating test reliability. Educational and Psychological Measurement, 15, 325-336.
74. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
75. Lumsden, J. (1976). Test theory. Annual Review of Psychology, 27, 251-280.
76. Marcoulides, G. A. (1990). An alternative method for estimating variance components in generalizability theory. Psychological Reports, 66, 102-109.
77. Marcoulides, G. A. (1996). Estimating variance components in generalizability theory: The covariance structure analysis approach. Structural Equation Modeling, 3, 290-299.
78. Marsh, H. W., & Grayson, D. (1994). Longitudinal confirmatory factor analysis: Common, time-specific, item-specific, and residual-error components of variance. Structural Equation Modeling, 1, 116-146.
79. Maxwell, A. E. (1968). The effect of correlated error on reliability coefficients. Educational and Psychological Measurement, 28, 803-811.
80. Mc Donald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
81. Mc Graw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46.
82. Mc Phail, S. M. (Ed.) (2007). Alternative validation strategies: Developing new and leveraging existing validity evidence. San Francisco, CA: John Wiley and Sons.
83. Merenda, P. F. (2007). Psychometrics and psychometricians in the 20th and 21st centuries: How it was in the 20th century and how it is now. Perceptual and Motor Skills, 104, 3-20.
84. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York: American Council on Education and Macmillan.
85. Mosier, C. I. (1943). On the reliability of a weighted composite. Psychometrika, 8, 161-168.
86. Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., & Hezlett, S. A. (1998). Trait rater and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557-576.
87. Murphy, K. R., & DeShon, R. (2000). Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53, 873-900.
88. Ng, K. T. (1974). Spearman’s test score model: A restatement. Educational and Psychological Measurement, 34, 487-498.
89. Novick, M. R., & Lewis, C. (1967). Coefficient alpha and the reliability of composite measurements. Psychometrika, 32, 1-13.
90. Nunnally, J. C. (1978). Psychometric theory. New York: McGraw-Hill.
91. Patz, R. J., Junker, B. W., Johnson, M. S., & Mariano, L. T. (2002). The hierarchical rater model for rated test items and its application to large-scale educational assessment data. Journal of Educational and Behavioral Statistics, 27, 341-384.
92. Ployhart, R. E., Schneider, B., & Schmitt, N. (2006). Staffing organizations: Contemporary practice and theory (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
93. Putka, D. J., Ingerick, M., & McCloy, R. A. (2008a, April). Integrating reliability and validity-based perspectives on error in performance ratings. Presentation in J. Cortina (Chair), Write, for these words are true: Uncovering complexity in I/O. Symposium conducted at the 23rd Annual Society for Industrial and Organizational Psychology Conference, San Francisco, CA.
94. Putka, D. J., Ingerick, M., & McCloy, R. A. (2008b). Integrating traditional perspectives on error in ratings: Capitalizing on advances in mixed effects modeling. Industrial and Organizational Psychology, 1, 167-173.
95. Putka, D. J., Le, H., McCloy, R. A., & Diaz, T. (2008). Ill-structured measurement designs in organizational research: Implications for estimating interrater reliability. Journal of Applied Psychology, 93, 959-981.
96. Putka, D. J., & McCloy, R. A. (2008, February). Estimating variance components in SPSS and SAS: An annotated reference guide. Retrieved March 23, 2009, from http://www.humrro.org/djp_archive/Estimating Variance Components in SPSS and SAS.pdf
97. Qualls-Payne, A. L. (1992). A comparison of score level estimates of the standard error of measurement. Journal of Educational Measurement, 29, 213-225.
98. Rae, G. (2007). A note on using stratified alpha to estimate the composite reliability of a test composed of interrelated nonhomogeneous items. Psychological Methods, 12, 177-184.
99. Raykov, T. (1998). Coefficient alpha and composite reliability with interrelated nonhomogeneous items. Applied Psychological Measurement, 22, 375-385.
100. Raykov, T. (2001a). Bias of coefficient a for fixed congeneric measures with correlated errors. Applied Psychological Measurement, 25, 69-76.
101. Raykov, T. (2001b). Estimation of congeneric scale reliability using covariance structure analysis with nonlinear constraints. British Journal of Mathematical and Statistical Psychology, 54, 315-323.
102. Raykov, T., & Shrout, P. E. (2002). Reliability of scales with general structure: Point and interval estimation using a structural equation modeling approach. Structural Equation Modeling, 9, 195-212.
103. Reuterberg, S. E., & Gustafsson, J. E. (1992). Confirmatory factor analysis and reliability: Testing measurement model assumptions. Educational and Psychological Measurement, 52, 795-811.
104. Rogers, W. M., Schmitt, N., & Mullins, M. E. (2002). Correction for unreliability of multifactor measures: Comparison of alpha and parallel forms of approaches. Organizational Research Methods, 5, 184-199.
105. Rozeboom W. W. (1966). Foundations of the theory of prediction. Homewood, IL: Dorsey.
106. Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199-223.
107. Schmidt, F. L., Le, H., & Ilies, R. (2003). Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual-differences constructs. Psychological Methods, 8, 206-224.
108. Schmidt, F. L., Viswesvaran, C., & Ones, D. S. (2000). Reliability is not validity and validity is not reliability. Personnel Psychology, 53, 901-912.
109. Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353
110. Schmitt, N., & Landy, F. J. (1993). The concept of validity. In N. Schmitt, W.C. Borman & Associates (Eds.), Personnel selection in organizations (pp. 275-309). San Francisco, CA: Jossey-Bass.
111. Scullen, S. E. (1999). Using confirmatory factor analyses of correlated uniquenesses to estimate method variance in multitrait-multimethod matrices. Organizational Research Methods, 2, 275-292.
112. Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
113. Searle, S. R., Casella, G., & McCulloch, C. E. (1992). Variance components. New York, NY: Wiley.
114. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
115. Shrout, P. E. (1993). Analyzing consensus in personality judgments: A variance components approach. Journal of Personality, 61, 769-788.
116. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
117. Smith, P. L., & Luecht, R. M. (1992). Correlated effects in generalizability studies. Applied Psychological Measurement, 36, 229-235.
118. Society for Industrial and Organizational Psychology. (2003). Principles for the validation and use of personnel selection procedures (4th ed.). Bowling Green, OH: Author.
119. Spearman, C. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72-101.
120. Spearman, C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3, 271-295.
121. SPSS. (2005). Linear mixed-effect modeling in SPSS: An introduction to the mixed procedure (Technical Report LMEMWP-0305). Chicago, IL: Author.
122. Thorndike, R. L. (1951). Reliability. In E. F. Lindquist (Ed.), Educational measurement (pp. 560-620). Washington, DC: American Council on Education.
123. Tryon, R. C. (1957). Reliability and behavior domain validity: Reformulation and historical critique. Psychological Bulletin, 54, 229-249.
124. Van Iddekinge, C. H., Putka, D. J., Raymark, P. H., & Eidson, C. E., Jr. (2005). Modeling error variance in job specification ratings: The influence of rater, job, and organization-level factors. Journal of Applied Psychology, 90, 323-334.
125. Van Iddekinge, C. H., Raymark, P. H., Eidson, C. E., Jr., & Attenweiler, W. J. (2004). What do structured selection interviews really measure? The construct validity of behavior description interviews. Human Performance, 17, 71-93.
126. Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108-131.
127. Webb, N. M., & Shavelson, R. J. (1981). Multivariate generalizability of general educational development ratings. Journal of Educational Measurement, 18, 13-22.
128. Werts, C. E., Linn, R. L., & Joreskog, K. G. (1974). Intraclass reliability estimates: Testing structural assumptions. Educational and Psychological Measurement, 34, 25-32.
129. Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1-26.
130. th ed., pp. 111-153). Westport, CT: American Council on Education and Praeger.
131. Zimmerman, D. W., & Williams, R. H. (1980). Is classical test theory “robust” under violation of the assumption of uncorrelated errors? Canadian Journal of Psychology, 34, 227-237.
132. Zimmerman, D. W., & Williams, R. H. (1997). Properties of the Spearman correction for attenuation for normal and realistic non-normal distributions. Applied Psychological Measurement, 21, 253-279.
133. Zimmerman, D. W., Zumbo, B. D., & LaLonde, C. (1993). Coefficient alpha as an estimate of test reliability under violation of two assumptions. Educational and Psychological Measurement, 53, 33-49.