Ill-Structured Measurement Designs in Organizational Research: Implications for Estimating Interrater Reliability

Journal of Applied Psychology

Volumn 93, Issue 5, 2008, Pages 959-981

  Putka, Dan J ^a   Le, Huy ^b   McCloy, Rodney A ^c   Diaz, Tirso ^a  

^a HUMAN RESOURCES RESEARCH ORGANIZATION   (United States)

^b UNIVERSITY OF CENTRAL FLORIDA   (United States)

^c Human Resources Research Organization   (United States)

Author keywords

measurement design; ratings; reliability

Indexed keywords

ARTICLE; HUMAN; OBSERVER VARIATION; ORGANIZATION; PSYCHOLOGICAL MODEL; PSYCHOMETRY; REPRODUCIBILITY; VALIDATION STUDY;

HUMANS; MODELS, PSYCHOLOGICAL; OBSERVER VARIATION; ORGANIZATIONAL CULTURE; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS;

EID: 54449084709     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/0021-9010.93.5.959     Document Type: Article

References (59)

1. American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. Standards for educational and psychological testing (1999), American Educational Research Association, Washington, DC
2. Bates D. Linear mixed model implementation in lme4 (2007). http://cran.cnr.berkeley.edu/doc/vignettes/lme4/Implementation.pdf Retrieved August 3, 2007, from http://cran.cnr.berkeley.edu/doc/vignettes/lme4/Implementation.pdf
3. Bliese P.D. Multilevel random coefficient modeling in organizational research. In: Drasgow F., and Schmitt N. (Eds). Measuring and analyzing behavior in organizations (2002), Jossey-Bass, San Francisco 401-445
4. Bost J.E. The effects of correlated errors on generalizability and dependability coefficients. Applied Psychological Measurement 19 (1995) 191-203
5. Brennan R.L. Generalizability theory (2001), Springer-Verlag, New York
6. Campbell J.P. Project A: The U.S. Army selection and classification project [Special issue]. Personnel Psychology 43 2 (1990)
7. Campion M.A., Campion J.E., and Hudson J.P. Structured interviewing: A note on incremental validity and alternative question types. Journal of Applied Psychology 79 (1994) 998-1002
8. Campion M.A., Outtz J.L., Zedeck S., Schmidt F.L., Kehoe J.F., Murphy K.R., and Guion R.M. The controversy over score banding in personnel selection: Answers to 10 key questions. Personnel Psychology 54 (2001) 149-185
9. Cascio W.F., Outtz J., Zedeck S., and Goldstein I.L. Statistical implications of six methods of score use in personnel selection. Human Performance 4 (1991) 233-264
10. Conway J.M., Jako R.A., and Goodman D.F. A meta-analysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology 80 (1995) 565-579
11. Cook T.D., and Campbell D.T. Quasi-experimentation: Design and analysis issues for field settings (1979), Rand McNally, Chicago
12. Crocker L., and Algina L. Introduction to classical and modern test theory (1986), Holt, Rinehart, and Winston, New York
13. Cronbach L.J., Gleser G.C., Nanda H., and Rajaratnam N. The dependability of behavioral measurements: Theory of generalizability for scores and profiles (1972), Wiley, New York
14. DeShon R.P. Generalizability theory. In: Drasgow F., and Schmitt N. (Eds). Measuring and analyzing behavior in organizations (2002), Jossey-Bass, San Francisco 189-220
15. Enders C.K. Using the expectation maximization algorithm to estimate coefficient alpha for scales with item-level missing data. Psychological Methods 8 (2003) 322-337
16. Feldt L.S., and Brennan R.L. Reliability. In: Linn R.L. (Ed). Educational measurement. 3rd ed. (1989), American Council on Education and Macmillan, New York 105-146
17. Greguras G.J., and Robie C. A new look at within-source interrater reliability of 360 degree feedback ratings. Journal of Applied Psychology 83 (1998) 960-968
18. Guilford J.P. Psychometric methods. 2nd ed. (1954), McGraw Hill, New York
19. Harris M.M., Becker A.S., and Smith D.E. Does the assessment center scoring method affect the cross-situational consistency of ratings?. Journal of Applied Psychology 78 (1993) 675-678
20. Hoyt W.T. Rater bias in psychological research: When is it a problem and what can we do about it?. Psychological Methods 5 (2000) 64-86
21. Hoyt W.T., and Kerns M. Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods 4 (1999) 403-424
22. Kenny D.A., and Judd C.M. Consequences of violating the independence assumption in analysis of variance. Psychological Bulletin 99 (1986) 422-431
23. Knapp D.J., McCloy R.A., and Heffner T. Validation of measures designed to maximize 21st-century Army NCO performance (Technical Report 1145) (2004), U.S. Army Research Institute for the Behavioral and Social Sciences, Arlington, VA
24. Knapp D.J., and Tremble T.R. Concurrent validation of experimental Army enlisted personnel selection and classification measures (Technical Report 1205) (2007), U.S. Army Research Institute for the Behavioral and Social Sciences, Arlington, VA
25. Lance C.E., Foster M.R., Gentry W.A., and Thoresen J.D. Assessor cognitive processes in an operational assessment center. Journal of Applied Psychology 89 (2004) 22-35
26. Landy F.J., and Farr J.L. Performance rating. Psychological Bulletin 87 (1980) 72-107
27. Latham G.P., Saari L.M., Pursell E.D., and Campion M.A. The situational interview. Journal of Applied Psychology 65 (1980) 422-427
28. LeBreton J.M., Burgess J.R.D., Kaiser R.B., Atchley E.K., and James L.R. The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar. Organizational Research Methods 6 (2003) 80-128
29. Littell R.C., Milliken G.A., Stroup W.W., and Wolfinger R.D. SAS system for mixed models (1996), SAS Institute Inc, Cary, NC
30. Lord F.M., and Novick M.R. Statistical theories of mental test scores (1968), Addison-Wesley, Reading, MA
31. Marcoulides G.A. An alternative method for estimating variance components in generalizability theory. Psychological Reports 66 (1990) 102-109
32. Maxwell A.E. The effect of correlated error on reliability coefficients. Educational and Psychological Measurement 28 (1968) 803-811
33. McCloy, R. A., & Putka, D. J. (2004, April). Estimating interrater reliability: Conquering the messiness of real-world data. Master tutorial conducted at the 19th annual Society for Industrial and Organizational Psychology conference, Chicago, IL.
34. McDonald R.P. Test theory: A unified treatment (1999), Erlbaum, Mahwah, NJ
35. McGraw K.O., and Wong S.P. Forming inferences about some intraclass correlation coefficients. Psychological Methods 1 (1996) 30-46
36. Miller I., and Miller M. John E. Freund's mathematical statistics. 6th ed. (1999), Prentice Hall, Saddle River, NJ
37. Mount M.K., Judge T.A., Scullen S.E., Sytsma M.R., and Hezlett S.A. Trait rater and level effects in 360-degree performance ratings. Personnel Psychology 51 (1998) 557-576
38. Murphy K.R., and DeShon R. Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology 53 (2000) 873-900
39. Nathan B.R., and Tippins N. The consequences of halo "error" in performance ratings: A field study of the moderating effect of halo on test validation results. Journal of Applied Psychology 75 (1990) 290-296
40. Nunnally J.C., and Bernstein I.H. Psychometric theory. 3rd ed. (1994), McGraw Hill, New York
41. Pulakos E.D., and Schmitt N. Experience-based and situational interview questions: Studies of validity. Personnel Psychology 48 (1995) 289-308
42. R Development Core Team. R Language Definition, Version 2.5.1 (Draft) (2007). http://www.r-project.org Retrieved August 8, 2007, from http://www.r-project.org
43. Rothstein H.R. Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology 75 (1990) 322-327
44. Saal F.E., Downey R.G., and Lahey M.A. Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin 88 (1980) 413-428
45. Sackett P.R., and Dreher G.F. Constructs and assessment center dimensions: Some troubling empirical findings. Journal of Applied Psychology 67 (1982) 401-410
46. Schmidt F.L., and Hunter J.E. Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods 1 (1996) 199-223
47. Schmidt F.L., Viswesvaran C., and Ones D.S. Reliability is not validity and validity is not reliability. Personnel Psychology 53 (2000) 901-912
48. Scullen S.E., Mount M.K., and Goff M. Understanding the latent structure of job performance ratings. Journal of Applied Psychology 85 (2000) 956-970
49. Searle S.R., Casella G., and McCulloch C.E. Variance components (1992), Wiley, New York
50. Shavelson R.J., and Webb N.M. Generalizability theory: A primer (1991), Sage, Newbury Park, CA
51. Shrout P.E., and Fleiss J.L. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin 86 (1979) 420-428
52. Society for Industrial and Organizational Psychology. Principles for the validation and use of personnel selection procedures. 4th ed. (2003), Author, Bowling Green, OH
53. Sussmann M., and Robertson D.U. The validity of validity: An analysis of validation study designs. Journal of Applied Psychology 71 (1986) 461-468
54. Tsacoumis S. Assessment centers. In: Whetzel D.L., and Wheaton G.R. (Eds). Applied measurement: Industrial psychology in human resources management (2007), Erlbaum, Mahwah, NJ 259-292
55. Tziner A., and Dolan S. Validity of an assessment center for identifying future female officers in the military. Journal of Applied Psychology 67 (1982) 728-736
56. Van Iddekinge C.H., Raymark P.H., Eidson C.E., and Attenweiler W.J. What do structured selection interviews really measure? The construct validity of behavior description interviews. Human Performance 17 (2004) 71-93
57. Viswesvaran C., Ones D.S., and Schmidt F.L. Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology 81 (1996) 557-574
58. Viswesvaran C., Schmidt F.L., and Ones D.S. Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology 90 (2005) 108-131
59. Winer B.J. Statistical principles in experimental design. 2nd ed. (1971), McGraw Hill, New York