The Use of Random Coefficient Modeling for Understanding and Predicting Job Performance Ratings: An Application With Field Data

Organizational Research Methods

Volumn 15, Issue 3, 2012, Pages 436-462

  O'Neill, Thomas A ^a   Goffin, Richard D ^b   Gellatly, Ian R ^c  

^a UNIVERSITY OF CALGARY   (Canada)

^b UNIVERSITY OF WESTERN ONTARIO   (Canada)

^c UNIVERSITY OF ALBERTA   (Canada)

Author keywords

construct validation procedures; method variance; multilevel research; quantitative research; reliability and validity

Indexed keywords

EID: 84862220910     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428112438699     Document Type: Article

References (95)

1. Algina J.,Swaminathan H.Handbook of advanced multilevel analysis. Hox J. J.Roberts J. K., ed. New York: Routledge; 2011:285-312.
2. Barrick M. R.,Mount M. K.,Judge T. A.Personality and performance at the beginning of the new millennium: What do we know and where do we go next?.International Journal of Selection and Assessment. 2001;9:9-30.
3. Bauer D. J.,Curran P. J.Probing interactions in fixed and multilevel regression: Inferential and graphical techniques.Multivariate Behavioral Research. 2005;40:373-400.
4. Bennett W.,Lance C. E.,Woehr D. J.Performance measurement: Current perspectives and future challenges. Mahwah, NJ: Lawrence Erlbaum; 2006:.
5. Beretvas S. N.Handbook of advanced multilevel analysis. Hox J. J.Roberts J. K., ed. New York: Routledge; 2011:313-334.
6. Biesanz J. C.The social accuracy model of interpersonal perception: Assessing differences in perceptive and expressive accuracy.Multivariate Behavioral Research. 2010;45:853-885.
7. Bliese P. D.Multi-level theory, research, and methods in organizations. Klein K.Kozlowski S., ed. San Francisco: Jossey-Bass; 2000:349-381.
8. Bliese P. D.,Hanges P. J.Both being too liberal and too conservative: The perils of treating grouped data as though they were independent.Organizational Research Methods. 2004;7:400-417.
9. Bliese P. D.,Ployhart R. E.Growth modeling using random coefficient models: Model building testing, and illustrations.Organizational Research Methods. 2002;5:362-387.
10. Borkenau P.,Liebler A.Convergence of stranger ratings of personality and intelligence with self-ratings, partner ratings, and measured intelligence.Journal of Personality and Social Psychology. 1993;65:546-553.
11. Brennan R. L.Generalizability theory. New York: Springer-Verlag; 2001:.
12. Bryk A. S.,Raudenbush S. W.Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage; 1992:.
13. Campbell D. T.,Fiske D.Convergent and discriminant validation by the multitrait-multimethod matrix.Psychological Bulletin. 1959;56:81-105.
14. Campbell J. P.,Gasser M. B.,Oswald F. L.Individual differences and behavior in organizations. Murphy K. R., ed. San Francisco: Jossey-Bass; 1996:258-299.
15. Christiansen N. D.,Robie C.Further consideration of the use of narrow trait scales.Canadian Journal of Behavioral Science. 2011;43:183-194.
16. Conway J. M.Understanding method variance in multitrait-multirater performance appraisal matrices: Examples using general impressions and interpersonal affect as measured method factors.Human Performance. 1998;11:29-55.
17. Cronbach L. J.,Gleser G. C.,Nanda H.,Rajaratnam N.The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley; 1972:.
18. Davison M. L.,Kwak N.,Seo Y. S.,Choi J.Using hierarchical linear models to examine moderator effects: Person-by-organization interactions.Organizational Research Methods. 2002;5:231-254.
19. DeShon R. P.Measuring and analyzing behavior in organizations. Drasgow F.Schmitt N., ed. San Francisco: Jossey-Bass; 2002:189-220.
20. Dierdorff E. C.,Morgeson F. P.Effects of descriptor specificity and observability on incumbent work analysis ratings.Personnel Psychology. 2009;62:601-628.
21. Enders C. K.,Tofighi D.Centering predictor variables in cross-sectional multilevel models: A new look at an old issue.Psychological Methods. 2007;12:121-138.
22. Funder D. C.On the accuracy of personality judgment: A realistic approach.Psychological Review. 1995;102:652-670.
23. Funder D. C.,Kolar D. C.,Blackman M. C.Agreement among judges of personality: Interpersonal relations, similarity, and acquaintanceship.Journal of Personality and Social Psychology. 1995;69:656-672.
24. Galwey N. W.Introduction to mixed modelling. Hoboken, NJ: Wiley; 2006:.
25. Gellatly I. R.,Paunonen S. V.,Meyer J. P.,Jackson D. N.,Goffin R. D.Personality, vocational interests, and cognitive predictors of managerial job performance and satisfaction.Personality and Individual Differences. 1991;12:221-231.
26. Goffin R. D.,Gellatly I. R.,Paunonen S. V.,Jackson D. N.,Meyer J. P.Criterion validation of two approaches to performance appraisal: The behavioral observation scale and the relative percentile method.Journal of Business and Psychology. 1996;11:23-33.
27. Goffin R. D.,Jelley R. B.,Powell D. M.,Johnston N. G.Taking advantage of social comparisons in performance appraisal: The relative percentile method.Human Resource Management. 2009;48:251-268.
28. Goffin R. D.,Olson J. M.Is it all relative? Comparative judgments and the possible improvement of self-ratings and ratings of others.Perspectives on Psychological Science. 2011;6:48-60.
29. Hammond K. R.Probabilistic functioning in the clinical method.Psychological Review. 1955;62:255-262.
30. Han J.Crossover linear modeling: Combining multilevel heterogeneities in crossover relationships.Organizational Research Methods. 2005;8:290-316.
31. HarveyR. J. (1993). Research monograph: The development of the Common-Metric Questionnaire (CMQ). Retrieved from the Personnel Systems and Technologies Corporation website: http://www.pstc.com/documents/monograph.pdf.
32. Hauenstein N. M. A.An information processing approach to leniency in performance judgments.Journal of Applied Psychology. 1992;77:485-493.
33. Hoffman B.,Lance C. E.,Bynum B.,Gentry W. A.Rater source effects are alive and well after all.Personnel Psychology. 2010;63:119-151.
34. Hofmann D. A.An overview of the logic and rationale of hierarchical linear models.Journal of Management. 1997;6:723-744.
35. Hofmann D. A.,Gavin M. B.Centering decisions in hierarchical linear models: Implications for research in organizations.Journal of Management. 1998;24:623-641.
36. Hox J. J.Hierarchical regression models for interviewer and respondent effects.Sociological Methods and Research. 1994;22:300-318.
37. Hox J. J.Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum; 2002:.
38. Hoyt W. T.Rater bias in psychological research: When is it a problem and what can we do about it?.Psychological Methods. 2000;5:64-86.
39. Hoyt W. T.,Kerns M.Magnitude and moderators of bias in observer ratings: A meta-analysis.Psychological Methods. 1999;4:403-424.
40. Jackson D. N.,Paunonen S. V.,Fraboni M.,Goffin R. D.A five-factor versus six-factor model of personality structure.Personality and Individual Differences. 1996;20:33-45.
41. Jackson D. N.,Paunonen S. V.,Tremblay P. F.Six Factor Personality Questionnaire manual. Port Huron, MI: Sigma Assessment Systems; 2000:.
42. Kenny D. A.,Kashy D. A.Analysis of the multitrait-multimethod matrix by confirmatory factor analysis.Psychological Bulletin. 1992;112:165-172.
43. Kingstrom P. O.,Mainstone L. E.An investigation of the rater-ratee acquaintance and rater bias.Academy of Management Journal. 1985;28:641-653.
44. Kozlowski S. W. J.,Kirsch M. P.,Chao G. T.Job knowledge, ratee familiarity, conceptual similarity and halo error: An exploration.Journal of Applied Psychology. 1986;71:45-49.
45. Kreft I. G. G.,de Leeuw J.,Aiken L. S.The effect of different forms of centering in hierarchical linear models.Multivariate Behavioral Research. 1995;30:1-21.
46. LaHuis D. M.,Avis J. M.Using multilevel random coefficient modeling to investigate rater effects in performance ratings.Organizational Research Methods. 2007;10:97-107.
47. LaHuis D. M.,Ferguson M. W.The accuracy and significance tests for slope variance components in multilevel random coefficient models.Organizational Research Methods. 2009;12:418-435.
48. Lance C. E.Test of a latent structure of performance ratings derived from Wherry's (1952) theory of rating.Journal of Management. 1994;20:757-771.
49. Lance C. E.,Baranik L. E.,Lau A. R.,Scharlau E. A.Statistical and methodological myths and urban legends. Lance C. E.Vandenberg R. J., ed. New York: Routledge; 2009:337-360.
50. Lance C. E.,Noble C. L.,Scullen S. E.A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data.Psychological Methods. 2002;7:228-244.
51. Lance C. E.,Woehr D. J.,Meade A. W.Case study: A Monte Carlo investigation of assessment center construct validity models.Organizational Research Methods. 2007;10:430-448.
52. Levine F.,Sanchez J. I.Can rater training improve the quality of inferences made by raters in competency modeling? A quasi experiment.Journal of Applied Psychology. 2007;92:812-819.
53. Littell R. C.,Milliken G. A.,Stroup W. W.,Wolfinger R. D.,Schabenberger O.SAS for mixed models. Cary, NC: SAS Institute; 2006:.
54. Luo W.,Kwok O.The impacts of ignoring a crossed factor in analyzing cross-classified data.Multivariate Behavioral Research. 2009;44:182-212.
55. Marsh H. W.Confirmatory factor analysis of multitrait-multimethod data: Many problems and a few solutions.Applied Psychological Measurement. 1989;15:47-70.
56. McGraw K. O.,Wong S. P.Forming inferences about some intraclass correlation coefficients.Psychological Methods. 1996;1:30-46.
57. Morrell C. H.Likelihood ratio testing of variance components in the linear mixed-effects models using restricted maximum likelihood.Biometrics. 1998;54:1560-1568.
58. Mount M. K.,Judge T. A.,Scullen S. E.,Sytsma M. R.,Hezlett S. A.Trait, rater, and level effects in 360-degree performance ratings.Personnel Psychology. 1998;51:557-576.
59. Murphy K. R.Explaining the relationship between job performance and ratings of job performance.Industrial and Organizational Psychology. 2008;1:148-160.
60. Murphy K. R.,Cleveland J. N.Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage; 1995:.
61. Murphy K. R.,DeShon R.Interrater correlations do not estimate the reliability of job performance ratings.Personnel Psychology. 2000;53:873-900.
62. Olson J. M.,Goffin R. D.,Haynes G.Relative versus absolute measures of explicit attitudes: Implications for predicting diverse attitude-relevant criteria.Journal of Personality and Social Psychology. 2007;93:907-926.
63. Paunonen S. V.Consensus in personality judgments: Moderating effects of target-rater acquaintanceship and behavior observability.Journal of Personality and Social Psychology. 1989;56:823-833.
64. Paunonen S. V.,O'Neill T. A.Self-reports, peer ratings, and construct validity.European Journal of Personality. 2010;24:189-206.
65. Pollack B. N.Hierarchical linear modeling and the "unit of analysis" problem: A solution for analyzing responses of intact group members.Group Dynamics, Theory, Research, and Practice. 1998;2:299-312.
66. Preacher K. J.,Curran P. J.,Bauer D. J.Computational tools for probing interactions in multiple linear regression, multilevel modeling, and latent curve analysis.Journal of Educational and Behavioral Statistics. 2006;31:437-448.
67. Pulakos E. D.A comparison of rater training programs: Error training and accuracy training.Journal of Applied Psychology. 1984;69:581-588.
68. Putka D. J.,Ingerick M.,McCloy R. A.Integrating traditional perspectives on error in ratings: Capitalizing on advances in mixed-effects modeling.Industrial and Organizational Psychology. 2008;1:167-173.
69. Putka D. J.,Lance C. E.,Le H.,McCloy R. A.A cautionary note on modeling multitrait-multirater data arising from ill-structured measurement designs.Organizational Research Methods. 2011;14:503-529.
70. Putka D. J.,Le H.,McCloy R. A.,Diaz T.Ill-structured measurement designs in organizational research: Implications for estimating interrater reliability.Journal of Applied Psychology. 2008;93:959-981.
71. Putka D. J.,Sackett P. R.Handbook of employee selection. Farr J. L.Tippins N. T., ed. New York: Routledge; 2010:9-49.
72. Rasbash J.,Browne W. J.Handbook of multilevel analysis. Leeuw J.Meijer E., ed. New York: Springer Science+Business Media; 2008:301-334.
73. Raudenbush S. W.,Bryk A. S.Hierarchical linear models: Application and data analysis methods. 2002). Hierarchical linear models: Application and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.Thousand Oaks, CA: Sage; 2002:.
74. Schafer J. L.,Graham J. W.Missing data: Our view of the state of the art.Psychological Methods. 2002;7:147-177.
75. Schmidt F. L.,Hunter J. E.Measurement error in psychological research: Lessons from 26 research scenarios.Psychological Methods. 1996;1:199-223.
76. Schmidt F. L.,Hunter J. E.Theory testing and measurement error.Intelligence. 1999;27:183-198.
77. Schmidt F. L.,Viswesvaran C.,Ones D. S.Reliability is not validity and validity is not reliability.Personnel Psychology. 2000;53:901-912.
78. Scullen S. E.Using confirmatory factor analysis of correlated uniquenesses to estimate method variance in multitrait-multimethod matrices.Organizational Research Methods. 1999;2:275-292.
79. Scullen S. E.,Mount M. K.,Goff M.Understanding the latent structure of job performance ratings.Journal of Applied Psychology. 2000;85:956-970.
80. Scullen S. E.,Mount M. K.,Judge T. A.Evidence of the construct validity of developmental ratings of managerial performance.Journal of Applied Psychology. 2003;88:50-66.
81. Shavelson R. J.,Webb N. M.Generalizability theory: A primer. Newbury Park, CA: Sage; 1991:.
82. Short J. C.,Ketchen D. J.,Bennett N.,du Toit M.An examination of firm, industry, and time effects on performance using random coefficients modeling.Organizational Research Methods. 2006;9:259-284.
83. Shrout P. E.,Fleiss J. L.Intraclass correlations: Uses in assessing rater reliability.Psychological Bulletin. 1979;86:420-428.
84. Singer J. D.Using SAS PROC MIXED to fit multilevel models, hierarchical models, and individual growth models.Journal of Educational and Behavioral Statistics. 1998;23:323-355.
85. Singer J. D.,Willett J. B.Applied longitudinal data analysis: Modeling change and event occurrence. New York: Oxford University Press; 2003:.
86. Snijders T.,Bosker R.Multilevel analysis. London: Sage; 1999:.
87. Linear mixed-effects modeling in SPSS: An introduction to the mixed procedure. Chicago: Author; 2005:.
88. Van Iddekinge C. H.,Putka D. J.,Raymark P. H.,Eidson C. E.,.Modeling error variance in job specification ratings: The influence of rater, job, and organization-level factors.Journal of Applied Psychology. 2005;90:323-334.
89. Verbeke G.,Molenberghs G.Linear mixed models for longitudinal data. New York: Springer-Verlag; 2000:.
90. Viswesvaran C.,Schmidt F. L.,Ones D. S.Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences.Journal of Applied Psychology. 2005;90:108-131.
91. Wagner S. H.,Goffin R. D.Differences in accuracy of individual and comparative performance appraisal methods.Organizational Behavior and Human Decision Processes. 1997;70:95-103.
92. West B. T.Analyzing longitudinal data with the linear mixed models procedure in SPSS.Evaluation and Health Professions. 2009;32:207-228.
93. West B. T.,Welch K. B.,Galecki A. T.Linear mixed models: A practical guide to statistical software. New York: Chapman & Hall/CRC; 2007:.
94. Widaman K. F.Hierarchically nested covariance structure models for multitrait-multimethod data.Applied Psychological Measurement. 1985;9:1-26.
95. Woehr D. J.,Putka D. J.,Bowler M. C.An examination of G-theory methods for modeling multitrait-multimethod data: Clarifying links to construct validity and confirmatory factor analysis.Organizational Research Methods. 2012;15:134-161.