The impact of job complexity and performance measurement on the temporal consistency, stability, and test-retest reliability of employee job performance ratings

Journal of Applied Psychology

Volumn 90, Issue 2, 2005, Pages 269-283

  Sturman, Michael C ^a,d   Cheramie, Robin A ^b,f   Cashen, Luke H ^c,e,g  

^a Department of Obstetrics Gynecology   (United States)

^b WINTHROP UNIVERSITY   (United States)

^c LOUISIANA STATE UNIVERSITY   (United States)

^d 335 Statler Hall   (United States)

^e LOUISIANA STATE UNIVERSITY   (United States)

^f KENNESAW STATE UNIVERSITY   (United States)

^g NICHOLLS STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; CLINICAL TRIAL; HUMAN; LONGITUDINAL STUDY; META ANALYSIS; OCCUPATION; PERSONNEL MANAGEMENT; REPRODUCIBILITY; TASK PERFORMANCE; THEORETICAL MODEL; TIME;

EMPLOYEE PERFORMANCE APPRAISAL; HUMANS; LONGITUDINAL STUDIES; MODELS, THEORETICAL; OCCUPATIONS; REPRODUCIBILITY OF RESULTS; TASK PERFORMANCE AND ANALYSIS; TIME FACTORS;

EID: 17444387775     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/0021-9010.90.2.269     Document Type: Article

References (110)

1. Ackerman, P. L. (1987). Individual differences in skill learning: An integration of psychometrics and information processing perspectives. Psychological Bulletin, 102, 3-27.
2. Ackerman, P. L. (1988). Determinants of individual differences during skill acquisition: Cognitive abilities and information processing. Journal of Experimental Psychology: General, 177, 288-318.
3. Ackerman, P. L. (1989). Within-task intercorrelations of skilled performance: Implications for predicting individual differences? Journal of Applied Psychology, 74, 360-364.
4. *Adkins, C. L., & Naumann, S. E. (2001). Situational constraints on the achievement-performance relationship: A service sector study. Journal of Organizational Behavior, 22, 453-465.
5. Alvares, K. M., & Hulin, C. L. (1972). Two explanations of temporal changes in ability-skill relationships: A literature review and theoretical analysis. Human Factors, 14, 295-308.
6. Alvares, K. M., & Hulin, C. L. (1973). An experimental evaluation of a temporal decay in the prediction of performance. Organizational Behavior and Human Performance, 9, 169-185.
7. Austin, J. T., Humphreys, L. G., & Hulin, C. L. (1989). A critical reanalysis of Barrett, Caldwell, and Alexander. Personnel Psychology, 42, 583-596.
8. Barrett, G. V., & Alexander, R. A. (1989). Rejoinder to Austin, Humphreys, and Hulin: Critical analysis of Barrett, Caldwell, and Alexander. Personnel Psychology, 42, 597-612.
9. Barrett, G. V., Caldwell, M. S., & Alexander, R. A. (1985). The concept of dynamic criteria: A critical reanalysis. Personnel Psychology, 38, 41-56.
10. Barrick, M. R., & Mount, M. K. (1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1-26.
11. *Bass, B. M. (1962). Further evidence on the dynamic character of criteria. Personnel Psychology, 15, 93-97.
12. Bayley, N. (1955). On the growth of intelligence. American Psychologist, 10, 805-818.
13. Bazerman, M. H. (1998). Judgment in managerial decision making (4th ed.). New York: Wiley.
14. Becker, G. (2000). How important is transient error in estimating reliability? Going beyond simulation studies. Psychological Methods, 5, 370-379.
15. Blumberg, M., & Pringle, C. D. (1982). The missing opportunity in organizational research: Some implications for a theory of work performance. Academy of Management Review, 7, 560-569.
16. Bommer, W. H., Johnson, J. L., Rich, G. A., Podsakoff, P. M., & Mac-Kenzie, S. B. (1995). On the interchangeability of objective and subjective measures of employee performance: A meta-analysis. Personnel Psychology, 48, 587-605.
17. *Breaugh, J. A. (1981). Predicting absenteeism from prior absenteeism and work attitudes. Journal of Applied Psychology, 66, 555-560.
18. Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.
19. Campbell, J. P. (1990). Modeling the performance prediction problem in industrial and organizational psychology. In M. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 1, pp. 687-732). Palo Alto, CA: Consulting Psychologists Press.
20. Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Newbury Park, CA: Sage.
21. Cascio, W. F. (1998). Applied psychology in human resource management (5th ed.). Upper Saddle River, NJ: Prentice Hall.
22. Conway, J. M. (1999). Distinguishing contextual performance from task performance for managerial jobs. Journal of Applied Psychology, 84, 3-13.
23. Costa, P. T., Jr., & McCrae, R. R. (1988). Personality in adulthood: A six-year longitudinal study of self-reports and spouse ratings on the NEO Personality Inventory. Journal of Personality and Social Psychology, 54, 853-863.
24. Costa, P. T., Jr., & McCrae, R. R. (1992). Four ways five factors are basic. Personality and Individual Differences, 13, 653-665.
25. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
26. Deadrick, D. L., Bennett, N., & Russell, C. J. (1997). Using hierarchical linear modeling to examine dynamic performance criteria over time. Journal of Management, 23, 745-757.
27. *Deadrick, D. L., & Madigan, R. M. (1990). Dynamic criteria revisited: A longitudinal study of performance stability and predictive validity. Personnel Psychology, 43, 717-744.
28. De Shon, R. P., Ployhart, R. E., & Sacco, J. M. (1998). The estimation of reliability in longitudinal models. International Journal of Behavioral Development, 22, 493-515.
29. Epstein, S. (1979). The stability of behavior: I. On predicting most of the people most of the time. Journal of Personality and Social Psychology, 37, 1097-1126.
30. Erez, A., Bloom, M. C., & Wells, M. T. (1996). Using random rather than fixed effects models in meta-analysis: Implications for situational specificity and validity generalization. Personnel Psychology, 49, 275-306.
31. Farrell, J. N., & McDaniel, M. A. (2001). The stability of validity coefficients over time: Ackerman's (1988) model and the general aptitude battery. Journal of Applied Psychology, 86, 60-79.
32. Feldman, J. M. (1981). Beyond attribution theory: Cognitive processes in performance appraisal. Journal of Applied Psychology, 66, 127-148.
33. Fisher, R. A. (1932). Statistical methods for research workers. London: Oxford University Press.
34. Gandy, J. A. (1986, June). Job complexity, aggregated samples, and aptitude test validity: Meta-analysis of the General Aptitude Test Battery Data Base. Paper presented at the meeting of the International Personnel Management Association, San Francisco.
35. Ghiselli, E. E. (1956). Dimensional problems of criteria. Journal of Applied Psychology, 40, 1-4.
36. Ghiselli, E. E., & Haire, M. (1960). The validation of selection tests in the light of the dynamic character of criteria. Personnel Psychology, 13, 225-231.
37. Green, S. B. (2003). A coefficient alpha for test-retest data. Psychological Methods, 8, 88-101.
38. *Griffin, R. W. (1991). Effects of work redesign on employee perceptions, attitudes, and behaviors: A long-term investigation. Academy of Management Journal, 34, 425-435.
39. Gutenberg, R. L., Arvey, R. D., Osburn, H. G., & Jeanneret, P. R. (1983). Moderating effects of decision-making/information-processing job dimensions on test validities. Journal of Applied Psychology, 68, 602-608.
40. *Hanges, P. J., Schneider, B., & Niles, K. (1990). Stability of performance: An interactionist perspective. Journal of Applied Psychology, 75, 658-667.
41. *Harris, M. M., Gilbreath, B., & Sunday, J. A. (1998). A longitudinal examination of a merit pay system: Relationships among performance ratings, merit increases, and total pay increases. Journal of Applied Psychology, 83, 825-831.
42. *Harrison, D. A., Virick, M., & William, S. (1996). Working without a net: Time, performance, and turnover under maximally contingent rewards. Journal of Applied Psychology, 81, 331-345.
43. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press.
44. Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486-504.
45. Heise, D. R. (1969). Separating reliability and stability in test-retest correlation. American Sociological Review, 34, 93-101.
46. Heneman, R. L. (1986). The relationship between supervisory ratings and results-oriented measures of performance: A meta-analysis. Personnel Psychology, 39, 811-826.
47. Henry, R. A., & Hulin, C. L. (1987). Stability of skilled performance across time: Some generalizations and limitations on utility. Journal of Applied Psychology, 72, 457-462.
48. *Huffman, C. C., Nathan, B. R., & Holden, L. M. (1991). A comparison of validation criteria: Objective versus subjective performance measures and self- versus supervisor ratings. Personnel Psychology, 44, 601-619.
49. *Hofmann, D. A., Jacobs, R., & Baratta, J. E. (1993). Dynamic criteria and the measurement of change. Journal of Applied Psychology, 78, 194-204.
50. Hofmann, D. A., Jacobs, R., & Gerras, S. J. (1992). Mapping individual performance over time. Journal of Applied Psychology, 77, 185-195.
51. Hulin, C. L., Henry, R. A., & Noon, S. L. (1990). Adding a dimension: Time as a factor in the generalizability of predictive relationships. Psychological Bulletin, 107, 328-340.
52. Humphreys, L. G. (1960). Investigations of the simplex. Psychometrika, 25, 313-323.
53. Hunter, J. E. (1983). A causal analysis of cognitive ability, job knowledge, job performance, and supervisory ratings. In F. Landy, S. Zedeck, & J. Cleveland (Eds.), Performance measurement and theory (pp. 257-266). Hillsdale, NJ: Erlbaum.
54. Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72-98.
55. Hunter, J. E., & Schmidt, F. L. (1990). Method of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
56. Jackson, S. E., & Schuler, R. S. (1985). A meta-analysis and conceptual critique of research on role ambiguity and role conflict in work settings. Organizational Behavior and Human Decision Processes, 36, 16-78.
57. Jensen, A. R. (1980). Bias in mental testing. New York: Free Press.
58. Judge, T. A., Higgins, C. A., Thoresen, C. J., & Barrick, M. R. (1999). The Big Five personality traits, general mental ability, and career success across the life span. Personnel Psychology, 52, 621-652.
59. Kahneman, D., Slovic, P., & Tversky, A. (1982). Judgment under uncertainty: Heuristics and biases. Cambridge, England: Cambridge University Press.
60. Keil, C. T., & Cortina, J. M. (2001). Degradation of validity over time: A test and extension of Ackerman's model. Psychological Bulletin, 127, 673-697.
61. Kerlinger, F. N. (1986). Foundations of behavioral research. New York: Holt, Rinehart & Winston.
62. Lance, C. E., Teachout, M. S., & Donnelly, T. M. (1992). Specification of the criterion construct space: An application of hierarchical confirmatory factor analysis. Journal of Applied Psychology, 77, 437-452.
63. McDaniel, M. A., Schmidt, F. L., & Hunter, J. E. (1988). Job experience correlates of job performance. Journal of Applied Psychology, 73, 327-330.
64. *McEvoy, G. M., & Beatty, R. W. (1989). Assessment centers and subordinate appraisals of managers: A seven-year examination of predictive validity. Personnel Psychology, 42, 37-52.
65. Mendoza, J. L. (1993). Fisher transformations for correlations corrected for selection and missing data. Psychometrika, 58, 601-615.
66. *Mitchel, J. O. (1975). Assessment center validity: A longitudinal study. Journal of Applied Psychology, 60, 573-579.
67. Mitchell, T. R. (1997). Matching motivational strategies with organizational contexts. Research in Organizational Behavior, 19, 57-149.
68. Mitchell, T. R., & James, L. W. (2001). Building better theory: Time and the specification of when things happen. Academy of Management Review, 26, 530-547.
69. Motowidlo, S. J., & Van Scotter, J. R. (1994). Evidence that task perfor mance should be distinguished from contextual performance. Journal of Applied Psychology, 79, 475-480.
70. Muckler, F. A., & Seven, S. A. (1992). Selection performance measures: "Objective" versus "subjective" measurement. Human Factors, 34, 441-455.
71. Murphy, K. R. (1989). Is the relationship between cognitive ability and job performance stable over time? Human Performance, 2, 183-200.
72. Murphy, K. R., & De Shon, R. (2000). Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53, 873-900.
73. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
74. Oldham, G. R., & Cummings, A. (1996). Employee creativity: Personal and contextual factors at work. Academy of Management Journal, 39, 607-634.
75. Overton, R. C. (1998). A comparison of fixed-effects and mixed (random-effects) models for meta-analysis tests of moderator variable effects. Psychological Methods, 3, 354-379.
76. Pedhazur, E. J., & Schmelkin, L. P. (1991). Measure, design, and analysis: An integrated approach. Hillsdale, NJ: Erlbaum.
77. Peters, L. H., & O'Connor, E. J. (1980). Situational constraints and work outcomes: The influence of a frequently overlooked construct. Academy of Management Review, 5, 391-397.
78. *Ployhart, R. E., & Hakel, M. D. (1998). The substantive nature of performance variability: Predicting interindividual differences in intra-individual performance. Personnel Psychology, 51, 859-901.
79. Raju, N. S., & Brand, P. A. (2003). Determining the significance of correlations corrected for unreliability and range restriction. Applied Psychological Measurement, 27, 52-71.
80. Raju, N. S., Burke, M. J., Normand, J., & Langlois, G. M. (1991). A new meta-analytic approach. Journal of Applied Psychology, 76, 432-446.
81. Rambo, W. W., Chomiak, A. M., & Price, J. M. (1983). Consistency of performance under stable conditions of work. Journal of Applied Psychology, 68, 78-87.
82. *Ravlin, E. C., Adkins, C. L., & Meglino, B. M. (1994, August). Organizational definition of performance and individual value orientation: Interactive effects on performance and absence. In T. Welchans (Chair), New directions in fit; Interaction and process. Symposium presented at the meeting of the Academy of Management, Dallas, TX.
83. Ree, M. J., Earles, J. A., & Teachout, M. S. (1994). Predicting job performance: Not much more than g. Journal of Applied Psychology, 79, 518-524.
84. *Rally, R. R., Smither, J. W., & Vasilopoulos, N. L. (1996). A longitudinal study of upward feedback. Personnel Psychology, 49, 599-612.
85. *Rothe, H. F. (1947). Output rates among machine operators: Distributions and their reliability. Journal of Applied Psychology, 31, 484-489.
86. *Rothe, H. F. (1970). Output rates among welders: Productivity and consistency of following removal of a financial incentive system. Journal of Applied Psychology, 54, 549-551.
87. Rotundo, M., & Sackett, P. R. (2002). The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy-capturing approach. Journal of Applied Psychology, 87, 66-80.
88. *Russell, C. J. (2001). A longitudinal study of top-level executive performance. Journal of Applied Psychology, 86, 560-573.
89. Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., & Fruyt, F. (2003). International validity generalization of GMA and cognitive abilities: A European community meta-analysis. Personnel Psychology, 56, 573-605.
90. Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 scenarios. Psychological Methods, 1, 199-223.
91. Schmitt, N., Gooding, R. Z., Noe, R. A., & Kirsch, M. (1984). Meta-analyses of validity studies published between 1864 and 1982 and the investigation of study characteristics. Personnel Psychology, 37, 407-422.
92. Schwab, D. P. (1999). Research methods for organizational studies. Mahwah, NJ: Erlbaum.
93. Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
94. Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Thousand Oaks, CA: Sage.
95. Steel, R. P., & Mento, A. J. (1986). Impact of situational constraints on subjective and objective criteria of managerial job performance. Organizational Behavior and Human Decision Processes, 37, 254-265.
96. *Steel, R. P., & Van Scotter, J. R. (2003). The organizational performance cycle: Longitudinal assessment of key factors. Journal of Business and Psychology, 18, 31-50.
97. Sturman, M. C. (2003). Searching for the inverted U-shaped relationship between time and performance: Meta-analyses of the experience/performance, tenure/performance, and age/performance relationships. Journal of Management, 29, 609-640.
98. *Sturman, M. C., & Trevor, C. O. (2001). The implications of linking the dynamic performance and turnover literatures. Journal of Applied Psychology, 86, 684-696.
99. Trevor, C. O., Gerhart, B., & Boudreau, J. W. (1997). Voluntary turnover and job performance: Curvilinearity and the moderating influences of salary growth and promotions. Journal of Applied Psychology, 82, 44-61.
100. Tubre, T. C., & Collins, J. M. (2000). Jackson and Schuler (1985) revisited: A meta-analysis of the relationships between role ambiguity, role conflict, and job performance. Journal of Management, 26, 155-169.
101. Tversky, A., & Kahneman, D. (1973). Availability: A heuristic for judging frequency and probability. Cognitive Psychology, 5, 207-232.
102. *Tziner, A., Ronen, S., & Hacohen, D. (1993). A four-year validation study of an assessment center in a financial corporation. Journal of Organizational Behavior, 14, 225-237.
103. U.S. Department of Labor. (1991). Dictionary of occupational titles (4th ed.). Washington, DC: Author.
104. Vance, R. J., MacCallum, R. C., Coovert, M. D., & Hedge, J. W. (1988). Construct validity of multiple job performance measures using confirmatory factor analysis. Journal of Applied Psychology, 73, 74-80.
105. Van Scotter, J. R., Motowidlo, S. J., & Cross, T. C. (2000). Effects of task performance and contextual performance on systematic rewards. Journal of Applied Psychology, 85, 526-535.
106. Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.
107. Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2002). The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty. Journal of Applied Psychology, 87, 345-354.
108. Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108-131.
109. *Warr, P., & Bunce, D. (1995). Trainee characteristics and the outcomes of open learning. Personnel Psychology, 48, 347-375.
110. Williams, C. R., & Livingstone, L. P. (1994). Another look at the relationship between performance and voluntary turnover. Academy of Management Journal, 37, 269-298.