Scale coarseness as a methodological artifact: Correcting correlation coefficients attenuated from using coarse scales

Organizational Research Methods

Volumn 12, Issue 4, 2009, Pages 623-652

  Aguinis, Herman ^a   Pierce, Charles A ^b   Culpepper, Steven A ^c  

^a UNIVERSITY OF COLORADO   (United States)

^b UNIVERSITY OF MEMPHIS   (United States)

^c METROPOLITAN STATE UNIVERSITY OF DENVER   (United States)

Author keywords

Artifact; Correlation coefficient; Likert type scale; Meta analysis; Scale coarseness

Indexed keywords

EID: 70350056242     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428108318065     Document Type: Article

References (105)

1. Aguinis, H. (2001). Estimation of sampling variance of correlations in meta-analysis. Personnel Psychology, 54, 569-590.
2. Aguinis, H. (2009). Performance management (2nd ed.). Upper Saddle River, NJ: Pearson Prentice Hall.
3. Aguinis, H., Beaty, J.C., Boik, R.J., & Pierce, C.A. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology, 90, 94-107.
4. Aguinis, H., Bommer, W.H., & Pierce, C.A. (1996). Improving the estimation of moderating effects by using computer-administered questionnaires. Educational and Psychological Measurement, 56, 1043-1047.
5. Aguinis, H., & Harden, E.E. (in press). Sample size rules of thumb: Evaluating three common practices. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Received doctrine, verity, and fable in the organizational and social sciences. Mahwah, NJ: Lawrence Erlbaum.
6. Aguinis, H., Henle, C.A., & Ostroff, C. (2001). Measurement in work and organizational psychology. In N. Anderson, D.S. Ones, H. K. Sinangil, & C. Viswesvaran (Eds.), Handbook of industrial, work and organizational psychology (Vol. 1, pp. 27-50). London: Sage.
7. Aguinis, H., & Pierce, C.A. (1998). Testing moderator variable hypotheses meta-analytically. Journal of Management, 24, 577-592.
8. Aguinis, H., & Smith, M.A. (2007). Understanding the impact of test validity and bias on selection errors and adverse impact in human resource selection. Personnel Psychology, 60, 165-199.
9. Aguinis, H., Sturman, M.C., & Pierce, C.A. (2008). Comparison of three meta-analytic procedures for estimating moderating effects of categorical variables. Organizational Research Methods, 11, 9-34.
10. Aguinis, H., & Whitehead, R. (1997). Sampling variance in the correlation coefficient under indirect range restriction: Implications for validity generalization. Journal of Applied Psychology, 82, 528-538.
11. Aiken, L.R. (1983). Number of response categories and statistics on a teacher rating scale. Educational and Psychological Measurement, 43, 397-401.
12. Allen, T.D., Eby, L.T., & Lentz, E. (2006). The relationship between formal mentoring program characteristics and perceived program effectiveness. Personnel Psychology, 59, 125-153.
13. Andrews, F.M., & Farris, G.F. (1972). Time pressure and performance of scientists and engineers: A 5-year panel study. Organizational Behavior and Human Performance, 8, 185-200.
14. Arnold, H.J. (1981). A test of the multiplicative hypothesis of expectancy valence theories of work motivation. Academy of Management Journal, 24, 128-141.
15. Ash, P. (1971). Screening employment applicants for attitudes toward theft. Journal of Applied Psychology, 55, 161-164.
16. Bandalos, D.L., & Enders, C.K. (1996). The effects of nonnormality and number of response categories on reliability. Applied Measurement in Education, 9, 151-160.
17. Barrick, M.R., Bradley, B.H., Kristof-Brown, A.L., & Colbert, A.E. (2007). The moderating role of top management team interdependence: Implications for real teams and working groups. Academy of Management Journal, 50, 544-557.
18. Bendig, A.W. (1953). The reliability of self-ratings as a function of the amount of verbal anchoring and the number of categories on the scales. Journal of Applied Psychology, 37, 38-41.
19. Bendig, A.W. (1954). Reliability and the number of rating scale categories. Journal of Applied Psychology, 38, 38-40.
20. Bedrick, E.J., & Breslin, F.C. (1996). Estimating the polyserial correlation coefficient. Psychometrika, 61, 427-443.
21. Bernstein, I.H., & Teng, G. (1989). Factoring items and factoring scales are different: Spurious evidence for multidimensionality due to item categorization. Psychological Bulletin, 105, 467-477.
22. Blanton, H., & Jaccard, J. (2006). Arbitrary metrics in Psychology. American Psychologist, 61, 27-41.
23. Bobko, P. (1983). An analysis of correlations corrected for attenuation and range restriction. Journal of Applied Psychology, 68, 584-589.
24. Bobko, P., & Rieck, A. (1980). Large sample estimators for standard errors of functions of correlation coefficients. Applied Psychological Measurement, 4, 385-398.
25. Bobko, P., & Roth, P.L. (2008). Psychometric accuracy and (the continuing need for) quality thinking in meta-analysis. Organizational Research Methods, 11, 114-126.
26. Bollen, K.A., & Barb, K.H. (1981). Pearson's r and coarsely categorized measures. American Sociological Review, 46, 232-239.
27. Bowling, N.A., & Beehr, T.A. (2006). Workplace harassment from the victim's perspective: A theoretical model and meta-analysis. Journal of Applied Psychology, 91, 998-1012.
28. Brown, D.J., Cober, R.T., Kane, K., Levy, P.E., & Shalhoop, J. (2006). Proactive personality and the successful job search: A field investigation with college graduates. Journal of Applied Psychology, 91, 717-726.
29. Buel, W.D. (1965). Biographical data and the identification of creative research personnel. Journal of Applied Psychology, 49, 318-321.
30. Buel, W.D. (1969). A reply to "personal data as a predictor of small finance company credit risk" by Moran, Walsh, Clement, and Bumbeck. Personnel Psychology, 22, 57-60.
31. Carlson, R.E. (1967). Selection interview decisions: Relative influence of appearance and factual written information on an interviewer's final rating. Journal of Applied Psychology, 51, 461-468.
32. Carlson, R.E. (1969). Relative influence of a photograph vs. factual written information on an interviewer's employment decision. Personnel Psychology, 22, 45-56.
33. Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249-253.
34. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
35. Corrigan, P.W., Holmes, E.P., Luchins, D., Buican, B., Basit, A., & Parks, J.J. (1994). Staff burnout in a psychiatric-hospital: A cross-lagged panel design. Journal of Organizational Behavior, 15, 65-74.
36. Cramton, C.D., Orvis, K.L., & Wilson, J.M. (2007). Situation invisibility and attribution in distributed collaborations. Journal of Management, 33, 525-546.
37. Dalton, D.R., & Dalton, C.M. (2008). Meta-analyses: Some very good steps toward a bit longer journey. Organizational Research Methods, 11, 127-147.
38. Dierdorff, E.C., & Surface, E.A. (2007). Placing peer ratings in context: Systematic influences beyond ratee performance. Personnel Psychology, 60, 93-126.
39. Dolan, C.V. (1994). Factor analysis of variables with 2, 3, 5, and 7 response categories: A comparison of categorical variable estimators using simulated data. British Journal of Mathematical and Statistical Psychology, 47, 309-326.
40. Edwards, J.R. (2001). Ten difference score myths. Organizational Research Methods, 4, 265-287.
41. Fedor, D.B., Caldwell, S., & Herold, D.M. (2006). The effects of organizational changes on employee commitment: A multilevel investigation. Personnel Psychology, 59, 1-29.
42. Ferrin, D.L., Dirks, K.T., & Shah, P.P. (2006). Direct and indirect effects of third party relationships on interpersonal trust. Journal of Applied Psychology, 91, 870-883.
43. Flora, D.B., Finkel, E.J., & Foshee, V.A. (2003). Higher order factor structure of a self-control test: Evidence from confirmatory factor analysis with polychoric correlations. Educational and Psychological Measurement, 63, 112-127.
44. Fong, E.A., & Tosi, H.L. (2007). Effort, performance, and conscientiousness: An agency theory perspective. Journal of Management, 33, 161-179.
45. Fuller, J.A., Stanton, J.M., Fisher, G.G., Spitzmüller, C., Russell, S.S., & Smith, P.C. (2003). A lengthy look at the daily grind: Time series analysis of events, mood, stress, and satisfaction. Journal of Applied Psychology, 88, 1019-1033.
46. Garner, W.R. (1960). Rating scales, discriminability, and information transmission. Psychological Review, 67, 343-352.
47. George, J.M., & Zhou, J. (2007). Dual tuning in a supportive context: Joint contributions of positive mood, negative mood, and supervisory behaviors to employee creativity. Academy of Management Journal, 50, 605-622.
48. Griffin, M.A., Neal, A., & Parker, S.K. (2007). A new model of work role performance: Positive behavior in uncertain and interdependent contexts. Academy of Management Journal, 50, 327-347.
49. Gross, I. (1972). Creative aspects of advertising. Sloan Management Review, 14(1), 83-109.
50. Guilford, J.P. (1954). Psychometric methods (2nd ed.). New York: McGraw-Hill.
51. Halbesleben, J.R.B. (2006). Sources of social support and burnout: A meta-analytic test of the conservation of resources model. Journal of Applied Psychology, 91, 1134-1145.
52. Hedges, L.V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
53. Heslin, P.A., Vandewalle, D., & Latham, G.P. (2006). Keen to help? Managers' implicit person theories and their subsequent employee coaching. Personnel Psychology, 59, 871-902.
54. Holgado Tello, F.P., Chacon Moscoso, S., Barbero Garcia, I., & Sanduvete Chaves, S. (2006). Training satisfaction rating scale: Development of a measurement model using polychoric correlations. European Journal of Psychological Assessment, 22, 268-279.
55. Homer, P., & O'Brien, R.M. (1988). Using LISREL models with crude rank category measures. Quality & Quantity, 22, 191-201.
56. Hunter, J.E., & Schmidt, F.L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
57. Hunter, J.E., Schmidt, F.L. & Le, H. (2006). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91, 594-612.
58. James, L.R., Demaree, R.G., & Mulaik, S.A. (1986). A note on validity generalization procedures. Journal of Applied Psychology, 71, 440-450.
59. James, L.R., Demaree, R.G., Mulaik, S.A., & Mumford, M.D. (1988). Validity generalization: Rejoinder to Schmidt, Hunter, and Raju (1988). Journal of Applied Psychology, 73, 673-678.
60. Judge, T.A., Heller, D., & Mount, M.K. (2002). Five-factor model of personality and job satisfaction: A meta-analysis. Journal of Applied Psychology, 87, 530-541.
61. Kemery, E.R., Dunlap, W.P., & Griffeth, R.W. (1988). Correction for variance restriction in point-biserial correlations. Journal of Applied Psychology, 73, 688-691.
62. Kisamore, J.L., & Brannick, M.T. (2008). An illustration of the consequences of meta-analysis model choice. Organizational Research Methods, 11, 35-53.
63. Klein, H.J., Noe, R.A., & Wang, C. (2007). Motivation to learn and course outcomes: The impact of delivery mode, learning goal orientation, and perceived barriers and enablers. Personnel Psychology, 59, 665-702.
64. Komorita, S.S., & Graham, W.K. (1965). Number of scale-points and the reliability of scales. Educational and Psychological Measurement, 25, 987-995.
65. Krieg, E.F. (1999). Biases induced by coarse measurement scales. Educational and Psychological Measurement, 59, 749-766.
66. Lam, W., Huang, X., & Snape, E. (2007). Feedback-seeking behavior and leader-member exchange: Do supervisor-attributed motives matter? Academy of Management Journal, 50, 348-363.
67. Le, H., Oh, I., Shaffer, J., & Schmidt, F.L. (2007). Implications of methodological advances for the practice of personnel selection: How practitioners benefit from meta-analysis. Academy of Management Perspectives, 21(3), 6-15.
68. Lee, S., & Poon, W. (1986). Maximum likelihood estimation of polyserial correlations. Psychometrika, 51, 113-121.
69. Lyness, K.S., & Heilman, M.E. (2006). When fit is fundamental: Performance evaluations and promotions of upper-level female and male managers. Journal of Applied Psychology, 91, 777-785.
70. Marcus, B., Lee, K., & Ashton, M.C. (2007). Personality dimensions explaining relationships between integrity tests and counterproductive behavior: Big Five, or one in addition? Personnel Psychology, 60, 1-34.
71. Marler, J.H., Liang, X., & Dulebohn, J.H. (2006). Training and effective employee information technology use. Journal of Management, 32, 721-743.
72. Martell, R.F., Lane, D.M., & Emrich, C. (1996). Male-female differences: A computer simulation. American Psychologist, 51, 157-158.
73. Martin, W.S. (1973). The effects of scaling on the correlation coefficient: A test of validity. Journal of Marketing Research, 10, 316-318.
74. Martin, W.S. (1978). Effects of scaling on the correlation coefficient: Additional considerations. Journal of Marketing Research, 15, 304-308.
75. Mattell, M.S., & Jacoby, J. (1971). Is there an optimal number of alternatives for Likert scale items? Study I: Reliability and validity. Educational and Psychological Measurement, 31, 657-674.
76. McKay, P.F., Avery, D.R., Tonidandel, S., Morris, M.A., Hernandez, M., & Hebl, M.R. (2007). Racial differences in employee retention: Are diversity climate perceptions the key? Personnel Psychology, 60, 35-62.
77. Meglino, B.M., & Korsgaard, M.A. (2007). The role of other orientation in reactions to job characteristics. Journal of Management, 33, 57-83.
78. O'Brien, R.M. (1979). The use of Pearson's r with ordinal data. American Sociological Review, 44, 851-857.
79. Olsson, U. (1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44, 443-460.
80. Pearson, K., & Pearson, E.S. (1922). On polychoric coefficients of correlation. Biometrika, 14, 127-156.
81. Peters, C.C., & van Voorhis, W.R. (1940). Statistical procedures and their mathematical bases. New York: McGraw-Hill.
82. Poon, W., & Lee, S. (1987). Maximum likelihood estimation of multivariate polyserial and polychoric correlation coefficients. Psychometrika, 52, 409-430.
83. Ritchie-Scott, A. (1918). On polychoric coefficients of correlation. Biometrika, 18, 342-353.
84. Rogelberg, S.G., Leach D.J., Warr, P.B., & Burnfield, J.L. (2006). "Not another meeting!" Are meeting time demands related to employee well-being? Journal of Applied Psychology, 91, 83-96.
85. Rogelberg, S.C., Spitzmüeller, C., Little, I., & Reeve, C.L. (2006). Understanding response behavior to an online special topics organizational satisfaction survey. Personnel Psychology, 59, 903-923.
86. Russell, C.J., & Bobko, P. (1992). Moderated regression analysis and Likert scales: Too coarse for comfort. Journal of Applied Psychology, 77, 336-342.
87. Russell, C.J., Pinto, J., & Bobko, P. (1991). Appropriate moderated regression and inappropriate research strategy: A demonstration of the need to give your respondents space. Applied Psychological Measurement, 15, 257-266.
88. Schmidt, F.L. (2008). Meta-analysis: A constantly evolving research integration tool. Organizational Research Methods, 11, 96-113.
89. Schmidt, F.L., & Hunter, J.E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199-223.
90. Schmidt, F.L., Hunter, J.E., McKenzie, R.C., & Muldrow, T.W. (1979). Impact of valid selection procedures on work-force productivity. Journal of Applied Psychology, 64, 609-626.
91. Schmidt, F.L., Le, H., & Ilies, R. (2003). Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. Psychological Methods, 8, 206-224.
92. Schmidt, F.L., Oh, I., & Le, H. (2006). Increasing the accuracy of corrections for range restriction: Implications for selection procedure validities and other research results. Personnel Psychology, 59, 281-305.
93. Sheppard, W.F. (1897). On the calculation of the average square, cube, of a large number of magnitudes. Journal of the Royal Statistical Society, 60, 698-703.
94. Sheppard, W.F. (1907). The calculation of moments of a frequency-distribution. Biometrika, 5, 450-459.
95. Song, X., & Lee, S. (2003). Full maximum likelihood estimation of polychoric and polyserial correlations with missing data. Multivariate Behavioral Research, 38, 57-79.
96. Steel, P.D.G., & Kammeyer-Mueller, J. (2008). Bayesian variance estimation for meta-analysis: Quantifying our uncertainty. Organizational Research Methods, 11, 54-78.
97. Symonds, P.M. (1924). On the loss of reliability in ratings due to coarseness of the scale. Journal of Experimental Psychology, 7, 456-461.
98. Taylor, E.K., & Griess, T. (1976). The missing middle in validation research. Personnel Psychology, 29, 5-11.
99. Vandenberg, R.J. (2008). Editorial. Organizational Research Methods, 11, 6-8.
100. Vecchio, R.P., & Brazil, D.M. (2007). Leadership and sex-similarity: A comparison in a military setting. Personnel Psychology, 60, 303-335.
101. Williams, C.R. (1990). Deciding when, how, and if to correct turnover correlations. Journal of Applied Psychology, 75, 732-737.
102. Wood, J.A. (2008). Methodology for dealing with duplicate study effects in a meta-analysis. Organizational Research Methods, 11, 79-95.
103. Wright, T.A., & Bonett, D.G. (2007). Job satisfaction and psychological well-being as nonadditive predictors of workplace turnover. Journal of Management, 33, 141-160.
104. Wylie, P.B. (1976). Effects of coarse grouping and skewed marginal distributions on the Pearson product moment correlation coefficient. Educational and Psychological Measurement, 36, 1-7.
105. Zhang, G., & Browne, M.W. (2006). Bootstrap fit testing, confidence intervals, and standard error estimation in the factor analysis of polychoric correlation matrices. Behaviormetrika, 33, 61-74.