The sources of four commonly reported cutoff criteria: What did they really say?

Organizational Research Methods

Volumn 9, Issue 2, 2006, Pages 202-220

  Butts, Marcus M ^a,b   Michels, Lawrence C ^a,c  

^a UNIVERSITY OF GEORGIA   (United States)

^b Applied Psychology Program   (United States)

^c Measurement Track of the Applied Psychology Program

Author keywords

Citation analysis; Cutoff criteria; Factor analysis; Goodness of fit; Reliability

Indexed keywords

EID: 33745221864     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428105284919     Document Type: Article

References (127)

1. Allen, S. J., & Hubbard, R. (1986). Regression equations for the latent roots of random data correlation matrices with unities on the diagonal. Multivariate Behavioral Research, 21, 393-398.
2. Ambrose, M. L., & Schminke, M. (2003). Organizational structure as a moderator of the relationship between procedural justice, interactional justice, perceived organizational support, and supervisory trust. Journal of Applied Psychology, 88, 295-305.
3. Anderson, J., & Gerbing, D. (1988). Structural equation modeling in practice: A review and recommended twostep approach. Psychological Bulletin, 103, 411-423.
4. Aryee, S., Chen, Z. X., & Budhwar, P. S. (2004). Exchange fairness and employee performance: An examination of the relationship between organizational politics and procedural justice. Organizational Behavior and Human Decision Processes, 94, 1-14.
5. Barrett, G. V. (1972). Research models of the future for industrial and organizational psychology. Personnel Psychology, 25, 1-17.
6. Bartlett, M. S. (1950). Tests of significance in factor analysis. British Journal of Psychology, 3, 77-85.
7. Bass, B. M., Avolio, B. J., Jung, D. I., & Berson, Y. (2003). Predicting unit performance by assessing transformational and transactional leadership. Journal of Applied Psychology, 88, 207-218.
8. Bentler, P. M. (1980). Multivariate analysis with latent variables: Causal modeling. Annual Review of Psychology, 31, 419-456.
9. Bentler, P. M. (1990). Comparative fit indices in structural models. Psychological Bulletin, 107, 238-246.
10. Bentler, P. M. (1992). On the fit of models to covariances and methodology to the Bulletin. Psychological Bulletin, 112, 400-404.
11. Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588-606.
12. Bliese, P. D., & Halverson, R. R. (1998). Group size and measures of group-level properties: An examination of eta-squared and ICC values. Journal of Management, 24, 157-172.
13. Brown, R. D., & Hauenstein, N. M. A. (2005). Interrater agreement reconsidered: An alternative to the rwg indices. Organizational Research Methods, 8, 165-184.
14. Browne, M. W. (1968). A note on lower bounds for the number of common factors. Psychometrika, 33, 233-236.
15. Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Newbury Park, CA: Sage.
16. Bunderson, J. S. (2003). Team member functional background and involvement in management teams: Direct effects and the moderating role of power centralization. Academy of Management Journal, 46, 458-474.
17. Bycio, P., Alvares, K. M., & Hahn, J. (1987). Situational specificity in assessment center ratings: A confirmatory factor analysis. Journal of Applied Psychology, 72, 463-474.
18. Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Newbury Park, CA: Sage.
19. Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1, 245-276.
20. Castro, S. L. (2003). Data analytic methods for the analysis of multilevel questions: A comparison of intraclass coefficients, rwg(j), hierarchical linear modeling, withinand between-analysis, and random group resampling. Leadership Quarterly, 13, 69-93.
21. Charnes, J. M., & Schriesheim, C. A. (1995). Estimation of quantiles for the sampling distribution of the rwg within-group agreement index. Educational and Psychological Measurement, 55, 435-437.
22. Cliff, N. (1988). The eigenvalues-greater-than-one rule and the reliability of components. Psychological Bulletin, 103, 276-279.
23. Cohen, A., Doveh, E., & Eick, U. (2001). Statistical properties of the rWG(J) index of agreement. Psychological Methods, 6, 297-310.
24. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
25. Conway, J. M., & Huffcutt, A. I. (2003). A review and evaluation of exploratory factor analysis practices in organizational research. Organizational Research Methods, 6, 147-168.
26. Crocker, J., Luhtanen, R. K., & Cooper, M. L. (2003). Contingencies of self-worth in college students: Theory and measurement. Journal of Personality and Social Psychology, 85, 894-908.
27. Demerouti, E., Bakker, A. B., Nachreiner, F., & Schaufeli, W. B. (2001). The job demands-resources model of burnout. Journal of Applied Psychology, 86, 499-512.
28. Dixon, W. J. (Ed.). (1992). BMDP statistical software manual. Los Angeles: University of California Press.
29. Dunlap, W. P., Burke, M. J., & Smith-Crowe, K. (2003). Accurate tests of statistical significance for rwg and average deviation interrater agreement indexes. Journal of Applied Psychology, 88, 356-362.
30. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299.
31. Fava, J. L., & Velicer, W. F. (1992). The effects of overextraction on factor and component analysis. Multivariate Behavioral Research, 27, 387-415.
32. Fava, J. L., & Velicer, W. F. (1996). The effects of underextraction on factor and component analysis. Educational and Psychological Measurement, 56, 907-929.
33. Fraas, J. W., & Newman, I. (1994). A binomial test of model fit. Structural Equation Modeling, 1, 268-273.
34. George, J. M. (1990). Personality, affect, and behavior in groups. Journal of Applied Psychology, 75, 107-116.
35. George, J. M., & Bettenhausen, K. (1990). Understanding prosocial behavior, sales performance, and turnover: A group-level analysis in a service context. Journal of Applied Psychology, 75, 698-709.
36. Geurts, S. A. E., Kompier, M. A. J., Roxburgh, S., & Houtman, I. L. D. (2003). Does home-work interference mediate the relationship between workload and well-being? Journal of Vocational Behavior, 63, 532-559.
37. Gibbs, J. C., Basinger, K. S., & Fuller, D. (1992). Moral maturity: Measuring the development of sociomoral reflection. Hillsdale, NJ: Lawrence Erlbaum.
38. Glorfeld, L. W. (1995). An improvement on Horn's parallel analysis methodology for selecting the correct number of factors to retain. Educational and Psychological Measurement, 55, 377-393.
39. Grawitch, M. J., Munz, D. C., Elliott, E. K., & Mathis, A. (2003). Promoting creativity in temporary problemsolving groups: The effects of positive mood and autonomy in problem definition on idea-generating performance. Group Dynamics: Theory, Research, and Practice, 7, 200-213.
40. Greenberg, J. (2002). Who stole the money, and when? Individual and situational determinants of employee theft. Organizational Behavior and Human Decision Processes, 89, 985-1003.
41. Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
42. Gorsuch, R. L. (1990). Common factor analysis versus component analysis: Some well and little known facts. Multivariate Behavioral Research, 25, 33-39.
43. Guttman, L. (1954). Some necessary conditions for common-factor analysis. Psychometrika, 19, 149-161.
44. Harvey, R. J., & Hollander, E. (2004, April). Benchmarking rWG interrater agreement indices: Let's drop the.70 rule of thumb. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Chicago.
45. Harzing, A.-W. (2002). Are our referencing errors undermining our scholarship and credibility? The case of expatriate failure rates. Journal of Organizational Behavior, 23, 127-148.
46. Hayduk, L. A., & Glaser, D. N. (2000). Jiving the four-step, waltzing around factor analysis, and other serious fun. Structural Equation Modeling, 7, 1-35.
47. Hays, R. D., Widaman, K. F., DiMatteo, M. R., & Stacy, A. W. (1987). Structural-equation models of current drug use: Are appropriate models so simple(x)? Journal of Personality and Social Psychology, 52, 134-144.
48. Hays, W. H. (1994). Statistics (5th ed.). Fort Worth, TX: Harcourt Brace.
49. Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7, 191-205.
50. Hodgkinson, G. P., & Sadler-Smith, E. (2003). Complex or unitary? A critique and empirical re-assessment of the Allinson-Hayes Cognitive Styles Index. Journal of Occupational and Organizational Psychology, 76, 243-268.
51. Hoelter, J. W. (1983). The analysis of covariance structures: Goodness-of-fit indices. Sociological Methods & Research, 11, 325-344.
52. Homer, P. M., & Kahle, L. R. (1988). A structural equation test of the value-attitude-behavior hierarchy. Journal of Personality and Social Psychology, 54, 638-646.
53. Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185.
54. Houkes, I., Janssen, P. P. M., & de Jonge, J. (2003). Specific determinants of intrinsic work motivation, emotional exhaustion and turnover intention: A multisample longitudinal study. Journal of Occupational and Organizational Psychology, 76, 427-450.
55. Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterization model misspecification. Psychological Methods, 3, 424-453.
56. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
57. James, L. R. (1982). Aggregation bias in estimates of perceptual agreement. Journal of Applied Psychology, 67, 219-229.
58. James, L. R. (1988). Organizational climate: Another look at a potentially important construct. In S. G. Cole & R. G. Demaree (Eds.), Applications of interactionist psychology: Essays in honor of Saul B. Sells (pp. 253-282). Hillsdale, NJ: Lawrence Erlbaum.
59. James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
60. James, L. R., Demaree, R. G., & Wolf, G. (1993). RWG: An assessment of within group interrater agreement. Journal of Applied Psychology, 78, 306-309.
61. Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141-151.
62. Kirkman, B. L., Tesluk, P. E., & Rosen, B. (2001). Assessing the incremental validity of team consensus ratings over aggregation of individual-level data in predicting team effectiveness. Personnel Psychology, 54, 645-667.
63. Klein, K. J., & Kozlowski, S. W. J. (2000). Multilevel theory, research and methods in organizations: Foundations, extensions, and directions. San Francisco: Jossey-Bass.
64. Kozlowski, S. W. J., & Hattrup, K. (1992). A disagreement about within group agreement: Disentangling issues of consistency vs. consensus. Journal of Applied Psychology, 77, 161-167.
65. Lance, C. E., Foster, M. R., Gentry, W. A., & Thoresen, J. D. (2004). Assessor cognitive processes in an operational assessment center. Journal of Applied Psychology, 89, 22-35.
66. Lance, C. E., LaPointe, J. A., & Stewart, A. M. (1994). A test of the context dependency of three causal models of halo rater error. Journal of Applied Psychology, 79, 332-340.
67. Lautenschlager, G. J. (1989). A comparison of alternatives to conducting Monte Carlo analyses for determining parallel analysis criteria. Multivariate Behavioral Research, 24, 365-395.
68. LeBreton, J. M., Burgess, J. R. D., Kaiser, R. B., Atchley, E. K., & James, L. R. (2003). The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar? Organizational Research Methods, 6, 80-128.
69. LeBreton, J. M., James, L. R., & Lindell, M. K. (2005). Recent issues regarding r WG, r*WG, r WG(J), and r*WG(J). Organizational Research Methods, 8, 128-138.
70. Lindell, K. K., & Brandt, C. J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21, 271-278.
71. Lindell, K. K., Brandt, C. J., & Whitney, D. J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23, 127-135.
72. Linn, R. L. (1968). A Monte Carlo approach to the number of factors problem. Psychometrika, 33, 37-72.
73. Longman, R. S., Holden, R. R., & Fekken, G. C. (1991). Anomalies in the Allen and Hubbard parallel analysis procedure. Applied Psychological Measurement, 15, 95-97.
74. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
75. Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103, 31-410.
76. Marsh, H. W., Hau, K., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler's (1999) findings. Structural Equation Modeling, 11, 320-341.
77. McAllister, D. J., & Bigley, G. A. (2002). Work context and the definition of self: How organizational care influences organization-based self-esteem. Academy of Management Journal, 45, 894-904.
78. McAnally, M. (1999). Semi-true stories. On Beach house on the moon [CD]. Key West, FL: Margaritaville/Island.
79. McCue, P., Martin, C. R., & Buchanan, T. (2003). An investigation into the psychometric properties of the Hospital Anxiety and Depression Scale in individuals with chronic fatigue syndrome. Psychology, Health, and Medicine, 8, 425-439.
80. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46.
81. Mohammed, S., Mathieu, J. E., & Bartlett, A. L. (2002). Technical-administrative task performance, leadership task performance, and contextual performance: Considering the influence of teamand task-related composition variables. Journal of Organizational Behavior, 23, 795-814.
82. Montenelli, R. G., Jr., & Humphreys, L. G. (1976). Latent roots of random data correlation matrices with squared multiple correlations on the diagonal: A Monte Carlo study. Psychometrika, 41, 341-348.
83. Moritz, S. E., & Watson, C. B. (1998). Levels of analysis issues in group psychology: Using efficacy as an example of a multilevel model. Group Dynamics: Theory, Research, and Practice, 2, 285-298.
84. Muchinsky, P. M. (1996). The correction for attenuation. Educational and Psychological Measurement, 56, 63-75.
85. Mulaik, S. A. (1972). Foundations of factor analysis. New York: McGraw-Hill.
86. Mulaik, S. A., James, L. R., Van Alstine, J., Bennett, N., Lind, S., & Stilwell, C. D. (1989). Evaluation of goodnessof-fit indices for structural equation models. Psychological Bulletin, 105, 430-445.
87. Mulaik, S. A., & Milsap, R. E. (2000). Doing the four-step right. Structural Equation Modeling, 7, 36-73.
88. Nie, N. H., Hull, C. H., Jenkins, J. G., Steinbrenner, K., & Bent, D. H. (1975). Statistical package for the social sciences (2nd ed.). New York: McGraw-Hill.
89. Nunnally, J. C. (1967). Psychometric theory. New York: McGraw-Hill.
90. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
91. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
92. Patterson, M. G., West, M. A., & Wall, T. D. (2004). Integrated manufacturing, empowerment, and company performance. Journal of Organizational Behavior, 25, 458-474.
93. Penley, J. A., Wiebe, J. S., & Nwosu, A. (2003). Psychometric properties of the Spanish Beck Depression Inventory-II in a medical sample. Psychological Assessment, 15, 569-577.
94. Plomplun, M., & Omar, H. (2003). Do minority representative reading passages provide factorially invariant scores for all students? Structural Equation Modeling, 10, 276-288.
95. Posig, M., & Kickul, J. (2003). Extending our understanding of burnout: Test of an integrated model in nonservice occupations. Journal of Occupational Health Psychology, 8, 3-19.
96. Postmes, T., & Branscombe, N. R. (2002). Influence of long-term racial environmental composition on subjective well-being in African Americans. Journal of Personality and Social Psychology, 83, 735-751.
97. Probst, T. M. (2003). Development and validation of the Job Security Index and the Job Security Satisfaction Scale: A classical test theory and IRT approach. Journal of Occupational and Organizational Psychology, 76, 451-467.
98. Pruchno, R. A. (2003). Enmeshed lives: Adult children with developmental disabilities and their aging mothers. Psychology and Aging, 18, 851-857.
99. Rentsch, J. R., & Klimoski, R. J. (2001). Why do "great minds" think alike? Antecedents of team members schema agreement. Journal of Organizational Behavior, 22, 107-120.
100. Rothbard, N. P., & Edwards, J. R. (2003). Investment in work and family roles: A test of identity and utilitarian motives. Personnel Psychology, 56, 699-729.
101. Ryan, A. M., West, B. J., & Carr, J. Z. (2003). Effects of the terrorist attacks of 9/11/01 on employee attitudes. Journal of Applied Psychology, 88, 647-659.
102. Sackett, P. R., Schmitt, N., Ellington, J. E., & Kabin, M. B. (2001). High-stakes testing in employment, credentialing, and higher education: Prospects in a post-affirmative-action world. American Psychologist, 56, 302-318.
103. SAS Institute Inc. (1985). SAS user's guide: Basics. (Version 5 ed.). Cary, NC: Author.
104. SAS Institute Inc. (1999). SAS online Doc® [Computer software and manual]. Cary, NC: Author.
105. Schilling, M. A. (2002). Technology success and failure in winner-take-all markets: The impact of learning orientation, timing, and network externalities. Academy of Management Journal, 45, 387-398.
106. Schmidt, F. L., & Hunter, J. E. (1989). Interrater reliability coefficients cannot be computed when only one stimulus is rated. Journal of Applied Psychology, 74, 368-370.
107. Schneider, B., Hanges, P. J., Smith, D. B., & Salvaggio, A. N. (2003). Which comes first: Employee attitudes or organizational financial and market performance? Journal of Applied Psychology, 88, 836-851.
108. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
109. Sommers, C. H. (1994). Who stole feminism? How women have betrayed women. New York: Simon & Schuster.
110. Spector, P. E., Cooper, C. L., Sanchez, J. I., O'Driscoll, M., Sparks, K., Bernin, P., et al. (2002). Locus of control and well-being at work: How generalizable are Western findings? Academy of Management Journal, 45, 453-470.
111. SPSS. (1999). SPSS 10.0 syntax reference guide. Chicago: Author.
112. Susskind, A. M., Kacmar, K. M., & Borchgrevink, C. P. (2003). Customer service providers'attitudes relating to customer service and customer satisfaction in the customer-server exchange. Journal of Applied Psychology, 88, 179-187.
113. Thill, A. D. W., Holmbeck, G. N., & Bryant, F. B. (2003). Assessing the factorial invariance of Harter's selfconcept measures: Comparing preadolescents with and without spina bifida using child, parent, and teacher report. Journal of Personality Assessment, 81, 111-122.
114. Tinsley, H. E. A., & Tinsley, D. J. (1987). Uses of factor analysis in counseling psychology research. Journal of Counseling Psychology, 34, 414-424.
115. Tjosvold, D., Hui, C., & Yu, Z. (2003). Conflict management and task reflexivity for team in-role and extra-role performance in China. International Journal of Conflict Management, 14, 141-163
116. Vance, R. J., MacCallum, R. C., Coovert, M. D., & Hedge, J. W. (1988). Construct validity of multiple job performance measures using confirmatory factor analysis. Journal of Applied Psychology, 73, 74-80.
117. Van der Vegt, G. S., Emans, B. J. M., & Van der Vliert, E. (2001). Patterns of interdependence in work teams: A two-level investigation of the relations with job and team satisfaction. Personnel Psychology, 54, 51-69.
118. Van der Vegt, G. S., & Janssen, O. (2003). Joint impact of interdependence and group diversity on innovation. Journal of Management, 29, 729-751.
119. Velicer, W. F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41, 321-327.
120. Velicer, W. F., Eaton, C. A., & Fava, J. L. (2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In R. D. Goffin & E. Helmes (Eds.), Problems and solutions in human assessment: Honoring Douglas N. Jackson at seventy (pp. 41-71). Boston: Kluwer.
121. Velicer, W. F., & Jackson, D. N. (1990). Component analysis versus common factor analysis: Some issues in selecting an appropriate procedure. Multivariate Behavioral Research, 25, 1-28.
122. Wheaton, B. (1987). Assessment of fit in overidentified models with latent variables. Sociological Methods & Research, 16, 118-154.
123. Williams, L. J., & Holahan, P. J. (1994). Parsimony-based fit indices for multiple-indicator models: Do they work? Structural Equation Modeling, 1, 161-189.
124. Wood, J. M., Tataryn, D. J., & Gorsuch, R. L. (1996). Effects of underand overextraction on principal axis factor analysis with varimax rotation. Psychological Methods, 1, 354-365.
125. Wright, P. M., Gardner, T. M., Moynihan, L. M., Park, H. J., Gerhart, B., & Delery, J. E. (2001). Measurement error in research on human resources and firm performance: Additional data and suggestions for future research. Personnel Psychology, 54, 875-901.
126. Zwick, W. R., & Velicer, W. F. (1982). Factors influencing four rules for determining the number of components to retain. Multivariate Behavioral Research, 17, 253-269.
127. Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432-442.