Internal consistency, retest reliability, and their implications for personality scale validity

Personality and Social Psychology Review

Volumn 15, Issue 1, 2011, Pages 28-50

  McCrae, Robert R ^a   Kurtz, John E ^b   Yamagata, Shinji ^c   Terracciano, Antonio ^a  

^a NATIONAL INSTITUTE ON AGING   (United States)

^b VILLANOVA UNIVERSITY   (United States)

^c KEIO UNIVERSITY   (Japan)

Author keywords

cross national; five factor model; personality traits; reliability; validity

Indexed keywords

EID: 78650981864     PISSN: 10888683     EISSN: None     Source Type: Journal    
DOI: 10.1177/1088868310366253     Document Type: Article

References (101)

1. Allik, J., Laidra, K., Realo, A., & Pullmann, H. (2004). Personality development from 12 to 18 years of age: Changes in mean levels and structures of traits. European Journal of Personality, 18, 445-462.
2. American Education Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
3. Ando, J., Suzuki, A., Yamagata, S., Kijima, N., Maekawa, H., Ono, Y., & Jang, K.L. (2004). Genetic and environmental structure of Cloninger's temperament and character dimensions. Journal of Personality Disorders, 18, 379-393.
4. Ashton, S.G., & Goldberg, L.R. (1973). In response to Jackson's challenge: The comparative validity of personality scales constructed by the external (empirical) strategy and scales developed intuitively by experts, novices, and laymen. Journal of Research in Personality, 7, 1-20.
5. Beatty, M.J., Heisel, A.D., Hall, A.E., Levine, T.R., & La France, B.H. (2002). What can we learn from the study of twins about genetic and environmental influences on interpersonal affiliation, aggressiveness, and social anxiety? A meta-analytic study. Communication Monographs, 69, 1-18.
6. Bleidorn, W., Kandler, C., Riemann, R., & Angleitner, A. (2009). Patterns and sources of adult personality development: Growth curve analyses of the NEO-PI-R scales in a longitudinal twin study. Journal of Personality and Social Psychology, 97, 142-155.
7. Borkenau, P., & Liebler, A. (1995). Observable attributes as manifestations and cues of personality and intelligence. Journal of Personality, 63, 1-25.
8. Bouchard, T.J., Jr., McGue, M., Hur, Y.-M., & Horn, J.M. (1998). A genetic and environmental analysis of the California Psychological Inventory using adult twins reared apart and together. European Journal of Personality, 12, 307-320.
9. Boyle, G.J. (1991). Does item homogeneity indicate internal consistency or item redundancy in psychometric scales? Personality and Individual Differences, 12, 291-294.
10. Carey, G., Goldsmith, H.H., Tellegen, A., & Gottesman, I.I. (1978). Genetics and personality inventories: The limits of replication with twin data. Behavior Genetics, 8, 299-313.
11. Carter, J.A., Herbst, J.H., Stoller, K.B., King, V.L., Kidorf, M.S., Costa, P.T., Jr., & Brooner, R.K. (2001). Short-term stability of NEO-PI-R personality trait scores in opioid-dependent outpatients. Psychology of Addictive Behaviors, 15, 255-260.
12. Cattell, R.B., Eber, H.W., & Tatsuoka, M.M. (1970). The handbook for the Sixteen Personality Factor Questionnaire. Champaign, IL: Institute for Personality and Ability Testing.
13. Cheung, F., Cheung, S.F., Zhang, J., Leung, K., Leong, F., & Yeh, K.-H. (2008). Relevance of Openness as a personality dimension in Chinese culture: Aspects of its cultural relevance. Journal of Cross-Cultural Psychology, 39, 81-108.
14. Chmielewski, M., & Watson, D. (2009). What is being assessed and why it matters: The impact of transient error on trait research. Journal of Personality and Social Psychology, 97, 186-202.
15. Churchill, G.A., Jr., & Peter, J.P. (1984). Research design effects on the reliability of rating scales: A meta-analysis. Journal of Marketing Research, 21, 360-375.
16. Cloninger, C.R., Przybeck, T.R., Svrakic, D.M., & Wetzel, R.D. (1994). The Temperament and Character Inventory (TCI): A guide to its development and use. St. Louis, MO: Author.
17. Comrey, A.L. (1970). Manual for the Comrey Personality Scales. San Diego, CA: EdITS.
18. Conley, J.J. (1984). The hierarchy of consistency: A review and model of longitudinal findings on adult individual differences in intelligence, personality, and self-opinion. Personality and Individual Differences, 5, 11-26.
19. Conn, S. R., & Rieke, M. L. (Eds.). (1994). 16PF Fifth Edition technical manual. Champaign, IL: Institute for Personality and Ability Testing.
20. Costa, P.T., Jr., Bagby, R.M., Herbst, J.H., & McCrae, R.R. (2005). Personality self-reports are concurrently reliable and valid during acute depressive episodes. Journal of Affective Disorders, 89, 45-55.
21. Costa, P.T., Jr., Herbst, J.H., McCrae, R.R., & Siegler, I.C. (2000). Personality at midlife: Stability, intrinsic maturation, and response to life events. Assessment, 7, 365-378.
22. Costa, P.T., Jr., & McCrae, R.R. (1978). Objective personality assessment. In M. Storandt, I. C. Siegler, & M. F. Elias (Eds.), The clinical psychology of aging (pp. 119-143). New York, NY: Plenum.
23. Costa, P.T., Jr., & McCrae, R.R. (1985). The NEO Personality Inventory manual. Odessa, FL: Psychological Assessment Resources.
24. Costa, P.T., Jr., & McCrae, R.R. (1988). Personality in adulthood: A six-year longitudinal study of self-reports and spouse ratings on the NEO Personality Inventory. Journal of Personality and Social Psychology, 54, 853-863.
25. Costa, P.T., Jr., & McCrae, R.R. (1992). Revised NEO Personality Inventory (NEO-PI-R) and NEO Five-Factor Inventory (NEO-FFI) professional manual. Odessa, FL: Psychological Assessment Resources.
26. Costa, P.T., Jr., McCrae, R.R., & Dye, D.A. (1991). Facet scales for Agreeableness and Conscientiousness: A revision of the NEO Personality Inventory. Personality and Individual Differences, 12, 887-898.
27. Cronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
28. Cronbach, L.J., Nageswari, R., & Gleser, G.C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137-163.
29. Cronbach, L.J., & Shavelson, R.J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, 391-418.
30. Dai, X.-Y., & Wu, Y.-Q. (2005). A study on NEO-PI-R used in 16-20 years old people [in Chinese]. Chinese Journal of Clinical Psychology, 13, 14-18.
31. DeShon, R.P. (1998). A cautionary note on measurement error corrections in structural equation models. Psychological Methods, 3, 412-423.
32. Eichler, W.C., & Kurtz, J.E. (2008, March). Coefficient alpha and the convergent validity of multi-item personality scales. Poster presented at the annual meeting of the Society of Personality Assessment, New Orleans, LA.
33. Fekken, G.C., Holden, R.R., Jackson, D.N., & Guthrie, G.M. (1987). An evaluation of the validity of the Personality Research Form with Filipino university students. International Journal of Psychology, 22, 399-407.
34. Fleeson, W. (2001). Toward a structure- and process-integrated view of personality: Traits as density distributions of states. Journal of Personality and Social Psychology, 80, 1011-1027.
35. Gottesman, I.I. (1966). Genetic variance in adaptive personality traits. Journal of Child Psychology and Psychiatry, 7, 199-208.
36. Gough, H.G. (1987). California Psychological Inventory administrator's guide. Palo Alto, CA: Consulting Psychologists Press.
37. Gough, H.G., & Heilbrun, A.B., Jr. (1983). Adjective Check List manual. Palo Alto, CA: Consulting Psychologists Press.
38. Grucza, R.A., & Goldberg, L.R. (2007). The comparative validity of 11 modern personality inventories: Behavioral acts, informant reports, and clinical indicators. Journal of Personality Assessment, 89, 167-187.
39. Guadiano, B.A. (2006). Is symptomatic improvement in clinical trials of cognitive-behavior therapy for psychosis clinically significant? Journal of Psychiatric Practice, 12, 11-23.
40. Guilford, J.S., Zimmerman, W.S., & Guilford, J.P. (1976). The Guilford-Zimmerman Temperament Survey Handbook: Twenty-five years of research and application. San Diego, CA: EdITS.
41. Harris, J.A., Vernon, P.A., & Jang, K.L. (1995). A possible genetic basis of accuracy in personality perception. Personality and Individual Differences, 18, 791-792.
42. Heise, D.R. (1969). Separating reliability and stability in test- retest correlation. American Sociological Review, 34, 93-101.
43. Helson, R., & Moane, G. (1987). Personality change in women from college to midlife. Journal of Personality and Social Psychology, 53, 176-186.
44. Horn, J.M., Plomin, R., & Rosenman, R. (1976). Heritability of personality traits in adult male twins. Behavior Genetics, 6, 17-30.
45. Jackson, D.N. (1975). The relative validity of scales prepared by naive item writers and those based on empirical methods of personality scale construction. Educational and Psychological Measurement, 35, 361-370.
46. Jackson, D.N. (1984). Personality Research Form manual (3rd ed.). Port Huron, MI: Research Psychologists Press.
47. Jacobson, N.S., Wilson, L., & Tupper, C. (1988). The clinical significance of treatment gains resulting from exposure-based intervention for agoraphobia: A reanalysis of outcome data. Behavior Therapy, 19, 539-554.
48. Jang, K.L., McCrae, R.R., Angleitner, A., Riemann, R., & Livesley, W.J. (1998). Heritability of facet-level traits in a cross-cultural twin sample: Support for a hierarchical model of personality. Journal of Personality and Social Psychology, 74, 1556-1565.
49. John, O.P., & Robins, R.W. (1993). Determinants of interjudge agreement on personality traits: The Big Five domains, observability, evaluativeness, and the unique perspective of the self. Journal of Personality, 61, 521-551.
50. John, O.P., & Soto, C.J. (2007). The importance of being valid: Reliability and the process of construct validation. In R. W. Robins, R. C. Fraley, & R. F. Krueger (Eds.), Handbook of research methods in personality psychology (pp. 461-494). New York, NY: Guilford.
51. Johnson, A.M., Vernon, P.A., Harris, J.A., & Jang, K.L. (2004). A behavior genetic investigation of the relationship between leadership and personality. Twin Research, 7, 27-32.
52. Kurtz, J.E., Lee, P.A., & Sherker, J.L. (1999). Internal and temporal reliability estimates for informant ratings of personality using the NEO-PI-R and IAS. Assessment, 6, 103-113.
53. Kurtz, J.E., & Parrish, C.L. (2001). Semantic response consistency and protocol validity in structured personality assessment: The case of the NEO-PI-R. Journal of Personality Assessment, 76, 315-332.
54. Löckenhoff, C.E., Terracciano, A., Bienvenu, O.J., Particiu, N.S., Nestadt, G., McCrae, R.R.,et al.(2008). Ethnicity, education, and the temporal stability of personality traits in the East Baltimore Epidemiologic Catchment Area study. Journal of Research in Personality, 42, 577-598.
55. Loehlin, J.C. (1992). Genes and environment in personality development. Newbury Park, CA: Sage.
56. Loevinger, J. (1954). The attenuation paradox in test theory. Psychological Bulletin, 51, 493-504.
57. Markon, K.E., Krueger, R.F., & Watson, D. (2005). Delineating the structure of normal and abnormal personality: An integrative hierarchical approach. Journal of Personality and Social Psychology, 88, 139-157.
58. Marsella, A.J., Dubanoski, J., Hamada, W.C., & Morse, H. (2000). The measurement of personality across cultures: Historical, conceptual, and methodological issues and considerations. American Behavioral Scientist, 44, 41-62.
59. Martin, T.A., Costa, P.T., Jr., Oryol, V.E., Rukavishnikov, A.A., & Senin, I.G. (2002). Applications of the Russian NEO-PI-R. In R. R. McCrae & J. Allik (Eds.), The five-factor model of personality across cultures (pp. 253-269). New York, NY: Kluwer/ Plenum.
60. Matthews, G. (1989). The factor structure of the 16PF: Twelve primary and three secondary factors. Personality and Individual Differences, 10, 931-940.
61. McCrae, R.R. (1982). Consensual validation of personality traits: Evidence from self-reports and ratings. Journal of Personality and Social Psychology, 43, 293-303.
62. McCrae, R.R., & Costa, P.T., Jr. (1992). Discriminant validity of NEO-PI-R facets. Educational and Psychological Measurement, 52, 229-237.
63. McCrae, R.R., & Costa, P.T., Jr. (2008). The five-factor theory of personality. In O. P. John, R. W. Robins, & L. A. Pervin (Eds.), Handbook of personality: Theory and research (3rd ed., pp. 157-180). New York, NY: Guilford.
64. McCrae, R.R., & Costa, P.T., Jr. (in press). Professional manual for the NEO Inventories: NEO-PI-3, NEO-PI-R, and NEO-FFI-3. Odessa, FL: Psychological Assessment Resources.
65. McCrae, R.R., Costa, P.T., Jr., Lima, M.P., Simões, A., Ostendorf, F., Angleitner, A.,et al.(1999). Age differences in personality across the adult life span: Parallels in five cultures. Developmental Psychology, 35, 466-477.
66. McCrae, R.R., Costa, P.T., Jr., & Martin, T.A. (2005). The NEO-PI-3: A more readable Revised NEO Personality Inventory. Journal of Personality Assessment, 84, 261-270.
67. McCrae, R.R., Costa, P.T., Jr., Martin, T.A., Oryol, V.E., Rukavishnikov, A.A., Senin, I.G.,et al.(2004). Consensual validation of personality traits across cultures. Journal of Research in Personality, 38, 179-201.
68. McCrae, R.R., Martin, T.A., & Costa, P.T., Jr. (2005). Age trends and age norms for the NEO Personality Inventory-3 in adolescents and adults. Assessment, 12, 363-373.
69. McCrae, R.R., Martin, T.A., Hřebíčková, M., Urbánek, T., Boomsma, D.I., Willemsen, G., & Costa, P.T., Jr. (2008). Personality trait similarity between spouses in four cultures. Journal of Personality, 76, 1137-1163.
70. McCrae, R.R., Stone, S.V., Fagan, P.J., & Costa, P.T., Jr. (1998). Identifying causes of disagreement between self-reports and spouse ratings of personality. Journal of Personality, 66, 285-313.
71. McCrae, R.R., Terracciano, A., & 78 Members of the Personality Profiles of Cultures Project. (2005). Universal features of personality traits from the observer's perspective: Data from 50 cultures. Journal of Personality and Social Psychology, 88, 547-561.
72. McCrae, R.R., Terracciano, A., & 79 Members of the Personality Profiles of Cultures Project. (2005). Personality profiles of cultures: Aggregate personality traits. Journal of Personality and Social Psychology, 89, 407-425.
73. McCrae, R.R., Yik, M.S.M., Trapnell, P.D., Bond, M.H., & Paulhus, D.L. (1998). Interpreting personality profiles across cultures: Bilingual, acculturation, and peer rating studies of Chinese undergraduates. Journal of Personality and Social Psychology, 74, 1041-1055.
74. McDonald, R.P. (1999). Test theory: A unified treatment. Mahwah, NJ: Erlbaum.
75. Millon, T. (1994). Millon Index of Personality Styles manual. San Antonio, TX: Psychological Corporation.
76. Nichols, R.C. (1966). The resemblance of twins in personality and interests. National Merit Scholarship Corporation Research Reports, 2, 1-23.
77. Nichols, R.C. (1967). Personality change and college. American Educational Research Journal, 4, 173-190.
78. Nunnally, J.C., & Bernstein, I. (1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.
79. Ostendorf, F., & Angleitner, A. (2004). NEO-Persönlichkeitsinventar, revidierte Form, NEO-PI-R nach Costa und McCrae [Revised NEO Personality Inventory, NEO-PI-R of Costa and McCrae]. Göttingen, Germany: Hogrefe.
80. Peter, J.P., & Churchill, G.A. (1986). Relationships among research design choices and psychometric properties of rating scales: A meta-analysis. Journal of Marketing Research, 23, 1-10.
81. Piedmont, R.L. (1994). Validation of the NEO-PI-R observer form for college students: Toward a paradigm for studying personality development. Assessment, 1, 259-268.
82. Piedmont, R.L. (2001). Cracking the plaster cast: Big Five personality change during intensive outpatient counseling. Journal of Research in Personality, 35, 500-520.
83. Piedmont, R.L., Bain, E., McCrae, R.R., & Costa, P.T., Jr. (2002). The applicability of the five-factor model in a sub-Saharan culture: The NEO-PI-R in Shona. In R. R. McCrae & J. Allik (Eds.), The five-factor model of personality across cultures (pp. 155-173). New York, NY: Kluwer/Plenum.
84. Pilia, G., Chen, W.-M., Scuteri, A., Orrú, M., Albai, G., Deo, M.,et al.(2006). Heritability of cardiovascular and personality traits in 6,148 Sardinians. PLoS Genetics, 2, 1207-1223.
85. Riemann, R., Angleitner, A., & Strelau, J. (1997). Genetic and environmental influences on personality: A study of twins reared together using the self- and peer report NEO-FFI scales. Journal of Personality, 65, 449-475.
86. Roberts, B.W., & DelVecchio, W.F. (2000). The rank-order consistency of personality traits from childhood to old age: A quantitative review of longitudinal studies. Psychological Bulletin, 126, 3-25.
87. Rushton, J.P., Fulker, D.W., Neale, M.C., Nias, D.K.B., & Eysenck, H.J. (1986). Altruism and aggression: The heritability of individual differences. Journal of Personality and Social Psychology, 50, 1192-1198.
88. Scarr, S. (1966). The origins of individual differences in adjective check list scores. Journal of Consulting Psychology, 30, 354-357.
89. Schinka, J.A., & Borum, R. (1994). Readability of normal personality inventories. Journal of Personality Assessment, 62, 95-101.
90. Schmidt, F.L., Le, H., & Ilies, R. (2003). Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. Psychological Methods, 8, 206-224.
91. Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353.
92. Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74, 107-120.
93. Snell, M.N., Mallinckrodt, B., Hill, R.D., & Lambert, M.J. (2001). Predicting counseling center clients' response to counseling: A 1-year follow-up. Journal of Counseling Psychology, 48, 463-473.
94. Streiner, D.L. (2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80, 99-103.
95. Sung, M.S., Kim, J.H., Yang, E., Abrams, K.Y., & Lyoo, I.K. (2002). Reliability and validity of the Korean version of the Temperament and Character Inventory. Comprehensive Psychiatry, 43, 236-243.
96. Terracciano, A., Costa, P.T., Jr., & McCrae, R.R. (2006). Personality plasticity after age 30. Personality and Social Psychology Bulletin, 32, 999-1009.
97. Vernon, P.A., Jang, K.L., Harris, J.A., & McCarthy, J.M. (1997). Environmental predictors of personality differences: A twin and sibling study. Journal of Personality and Social Psychology, 72, 177-183.
98. Vul, E., & Pashler, H. (2008). Measuring the crowd within: Probabilistic representations within individuals. Psychological Science, 19, 645-647.
99. Watson, D. (2004). Stability versus change, dependability versus error: Issues in the assessment of personality over time. Journal of Research in Personality, 38, 319-350.
100. Yamagata, S., Ando, J., Ostendorf, F., Angleitner, A., Riemann, R., Spinath, F.,et al.(2006, September). Cross-cultural differences in heritability of personality traits: Using behavioral genetics to study culture. Paper presented at the 4th CEFOM/21 International Symposium, Tokyo, Japan.
101. Yang, J., McCrae, R.R., Costa, P.T., Jr., Dai, X., Yao, S., Cai, T., & Gao, B. (1999). Cross-cultural personality assessment in psychiatric populations: The NEO-PI-R in the People's Republic of China. Psychological Assessment, 11, 359-368.