Score reliability: A retrospective look back at 12 years of reliability generalization studies

Measurement and Evaluation in Counseling and Development

Volumn 44, Issue 3, 2011, Pages 159-168

  Vacha Haase, Tammi ^a   Thompson, Bruce ^b,c  

^a Department of Environmental and Radiological Health Sciences   (United States)

^b TEXAS A AND M UNIVERSITY   (United States)

^c BAYLOR COLLEGE OF MEDICINE   (United States)

Author keywords

measurement; meta analysis; psychometrics; reliability; reliability generalization

Indexed keywords

EID: 79958292799     PISSN: 07481756     EISSN: 19476302     Source Type: Journal    
DOI: 10.1177/0748175611409845     Document Type: Article

References (77)

1. Bachner, Y.G., & O'Rourke, N. (2007). Reliability generalization of responses by care providers to the Zarit Burden Interview. Aging & Mental Health, 11, 678-685. doi:10.1080/ 13607860701529965 (Pubitemid 350249317)
2. Barnes, L.L.B., Harp, D., & Jung, W.S. (2002). Reliability generalization of scores on the Spielberger State-Trait Anxiety Inventory. Educational and Psychological Measurement, 62, 603-618. doi:10.1177/ 0013164402062004005 (Pubitemid 36688778)
3. Bem, S.L. (1981). Bem Sex-Role Inventory: Professional manual. Palo Alto, CA: Consulting Psychologists Press.
4. Beretvas, S.N., Meyers, J.L., & Leite, W.L. (2002). A reliability generalization study of the Marlowe- Crowne Social Desirability Scale. Educational and Psychological Measurement, 62, 570-589. doi:10.1177/ 0013164402062004003 (Pubitemid 36688776)
5. Beretvas, S.N., Suizzo, M.-A., Durham, J.A., & Yarnell, L.M. (2008). A reliability generalization study of scores on Rotter's and Nowicki-Strickland's locus of control scales. Educational and Psychological Measurement, 68, 97-119. doi:10.1177/0013164407301529
6. Bonnett, D.G. (2010). Varying coefficient alpha meta-analytic methods for alpha reliability. Psychological Methods, 15, 368-385. doi:10.1037/ a0020142
7. Campbell, J.S., Pulos, S., Hogan, M., & Murry, F. (2005). Reliability generalization of the Psychopathy Checklist Applied in youthful samples. Educational and Psychological Measurement, 65, 639-656. doi:10.1177/ 0013164405275666 (Pubitemid 41111849)
8. Capraro, R.M., & Capraro, M.M. (2002). Myers-Briggs Type Indicator score reliability across studies: A meta-analytic reliability generalization study. Educational and Psychological Measurement, 62, 590-602. doi:10.1177/ 0013164402062004004 (Pubitemid 36688777)
9. Capraro, M.M., Capraro, R.M., & Henson, R.K. (2001). Measurement error of scores on the Mathematics Anxiety Rating Scale across studies. Educational and Psychological Measurement, 61, 373-386. doi:10.1177/00131640 121971266
10. Capraro, R.M., & Thompson, B. (2008). The educational researcher defined: What will future researchers be trained to do ? Journal of Educational Research, 101, 247-253. doi:10.3200/ JOER.101.4.247-253
11. Caruso, J.C. (2000). Reliability generalization of the NEO Personality Scales. Educational and Psychological Measurement, 60, 236-254. doi:10.1177/00131640021970484
12. Caruso, J.C., & Edwards, S. (2001). Reliability generalization of the Junior Eysenck Personality Questionnaire. Personality and Individual Differences, 31, 173-184. doi:10.1016/S091-8869(00)00126-4 (Pubitemid 33619558)
13. Caruso, J.C., Witkiewitz, K., Belcourt-Dittloff, A., & Gottlieb, J.D. (2001). Reliability of scores from the Eysenck Personality Questionnaire: A reliability generalization study. Educational and Psychological Measurement, 61, 675-689. doi:10.1177/00131640121971437 (Pubitemid 33403790)
14. Cohen, J. (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70, 426-433. doi:10.1037/h0026714
15. Dandan, G., & Houcan, Z. (2004). A redefinition of reliability and the study of reliability generalization. Psychological Science (China). 27, 445-448.
16. Deditius-Island, H.K., & Caruso, J.C. (2002). An examination of the reliability of scores from Zuckerman's Sensation Seeking Scales, Form V. Educational and Psychological Measurement, 62, 728-734. doi:10.1177/ 0013164402062004012 (Pubitemid 36688785)
17. Dimitrov, D.M. (2002). Reliability: Arguments for multiple perspectives and potential problems with generalizability across studies. Educational and Psychological Measurement, 62, 783-801. doi:10.1177/001316402236878 (Pubitemid 36688768)
18. Dunn, T.W., Smith, T.B., & Montoya, J.A. (2006). Multicultural competency instrumentation: A review and analysis of reliability generalization. Journal of Counseling & Development, 84, 471-482. (Pubitemid 44393090)
19. Graham, J.M., & Christiansen, K. (2009). The reliability of romantic love: A reliability generalization meta-analysis. Personal Relationships, 16, 49-66. doi:10.1111/j.1475-6811.2009.01209.x
20. Graham, J.M., Liu, Y.J., & Jeziorski, J.L. (2006). The Dyadic Adjustment Scale: A reliability generalization meta-analysis. Journal of Marriage and Family, 68, 701-717. doi:10.1111/ j.1741-3737.2006.00284.x (Pubitemid 44022590)
21. Gronlund, N.E., & Linn, R.L. (1990). Measurement and evaluation in teaching (6 th ed.). New York, NY: Macmillan.
22. Hanson, W.E., Curry, K.T., & Bandalos, D.L. (2002). Reliability generalization of Working Alliance Inventory scale scores. Educational and Psychological Measurement, 62, 659-673. doi:10.1177/0013164402062004008 (Pubitemid 36688781)
23. Hellman, C.M., Fuqua, D.R., & Worley, J. (2006). A reliability generalization study on the Survey of Perceived Organizational Support: The effects of mean age and number of items on score reliability. Educational and Psychological Measurement, 66, 631-642. doi:10.1177/ 0013164406288158 (Pubitemid 44060238)
24. Hellman, C.M., Muilenburg-Trevino, E.M., & Worley, J.A. (2008). The belief in a just world: An examination of reliability estimates across three measures. Journal of Personality Assessment, 90, 399-401. doi:10.1080/ 00223890802108238 (Pubitemid 351918508)
25. Henson, R.K., & Hwang, D.-Y. (2002). Variability and prediction of measurement error in a Kolb's Learning Style Inventory Scores: A reliability generalization study. Educational and Psychological Measurement, 62, 712-727. doi:10.1177/ 0013164402062004011 (Pubitemid 36688784)
26. Henson, R.K., Kogan, L.R., & Vacha-Haase, T. (2001). A reliability generalization study of the Teacher Efficacy Scale and related instruments. Educational and Psychological Measurement, 61, 404-420. doi:10.1177/ 00131640121971284
27. Henson, R.K., & Thompson, B. (2002). Characterizing measurement error in scores across studies: Some recommendations for conducting "reliability generalization" (RG) studies. Measurement and Evaluation in Counseling and Development, 35, 113-127.
28. Hogan, T.P., Benjamin, A., & Brezinski, K.L. (2000). Reliability methods: A note on the frequency of use of various types. Educational and Psychological Measurement, 60, 523-531.
29. Hunter, J.E., & Schmidt, F.L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
30. Huynh, Q.-L., Howell, R.T., & Benet-Martinez, V. (2009). Reliability of bidimensional acculturation scores: A meta-analysis. Journal of Cross-Cultural Psychology, 40, 256-274. doi:10.1177/0022022108328919
31. Kieffer, K.M., Cronin, C., & Fister, M.C. (2004). Exploring variability and sources of measurement error in Alcohol Expectancy Questionnaire reliability coefficients: A meta-analytic reliability generalization study. Journal of Studies on Alcohol, 65, 663-671. (Pubitemid 39383199)
32. Kieffer, K.M., & Reese, R.J. (2002). A reliability generalization study of the Geriatric Depression Scale. Educational and Psychological Measurement, 62, 969-994. doi:10.1177/0013 164402238085 (Pubitemid 36688759)
33. Knapp, T.R. (1978). Canonical correlation analysis: A general parametric significance testing system. Psychological Bulletin, 85, 410-416. doi:10.1037//0033-2909.85.2.410
34. Lane, G.G., White, A.E., & Henson, R.K. (2002). Expanding reliability generalization methods with KR-21 estimates: An RG study of the Coopersmith Self-Esteem Inventory. Educational and Psychological Measurement, 62, 685-711. doi:10.1177/0013164402062004010 (Pubitemid 36688783)
35. Leach, L.F., Henson, R.K., Odom, L.R., & Cagle, L.S. (2006). A reliability generalization study of the Self-Description Questionnaire. Educational and Psychological Measurement, 66, 285-304. doi:10.1177/ 0013164405284030 (Pubitemid 43344144)
36. Li, A., & Bagger, J. (2007). The Balanced Inventory of Desirable Responding (BIDR): A reliability generalization study. Educational and Psychological Measurement, 67, 525-544. doi:10.1177/001316440292087
37. Lopez-Pina, J.A., Sanchez-Meca, J., & Rosa-Alcazar, A.I. (2009). The Hamilton Rating Scale for Depression: A meta-analytic reliability generalization study. International Journal of Clinical and Health Psychology, 9, 143-159.
38. McCrae, R.R., Kurtz, J.E., Yamagata, S., & Terracciano, A. (2011). Internal consistency, retest reliability, and their implications for personality scale validity. Personality and Social Psychology Review, 15, 28-50. doi:10.1177/1088868310366253
39. Miller, B.K., & Byrne, Z.S. (2009). Perceptions of organizational politics: A demonstration of the reliability generalization technique. Journal of Managerial Issues, 21, 280-300.
40. Miller, C.S., Shields, A.L., Campfield, D., Wallace, K.A., & Weiss, R.D. (2007). Substance use scales of the Minnesota Multiphasic Personality Inventory: An exploration of score reliability via meta-analysis. Educational and Psychological Measurement, 67, 1052-1065. doi:10.1177/0013164406299130
41. Miller, C.S., Woodson, J., Howell, R.T., & Shields, A.L. (2009). SASSI: Assessing the reliability of scores produced by the Substance Abuse Subtle Screening Inventory. Substance Use & Misuse, 44, 1090-1100.
42. Mji, A., & Alkhateeb, H.M. (2005). Combining reliability coefficients: Toward reliability generalization of the Conceptions of Mathematics Questionnaire. Psychological Reports, 96, 627-634. doi:10.2466/pr0.96.3.627-634 (Pubitemid 41053258)
43. Nilsson, J.E., Schmidt, C.K., & Meek, W.D. (2002). Reliability generalization: An examination of the Career Decision-Making Self-Efficacy Scale. Educational and Psychological Measurement, 62, 647-658. doi:10.1177/ 0013164402062004007 (Pubitemid 36688780)
44. O'Rourke, N. (2004). Reliability generalization of responses by care providers to the Center for Epidemiologic Studies-Depression Scale. Educational and Psychological Measurement, 64, 973-990. doi:10.1177/0013164404268668 (Pubitemid 39604745)
45. Pedhazur, E.J., & Schmelkin, L.P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Erlbaum.
46. Reese, R.J., Kieffer, K.M., & Briggs, B.K. (2002). A reliability generalization study of select measures of adult attachment style. Educational and Psychological Measurement, 62, 619-646. doi:10.1177/0013164402062004006 (Pubitemid 36688779)
47. Reinhardt, B. (1996). Factors affecting coefficient alpha: A mini Monte Carlo study. In B. Thompson (Ed.), Advances in social science methodology (Vol. 4, pp. 3-20). Greenwich, CT: JAI Press.
48. Rexrode, K.R., Petersen, S., & O'Toole, S. (2008). The Ways of Coping Scale: A reliability generalization study. Educational and Psychological Measurement, 68, 262-280. doi:10.1177/ 0013164407310128
49. Rodriguez, M.C., & Maeda, Y. (2006). Meta-analysis of coefficient alpha. Psychological Methods, 11, 306-322. doi:10.1037/1082-989X.11.3.306 (Pubitemid 44414074)
50. Ross, M.E., Blackburn, M., & Forbes, S. (2005). Reliability generalization of the Patterns of Adaptive Learning Survey Goal Orientation Scales. Educational and Psychological Measurement, 65, 451-464. doi:10.1177/0013164404272496 (Pubitemid 40856004)
51. Rouse, S.V. (2007). Using reliability generalization methods to explore measurement error: An illustration using the MMPI-2 PSY-5 Scales. Journal of Personality Assessment, 88, 264-275. (Pubitemid 46917722)
52. Ryngala, D.J., Shields, A.L., & Caruso, J.C. (2005). Reliability generalization of the Revised Children's Manifest Anxiety Scale. Educational and Psychological Measurement, 65, 259-271. doi:10.1177/0013164404272495
53. Schmidt, F.L., & Hunter, J.E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540. doi:10.1037//0021-9010.62.5.529
54. Shields, A.L., & Caruso, J.C. (2003). Reliability generalization of the Alcohol Use Disorders Identification Test. Educational and Psychological Measurement, 63, 404-413. doi:10.1177/0013164403063003004 (Pubitemid 36751230)
55. Shields, A.L., & Caruso, J.C. (2004). A reliability induction and reliability generalization study of the CAGE Questionnaire. Educational and Psychological Measurement, 64, 254-270. doi:10.1177/0013164403261814 (Pubitemid 38430614)
56. Thompson, B. (1992). Two and one-half decades of leadership in measurement and evaluation. Journal of Counseling and Development, 70, 434-438.
57. Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.
58. Thompson, B. (Ed.). (2003). Score reliability: Contemporary thinking on reliability issues. Thousand Oaks, CA: Sage.
59. Thompson, B., & Cook, C. (2002). Stability of the reliability of LibQUAL + TM scores: A reliability generalization meta-analysis study. Educational and Psychological Measurement, 62, 735-743. doi:10.1177/ 0013164402062004013 (Pubitemid 36688786)
60. Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174-195. doi:10.1177/00131640021970448
61. Urbina, S. (2004). Essentials of psychological testing. Hoboken, NJ: John Wiley.
62. Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20. doi:10.1177/00131640121971059
63. Vacha-Haase, T., Henson, R.K., & Caruso, J.C. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational and Psychological Measurement, 62, 562-569. doi:10.1177/ 0013164402062004002 (Pubitemid 36688775)
64. Vacha-Haase, T., Kogan, L.R., Tani, C.R., & Woodall, R.A. (2001). Reliability generalization: Exploring variation of reliability coefficients of MMPI clinical scales scores. Educational and Psychological Measurement, 61, 45-59. doi:10.1177/00131640121971059
65. Vacha-Haase, T., Kogan, L.R., & Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60, 509-522. doi:10.1177/00131640021970682
66. Vacha-Haase, T., Ness, C.M., Nilsson, J., & Reetz, D. (1999). Practices regarding reporting of reliability coefficients: A review of three journals. Journal of Experimental Education, 67, 335-341. doi:10.1080/ 00220979909598487
67. Vacha-Haase, T., Tani, C.R., Kogan, L.R., Woodall, R.A., & Thompson, B. (2001). Reliability generalization: Exploring reliability variations on MMPI/MMPI-2 validity scale scores. Assessment, 8, 391-401. doi:10.1177/ 107319110100800404 (Pubitemid 33595959)
68. Vassar, M., & Crosby, J.W. (2008). A reliability generalization study of coefficient alpha for the UCLA Loneliness Scale. Journal of Personality Assessment, 90, 601-607. doi:10.1080/ 00223890802388624
69. Victorson, D., Barocas, J., Song, J., & Cella, D. (2008). Reliability across studies from the Functional Assessment of Cancer Therapy-General (FACT-G) and its subscales: A reliability generalization. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care & Rehabilitation, 17, 1137-1146. doi:10.1007/s11136-008-9398-2
70. Wallace, K.A., & Wheeler, A.J. (2002). Reliability generalization of the Life Satisfaction Index. Educational and Psychological Measurement, 62, 674-684. doi:10.1177/001316 4402062004009 (Pubitemid 36688782)
71. Wilkinson, L., & American Psychological Association (APA) Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604. doi:10.1037//0003066X.54.8.594
72. Willson, V.L. (1980). Research techniques in AERJ articles: 1969 to 1978. Educational Researcher, 9 (6). 5-10. doi:10.2307/1175221
73. Yetkiner, Z.E., & Thompson, B. (in press). Demonstration of how score reliability is integrated into SEM and how reliability affects all statistical analyses. Multiple Linear Regression Viewpoints, 36 ( 2 ).
74. Yin, P., & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60, 201-223. doi:10.1177/ 00131640021970466
75. Youngstrom, E.A., & Green, K.W. (2003). Reliability generalization of self-report of emotions when using the Differential Emotions Scale. Educational and Psychological Measurement, 63, 279-295. doi:10.1177/0013164403253226
76. Zangaro, G.A., & Soeken, K.L. (2005). Meta-analysis of the reliability and validity of Part B of the Index of Work Satisfaction across studies. Journal of Nursing Measurement, 13, 7-22. doi:10.1891/jnum.2005.13.1.7 (Pubitemid 41551747)
77. Zientek, L.R., & Thompson, B. (2009). Matrix summaries improve research reports: Secondary analyses using published literature. Educational Researcher, 38, 343-352. doi:10.3102/0013189X09339056