What are the consequences if the assumption of independent observations is violated in reliability generalization meta-analysis studies?

Educational and Psychological Measurement

Volumn 69, Issue 3, 2009, Pages 404-428

  Romano, Jeanine L ^a   Kromrey, Jeffrey D ^b  

^a AMERICAN UNIVERSITY OF SHARJAH   (United Arab Emirates)

^b UNIVERSITY OF SOUTH FLORIDA   (United States)

Author keywords

Intraclass correlation; Meta analysis; Reliability generalization; Research design assumptions; Simulation studies

Indexed keywords

EID: 65649086039     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164408323237     Document Type: Article

References (52)

1. Barcikowski, R.S. (1981). Satistical power with group mean as the unit of analysis. Journal of Educatonal Statistics, 6, 267-285.
2. Barnes, L.L.B., Harp, D., & Jung, W.S. (2002). Reliability generalization of scores on the Spielberger State-Trait Anxiety Inventory. Educational and Psychological Measurement, 62, 603-618. (Pubitemid 36688778)
3. Beck, A.T., Ward, C.H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 53-63.
4. Becker, B.J. (2000). Multivariate meta-analysis. In H. E. A. Tinsley & S. Brown (Eds.), Handbook of applied and multivariate statistics and mathematical modeling (pp. 499-525). San Diego, CA: Academic Press.
5. Becker, B.J., & Kim, J. (2002, April). Effects of ignoring nonindependence of effect sizes on Hedge's homogeneity test. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
6. Bem, S.L. (1974). The measurement of psychological androgyny. Journal of Consulting and Clinical Psychology, 42, 155-162.
7. Beretevas, S.N., & Pastor, D.A. (2003). Using mixed-effects models in reliability generalization studies. Educational and Psychological Measurement, 63, 75-95. (Pubitemid 36218407)
8. Bock, R.D. (1975). Multivariate statistical methods in behavioral research. New York: McGraw-Hill.
9. Bonett, D.G. (2002). Sample size requirements for testing and estimating coefficient alpha. Journal of Educational and Behavioral Statistics, 27, 335-340. (Pubitemid 36385299)
10. Bradley, J.V. (1978). Robustness ? British Journal of Mathematics and Statistical Psychology, 31, 144-152.
11. Capraro, R.M., & Capraro, M.M. (2002). Myers-Briggs Type Indicator score reliability across studies: A meta-analytic reliability generalization study. Educational and Psychological Measurement, 62, 590-602. (Pubitemid 36688777)
12. Caruso, J.C. (2000). Reliability generalization of NEO personality scales. Education and Psychological Measurement, 60, 236-254.
13. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2 nd ed.). New York: Academic Press.
14. Cooper, H. (1979). Statistically combining independent studies: A meta-analysis of sex differences in conformity research. Journal of Personality and Social Psychology, 37, 131-146.
15. Costa, P.T., Jr., & McCrae, R.R. (1985). The NEO Personality Inventory: Professional manual. Odessa, FL: Psychological Assessment Resources.
16. Crocker. L. & Angina, J. (1986). Introduction to classical and modern test theory. Toronto, Ontario, Canada: Holt Rinehart & Winston.
17. Feldt, L.S. (1965). The approximate sampling distribution of Kuder-Richardson reliability coefficient twenty. Psychometrika, 30, 357-370.
18. Fisher, R.A. (1950). Statistical methods for research workers. Edinburgh, UK: Oliver and Boyd.
19. Fuller, J.B., & Hester, K. (1999). Comparing the sample-weighted and unweighted meta-analysis: An applied perspective. Journal of Management, 25, 803-828.
20. Greenhouse, J.B., & Iyengar, S. (1994). Sensitivity analysis and diagnostics. In H. Cooper & L. Hedges (Eds.), The handbook of research synthesis (pp. 383-398). New York: Russell Sage.
21. Hakstian, A.R., & Whalen, T.E. (1976). A k-sample significance test for independent alpha coefficients. Psychometrika, 41, 219-231.
22. Hanson, B.A., & Beguin, A.A. (1999, April). Obtaining a common scale for IRT item parameters using separate versus concurrent estimation in the common item nonequivalent groups equating design. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Québec, Canada.
23. Hedges, L.V., & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press, Inc.
24. Helms, J.E. (1999). Another meta-analysis of the White Racial Identity Attitude Scale's Cronbach alphas: Implication for validity. Measurement and Evaluation in Counseling and Development, 32, 122-137.
25. Henson, R.K. (2001). Understanding internal consistency reliabilities estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34, 177-189. (Pubitemid 33367275)
26. Hewitt-Gervais, C.M., & Kromrey, J.D. (1999, April). Analysis strategies for conducting F-tests with non-independent observations: An empirical investigation. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Québec, Canada.
27. Hunter, J.E., & Schmidt, F.L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
28. Iacobucci, D., & Duhachek, A. (2003). Advancing alpha: Measuring reliability with confidence. Journal of Consumer Psychology, 13, 478-487. (Pubitemid 37550799)
29. Kenny, D.A., & Judd, C.M. (1986). Consequences of violating the independence assumption in analysis of variance. Psychological Bulletin, 99, 422-431.
30. Koning, A.J., & Franses, P.H. (2003). Confidence intervals for Cronbach's coefficient alpha values (ERIM Report No. ERS-2003-041-MKT). Available at http://ssrn.com/abstract=423658
31. Kromrey, J.D., & Dickinson, W.B. (1996). Detecting unit of analysis problems in nested designs: Statistical power and Type I error rates of the F test for groups-within-treatments effects. Educational and Psychological Measurement, 56, 215-231. (Pubitemid 126336538)
32. Landman, J.T., & Dawes, R.M. (1982). Psychotherapy outcomes: Smith and Glass' conclusions stand up to scrutiny. American Psychologist, 37 (5). 504-516.
33. Maydeu-Olivares, A., Coffman, D.L., & Hartmann, W.M. (2007). Asymptotically distribution-free (ADF) interval estimation of coefficient alpha. Psychological Methods, 12, 157-176.
34. Myers, I.B., & McCaulley, M.H. (1985). Manual: A guide to the development and use of the Myers-Briggs Type Indicator. Palo Alto, CA: Consulting Psychologists Press.
35. Raudenbush, S.W. (1994). Random effects models. In H. Cooper & L. Hedges (Eds.), The handbook of research synthesis (pp. 301-321). New York: Russell Sage.
36. Raudenbush, S.W., Becker, B.J., & Kalaian, H. (1988). Modeling multivariate effect sizes. Psychological Bulletin, 103, 111-120.
37. Raudenbush, S.W., & Bryk, A.S. (1987). Examining correlates of diversity. Journal of Educational Statistics, 12, 241-269.
38. Robey, R., & Barcikowski, R. (1992). Type I error and the number of iterations in Monte Carlo studies of robustness. British Journal of Mathematics and Statistical Psychology 45, 283-288.
39. Romano, J.L., Kromrey, J.D., & Hibbard, S.T. (2008, March). Confidence interval methods for coefficient alpha: A Monte Carlo study of eight methods proposed in the literature. Paper presented at the annual meeting of the American Educational Research Association, New York.
40. Rosenthal, R., & Rubin, D.B. (1986). Meta-analytic procedures for combining studies with multiple effect sizes. Psychological Bulletin, 99, 400-406.
41. Sawilowsky, S.S. (2000). Psychometrics versus datametrics: Comment on Vacha-Haase's ''reliability generalization'' method and some ERM editorial policies. Educational and Psychological Measurement, 60, 157-173.
42. Scariano, S.M., & Davenport, J.M. (1987). The effects of violations of independence assumptions in an one-way ANOVA. The American Statistician, 41, 123-129.
43. Schmidt, F.L., & Hunter, J.E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540.
44. Smith, M.L., Glass, G.V., & Miller, T.I. (1980). The benefits of psychotherapy, Baltimore: Johns Hopkins Unversity Press.
45. Spielberger, C.D., Gorsuch, R.L., & Lushene, R.E. (1970). Manual for the State-Trait Anxiety Inventory. Palo Alto, CA: Consulting Psychologists Press.
46. Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174-195.
47. Tracz, S.M., Elmore, P.B., & Pohlmann, J.T. (1992). Correlational meta-analysis: Independent and non independent cases. Educational and Psychological Measurement, 52, 879-888.
48. Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20. (Pubitemid 128020876)
49. Vacha-Haase, T., Henson, R.K., & Caruso, J. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational and Psychological Measurement, 62, 562-569. (Pubitemid 36688775)
50. Vacha-Haase, T., Kogan, L.R., & Thompson. B. (2000). Sample composition and variabilities in published studies versus the in test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60, 509-522.
51. Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
52. Yin, P., & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability Generalization across studies. Educational and Psychological Measurement, 61, 201-223.