Alternatives for Mixed-Effects Meta-Regression Models in the Reliability Generalization Approach: A Simulation Study

Journal of Educational and Behavioral Statistics

Volumn 38, Issue 5, 2013, Pages 443-469

  López López, José Antonio ^a   Botella, Juan ^b   Sánchez Meca, Julio ^a   Marín Martínez, Fulgencio ^a  

^a UNIVERSITY OF MURCIA   (Spain)

^b UNIVERSIDAD AUTÓNOMA DE MADRID   (Spain)

Author keywords

coefficient alpha; meta analysis; mixed effects meta regression; reliability; reliability generalization

Indexed keywords

EID: 84883027760     PISSN: 10769986     EISSN: None     Source Type: Journal    
DOI: 10.3102/1076998612466142     Document Type: Article

References (68)

1. Aguayo R.,Vargas C.,de la Fuente E. I.,Lozano L. M.A meta-analytic reliability generalization study of the Maslach Burnout Inventory.International Journal of Clinical and Health Psychology. 2011;11:343-361.
2. Aguinis H.,Gottfredson R. K.,Wright T. A.Best-practice recommendations for estimating interaction effects using meta-analysis.Journal of Organizational Behavior. 2011;32:1033-1043.
3. Beretvas S. N.,Pastor D. A.Using mixed-effects models in reliability generalization studies.Educational and Psychological Measurement. 2003;63:75-95.
4. Beretvas S. N.,Suizzo M.A.,Durham J. A.,Yarnell L. M.A reliability generalization study of scores on Rotter's and Nowicki-Strickland's Locus of Control Scales.Educational and Psychological Measurement. 2008;68:97-119.
5. Berkey C. S.,Hoaglin D. C.,Mosteller F.,Colditz G. A.A random-effects meta-regression model for meta-analysis.Statistics in Medicine. 1995;14:395-411.
6. Bonett D. G.Sample size requirements for testing and estimating coefficient alpha.Journal of Educational and Behavioral Statistics. 2002;27:335-340.
7. Bonett D. G.Varying coefficient meta-analytic methods for alpha reliability.Psychological Methods. 2010;15:368-385.
8. Borenstein M.,Hedges L. V.,Higgins J. P. T.,Rothstein H. R.A basic introduction to fixed-effect and random-effects models for meta-analysis.Research Synthesis Methods. 2010;1:97-111.
9. Botella J.,Ponte G.Effects of the heterogeneity of the variances on reliability generalization: An example with the Beck Depression Inventory.Psicothema. 2011;23:516-522.
10. Botella J.,Suero M.Managing heterogeneity of variances in studies of internal consistency generalization.Methodology: European Journal of Research Methods for the Behavioral and Social Sciences. 2012;8:71-80.
11. Botella J.,Suero M.,Gambara H.Psychometric inferences from a meta-analysis of reliability and internal consistency coefficients.Psychological Methods. 2010;15:386-397.
12. Brockwell S. E.,Gordon I. R.A comparison of statistical methods for meta-analysis.Statistics in Medicine. 2001;20:825-840.
13. Churchill G. A.,Peter J. P.Research design effects on the reliability of rating scales: A meta-analysis.Journal of Marketing Research. 1984;21:360-375.
14. Conway J. M.,Jako R. A.,Goodman D. F.A meta-analysis of interrater and internal consistency reliability of selection interviews.Journal of Applied Psychology. 1995;80:565-579.
15. Cooper H.,Hedges L. V.,Valentine J. C.The handbook of research synthesis and meta-analysis. 2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.New York, NY: Russell Sage Foundation; 2009:.
16. Crocker L.,Algina J.Introduction to classical and modern test theory. New York, NY: Holt, Rinehart & Winston; 1986:.
17. Dawis R. V.Scale construction.Journal of Counseling Psychology. 1987;34:481-489.
18. Edgington E. S.Statistical inference and nonrandom samples.Psychological Bulletin. 1966;66:485-487.
19. Feldt L. S.A test of the hypothesis that Cronbach's alpha or Kuder-Richardson coefficient twenty is the same for two tests.Psychometrika. 1969;34:363-373.
20. Feldt L. S.,Charter R. A.Averaging internal consistency reliability coefficients.Educational and Psychological Measurement. 2006;66:215-227.
21. Field A. P.The problems in using fixed-effects models of meta-analysis on real-world data.Understanding Statistics. 2003;2:105-124.
22. Field A. P.Is the meta-analysis of correlation coefficients accurate when population correlations vary?.Psychological Methods. 2005;10:444-467.
23. Frick R. W.Interpreting statistical testing: Process and propensity, not population and random sampling.Behavior Research Methods, Instruments, & Computers. 1998;30:527-535.
24. Graham J. M.Congeneric and (essentially) tau-equivalent estimates of score reliability: What they are and how to use them.Educational and Psychological Measurement. 2006;66:930-944.
25. Gronlund N. E.,Linn R. L.Measurement and evaluation in teaching. 1990). Measurement and evaluation in teaching, 6th ed. New York, NY: Macmillan.New York, NY: Macmillan; 1990:.
26. Hakstian A. R.,Whalen T. E.A k-sample significance test for independent alpha coefficients.Psychometrika. 1976;41:219-231.
27. Harbord R. M.,Higgins J. P. T.Meta-regression in Stata.The Stata Journal. 2008;8:493-519.
28. Hedges L. V.,Olkin I.Statistical methods for meta-analysis. Orlando, FL: Academic Press; 1985:.
29. Hedges L. V.,Vevea J. L.Fixed- and random-effects models in meta-analysis.Psychological Methods. 1998;3:486-504.
30. Henson R. K.,Thompson B.Characterizing measurement error in scores across studies: Some recommendations for conducting "reliability generalization" studies.Measurement and Evaluation in Counseling and Development. 2002;35:113-126.
31. Hunter J. E.,Schmidt F. L.Methods of meta-analysis: Correcting errors and bias in research findings. 2004). Methods of meta-analysis: Correcting errors and bias in research findings (2nd ed.). Newbury Park, CA: Sage.Newbury Park, CA: Sage; 2004:.
32. Kieffer K. M.,Reese R. J.A reliability generalization study of the Geriatric Depression Scale.Educational and Psychological Measurement. 2002;62:969-994.
33. Knapp G.,Hartung J.Improved tests for a random effects meta-regression with a single covariate.Statistics in Medicine. 2003;22:2693-2710.
34. Laird N. M.,Mosteller F.Some statistical methods for combining experimental results.International Journal of Technology Assessment in Health Care. 1990;6:5-30.
35. Leach L. F.,Henson R. K.,Odom L. R.,Cagle L. S.A reliability generalization study of the Self-Description Questionnaire.Educational and Psychological Measurement. 2006;66:285-304.
36. López-Pina J. A.,Sánchez-Meca J.,Rosa-Alcázar A. I.The Hamilton Rating Scale for Depression: A meta-analytic reliability generalization study.International Journal of Clinical and Health Psychology. 2009;9:143-159.
37. Marín-Martínez F.,Sánchez-Meca J.Weighting by inverse variance or by sample size in random-effects meta-analysis.Educational and Psychological Measurement. 2010;70:56-73.
38. Martin A. D.,Quinn K. M.,Park J. H.MCMCpack: Markov Chain Monte Carlo in R.Journal of Statistical Software. 2011;42:1-21.
39. Mason C.,Allam R.,Brannick M. T.How to meta-analyze coefficient-of-stability estimates.Educational and Psychological Measurement. 2007;67:765-783.
40. Combining information: Statistical issues and opportunities for research. Washington, DC: National Academy Press; 1992:.
41. Overton R. C.A comparison of fixed-effects and mixed (random-effects) models for meta-analysis tests of moderator variable effects.Psychological Methods. 1998;3:354-379.
42. Parker K.A meta-analysis of the reliability and validity of the Rorschach.Journal of Personality Assessment. 1983;47:227-231.
43. Parker K. C. H.,Hanson R. K.,Hunsley J.MMPI, Rorschach, and WAIS: A meta-analytic comparison of reliability, stability, and validity.Psychological Bulletin. 1988;103:367-373.
44. Pedhazur E. J.,Schmelkin L. P.Measurement, design, and analysis: An integrated analysis. Hillsdale, NJ: Lawrence Erlbaum; 1991:.
45. Peter J. P.,Churchill G. A.Relationships among research design choices and psychometric properties of rating scales: A meta-analysis.Journal of Marketing Research. 1986;23:1-10.
46. Raudenbush S. W.The handbook of research synthesis. Cooper H.Hedges L. V., ed. New York, NY: Russell Sage Foundation; 1994:301-321.
47. Raudenbush S. W.The handbook of research synthesis and meta-analysis. 2009). Analyzing effect sizes: Random-effects models. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 295-315). New York, NY: Russell Sage Foundation.Cooper H.Hedges L. V.Valentine J. C., ed. New York, NY: Russell Sage Foundation; 2009:295-315.
48. Ray J. W.,Shadish W. R.How interchangeable are different estimators of effect size?.Journal of Consulting and Clinical Psychology. 1996;64:1316-1325.
49. Rodriguez M. C.,Maeda Y.Meta-analysis of coefficient alpha.Psychological Methods. 2006;11:306-322.
50. Romano J. L.,Kromrey J. D.What are the consequences if the assumption of independent observations is violated in reliability generalization meta-analysis studies?.Educational and Psychological Measurement. 2009;69:404-428.
51. Salgado J. F.,Moscoso S.Meta-analysis of interrater reliability of job performance ratings in validity studies of personnel selection.Perceptual and Motor Skills. 1996;83:1195-1201.
52. Sánchez-Meca J.,López-López J. A.,López-Pina J. A.Some recommended statistical analytic practices when reliability generalization studies are conducted.British Journal of Mathematical and Statistical Psychology. in press;:.
53. Sánchez-Meca J.,López-Pina J. A.,López-López J. A.,Marín-Martínez F.,Rosa-Alcázar A. I.,Gómez-Conesa A.The Maudsley Obsessive-Compulsive Inventory: A reliability generalization meta-analysis.International Journal of Clinical and Health Psychology. 2011;11:473-493.
54. Sánchez-Meca J.,Marín-Martínez F.Confidence intervals for the overall effect size in random-effects meta-analysis.Psychological Methods. 2008;13:31-48.
55. Sawilowsky S.Psychometrics versus datametrics: Comment on Vacha-Haase's "reliability generalization" method and some EPM editorial policies.Educational and Psychological Measurement. 2000;60:157-173.
56. Schmidt F. L.Detecting and correcting the lies that data tell.Perspectives on Psychological Science. 2010;5:233-242.
57. Schmidt F. L.,Oh I.-S.,Hayes T. L.Fixed- versus random-effects models in meta-analysis: Model properties and an empirical comparison of differences in results.British Journal of Mathematical and Statistical Psychology. 2009;62:97-128.
58. Schulze R.Meta-analysis: A comparison of approaches. Cambridge, MA: Hogrefe & Huber; 2004:.
59. Sidik K.,Jonkman J. N.A note on variance estimation in random effects meta-regression.Journal of Biopharmaceutical Statistics. 2005;15:823-838.
60. Vacha-Haase T.Reliability generalization: Exploring variance in measurement error affecting score reliability across studies.Educational and Psychological Measurement. 1998;58:6-20.
61. Vacha-Haase T.,Thompson B.Score reliability: A retrospective look back at 12 years of reliability generalization studies.Measurement and Evaluation in Counseling and Development. 2011;44:159-168.
62. Van Houwelingen H. C.,Arends L. R.,Stijnen T.Advanced methods in meta-analysis: Multivariate approach and meta-regression.Statistics in Medicine. 2002;21:589-624.
63. Victorson D.,Barocas J.,Song J.Reliability across studies from the functional assessment of cancer therapy-general (FACT-G) and its subscales: A reliability generalization.Quality of Life Research. 2008;17:1137-1146.
64. Viechtbauer W.Bias and efficiency of meta-analytic variance estimators in the random-effects model.Journal of Educational and Behavioral Statistics. 2005;30:261-293.
65. Viechtbauer W.Conducting meta-analyses in R with the metafor package.Journal of Statistical Software. 2010;36:1-48.
66. Yarnold P. R.,Mueser K. T.Meta-analyses of the reliability of Type A behaviour measures.British Journal of Medical Psychology. 1989;62:43-52.
67. Yin P.,Fan X.Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies.Educational and Psychological Measurement. 2000;60:201-223.
68. Zangaro G. A.,Soeken K. L.Meta-analysis of the reliability and validity of Part B of the Index of Work Satisfaction across studies.Journal of Nursing Measurement. 2005;13:7-22.