Deficiencies of reporting in meta-analyses and some remedies

Journal of Experimental Education

Volumn 76, Issue 4, 2008, Pages 403-430

  Harwell, Michael ^a   Maeda, Yukiko ^a  

^a UNIVERSITY OF MINNESOTA   (United States)

Author keywords

Meta analysis; Reporting deficiencies; Synthesizing studies

Indexed keywords

EID: 46949087691     PISSN: 00220973     EISSN: None     Source Type: Journal    
DOI: 10.3200/JEXE.76.4.403-430     Document Type: Article

References (73)

1. Abrami, P. C., Cohen, P. A., & d'Apollonia, S. (1988). Implementation problems in meta-analysis. Review of Educational Research, 58, 151-179.
2. Algozzine, R., Browder, D., & Karvonen, M. (2001). Effects of interventions to promote self-determination for individuals with disabilities. Review of Educational Research, 71, 219-277.
3. Armor, D. J. (1972). School and family effects of black and white achievement: A reexamination of USOE data. In F. Mosteller & D. P. Moynihan (Eds.), On equality of educational opportunity (pp. 168-229). New York: Random House.
4. Bailar, J. C., & Mosteller, F. (1988). Guidelines for statistical reporting in articles for medical journals. Annals of Internal Medicine, 108, 266-273.
5. Bangert-Drowns, R. L., Wells-Parker, E., & Chevillard, I. (1997). Assessing the methodological qual-ity of research in narrative reviews and meta-analyses. In K. J. Bryant, M. Windle, & S. G. West (Eds.). The science of prevention: Methodological advances from alcohol and substance abuse (pp. 415-429). Washington, DC: American Psychological Association.
6. Becker, B. J., Hedges, L. V., & Pigott, T. D. (2001). Campbell collaboration statistical analysis policy brief: Executive summary. Retrieved December 23, 2005, from http://www.duke.edu/web/c2method/ StatsPolicyBrief.pdf
7. Begg, C. B. (1994). Publication bias. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 399-409). Thousand Oaks, CA: Sage.
8. Borman, G. D., Hewes, G. M., Overman, L. T, & Brown, S. (2003). Comprehensive school reform and achievement: A meta-analysis. Review of Educational Research, 73, 125-230.*
9. Brown, F. G. (1970). Principles of educational and psychological testing. Hinsdale, IL: Dryden Press.
10. Brown, R. (1986, April). The issue of independence of effect sizes in meta-analyses. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
11. Bus, A. G., van Ijzendoorn, M. H., & Pellegrini, A. D. (1995). Joint book reading makes for success in learning to read: A meta-analysis on intergenerational transmission of literacy. Review of Educational Research, 65, 1-21.*
12. Campbell Collaboration. (2001). Campbell collaboration guidelines. Retrieved December 23, 2005, from http://www.campbellcollaboration.org/ guidelines.asp
13. Chang, L. (1993). A power analysis of the test of homogeneity in effect-size meta-analysis. Unpublished doctoral dissertation, Michigan State University, East Lansing.
14. Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfield, F. D., et al. ( 1966). Equality of educational opportunity. (Vol. 2). Washington, DC: U.S. Government Printing Office.
15. Cook, S. A. (2004). A note on testing for homogeneity among effect sizes sharing a common control group. Psychological Methods, 9, 446-452.
16. Cooper, H. (1982). Scientific guidelines for conducting integrative research reviews. Review of Educational Research, 52, 291-302.
17. Cooper, H. (1998). Synthesizing research: A guide for literature reviews (3rd ed.). Thousand Oaks, CA: Sage.
18. Cooper, H., & Hedges, L. V. (1994). Research synthesis as a scientific enterprise. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 3-14). Thousand Oaks, CA: Sage.
19. Cooper, H., Nye, B., Charlton, K., Lindsay, J., & Greathouse, S. (1996). The effects of summer vacation on achievement test scores: A narrative and meta-analytic review. Review of Educational Research, 66, 227-268.*
20. Cooper, H., Valentine, J. C., & Charlton, K. (2003). The effects of modified school calendars on student achievement and on school and community attitudes. Review of Educational Research, 73, 1-52.*
21. D'Agostino, J. V. D., & Murphy, J. A. (2004). A meta-analysis of reading recovery in United States schools. Educational Evaluation and Policy Analysis, 26, 23-38.
22. Deci, E. L., Ryan, R. M., & Koestner, R. (2001). The pervasive negative effects of rewards on intrinsic motivation: Response to Cameron (2001). Review of Educational Research, 71, 43-52.*
23. Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70, 287-322.*
24. Farquhar, W. W., & Krumboltz, J. D. (1959). A check list for evaluating experimental research in psychology and education. Journal of Educational Research, 52, 353-354.
25. Fukkink, R. G., & de Glopper, K. (1998). Effects of instruction in deriving word meaning from con-text: A meta-analysis. Review of Educational Research, 68, 450-469.*
26. Gautam, S., Kimeldorf, G., & Sampson, A. R. (1996). Optimized scorings for ordinal data for the general linear model. Statistics & Probability Letters, 27, 231-239.
27. Georgia Department of Education-Testing. (2008). Criterion-Referenced Competency Tests (CRCT). Retrived April 15, 2008, from http://www.doe.kl2.ga.us/ ci_testing.aspx?PageReq=CI_TESTING.CRCT
28. Glass, G. V. (1976). Primary, secondary, and meta-analysis. Educational Researcher, 5, 3-8.
29. Gleser, L. J., & Olkin, I. (1994). Stochastically dependent effect sizes. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 339-355). Thousand Oaks, CA: Sage.
30. Greenhouse, J. B., & Iyengar, S. (1994). Sensitivity analysis and diagnostics. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 383-398). Thousand Oaks, CA: Sage.
31. Hall, B. W., Ward, A. W., & Comer, C. B. (1988). Published educational research: An empirical study of its quality. Journal of Educational Research, 81, 182-189.
32. Hall, J. A., Rosenthal, R., Tickle-Degnen, L., & Mosteller, F. (1994). Hypotheses and problems in research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 17-28). Thousand Oaks, CA: Sage.
33. Halvorsen, K. T. (1994). The reporting format. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 425-437). Thousand Oaks, CA: Sage.
34. Harwell, M. R. (1997). An empirical study of the Hedges (1982) homogeneity test. Psychological Methods, 2, 219-226.
35. Harwell, M. R. (1999). Evaluating the validity of educational rating data. Educational and Psychological Measurement, 59, 25-27.
36. Harwell, M. R., Maeda, Y, & Lee, K. (2004, April). Replicating and extending White's (1982) meta-analysis of the relationship between SES and student achievement. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
37. Hattie, J., & Marsh, H. W. (1996). The relationship between research and teaching: A meta-analysis. Review of Educational Research, 66, 507-542.
38. Hedges, L. V. (1982a). Fitting categorical models to effect sizes from a series of experiments. Journal of Educational Statistics, 7, 119-137.
39. Hedges, L. V. (1982b). Fitting continuous models to effect size data. Journal of Educational Statis-tics, 7, 245-270.
40. Hedges, L. V. (1990). Directions for future methodology. In K. W. Wachter & M. L. Straf (Eds.), The future of meta-analysis (pp. 11-26). Thousand Oaks, CA: Sage.
41. Hedges, L. V., & Olkin, O. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
42. Hedges, L. V., & Pigott, T. S. (2001). The power of statistical tests in meta-analyses. Psychological Bulletin, 6, 203-207.
43. Hedges, L. V., & Pigott, T. S. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Bulletin, 9, 426-445.
44. Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Bulletin, 3, 486-504.
45. Hedges, L. V., & Vevea, J. (2004). Selection model approaches to publication bias. In H. Rothstein & A. Sutton (Eds.), The handbook of publication bias (pp. 145-174). New York: Wiley.
46. Hyde, J. S., Fennema, E., & Lamon, S. J. (1990). Gender differences in mathematics performance: A meta-analysis. Psychological Bulletin, 107, 139-155.
47. Jackson, G. (1980). Methods for integrative reviews. Review of Educational Research, 50, 438-460.
48. La Paro, K. M., & Pianta, R. C. (2000). Predicting children's competence in the early school years: A meta-analytic review. Review of Educational Research, 70, 443-484.*
49. Light, R. J., & Pillemer, D. B. (1984). Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press.
50. MacColl, G. S., & White, K. D. (1998). Communicating educational research data to general, non-researcher audiences. Practical Assessment, Research, & Evaluation, 6. Retrieved December 23, 2005, from http://PAREonline.net/getvn.asp?v=6&n=7
51. Matt, G. E., & Cook, T. D. (1994). Threats of the validity of research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 503-520). Thousand Oaks, CA: Sage.
52. Orwin, R. G., & Cordray, D. S. (1985). Effects of deficient reporting on meta-analysis: A conceptual framework and reanalysis. Psychological Bulletin, 97, 134-147.
53. Pigott, T. D. (1994). Methods for handling missing data in research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 163-176). Thousand Oaks, CA: Sage.
54. Pigott, T. D. (2001). Methods for missing data in meta-analysis. Evaluation in the Health Professions, 24, 277-307.
55. Purdie, N., Hattie, J., & Carroll, A. (2002). A review of the research on interventions for attention deficit hyperactivity disorder: What works best? Review of Educational Research, 72, 61-99.*
56. Ritts, V., Patterson, M. L., & Tubbs, M. E. (1992). Expectations, impressions, and judgments of physically attractive students: A review. Review of Educational Research, 62, 413-426;*
57. Rubin, D. B. (1976). Inference and missing data. Biometrika, 63, 581-592.
58. Senk, S. L., & Thompson, D. R. (2003). Middle school mathematics curriculum reform. In S. L. Senk & D. R. Thompson (Eds.), Standards-based school mathematics curricula: What are they? What do students learn? (pp. 181-191). Mahwah, NJ: Erlbaum.
59. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 2, 420-428.
60. Slavin, R. E. (1986). Best-evidence synthesis: An alternative to meta-analytic and traditional reviews. Educational Researcher, 15, 5-11.
61. SPSS. (2005). SPSS for Windows [Computer software]. Chicago, IL: Author.
62. Swanborn, M. S. L., & de Glopper, K. (1999). Incidental word learning while reading: A meta-analysis. Review of Educational Research, 69, 261-285.*
63. Swanson, H. L., & Hoskyn, M. (1998). Experimental intervention research on students with learning disabilities: A meta-analysis of treatment outcomes. Review of Educational Research, 68, 277-321.*
64. Thompson, B. (1994, April). Common methodology mistakes in dissertations, revisited. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA. (ERIC Document Reproduction Service No. ED368771)
65. Tuckman, B. W. (1990). A proposal for improving the quality of published educational research. Educational Researcher, 19, 22-25.
66. Vockell, E. L., & Asher, W. (1974). Perceptions of document quality and use by educational decision makers and researchers. American Educational Research Journal, 11, 249-258.
67. Wandt, E., Adams, G. M., Collett, D. M., Michael, W. B., Ryan, D. G., & Shay, C. B. (1965). An evaluation of educational research published in journals. Retrieved December 23. 2005, from http://www.indiana.edu/ ∼educy520/readings/wandt65.pdf
68. Waxman, H. C., Connell, M. L., & Gray, J. (2002). A quantitative synthesis of recent research on the effects of teaching and learning with technology on student outcomes. Retrieved December 23, 2005, from http://www.ncrel.org/tech/effects/results.htm
69. White, K. R. (1982). The relation between socioeconomic status and academic achievement. Psychological Bulletin, 91, 461-481.
70. Wilkinson, L., & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604. Retrieved December 23, 2005, from http://www.loyola.edu/library/ref/articles/Wilkinson.pdf
71. Willig, A. C. (1985). A meta-analysis of selected studies on the effectiveness of bilingual education. Review of Educational Research, 55, 269-317."
72. Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Erlbaum.
73. Wortman, P. M. (1994). Judging research quality. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 97-109). Thousand Oaks, CA: Sage.