Of apples and oranges, file drawers and garbage: Why validity issues in meta-analysis will not go away

Clinical Psychology Review

Volumn 17, Issue 8, 1997, Pages 881-901

  Sharpe, Donald ^a  

^a UNIVERSITY OF REGINA   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

CLINICAL PSYCHOLOGY; CLINICAL TRIAL; HUMAN; INFORMATION PROCESSING; MEDICAL LITERATURE; META ANALYSIS; REVIEW; STATISTICAL ANALYSIS; VALIDATION PROCESS;

EID: 0031470784     PISSN: 02727358     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0272-7358(97)00056-1     Document Type: Article

References (111)

1. Abrami, P. C., Cohen, P. A., & d'Apollonia, S. (1988). Implementation problems in meta-analysis. Review of Educational Research, 58, 151-179.
2. Andersen, R. D. (1983). A consolidation and appraisal of science meta-analyses. Journal of Research in Science Teaching, 20, 497-509.
3. Bangert-Drowns, R. L. (1986). Review of developments in meta-analytic method. Psychological Bulletin, 99, 388-399.
4. Beaman, A. L. (1991). An empirical comparison of meta-analytic and traditional reviews. Personality and Social Psychology Bulletin, 17, 252-257.
5. Begg, C. B. (1994). Publication bias. In H. Cooper, & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 399-409). New York: Russell Sage.
6. Bergin, A. E., & Lambert, M. J. (1978). The evaluation of therapeutic outcomes. In A. E. Bergin & S. L. Garfield (Eds.), Handbook of psychotherapy and behaviour change: An empirical analysis (2nd ed., pp. 139-189). New York: John Wiley & Sons.
7. Bland, C. J., Meurer, L. N., & Maldonado, G. (1995). A systematic approach to conducting a non-statistical meta-analysis of research literature. Academic Medicine, 70, 642-653.
8. Boden, W. E. (1992). Meta-analysis in clinical trials reporting: Has a tool become a weapon? Editorial. American Journal of Cardiology, 69, 681.
9. Bowers, T., & Clum, G. (1988). Relative contributions of specific and nonspecific treatment effects: Meta-analysis of placebo-controlled behavior therapy research. Psychological Bulletin, 103, 315-323.
10. Brody, N. (1990). Behavior therapy versus placebo: Commentary on Bowers and Clum's meta-analysis. Psychological Bulletin, 107, 106-109.
11. Brown, S. A., & Hedges, L. V. (1994). Predicting metabolic control in diabetes: A pilot study using meta-analysis to estimate a linear model. Nursing Research, 43, 362-368.
12. Bushman, B. J. (1994). Vote-counting procedures. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 193-214). New York: Russell Sage.
13. Carson, K. P., Schriesheim, C. A., & Kinicki, A. J. (1990). The usefulness of the "fail-safe" statistic in meta-analysis. Educational and Psychological Measurement, 50, 233-243.
14. Chalmers, T. C., Berrier, J., Sacks, H. S., Levin, H., Reitman, D., & Nagalingam, R. (1987). Meta-analysis of clinical trials as a scientific discipline. II: Replicate variability and comparison of studies that agree and disagree. Statistics in Medicine, 6, 733-744.
15. Chalmers, T. C., Smith, H., Blackburn, B., Silverman, B., Schroeder, B., Reitman, D., & Ambroz, A. (1981). A method for assessing the quality of a randomized control trial. Controlled Clinical Trials, 2, 31-49.
16. Chow, S. L. (1987). Meta-analysis of pragmatic and theoretical research: A critique. Journal of Psychology, 121, 259-271.
17. Clum, G. A., & Bowers, T. G. (1990). Behavior therapy better than placebo treatment: Fact or artifact? Psychological Bulletin, 107, 110-113.
18. Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
19. Cook, D. J., Guyatt, G. H., Ryan, G., Clifton, J., Buckingham, L., Willan, A., McIlroy, W., & Oxman, A. D. (1993). Should unpublished data be included in meta-analyses? Current convictions and controversies. Journal of the American Medical Association, 269, 2749-2753.
20. Cook, D. T., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Mifflin.
21. Cook, T. D., Cooper, H., Cordray, D. S., Hartmann, H., Hedges, L. V., Light, R. J., Louis, T. A., & Mosteller, F. (1992). Meta-Analysis for explanation: A casebook. New York: Russell Sage.
22. Cook, T. D., & Leviton, L. C. (1980). Reviewing the literature: A comparison of traditional methods with meta-analysis. Journal of Personality, 48, 449-472.
23. Cooper, H. M., & Dorr, N. (1995). Race comparisons on need for achievement: A meta-analytic alternative to Graham's narrative review. Review of Educational Research, 65, 483-508.
24. Cooper, H., & Hedges, L. V. (1994). The handbook of research synthesis. New York: Russell Sage.
25. Cooper, H. M., & Lemke, K. M. (1991). On the role of meta-analysis in personality and social psychology. Personality and Social Psychology Bulletin, 17, 245-251.
26. Cordray, D. S. (1990). An assessment from the policy perspective. In K. W. Wachter & M. L. Straf (Eds.), The future of meta-analysis (pp. 99-122). New York: Russell Sage.
27. Dar, R., Serlin, R. C., & Omer, H. (1994). Misuse of statistical tests in three decades of psychotherapy research. Journal of Consulting and Clinical Psychology, 62, 75-82.
28. Dear, K. B. G., & Begg, C. B. (1992). An approach for assessing publication bias prior to performing a meta-analysis. Statistical Science, 7, 237-245.
29. Dion, K. L., & Evans, C. R. (1992). On cohesiveness: Reply to Keyton and other critics of the construct. Small Group Research, 23, 242-250.
30. Eagly, A. H., & Wood, W. (1994). Using research syntheses to plan future research. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 485-502). New York: Russell Sage.
31. Epstein, R. H. (1994). The number-crunchers drugmakers fear and love. Business Week, 3386, 70.
32. Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16, 319-324.
33. Eysenck, H. J. (1978). An exercise in mega-silliness. American Psychologist, 33, 517.
34. Eysenck, H. J. (1984). Meta-analysis: An abuse of research integration. Journal of Special Education, 18, 41-59.
35. Eysenck, H. J. (1995). Meta-analysis squared-Does it make sense? American Psychologist, 50, 110-111.
36. Feinstein, A. R. (1995). Meta-analysis: Statistical alchemy for the 21st century. Journal of Clinical Epidemiology, 48, 71-79.
37. Fumento, M. (1993). Is EPA blowing its own smoke? How much science is behind its tobacco finding? Investor's Business Daily, 9, 1.
38. Garfield, S. L. (1983). Meta-analysis and psychotherapy: Introduction to special section. Journal of Consulting and Clinical Psychology, 51, 3.
39. Gibbs, L. E. (1989). Quality of study rating form: An instrument for synthesizing evaluation studies. Journal of Social Work Education, 25, 55-67.
40. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Research, 5, 3-8.
41. Glass, G. (1978). In defense of generalization. Behavioral and Brain Sciences, 3, 394-395.
42. Glass, G. V. (1995). The next-to-last word on meta-analysis. Contemporary Psychology, 40, 736-738.
43. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Newbury Park: Sage.
44. Goldin, G. A. (1992). Meta-analysis of problem-solving studies: A critical response. Journal for Research in Mathematics Education, 23, 274-283.
45. Graham, S. (1995). Narrative versus meta-analytic reviews of race differences in motivation: A comment on Cooper and Dorr. Review of Educational Research, 65, 509-514.
46. Greenhouse, J. B., & Iyengar, S. (1994). Sensitivity analysis and diagnostics. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 383-398). New York: Russell Sage.
47. Greenwald, A. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20.
48. Gregoire, G., Derderian, F., & Le Lorier, J. (1995). Selecting the language of the publications included in a meta-analysis: Is there a tower of Babel? Journal of Clinical Epidemiology, 48, 159-163.
49. Grover, B. L. (1993). Trends in published meta-analyses. Paper presented al the Annual Meeting of the American Educational Research Association (Atlanta, GA, April 12-16, 1993). ERIC ED 358 115.
50. Guzzo, R. A., Jackson, S. E., & Katzell, R. A. (1987). Meta-analysis analysis. Research in Organizational Behavior, 9, 407-442.
51. Hale, J. L., & Dillard, J. P. (1991). The uses of meta-analysis: Making knowledge claims and setting research agendas. Communication Monographs, 58, 465-471.
52. Hall, J. A., & Rosenthal, R. (1991). Testing for moderator variables in meta-analysis: Issues and methods. Communication Monographs, 58, 437-448.
53. Hall, J. A., Rosenthal, R., Tickle-Degnen, L., & Mosteller, F. (1994). Hypotheses and problems in research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 17-28). New York: Russell Sage.
54. Hedges, L. V. (1988). Comment. Statistical Science, 3, 118-120.
55. Hedges, L. V. (1990). Directions for future methodology. In K. W. Wachter & M. L. Straf (Eds.), The future of meta-analysis (pp. 11-24). New York: Russell Sage.
56. Hedges, L. V. (1992a). Meta-analysis. Journal of Educational Statistics, 17, 279-296.
57. Hedges, L. V. (1992b). Modeling publication selection effects in meta-analysis. Statistical Science, 7, 246-255.
58. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
59. Hoyle, R. H. (1993). On the relation between data and theory. American Psychologist, 48, 1094-1096.
60. Hunter, J. E., Schmidt, F. L., & Jackson, G. B. (1982). Meta-analysis: Cumulating research findings across studies. Beverly Hills, CA: Sage.
61. Iyengar, S., & Greenhouse, J. B. (1988). Selection models and the file drawer problem. Statistical Science, 3, 109-135.
62. Jackson, S. (1991). Meta-analysis for primary and secondary data analysis: The super-experiment metaphor. Communication Monographs, 58, 449-462.
63. Johnson, B. T., Mullen, B., & Salas, E. (1995). Comparison of three major meta-analytic approaches. Journal of Applied Psychology, 80, 94-106.
64. Johnson, B. T., & Turco, R. M. (1992). The value of goodness-of-fit indices in meta-analysis: A comment on Hall and Rosenthal. Communication Monographs, 59, 388-396.
65. Kazdin, A. E. (1985). The role of meta-analysis in the evaluation of psychotherapy. Clinical Psychology Review, 5, 49-61.
66. Kulik, J. A., & Kulik, C. C. (1989). Meta-analysis in education. International Journal of Educational Research, 13, 221-340.
67. Kulik, J. A., & Kulik, C. L. (1990). Conventional and newer statistical methods in meta-analysis. Paper presented at the Annual Meeting of the American Educational Research Association (Boston, MA, April 16-20, 1990). ERIC ED 322218.
68. Kulik, J. A., Kulik, C. C., & Bangert-Drowns, R. L. (1990). Is there better evidence on mastery learning? A response to Slavin. Review of Educational Research, 60, 303-307.
69. Laird, N. M. (1990). A discussion of the Aphasia Study. In K. W. Wachter & M. L. Straf (Eds.), The future of meta-analysis (pp. 47-52). New York: Russell Sage.
70. Landman, J. T., & Dawes, R. M. (1982). Psychotherapy outcome: Smith and Glass conclusions stand up under scrutiny. American Psychologist, 37, 504-516.
71. Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181-1209.
72. Lipsey, M. W., & Wilson, D. B. (1995). Reply to comments on Lipsey and Wilson (1993). American Psychologist, 50, 113-115.
73. Mann, C. (1990). Meta-analysis in the breech. Science, 249, 476-480.
74. Matt, G. E. (1989). Decision rules for selecting effect sizes in meta-analysis: A review and reanalysis of psychotherapy outcome studies. Psychological Bulletin, 105, 106-115.
75. Matt, G. E., & Cook, T. D. (1994). Threats to the validity of research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 503-520). New York: Russell Sage.
76. Michelson, L. (1985). Editorial: Meta-analysis and clinical psychology. Clinical Psychology Review, 5, 1-2.
77. Miller, N., & Pollock, V. E. (1994). Meta-analysis synthesis for theory development. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 457-484). New York: Russell Sage.
78. Mullen, B. (1989). Advanced BASIC meta-analysis. Hillsdale, NJ: Lawrence Erlbaum.
79. Orwin, R. G. (1994). Evaluating coding decisions. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 139-162). New York: Russell Sage.
80. Petitti, D. B. (1994). Of babies and bathwater. American Journal of Epidemiology, 140, 779-782.
81. Rachman, S. J., & Wilson, G. T. (1980). The effects of psychological therapy (2nd ed.). Oxford: Pergaman.
82. Rosenthal, R. (1979). The "File Drawer Problem" and tolerance for null results. Psychological Bulletin, 86, 638-641.
83. Rosenthal, R. (1990). An evaluation of procedures and results. In K. W. Wachter & M. L. Straf (Eds.), The future of meta-analysis (pp. 123-134). New York: Russell Sage.
84. Rosenthal, R. (1991a). Meta-analytic procedures for social research (2nd ed). Newbury Park: Sage.
85. Rosenthal, R. (1991b). Quality-weighting of studies in meta-analytic research. Psychotherapy Research, 1, 25-28.
86. Rosenthal, R. (1995). Writing meta-analytic reviews. Psychological Bulletin, 118, 183-192.
87. Rosenthal, R. (1996). Meta-analysis: Concept, corollaries and controversies. Paper presented at the meeting of the International Congress of Psychology, Montreal, Quebec.
88. Rosenthal, R., & Rubin, D. B. (1986). Meta-analytic procedures for combining studies with multiple effect sizes. Psychological Bulletin, 99, 400-406.
89. Rosenthal, R., & Rubin, D. B. (1988). Comment: Assumptions and procedures in the file drawer problem. Statistical Science, 3, 120-125.
90. Sacks, H. S., Berrier, J., Reitman, D., Ancona-Berk, V. A., & Chalmers, T. C. (1987). Meta-analyses of randomized control trials. New England Journal of Medicine, 316, 450-455.
91. Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173-1181.
92. Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316.
93. Shapiro, S. (1994). Meta-analysis/Shmeta-analysis. American Journal of Epidemiology, 140, 771-778.
94. Slavin, R. E. (1984). Meta-analysis in education: How has it been used? Educational Researcher, 13, 6-15, 24-27.
95. Slavin, R. E. (1986). Best-evidence synthesis: An alternative to meta-analytic and traditional reviews. Educational Researcher, 15, 5-11.
96. Slavin, R. E. (1995). Best evidence synthesis: An intelligent alternative to meta-analysis. Journal of Clinical Epidemiology, 48, 9-18.
97. Smith, M. L. (1980). Publication bias and meta-analysis. Evaluation in Education, 4, 22-24.
98. Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752-760.
99. Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits of psychotherapy. Baltimore, MD: Johns Hopkins.
100. Sohn, D. (1996). Publication bias and the evaluation of psychotherapy efficacy in reviews of the research literature. Clinical Psychology Review, 16, 147-156.
101. Steiner, D. D., Lane, I. M., Dobbins, G. H., Schnur, A., & McConnell, S. (1991). A review of meta-analyses in organizational behavior and human resources management: An empirical assessment. Educational and Psychological Measurement, 51, 609-626.
102. Strube, M. J., Gardner, W., & Hartmann, D. P. (1985). Limitations, liabilities, and obstacles in reviews of the literature: The current status of meta-analysis. Clinical Psychology Review, 5, 63-78.
103. Victor, N. (1995). "The challenge of meta-analysis": Discussion. Indications and contra-indications for meta-analysis. Journal of Clinical Epidemiology, 48, 5-8.
104. Viswesvaran, C., & Ones, D. S. (1995). Theory testing: Combining psychometric meta-analysis and structural equations modeling. Personnel Psychology, 48, 865-883.
105. Wachter, K. W. (1988). Disturbed by meta-analysis? Science, 241, 1407-1408.
106. Wanous, J. P., Sullivan, S. E., & Malinak, J. (1989). The role of judgment calls in meta-analysis. Journal of Applied Psychology, 74, 259-264.
107. White, H. D. (1994). Scientific communication and literature retrieval. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 41-55). New York: Russell Sage.
108. Wilson, G. T. (1985). Limitations of meta-analysis in the evaluation of the effects of psychological therapy. Clinical Psychology Review, 5, 35-47.
109. Wilson, G. T., & Rachman, S. J. (1983). Meta-analysis and the evaluation of psychotherapy outcome: Limitations and liabilities. Journal of Consulting and Clinical Psychology, 51, 54-64.
110. Wolf, F. M. (1990). Methodological observations on bias. In K. W. Wachter & M. L. Straf (Eds), The future of meta-analysis (pp. 139-151). New York: Russell Sage.
111. Wortman, P. M. (1994). Judging research quality. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 139-152). New York: Russell Sage.