Threats to the validity of generalized inferences

The Hand. of Res. Synthesis and Meta-Analysis, 2nd Ed.

Volumn , Issue , 2009, Pages 537-560

  Matt, Georg E ^a   Cook, Thomas D ^b  

^a San Diego University   (United States)

^b NORTHWESTERN UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84902748631     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Chapter

References (104)

1. Becker, Betsy J. 1992. "Models of Science Achievement: Forces Affecting Male and Female Performance in School Science." in Meta-Analysis for Explanation: A Casebook, edited by Thomas D. Cook, Harris Cooper, David S. Cordray, Heidi Hartmann, Larry V. Hedges, Richard J. Light, Thomas A. Louis, and Frederick Mosteller. New York: Russell Sage Foundation.
2. -. 2001. "Examining Theoretical Models Through Research Synthesis: The Benefits of Model-Driven Meta-Analysis." Evaluation and the Health Professions. 24(2): 190-217.
3. Boruch, Robert F. 2005. "Comments on 'Can the Federal Government Improve Education Research?' by Brian Jacob and Jacob Ludwig." Brookings Papers on Education Policy 2005(1): 67-280.
4. Campbell, Donald T. 1957. "Factors Relevant to the Validity of Experiments in Social Settings." Psychological Bulletin 54(4): 297-312.
5. -. 1986. "Relabeling Internal and External Validity for Applied Social Scientist." in Advances in Quasi-Experimental Design and Analysis, edited by William M.K. Trochim. San Francisco: Jossey-Bass.
6. Campbell, Donald T., and Robert F. Boruch. 1975. "Making the Case for Randomized Assignment to Treatments by Considering the Alternatives: Six Ways in which Quasi-Experimental Evaluations Tend to Underestimate Effects." in Evaluation and Experiment: Some Critical Issues in Assessing Social Programs, edited by C. A. Bennett and A. A. Lumsdaine. New York: Academic Press.
7. Campbell, Donald T., and Julian C. Stanley. 1963. Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally.
8. Chalmers, Thomas C., P. Celano, Henry S. Sacks, and H. Smith Jr. 1983. "Bias in Treatment Assignment in Controlled Clinical Trials." New England Journal of Medicine 309(22): 1358-61.
9. Chalmers, Thomas C., Peg Hewett, Dinah Reitman, and Henry S. Sacks. 1989. "Selection and Evaluation of Empirical Research in Technology Assessment." International Journal of Technology Assessment in Health Care 5(4): 521-36.
10. Chalmers, Thomas C., and Joseph Lau. 1992. "Meta-Analysis of Randomized Control Trials Applied to Cancer Therapy." in Important Advances in Oncology, 1992, edited by Vincent T. Devita, Samuel Hellman and Steven A. Rosenberg. Philadephia, Pa.: Lippincott.
11. Cohn, Lawrence D., and Betsy J. Becker. 2003. "How Meta-Analysis Increases Statistical Power." Psychological Methods 8(3): 243-53.
12. Collingwood, Robin G. 1940. An Essay on Metaphysics. Oxford: Clarendon Press.
13. Cook, Thomas D. 1990. "The Generalization of Causal Connections." in Research Methodology: Strengthening Causal Interpretations of Nonexperimental Data, edited by Lee Sechrest, E. Perrin and J. Bunker. Washington, D.C.: U.S. Department of Health and Human Services.
14. -. 1993. "Understanding Causes and Generalizing About Them." in New Directions in Program Evaluation, edited by Lee B. Sechrest and A. G. Scott. San Francisco: Jossey Bass.
15. Cook, Thomas D., and Donald T. Campbell. 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Chicago: Rand McNally.
16. Cook, Thomas D., Harris Cooper, David S. Cordray, Heidi Hartmann, Larry V. Hedges, Richard J. Light, Thomas A. Louis, and Frederick Mosteller, eds. 1992. Meta-Analysis for Explanation: A Casebook. New York: Russell Sage Foundation.
17. Cronbach, Lee J. 1980. Toward Reform of Program Evaluation. San Francisco: Jossey-Bass.
18. -. 1982. Designing Evaluations of Educational and Social Programs. San Francisco: Jossey-Bass.
19. Devine, Elizabeth C. 1992. "Effects of Psychoeducational Care with Adult Surgical Patients: A Theory Probing Meta-Analysis of Intervention Studies." in Meta-Analysis for Explanation: A Casebook, edited by Thomas D. Cook, Harris Cooper, David S. Cordray, Heidi Hartmann, Larry V Hedges, Richard J. Light, Thomas A. Louis, and Frederick Mosteller. New York: Russell Sage Foundation.
20. Donner, Allan, Gilda Piaggio, and José Villar. 2001. "Statistical Methods for the Meta-Analysis of Cluster Randomization Trials." Statistical Methods in Medical Research 10(5): 325-38.
21. -. 2003. "Meta-Analyses of Cluster Randomization Trials: Power Considerations." Evaluation and the Health Professions 26(3): 340-51.
22. Duval, Sue, and Richard Tweedie. 2000. "Trim and Fill: A Simple Funnel-Plot-Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis." Biometrics 56(2): 455-63.
23. Eddy, David M., Vic Hasselblad, and Ross Shachter. 1990. "An Introduction to a Bayesian Method for Meta-Analysis: The Confidence Profile Method." Medical Decision Making 10(1): 15-23.
24. Egger, Matthias, George Davey Smith, Martin Schneider, and Christoph Minder. 1997. "Bias in Meta-Analysis Detected by a Simple, Graphical Test." British Medical Journal 315(7109): 629-34.
25. Furlow, Carolyn F., and S. Natasha Beretvas. 2005. "Meta-Analytic Methods of Pooling Correlation Matrices for Structural Equation Modeling Under Different Patterns of Missing Data." Psychological Methods 10(2): 227-54.
26. Gasking, Douglas. 1955. "Causation and Recipes." Mind 64: 479-87.
27. Glass, Gene V. 1976. "Primary, Secondary, and Meta-Analysis of Research." Educational Researcher 5(10): 3-8.
28. -. 1977. "Integrating Findings: The Meta-Analysis of Research." Review of Research in Education 5(1): 351-79.
29. Greenwald, Anthony G. 1975. "Consequences of Prejudice Against the Null Hypothesis." Psychological Bulletin 82(1): 1-20.
30. Hardy, Rebecca J., and Simon G. Thompson. 1998. "Detecting and Describing Heterogeneity in Meta-Analysis." Statistics in Medicine 17(8): 841-56.
31. Harris, Monica J., and Robert Rosenthal. 1985. "Mediation of Interpersonal Expectancy Effects: 31 Meta-Analyses." Psychological Bulletin 27(3): 363-86.
32. Harwell, Michael. 1997. "An Empirical Study of Hedge's Homogeneity Test." Psychological Methods 2(2): 219-31.
33. Hedges, Larry V. 1982. "Estimation of Effect Sizes from a Series of Independent Experiments." Psychological Bulletin 92(2): 490-99.
34. -. 1990. "Directions for Future Methodology." in The Future of Meta-Analysis, edited by Kenneth W. Wachter and Miron L. Straf. New York: Russell Sage Foundation.
35. Hedges, Larry V., and Betsy Jane Becker. 1986. "Statistical Methods in the Meta-Analysis of Research on Gender Differences." in The Psychology of Gender: Advances Through Meta-Analysis, edited by Janet Shibley Hyde and Marcia C. Linn. Baltimore, Md.: Johns Hopkins University Press.
36. Hedges, Larry V., and Ingram Olkin. 1985. Statistical Methods for Meta-Analysis. Orlando, FL: Academic Press.
37. Hedges, Larry V., and Therese D. Pigott. 2001. "The Power of Statistical Tests in Meta-Analysis." Psychological Methods 6(3): 203-17.
38. -. 2004. "The Power of Statistical Tests for Moderators in Meta-Analysis." Psychol Methods 9(4): 426-45.
39. Hedges, Larry V., and Jack L. Vevea. 1996. "Estimating Effect Size Under Publication Bias: Small Sample Properties and Robustness of a Random Effects Selection Model." Journal of Educational and Behavioral Statistics 21(4): 299-332.
40. Hox, Joop J., and Edith D. de Leeuw. 2003. "Multilevel Models for Meta-Analysis." in Multilevel Modeling: Methodological Advances, Issues, and Applications, edited by Steven P. Reise and Naihua Duan. Mahwah, N.J.: Lawrence Erlbaum.
41. Hunter, John E., and Frank L. Schmidt. 1990. Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. Newbury Park, Calif.: Sage Publications.
42. Jackson, Dan. 2006. "The Power of the Standard Test for the Presence of Heterogeneity in Meta-Analysis." Statistics in Medicine 25(15): 2688-99.
43. Jacob, Brian, and Jens Ludwig. 2005. "Can the Federal Government Improve Education Research?" Brookings Papers on Education Policy 2005(1): 47-87.
44. Kelley, George A., Kristi S. Kelley, and Zung Vu Tran. 2004. "Retrieval of Missing Data for Meta-Analysis: A Practical Example." International Journal of Technology Assessment in Health Care 20(3): 296-99.
45. Lipsey, Mark W. 1992. "Juvenile Delinquency Treatment: A Meta-Analytic Inquiry into the Variability of Effects." in Meta-Analysis for Explanation: A Casebook, edited by Thomas D. Cook, Harris Cooper, David S. Cordray, Heidi Hartmann, Larry V Hedges, Richard J. Light, Thomas A. Louis, and Frederick Mosteller. New York: Russell Sage Foundation.
46. -. 2003. "Those Confounded Moderators in Meta-Analysis: Good, Bad, and Ugly." Annals of the American Academy of Political and Social Science 587(May, Special Issue): 69-81.
47. Lipsey, Mark W., and David B. Wilson. 2001. Practical Meta-Analysis. Applied Social Research Methods Series, Vol. 49. Thousand Oaks, Calif.: Sage Publications.
48. Little, Roderick J. A., and Donald B. Rubin. 2002. Statistical Analysis with Missing Data, 2nd ed. New York: John Wiley & Sons.
49. Mackie, John L. 1974. The Cement of the Universe: A Study of Causation. Oxford: Oxford University Press.
50. Matt, Georg E. 1989. "Decision Rules for Selecting Effect Sizes in Meta-Analysis: A Review and Reanalysis of Psychotherapy Outcome Studies." Psychological Bulletin 105(1): 106-15.
51. -. 2003. "Will it Work in Münster? Meta-Analysis and the Empirical Generalization of Causal Relationships." in Meta-analysis, edited by H. Holling, D. Böhning and R. Schulze. Göttingen: Hogrefe and Huber.
52. -. 2005. "Uncertainty in the Multivariate World: A Fuzzy Look through Brunswik's Lens." in Multivariate Research Strategies. A Festschrift in Honor of Werner W. Wittmann, edited by Andre Beauducel, Bernhard Biehl, Michael Bosniak, Wolfgang Conrad, Gisela Schönberger and Deitrich Wagener. Maastricht: Shaker Publishing.
53. Matt, Georg E., and Thomas D. Cook. 1994. "Threats to the Validity of Research Syntheses." in The Handbook of Research Synthesis., edited by Harris Cooper and Larry V. Hedges. New York: Russell Sage Foundation.
54. Mick, Eric, Joseph Biederman, Gahan Pandina, and Stephen V. Faraone. 2003. "A Preliminary Meta-Analysis of the Child Behavior Checklist in Pediatric Bipolar Disorder." Biological Psychiatry 53(11): 1021-27.
55. Moyer, Anne, John W. Finney, Carolyn E. Swearingen, and Pamela Vergun. 2002. "Brief Interventions for Alcohol Problems: A Meta-Analytic Review of Controlled Investigations in Treatment-Seeking and Non Treatment-Seeking Populations." Addiction 97(3): 279-92.
56. Nam, In-Sum, Kerrie Mengersen, and Paul Garthwaite. 2003. "Multivariate Meta-Analysis." Statistics in Medicine 22(14): 2309-33.
57. Orwin, Robert G., and David S. Cordray. 1985. "The Effects of Deficient Reporting on Meta-Analysis: A Conceptual Framework and Reanalysis." Psychological Bulletin 97(1): 134-47.
58. Premack, Steven L., and John E. Hunter. 1988. "Individual Unionization Decisions." Psychological Bulletin 103(2): 223-34.
59. Prevost, Teresa C., Keith R. Abrams, and David R. Jones. 2000. "Hierarchical Models in Generalized Synthesis of Evidence: An Example Based on Studies of Breast Cancer Screening." Statistics in Medicine. 19(24): 3359-376.
60. Raudenbush, Stephen W., Betsy J. Becker, and Hripsime Kalaian. 1988. "Modeling Multivariate Effect Sizes." Psychological Bulletin 103(1): 111-20.
61. Raudenbush, Stephen W., and Anthony S. Bryk. 1985. "Empirical Bayes Meta-Analysis." Journal of Educational Statistics 10(2): 75-98.
62. Robinson, Leslie A., Jeffrey S. Berman, and Robert A. Neimeyer. 1990. "Psychotherapy for the Treatment of Depression: A Comprehensive Review of Controlled Outcome Research." Psychological Bulletin 108(1): 30-49.
63. Rosch, Eleanor. 1973. "Natural Categories." Cognitive Psychology 4: 328-50.
64. -. 1978. "Principles in Categorization." in Cognition and Categorization, edited by Eleanor Rosch and Barbara B. Lloyd. Hillsdale, N.J.: Lawrence Erlbaum.
65. Rosenthal, Robert. 1979. "The File Drawer Problem and Tolerance for Null Results." Psychological Bulletin 86(3): 638-41.
66. -. 1990. "An Evaluation of Procedure and Results." in The Future of Meta-Analysis, edited by Kenneth W. Wachter and Miron L. Straf. New York: Russell Sage Foundation.
67. -. 1991. Meta-Analytic Procedures for Social Research, 2nd ed. Newbury Park, Calif.: Sage Publications.
68. Rosenthal, Robert, and Donald B. Rubin. 1982. "Comparing Effect Sizes of Independent Studies." Psychological Bulletin 92(2): 500-4.
69. -. 1986. "Meta-Analytic Procedures for Combining Studies with Multiple Effect Sizes." Psychological Bulletin 99(3): 400-6.
70. Rothstein, Hannah R., Alexander J. Sutton, and Michael Borenstein, eds. 2005. Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. New York: John Wiley & Sons.
71. Rubin, Donald B. 1987. Multiple Imputation for Nonresponse in Surveys. New York: John Wiley & Sons.
72. Sacks, H. S., J. Berrier, D. Reitman, V. A. Ancona-Berk, and T. C. Chalmers. 1987. "Meta-Analyses of Randomized Controlled Trials." The New England Journal of Medicine 316(8): 450-55.
73. Saleh, A. K. Ehsanes, Khatab M. Hassanein, Ruth S. Hassanein, and H. M. Kim. 2006. "Quasi-Empirical Bayes Methodology for Improving Meta-Analysis." Journal of Biopharmaceutical Statistics 16(1): 77-90.
74. Sánchez-Meca, Julio, Fulgencio Marín-Martínez, and Salvador Chacon-Moscoso. 2003. "Effect-Size Indices for Dichotomized Outcomes in Meta-Analysis." Psychological Methods 8(4): 448-67.
75. Schulze, Ralf. 2004. Meta-Analysis: A Comparison of Approaches. New York: Hogrefe and Huber.
76. Scott, J. Cobb, Steven Paul Woods, Georg E. Matt, Rachel A. Meyer, Robert K. Heaton, J. Hampton Atkinson, and Igor Grant. 2007. "Neurocognitive Effects of Methamphetamine: A Critical Review and Meta-Analysis." Neuropsychology Review 17(3): 275-97.
77. Shadish, William R. 1992. "Do Family and Marital Therapies Change What People Do? A Meta-Analysis of Behavioral Outcomes." in Meta-Analysis for Explanation: A Casebook, edited by Thomas D. Cook, Harris Cooper, David S. Cordray, Heidi Hartmann, Larry V. Hedges, Richard J. Light, Thomas A. Louis, and Frederick Mosteller. New York: Russell Sage Foundation.
78. Shadish, William R., Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and Quasi-Experimental Design for Generalized Causal Inference. Boston, Mass.: Houghton Mifflin.
79. Shadish, William R., Xiangen Hu, Renita R. Glaser, Richard Kownacki, and Seok Wong. 1998. "A Method for Exploring the Effects of Attrition in Randomized Experiments with Dichotomous Outcomes." Psychological Methods 3(1): 3-22.
80. Shadish, William R., Jason K. Luellen, and M. H. Clark. 2006. "Propensity Scores and Quasi-Experiments: A Testimony to the Practical Side of Lee Sechrest." in Strengthening Research Methodology: Psychological Measurement and Evaluation, edited by Richard Bootzin and Patrick E. McKnight. Washington, D.C.: American Psychological Association.
81. Shadish, William R., Georg E. Matt, Ana M. Navarro, and Glenn Phillips. 2000. "The Effects of Psychological Therapies Under Clinically Representative Conditions: A Meta-Analysis." Psychological Bulletin 126(4): 512-29.
82. Shadish, William R., and Rebecca B. Sweeney. 1991. "Mediators and Moderators in Meta-Analysis: There's a Reason We Don't Let Dodo Birds Tell Us Which Psychotherapies Should Have Prizes." Journal of Consulting Clinical Psychology 59(6): 883-93.
83. Shapiro, David A., and Diana Shapiro. 1982. "Meta-Analysis of Comparative Therapy Outcome Studies: A Replication and Refinement." Psychological Bulletin 92(3): 581-604.
84. Sinharay, Sandip, Hal S. Stern, and Daniel Russell. 2001. "The Use of Multiple Imputation for the Analysis of Missing Data." Psychological Methods 6(4): 317-29.
85. Slavin, Robert E. 1986. "Best-Evidence Synthesis: An Alternative to Meta-Analytic and Traditional Reviews." Educational Researcher 15(9): 5-11.
86. Smith, Edward E., and Douglas L. Medin. 1981. Categories and Concepts. Cambridge, Mass.: Harvard University Press.
87. Smith, Mary Lee, Gene V. Glass, and Thomas I. Miller. 1980. The Benefits of Psychotherapy. Baltimore, Md.: Johns Hopkins University Press.
88. Sterne, Jonathan A. C, and Matthias Egger. 2001. "Funnel Plots for Detecting Bias in Meta-Analysis: Guidelines on Choice of Axis." Journal of Clinical Epidemiology 54(10): 1046-55.
89. Sutton, Alexander J. 2000. Methods for Meta-Analysis in Medical Research. Wiley Series in Probability and Mathematical Statistics. New York: John Wiley & Sons.
90. Sutton, Alexander J., and Keith R. Abrams. 2001. "Bayesian Methods in Meta-Analysis and Evidence Synthesis." Statistical Methods in Medical Research 10(4): 277-303.
91. Sutton, Alexander J., Susan J. Duval, Richard L. Tweedie, Keith R. Abrams, and David R. Jones. 2000. "Empirical Assessment of Effect of Publication Bias on Meta-Analyses." British Medical Journal 320(7249): 1574-77.
92. Sutton, Alexander J., Fujian Song, Simon M. Gilbody, and Keith R. Abrams. 2000. "Modelling Publication Bias in Meta-Analysis: A Review." Statistical Methods in Medical Research 9(5): 421-45.
93. Tabarrok, Alexander T. 2000. "Assessing the FDA via the Anomaly of Off-Label Drug Prescribing. The Independent Review V(1): 25-53.
94. Twenge, Jean M., and W. Keith Campbell. 2001. "Age and Birth Cohort Differences in Self-Esteem: A Cross-Temporal Meta-Analysis." Personality and Social Psychology Review 5(4): 321-44.
95. van Houwelingen, Hans C., Lidia R. Arends, and Theo Stijnen. 2002. "Advanced Methods in Meta-Analysis: Multivariate Approach and Meta-Regression." Statistics in Medicine 21(4): 589-624.
96. Vevea, Jack L., and Larry V. Hedges. 1995. "A General Linear Model for Estimating Effect Size in the Presence of Publication Bias." Psychometrika 60(3): 419-35.
97. Vevea, Jack L., and Carol M. Woods. 2005. "Publication Bias in Research Synthesis: Sensitivity Analysis Using a Priori Weight Functions." Psychological Methods 10(4): 428-43.
98. Vevea, Jack L., Nancy C. Clements, and Larry V. Hedges. 1993. "Assessing the Effects of Selection Bias on Validity Data for the General Aptitude Test Battery." Journal of Applied Psychology 78(6): 981-87.
99. Viechtbauer, Wolfgang. 2007. "Confidence Intervals for the Amount of Heterogeneity in Meta-Analysis." Statistics in Medicine 26(1): 37-52.
100. Webb, Eugene J., Donald T. Campbell, Richard D. Schwartz, Lee Sechrest, and Janet B. Grove. 1981. Nonreactive Measures in the Social Sciences, 2nd ed. Chicago: Rand McNally.
101. West, Stephen G., Jeremy C. Biesanz, and Steven C. Pitts. 2000. "Causal Inference and Generalization in Field Settings: Experimental and Quasi-Experimental Designs." in Handbook of Research Methods in Social and Personality Psychology, edited by H.T. Reid and CM. Judd. New York: Cambridge University Press.
102. Whitbeck, Caroline. 1977. "Causation in Medicine: The Disease Entity Model." Philosophy of Science 44(4): 619-37.
103. Wittmann, Werner W., and Georg E. Matt. 1986. "Meta-Analyse als Integration von Forschungsergebnissen am Beispiel deutschsprachiger Arbeiten zur Effektivität von Psychotherapie." Psychologische Rundschau 37(1): 20-40.
104. Wolf, Frederic M. 1990. "Methodological Observations on Bias." in The Future of Meta-Analysis, edited by Kenneth W. Wachter and Miron L. Straf. New York: Russell Sage Foundation.