Sample sizes and effect sizes are negatively correlated in meta-analyses: Evidence and implications of a publication bias against nonsignificant findings

Communication Monographs

Volumn 76, Issue 3, 2009, Pages 286-302

  Levine, Timothy R ^a   Asada, Kelli J ^b   Carpenter, Chris ^b  

^a MICHIGAN STATE UNIVERSITY   (United States)

^b WILLIAM PATERSON UNIVERSITY   (United States)

Author keywords

Effect Size; Meta Analysis; Publication Bias

Indexed keywords

EID: 70349286316     PISSN: 03637751     EISSN: 14795787     Source Type: Journal    
DOI: 10.1080/03637750903074685     Document Type: Article

References (86)

1. Abelson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: LEA.
2. Allen, M., & Preiss, R. W. (1998). Persuasion: Advances through meta-analysis. Cresskill, NJ: Hampton Press.
3. Allen, M., Hunter, J. E., & Donohue, W. A. (1989). Meta-analysis of self-report data on the effectiveness of public speaking anxiety treatment. Communication Education, 38, 54_76.
4. Allen, M., Preiss, R. W., Gayle, B. M., & Burrell, N. A. (2002). Interpersonal communication research: Advances through meta-analysis (pp. 263_279). Mahwah, NJ: LEA.
5. Becker, B. J. (2005). Failsafe N or file-drawer number. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments (pp. 111-125). Chichester, UK: Wiley.
6. Begg, C. B. (1994). Publication bias. In H. Cooper & L. V. Hedges (Eds.), The Handbook of Research Synthesis (pp. 399_409). New York: Russell Sage Foundation.
7. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088_1101.
8. Boster, F. J. (2002). On making progress in communication science. Human Communication Research, 28, 473_490.
9. Callaham, M. L., Wears, R. L., Weber, E. J., Barton, C., & Young, G. (2008). Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. Journal of the American Medical Association, 280, 254-257.
10. Cohen, J. (1994). The earth is round (pB.05). American Psychologist, 49, 997_1003.
11. Dickersin, K. (2005). Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments (pp. 12_33). Chichester, UK: Wiley.
12. Duval, S. (2005). The trim and fill method. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments (pp. 127-144). Chichester, UK: Wiley.
13. Duval, S., & Tweedie, R. L. (2000a). A non-parametric "trim and fill" method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89-98.
14. Duval, S., & Tweedie, R. L. (2000b). Trim and fill: A simple funnel plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 276-284.
15. Gerber, A. S., & Malhotra, N. (2008). Publication bias in empirical sociological research: Do arbitrary significance levels distort published results? Sociological Methods and Research, 37, 3-30.
16. Hunter, J., & Schmidt, F. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
17. Kromrey, D. J., & Rendina-Gobioff, G. (2006). On knowing what we do not know: An empirical comparison of methods to detect publication bias in meta-analysis. Educational and Psychological Measurement, 66, 357-373.
18. La France, B. H., Heisel, A. D., & Beatty, M. J. (2004). Is there empirical evidence for a nonverbal profile of extraversion?: A meta-analysis and critique of the literature. Communication Monographs, 71, 28-48.
19. Levine, T. R., Weber, R., Park, H. S., & Hullett, C. R. (2008). A communication researchers guide to null hypothesis significance testing and alternatives. Human Communication Research, 34, 188-209.
20. Levine, T. R., Weber, R., Hullett, C. R., Park, H. S., & Lindsey, L. (2008). A critical assessment of null hypothesis significance testing in quantitative communication research. Human Communication Research, 34, 171-187.
21. Light, R., & Pillemer, D. (1984). Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press.
22. Littner, Y., Mimouni, F. B., Dollberg, S., & Mandel, D. (2005). Negative results and impact factor: A lesson from neonatology. Archives of Pediatric and Adolescent Medicine, 159, 1036-1037.
23. Masaskill, P., Walter, S. D., & Irwig, L. (2001). A comparison of methods to detect publication bias in meta-analysis. Statistics in Medicine, 20, 641-654.
24. Meehl, P. E. (1986). What social scientists don't understand. In D. W. Fiske & R. A. Shweder (Eds.), Meta-Theory in Social Science (pp. 315-338). Chicago: University of Chicago Press.
25. Palmer, A. R. (1999). Detecting publication bias in meta-analysis: A case study of fluctuating asymmetry and sexual selection. American Naturalist, 154, 220-233.
26. Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007). Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Statistics in Medicine, 26, 4544-4562.
27. Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331-363.
28. Rosenthal, R. (1979). The "file drawer problem" and tolerance for null results. Psychological Bulletin, 86, 638-641.
29. Rothstein, H. R. (2008). Publication bias as a threat to the validity of meta-analytic results. Journal of Experimental Criminology, 4, 61-81.
30. Scargle, J. D. (1999). The "file-drawer" problem in scientific inference. Paper presented at the Sturrock Symposium. Stanford University.
31. Sterne, J. A. C., Becker, B. J., & Egger, M. (2005). The funnel plot. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments (pp. 75-98). Chichester, UK: Wiley.
32. Sutton, A. J., Duval, S. J., Tweedie, R. L., Abrams, K. R., & Jones, D. R. (2000). Empirical assessment of effect of publication bias on meta-analysis. British Medical Journal, 320, 1574-1577.
33. Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine, 22, 2113-2126.
34. Thornton, A., & Lee, P. (2000). Publication bias in meta-analysis: Its causes and consequences. Journal of Clinical Epidemiology, 53, 207-216.
35. Vevea, J. L., & Woods, C. M. (2005). Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods, 10, 428-443.
36. Allen, M., Bourhis, J., Emmers-Sommer, T., & Sahlstein, E. (1998). Reducing dating anxiety: A meta-analysis. Communication Reports, 11, 49-55.
37. Allen, M., D'Alessio, D., & Brezgel, K. (1995). A meta-analysis summarizing the effects of pornography II: Aggression after exposure. Human Communication Research, 22, 258-283.
38. Allen, M., D'Alessio, D., Emmers, T. M., & Gebhardt, L. (1996). The role of educational briefings in mitigating effects of experimental exposure to violent sexually explicit material: A metaanalysis. The Journal of Sex Research, 33, 135-141.
39. Allen, M., Emmers-Sommer, T. M., & Crowell, T. L. (2002). Couples negotiating safer sex behaviors: A meta-analysis of the impact of conversation and gender. In M. Allen, R. W. Preiss, B. M. Gayle, & N. A. Burrell (Eds.), Interpersonal communication research: Advances through metaanalysis (pp. 263-279). Mahwah, NJ: Lawrence Erlbaum Associates.
40. Allen, M., Hunter, J. E., & Donohue, W. A. (1989). Meta-analysis of self-report data on the effectiveness of public speaking anxiety treatment. Communication Education, 38, 54-76.
41. Allen, M., & Preiss, R. W. (1997). Comparing the persuasiveness of narrative and statistical evidence using meta-analysis. Communication Research Reports, 14, 125-131.
42. Beck, R., & Fernandez, E. (1998). Cognitive-behavioral therapy in the treatment of anger: A metaanalysis. Cognitive Therapy and Research, 23, 63-74.
43. Benoit, W. L. (1998). Forewarning and persuasion. In R. W. Allen & Preiss (Eds.), Persuasion: Advances through meta-analysis (pp. 139-154). Cresskill, NJ: Hampton Press.
44. Benoit, W. L., Hansen, G. J., & Verser, R. M. (2003). A meta-analysis of the effects of viewing US presidential debates. Communication Monographs, 70, 335-350.
45. Bettencourt, B. A., & Kernahan, C. (1997). A meta-analysis of aggression in the presence of violent cues: Effects of gender differences and aversive provocation. Aggressive Behavior, 23, 447-456.
46. Burrell, N. A., & Koper, R. J. (1998). The efficacy of powerful/powerless language on attitudes and source credibility. In R. W. Allen & Preiss (Eds.), Persuasion: Advances through meta-analysis (pp. 203-215). Cresskill, NJ: Hampton Press.
47. Cohn, L. D., & Westenberg, P. M. (2004). Intelligence and maturity: Meta-analytic evidence for the incremental and discriminant validity of Loevinger's measure of ego development. Journal of Personality and Social Psychology, 86, 760-772.
48. Cruz, M. G. (1998). Explicit and implicit conclusions in persuasive messages. In M. Allen & R. W. Preiss (Eds.), Persuasion: Advances through meta-analysis (pp. 217-230). Cresskill, NJ: Hampton Press.
49. DeCoster, J., & Claypool, H. M. (2004). A meta-analysis of priming effects on impression formation supporting a general model of informational biases. Personality and Social Psychology Review, 8, 2-27.
50. Dillard, J. P., Hunter, J. E., & Burgoon, M. (1984). Sequential-request persuasive strategies: Meta-analysis of foot-in-the-door and door-in-the-face. Human Communication Research, 10, 461-488.
51. Emmers-Sommer, T. M., Allen, M., Bourhis, J., Sahlstein, E., Laskowski, K., Falato, W. L., et al. (2004). A meta-analysis of the relationship between social skills and sexual offenders. Communication Reports, 17, 1-10.
52. Fejfar, M. C., & Holye, R. H. (2000). Effect of private self-awareness on negative affect and selfreferent attribution: A quantitative review. Personality and Social Psychology Review, 4, 132-142.
53. Gayle, B. M., Preiss, R. W., & Allen, M. (1998). Another look at the use of rhetorical questions. In M. Allen & R. W. Preiss (Eds.), Persuasion: Advances through meta-analysis (pp. 189-201). Cresskill, NJ: Hampton Press.
54. Georgesen, J. C., & Harris, M. J. (1998). Why's my boss always holding me down? A meta-analysis of power effects on performance evaluations. Personality and Social Psychology Review, 2, 184-195.
55. Gold, C., Voracek, M., & Wigram, T. (2004). Effects of music therapy for children and adolescents with psychopathology: A meta-analysis. Journal of Child Psychology and Psychiatry, 45, 1054-1063.
56. Hall, J. A., Halberstadt, A. G., & O'Brien, C. E. (1997). "Subordination" and nonverbal sensitivity: A study and synthesis of findings based on trait measures. Sex Roles, 37, 295-317.
57. Hamilton, M. A., & Hunter, J. E. (1998). The effect of language intensity on receiver evaluations of message, source, and topic. In M. Allen & R. W. Preiss (Eds.), Persuasion: Advances through meta-analysis (pp. 99-138). Cresskill, NJ: Hampton Press.
58. Hamilton, M. A., & Mineo, P. J. (2002). Argumentativeness and its effect on verbal aggressiveness: A meta-analytic review. In M. Allen, R. W. Preiss, B. M. Gayle, & N. A. Burrell (Eds.), Interpersonal communication research: Advances through meta-analysis (pp. 281-314). Mahwah, NJ: Lawrence Erlbaum Associates.
59. Hogben, M. (1998). Factors moderating the effect of televised aggression on viewer behavior. Communication Research, 25, 220-247.
60. Irvin, J. E., Bowers, C. A., Dunn, M. E., & Wang, M. C. (1999). Efficacy of relapse prevention: A meta-analytic review. Journal of Counseling and Clinical Psychology, 67, 563-570.
61. Karau, S. J., & Williams, K. D. (1993). Social loafing: A meta-analytic review and theoretical integration. Journal of Personality and Social Psychology, 65, 681-706.
62. Khaleque, A., & Rohner, R. P. (2002). Perceived parental acceptance-rejection and psychological adjustment: A meta-analysis of cross-cultural and intracultural studies. Journal of Marriage and Family, 64, 54-64.
63. Kierein, N. M., & Gold, M. A. (2000). Pygmalion in work organizations: A meta-analysis. Journal of Organizational Behavior, 21, 913-928.
64. Kim, M. S., & Hunter, J. E. (1993a). Attitude-behavior relations: A meta-analysis of attitudinal relevance and topic. Journal of Communication, 43, 101-142.
65. Kim, M. S., & Hunter, J. E. (1993b). Relationships among attitudes, behavioral intentions, and behavior: A meta-analysis of past research, part 2. Communication Research, 20, 331-364.
66. Knight, G. K., Guthrie, I. K., Page, M. C., & Fabes, R. A (2002). Emotional arousal and gender differences in aggression: A meta-analysis. Aggressive Behavior, 28, 366-393.
67. Kossek, E. E., & Ozeki, C. (1998). Work-family conflict, policies, and the job-life satisfaction relationship: A review and directions for organizational behavior-human resources research. Journal of Applied Psychology, 83, 139-149.
68. La France, B. H., Heisel, A. D., & Beatty, M. J. (2004). Is there empirical evidence for a nonverbal profile of extraversion?: A meta-analysis and critique of the literature. Communication Monographs, 71, 28-48.
69. Le, B., & Agnew, C. R. (2003). Commitment and its theorized determinants: A meta-analysis of the investment model. Personal Relationships, 10, 37-57.
70. Levine, T. R., Bresnahan, M. J., Park, H. S., Lapinski, M. K., Wittenbaum, G. M., & Shearman, S. M. et al. (2003). Self-construal scales lack validity. Human Communication Research, 29, 210-252.
71. Marcus-Newhall, A., Pedersen, W. C., Carlson, M., & Miller, N. (2000). Displaced aggression is alive and well: A meta-analytic review. Journal of Personality and Social Psychology, 78, 670-689.
72. Murnen, S. K., & Stockton, M. (1997). Gender and self-reported sexual arousal in response to sexual stimuli: A meta-analytic review. Sex Roles, 37, 135-153.
73. Murnen, S. K., Wright, C., & Kaluzny, G. (2002). If "boys will be boys," then girls will be victims? A meta-analytic review of the research that relates masculine ideology to sexual aggression. Sex Roles, 46, 359-375.
74. Preiss, R. W., & Allen, M. (1998). Performing counterattitudinal advocacy: The persuasive impact of incentives. In M. Allen & R. W. Preiss (Eds.), Persuasion: Advances through meta-analysis (pp. 241-242). Cresskill, NJ: Hampton Press.
75. Rafaeli-Mor, E., & Steinberg, J. (2002). Self-complexity and well-being: A review and research synthesis. Personality and Social Psychology Review, 6, 31-58.
76. Riggio, H. R., & Riggio, R. E. (2002). Emotional expressiveness, extraversion, and neuroticism: A meta-analysis. Journal of Nonverbal Behavior, 26, 195-218.
77. Roese, N. J., & Jamieson, D. W. (1993). Twenty years of bogus pipeline research: A critical review and meta-analysis. Psychological Bulletin, 114, 363-375.
78. Schutz, H., & Six, B. (1996). How strong is the relationship between prejudice and discrimination?: A meta-analytic answer. International Journal of Intercultural Relations, 20, 441-462.
79. Sheppard, B. H., Hartwick, J., & Warshaw, P. R. (1988). The theory of reasoned action: A metaanalysis of past research with recommendations for modifications and future research. Journal of Consumer Research, 15, 325-343.
80. Sherry, J. L. (2001). The effects of violent video games on aggression: A meta-analysis. Human Communication Research, 27, 409-431.
81. Silverman, I. W. (2003). Gender differences in delay of gratification: A meta-analysis. Sex Roles, 49, 451-463.
82. Stith, S. M., Rosen, K. H., Middleton, K. A., Busch, A. L., Lundeberg, K., & Carlton, R. P. (2000). The intergenerational transmission of spouse abuse: A meta-analysis. Journal of Marriage and the Family, 62, 640-654.
83. Tamres, L. K., Janicki, D., & Helgeson, V. S. (2002). Sex differences in coping behavior: A metaanalytic review and an examination of relative coping. Personality and Social Psychology Review, 6, 2-30.
84. Timmerman, L. M. (2002). Comparing the production of power in language on the basis of sex. In M. Allen, R. W. Preiss, B. M. Gayle, & N. A. Burrell (Eds.), Interpersonal communication research: Advances through meta-analysis (pp. 73-88). Mahwah, NJ: Lawrence Erlbaum Associates.
85. Whitely, B. E., Jr., Nelson, A. B., & Jones, C. J. (1999). Gender differences in cheating attitudes and classroom cheating behavior: A meta-analysis. Sex Roles, 41, 657-680.
86. Wolfe, D. A., Crooks, C. V., Lee, V., McIntyre-Smith, A., & Jaffe, P. G. (2003). The effects of children's exposure to domestic violence: A meta-analysis and critique. Clinical Child and Family Psychology Review, 6, 171-187.