Publication bias in empirical sociological research: Do arbitrary significance levels distort published results?

Sociological Methods and Research

Volumn 37, Issue 1, 2008, Pages 3-30

  Gerber, Alan S ^a   Malhotra, Neil ^b  

^a YALE UNIVERSITY   (United States)

^b STANFORD UNIVERSITY   (United States)

Author keywords

Caliper test; Hypothesis testing; Meta analysis; Publication bias

Indexed keywords

EID: 47349130247     PISSN: 00491241     EISSN: 15528294     Source Type: Journal    
DOI: 10.1177/0049124108318973     Document Type: Article

References (49)

1. Allen, Michael P. 2003. ''The 'Core Influence' of Journals in Sociology Revisited.'' Footnotes. 31: 7-10.
2. Allison, D.B., M.S. Faith, and B.S. Gorman. 1996. ''Publication Bias in Obesity Treatment Trials ? '' International Journal of Obesity 20: 931-37.
3. Ashenfelter, Orley, Colm Harmon, and Hessel Oosterbeek. 1999. ''A Review of Estimates of the Schooling/Earnings Relationship, With Tests for Publication Bias. '' Labour Economics 6: 453-70.
4. Begg, Colin B. and Jesse A. Berlin. 1988. ''Publication Bias: A Problem in Interpreting Medical Data.'' Journal of the Royal Statistical Society, Series A 151: 419-63.
5. Berlin, Jesse A., Colin B. Begg, and Thomas A. Louis. 1989. ''An Assessment of Publication Bias Using a Sample of Published Clinical Trials.'' Journal of the American Statistical Association 84: 381-92.
6. Campbell, Donald T. 1969. ''Reforms as Experiments.'' American Psychologist 24: 407-29.
7. Card, David and Alan B. Krueger. 1995. ''Time-Series Minimum Wage Studies: A Meta-Analysis.'' American Economic Review 85: 238-43.
8. Cleary, Richard J. and George Casella. 1997. ''An Application of Gibbs Sampling to Estimation in Meta-Analysis: Accounting for Publication Bias.'' Journal of Educational and Behavioral Statistics 22: 141-54.
9. Copas, John. 1999. ''What Works?: Selectivity Models and Meta-Analysis.'' Journal of the Royal Statistical Society, Series A 161: 95-105.
10. Correll, Shelley J. 2001. ''Gender and the Career Choice Process: The Role of Biased Self-Assessments.'' American Journal of Sociology 106: 1691-730.
11. Coursol, Allan and Edwin E. Wagner. 1986. ''Effect of Positive Findings on Submission and Acceptance Rates: A Note on Meta-Analysis Bias.'' Professional Psychology 17: 136-37.
12. Dear, Keith B. G. and Colin B. Begg. 1992. ''An Approach for Assessing Publication Bias Prior to Performing a Meta-Analysis.'' Statistical Science 7: 237-45.
13. De LongJ. Bradford, nd Kevin Lang. 1992. ''Are All Economic Hypotheses False ? '' Journal of Political Economy 100: 1257-72.
14. DiPrete, Thomas A. and Henriette Engelhart. 2004. ''Estimating Causal Effects With Matching Methods in the Presence and Absence of Bias Cancellation.'' Sociological Methods & Research 32: 501-28.
15. Doucouliagos, Chris. 2005. ''Publication Bias in the Economic Freedom and Economic Growth Literature. '' Journal of Economic Surveys 19: 367-87.
16. Doucouliagos, Chris, Patrice Laroche, and T.D. Stanley. 2005. ''Publication Bias in Union-Productivity Research ? '' Industrial Relations 60: 320-47.
17. Duval, Sue and Richard Tweedie. 2000. ''A Nonparametric 'Trim and Fill' Method of Accounting for Publication Bias in Meta-Analysis. '' Journal of the American Statistical Association 95: 89-98.
18. Easterbrook, P.J., J.C. Keruly, T. Creagh-Kirk, D.D. Richman, R.E. Chaisson, and R.D. Moore. 1991. ''Racial and Ethnic Differences in Outcome in Zidovudine-Treated Patients With Advanced HIV Disease.'' Journal of the American Medical Association 266: 2713-18.
19. Egger, Matthias, George D. Smith, Mariin Schneider, and Christoph Minder. 1997. ''Bias in Meta-Analysis Detected by a Simple, Graphical Test.'' British Medical Journal 315: 629-34.
20. Elman, Cheryl and Angela M. O'Rand. 2004. ''The Race Is to the Swift: Socioeconomic Origins, Adult Education, and Wage Attainment.'' American Journal of Sociology 110: 123-60.
21. Epstein, William M. 2000. ''Confirmational Response Bias Among Social Work Journals.'' Science, Technology, & Human Values 15: 9-38.
22. _. 2004. ''Confirmational Response Bias and the Quality of the Editorial Processes Among American Social Work Journals.'' Research on Social Work Practice 14: 450-58.
23. Gelman, Andrew, John B. Carlin, Hal S. Stern, and Donald B. Rubin. 2004. Bayesian Data Analysis. 2 d ed. Boca Raton, FL: Chapman & Hall.
24. Gerber, Alan S., Donald P. Green, and David Nickerson. 2000. ''Testing for Publication Bias in Political Science.'' Political Analysis 9: 385-92.
25. Gerber, Alan S. and Neil Malhotra. 2006. ''Can Political Science Literatures Be Believed? A Study of Publication Bias in the APSR and the AJPS.'' Presented at the annual meeting of the Midwest Political Science Association, April 20-23, Chicago, IL.
26. Glass, G.V., B. McGaw, and M.L. Smith. 1981. Meta Analysis in Social Research. Beverly Hills, CA: Sage.
27. Gleser, L.J. and I. Olkin. 1996. ''Models for Estimating the Number of Unpublished Studies. '' Statistics in Medicine 15: 2493-507.
28. Gorg, Holger and Eric Strobl. 2001. ''Multinational Companies and Productivity: A Meta-Analysis. Economic Journal 111: 723-39.
29. Greene, William H. 1996. Econometric Analysis. 3 d ed. Upper Saddle River, NJ: Prentice Hall.
30. Greenwald, Anthony G. 1975. ''Consequences of Prejudice Against the Null Hypothesis.'' Psychological Bulletin 82: 1-20.
31. Halaby, Charles N. 2004. ''Panel Models in Sociological Research: Theory Into Practice.'' Annual Review of Sociology 30: 507-44.
32. Hedges, Larry V. 1984. ''Estimation of Effect Size Under Nonrandom Sampling: The Effects of Censoring Studies Yielding Statistically Insignificant Mean Differences.'' Journal of Educational Statistics 9: 61-85.
33. _. 1992. ''Modeling Publication Selection Effects in Meta-Analysis. '' Statistical Science 7: 246-55.
34. Iyengar, Satish and Joel B. Greenhouse. 1988. ''Selection Models and the File Drawer Problem.'' Statistical Science 3: 109-35.
35. Leahey, Erin. 2005. ''Alphas and Asterisks: The Development of Statistical Significance Testing Standards in Sociology. '' Social Forces 84: 1-23.
36. Levois, M.E. and M.W. Layard. 1995. ''Publication Bias in the Environmental Tobacco Smoke/Coronary Heart Disease Epidemiologic Literature. '' Regulatory Toxicology and Pharmacology 21: 184-91.
37. Light, Richard J. and David B. Pillemar. 1984. Summing Up: The Science of Reviewing Research. Cambridge, MA: Harvard University Press.
38. McLeod, B.D. and J.R. Weisz. 2004. ''Using Dissertations to Examine Potential Bias in Child and Adolescent Clinical Trials. '' Journal of Consulting and Clinical Psychology 2: 235-51.
39. Mouw, Ted. 2006. ''Estimating the Causal Effect of Social Capital: A Review of Recent Research. '' Annual Review of Sociology 32: 79-102.
40. Pager, Devah. 2003. ''The Mark of a Criminal Record. '' American Journal of Sociology 108: 937-75.
41. Rosenthal, Robert. 1979. ''The File Drawer Problem and Tolerance for Null Results. '' Psychological Bulletin 86: 638-41.
42. Sigelman, Lee. 1999. ''Publication Bias Reconsidered. '' Political Analysis 8: 201-10.
43. Simes, R.J. 1986. ''Publication Bias: The Case for an International Registry of Clinical Trials. '' Journal of Clinical Oncology 4: 1529-41.
44. Stanley, T.D. 2005. ''Beyond Publication Bias. '' Journal of Economic Surveys 19: 309-37.
45. Stanley, T.D. and Stephen B. Jarrell. 1989. ''Meta-Regression Analysis: A Quantitative Method of Literature Surveys.'' Journal of Economic Surveys 3: 161-70.
46. Sterling, T.D. 1959. ''Publication Decision and the Possible Effects on Inferences Drawn From Tests of Significance-or Vice Versa. '' Journal of the American Statistical Association 54: 30-34.
47. Sterling, T.D., W.L. Rosenbaum, and J.J. Weinkam. 1995. ''Publication Bias Revisited: The Effect of the Outcome of Statistical Tests on the Decision to Publish and Vice Versa. '' American Statistician 49: 108-12.
48. Sutton, A.J., F. Song, S.M. Gilbody, and K.R. Abrams. 2000. ''Modelling Publication Bias in Meta-Analysis: A Review.'' Statistical Methods in Medical Research 9: 421-45.
49. Wilson, Franklin D., Gale L. Smoke, and J. David Martin. 1973. ''The Replication Problem in Sociology: A Report and a Suggestion.'' Sociological Inquiry 43: 141-49.