US studies may overestimate effect sizes in softer research

Proceedings of the National Academy of Sciences of the United States of America

Volumn 110, Issue 37, 2013, Pages 15031-15036

  Fanelli, Daniele ^a   Ioannidis, John P A ^b,c  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

^b STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

^c STANFORD UNIVERSITY   (United States)

Author keywords

Publish or perish; Questionable research practices; Research bias; Scientific misconduct; Soft science

Indexed keywords

ANALYTICAL PARAMETERS; ARTICLE; AUTHOR; BEHAVIORAL RESEARCH; BIOLOGY; CONSENSUS; EFFECT SIZE; HUMAN; HYPOTHESIS; MEDICAL RESEARCH; META ANALYSIS (TOPIC); NOISE; NORMAL HUMAN; PRIORITY JOURNAL; PUBLISHING; SAMPLE SIZE; SELF REPORT; UNITED STATES;

PUBLISH OR PERISH; QUESTIONABLE RESEARCH PRACTICES; RESEARCH BIAS; SCIENTIFIC MISCONDUCT; SOFT SCIENCE;

BEHAVIORAL RESEARCH; BIOMEDICAL RESEARCH; EDITORIAL POLICIES; HUMANS; META-ANALYSIS AS TOPIC; PUBLICATION BIAS; PUBLISHING; UNITED STATES;

EID: 84883829761     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.1302997110     Document Type: Article

References (46)

1. Song F, et al. (2010) Dissemination and publication of research findings: An updated review of related biases. Health Technol Assess 14(8):iii, ix-xi, 1-193.
2. Rosenthal R (1979) The file drawer problem and tolerance for null results. Psychol Bull 86(3):638-641.
3. Ioannidis JPA, Ntzani EE, Trikalinos TA, Contopoulos-Ioannidis DG (2001) Replication validity of genetic association studies. Nat Genet 29(3):306-309. (Pubitemid 33096456)
4. Ioannidis JPA, Trikalinos TA (2005) Early extreme contradictory estimates may appear in published research: The Proteus phenomenon in molecular genetics research and randomized trials. J Clin Epidemiol 58(6):543-549.
5. Schooler J (2011) Unpublished results hide the decline effect. Nature 470(7335):437-437.
6. Ioannidis JPA (2008) Why most discovered true associations are inflated. Epidemiology 19(5):640-648.
7. Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG (2004) Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA 291(20):2457-2465. (Pubitemid 38669189)
8. Chan AW, Krleza-Jerić K, Schmid I, Altman DG (2004) Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 171(7):735-740. (Pubitemid 39319090)
9. Chan AW, Altman DG (2005) Identifying outcome reporting bias in randomised trials on PubMed: Review of publications and survey of authors. BMJ 330(7494):753-756.
10. Fanelli D (2009) How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE 4(5):e5738.
11. Stroebe W, Postmes T, Spears R (2012) Scientific misconduct and the myth of self-correction in science. Perspect Psychol Sci 7(6):670-688.
12. John LK, Loewenstein G, Prelec D (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 23(5):524-532.
13. Fang FC, Steen RG, Casadevall A (2012) Misconduct accounts for the majority of retracted scientific publications. Proc Natl Acad Sci USA 109(42):17028-17033.
14. Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2(8):e124.
15. Rosenthal R (1976) Experimenter Effects in Behavioural Research. Enlarged Edition (Irvington Publishers, New York).
16. Burghardt GM, et al. (2012) Perspectives - Minimizing observer bias in behavioral studies: A review and recommendations. Ethology 118(6):511-517.
17. Fanelli D (2010) "Positive" results increase down the hierarchy of the sciences. PLoS ONE 5(4):e10068.
18. Qiu J (2010) Publish or perish in China. Nature 463(7278):142-143.
19. Osuna C, Cruz-Castro L, Sanz-Menéndez Luis (2011) Overturning some assumptions about the effects of evaluation systems on publication performance. Scientometrics 86:575-592.
20. De Rond M, Miller AN (2005) Publish or perish: Bane or boon of academic life? J Manage Inq 14(4):321-329. (Pubitemid 41795398)
21. de Meis L, Velloso A, Lannes D, Carmo MS, de Meis C (2003) The growing competition in Brazilian science: Rites of passage, stress and burnout. Braz J Med Biol Res 36(9):1135-1141. (Pubitemid 37236236)
22. van Dalen HP, Henkens K (2012) Intended and unintended consequences of a publish-or-perish culture: A worldwide survey. J Am Soc Inf Sci Technol 63(7):1282-1293.
23. Munafò MR, Attwood AS, Flint J (2008) Bias in genetic association studies: Effects of research location and resources. Psychol Med 38(8):1213-1214. (Pubitemid 351950155)
24. Doucouliagos H, Laroche P, Stanley TD (2005) Publication bias in union-productivity research? Relations Industrielles-Industrial Relations 60(2):320-347.
25. Fanelli D (2012) Negative results are disappearing from most disciplines and countries. Scientometrics 90(3):891-904.
26. Fanelli D (2010) Do pressures to publish increase scientists' bias? An empirical support from US States Data. PLoS ONE 5(4):e10271.
27. Simmons JP, Nelson LD, Simonsohn U (2011) False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 22(11):1359-1366.
28. Francis G (2012) Too good to be true: publication bias in two prominent studies from experimental psychology. Psychon Bull Rev 19(2):151-156.
29. Ferguson CJ, Brannick MT (2012) Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychol Methods 17(1):120-128.
30. Fiedler K (2011) Voodoo correlations are everywhere - not only in neuroscience. Perspect Psychol Sci 6(2):163-171.
31. Bakker M, Wicherts JM (2011) The (mis)reporting of statistical results in psychology journals. Behav Res Methods 43(3):666-678.
32. Fava GA (2007) Financial conflicts of interest in psychiatry. World Psychiatry 6(1):19-24.
33. Raudenbush SW, Spybrook J, Liu X, Congdon R (2005) Optimal Design for Longitudinal and Multilevel Research (Univ of Chicago Press, Chicago), Version 1.55.
34. Scherbaum CA, Ferreter JM (2009) Estimating statistical power and required sample sizes for organizational research using multilevel modeling. Organ Res Methods 12(2):347-367.
35. Nezlek JB (2011) Multilevel Modeling for Social and Personality Psychology (Sage, London).
36. Yousefi-Nooraie R, Shakiba B, Mortaz-Hejri S (2006) Country development and manuscript selection bias: A review of published studies. BMC Med Res Methodol 6:37.
37. Matías-Guiu J, García-Ramos R (2011) Editorial bias in scientific publications. Neurologia 26(1):1-5.
38. Link AM (1998) US and non-US submissions: An analysis of reviewer bias. JAMA 280(3):246-247.
39. Lynch JR, et al. (2007) Commercially funded and United States-based research is more likely to be published; good-quality studies with negative outcomes are not. J Bone Joint Surg Am 89(5):1010-1018. (Pubitemid 46724591)
40. Fanelli D, Glänzel W (2013) Bibliometric evidence for a hierarchy of the sciences. PLoS ONE 8(6):e66938
41. van Wilgenburg E, Elgar MA (2013) Confirmation bias in studies of nestmate recognition: a cautionary note for research into the behaviour of animals. PLoS ONE 8(1):e53548.
42. Jeng M (2006) A selected history of expectation bias in physics. Am J Phys 74(7):578-583.
43. Fanelli D (2012) When East meets West. . . does bias increase? A preliminary study on South Korea, United States and other countries. Proceedings of the 8th International Conference on Webometrics, Informetrics and Scientometrics and 13th COLLNET Meeting, eds Choi H-N, et al. (Korea Institute of Science and Technology Information, Seoul), pp 218-223.
44. Ioannidis JPA (2008) Interpretation of tests of heterogeneity and bias in meta-analysis. J Eval Clin Pract 14(5):951-957.
45. Lipsey MW, Wilson DB (2001) Practical Meta-Analysis (Sage Publications, Thousand Oaks, CA).
46. Viechtbauer W (2010) Conducting meta-analyses in {R} with the {metafor} package. J Stat Softw 36(3):1-48.