The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power

PLoS ONE

Volumn 9, Issue 10, 2014, Pages

  Fraley, R Chris ^a   Vazire, Simine ^b  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; EFFECT SIZE; INFORMATION PROCESSING; JOURNAL IMPACT FACTOR; MEDICAL RESEARCH; N PACT FACTOR; PUBLISHING; QUALITY CONTROL; QUANTITATIVE ANALYSIS; SAMPLE SIZE; SCIENTIFIC LITERATURE; STATISTICAL CONCEPTS; STATISTICAL POWER; STANDARDS;

JOURNAL IMPACT FACTOR; PUBLISHING; SAMPLE SIZE;

EID: 84907830999     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0109019     Document Type: Article

References (51)

1. Basken P (2013) Researchers and scientific groups make new push against impact factors. Available: http://chronicle.com/article/article-content/139337/Accessed 2013 October 12.
2. Brembs B, Button K, Munafo M (2013) Deep impact: Unintended consequences of journal rank. Frontiers in Human Neuroscience 7:291. doi: 10.3389/fnhum.2013.00291.
3. Davis P, Scholarlykitchen (17 October 2011) Gaming the Impact Factor puts journal in time-out. Available: http://scholarlykitchen.sspnet.org/2011/10/17/gaming-the-impact-factor-puts-journal-in-time-out/Accessed 2013 October 12.
4. Roediger HL III (September 2013) Journal impact factors: How much should we care? Observer, 26. Available: http://www.psychologicalscience.org/index.php/publications/observer/2013/september-13/journal-impact-factors.html. Accessed 2013 October 12.
5. Vanclay JK (2012) Impact factor: outdated artifact or stepping-stone to journal certification. Scientometrics. doi:10.1007/s11192-011-0561-0.
6. Bornmann L, Mutz R, Daniel H-D (2010) A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE 5(12): e14331. doi:10.1371/journal.pone.0014331.
7. Cohen J (1962) The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology 65:145-153.
8. Sedlmeier P, Gigerenzer G (1989) Do studies of statistical power have an effect on the power of studies? Psychological Bulletin 105: 309-316.
9. Cohen J (1990) Things I have learned (so far). American Psychologist 45: 1304- 1312.
10. Ioannidis JPA (2005) Why most published research findings are false. PLoS Med, 2(8): e124. doi:10.1371/journal.pmed.0020124.
11. Wacholder S, Chanock S, Garcia-Closas M, Elghormli L, Rothman N (2004) Assessing the probability that a positive report is false: An approach for molecular epidemiology studies. Journal of the National Cancer Institute 96: 434-442. doi: 10.1056/nejmsa070929.
12. Fanelli D (2012) Negative results are disappearing from most disciplines and countries. Scientometrics 90: 891-904.
13. Pashler H, Harris C (2012) Is the replicability crisis overblown? Three arguments examined Perspectives on Psychological Science 7: 531-536.
14. Oakes M (1986) Statistical inference: A commentary for the social and behavioral sciences. New York: Wiley. 196 p.
15. Aarts AA, Alexander A, Dreber A, Cramblet Alvarez LD, et al. (2014) Reproducibility Project: Psychology. Retrieved from Open Science Framework, http://osf.io/ezcuj.
16. Nosek BA, Lakens D, Bowman S (2014) Replications of Important Results in Social Psychology: Special Issue of Social Psychology. Retrieved from Open Science Framework, http://osf.io/hxeza.
17. Fraley RC, Marks MJ (2007) The null hypothesis significance testing debate and its implications for personality research. In R. W. Robins, R. C. Fraley, & R. F. Krueger (Eds.), Handbook of research methods in personality psychology (pp. 149-169). New York: Guilford.
18. Schmidt FL (1996) Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods 1: 115-129.
19. Yong E (10 March 2012) A failed replication attempt draws a scathing personal attack from a psychology professor. Available: http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed-replication-bargh-psychology-studydoyen/Accessed 2012 March 10.
20. Pashler H, Wagenmakers EJ (2012) Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science 7: 528-530.
21. Francis G (2012) Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin and Review 19: 151-156.
22. Schimmack U (2012) The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods 17: 551-566. doi: 10.1037/a0029487.
23. Cohen J (1977) Statistical power analysis for the behavioral sciences (2nd ed.). New York: Academic Press. 590 p.
24. Maxwell SE (2004) The persistence of underpowered studies in psychological research: causes, consequences, and remedies. Psychological Methods 9: 147.
25. Rossi JS (1990) Statistical power of psychological research: What have we gained in 20 years? Journal of Consulting and Clinical Psychology 58: 646-656.
26. LeBel EP, Peters KR (2011) Fearing the future of empirical psychology: Bem's (2011) evidence of psi as a case study in deficiencies in modal research practice. Review of General Psychology 15: 371-379.
27. Klein RA, Ratliff KA, Vianello M, Adams RB, Bahnik S, et al. (2014) Investigating variation in replicability: A "many labs" replication project. Retrieved from Open Science Framework, osf.io/wx7ck.
28. Tracy JL, Robins RW, Sherman JW (2009) The practice of psychological science: Searching for Cronbach's two streams in social-personality psychology. Journal of Personality and Social Psychology 96: 1206-1225.
29. Richard FD, Bond CF Jr, Stokes-Zoota JJ (2003) One hundred years of social psychology quantitatively described. Review of General Psychology 7: 331-363.
30. Funder DC, Ozer DJ (1983) Behavior as a function of the situation. Journal of Personality and Social Psychology 44: 107-121.
31. Sarason IG, Smith RE, Diener E (1975) Personality research: Components of variance attributable to the person and the situation. Journal of Personality and Social Psychology 32: 199-204.
32. Reis HT, Stiller J (1992) Publication trends in JPSP: A Three-decade review. Personality and Social Psychology Bulletin 18: 465-472.
33. Hoenig JM, Heisey DM (2001) The abuse of power: The pervasive fallacy of power calculations for data analysis. American Statistician 55: 19-24.
34. Yuan K-H, Maxwell S (2005) On the post hoc power in testing mean differences. Journal of Educational and Behavioral Statistics 30: 141-167.
35. Newman DA, Jacobs RR, Bartram D (2007) Choosing the best method for local validity estimation: relative accuracy of meta-analysis versus a local study versus Bayes-analysis. Journal of Applied Psychology 92: 1394-1413.
36. Simmons JP, Nelson LD, Simonsohn U (2011) False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22: 1359-1366.
37. John L, Loewenstein GF, Prelec D (2012) Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science 23: 524-532.
38. Pashler H, Coburn N, Harris CR (2012) Priming of social distance? Failure to replicate effects on social and food judgments. PloS one, 7(8), e42510.
39. Harris CR, Coburn N, Rohrer D, Pashler H (2013) Two failures to replicate high-performance-goal priming effects. PloS one, 8(8), e72467.
40. Shanks DR, Newell BR, Lee EH, Dalakrishnan D, Ekelund L, et al. (2013) Priming intelligent behavior: An elusive phenomenon. PloS one, 8(4), e56515.
41. Donnellan MB, Lucas RE, Cesario J (in press) On the association between loneliness and bathing habits: Nine replications of Bargh and Shalev (2012) study 1. Emotion.
42. Simmons JP, Nelson LD, Simonsohn U (2013) Life after p-hacking. Meeting of the Society for Personality and Social Psychology, New Orleans, LA, 17-19 January 2013. Available at SSRN: http://ssrn.com/abstract=2205186 or http://dx.doi.org/10.2139/ssrn.2205186.
43. Friston K (2012) Ten ironic rules for non-statistical reviewers. NeuroImage, 61, 1300-1310.
44. Meehl PE (1967) Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science 34: 103-115.
45. Meehl PE (1978) Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology 46: 806-834.
46. Ableson RP (1985) A variance explanation paradox: When a little is a lot. Psychological Bulletin 97: 128-132.
47. Rosenthal R, Rubin D (1983) A note on percent of variance explained as a measure of the importance of effects. Journal of Applied Social Psychology 9: 395-396.
48. Mook DG (1983) In defense of external invalidity. American Psychologist, 38, 379-387.
49. Prentice DA, Miller DT (1992). When small effects are impressive. Psychological Bulletin 112: 160-164.
50. Cumming G (2012) Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis. New York: Routledge. 535 p.
51. Kraus M (2013) Quality v. quantity in publication. Available: http://psychyour-mind.blogspot.com/2013/06/quality-v-quantity-in-publication.html. Accessed 2013 June 19.