The frequency of excess success for articles in Psychological Science

Psychonomic Bulletin and Review

Volumn 21, Issue 5, 2014, Pages 1180-1187

  Francis, Gregory ^a  

^a PURDUE UNIVERSITY   (United States)

Author keywords

Probabilistic reasoning; Statistical inference; Statistics

Indexed keywords

HUMAN; PROCEDURES; PSYCHOLOGY; PUBLICATION; SAMPLE SIZE; STANDARDS; STATISTICAL ANALYSIS; STATISTICAL BIAS; STATISTICAL MODEL; STATISTICS; STATISTICS AND NUMERICAL DATA;

BIAS (EPIDEMIOLOGY); DATA INTERPRETATION, STATISTICAL; HUMANS; MODELS, STATISTICAL; PERIODICALS AS TOPIC; PSYCHOLOGY, EXPERIMENTAL; SAMPLE SIZE; STATISTICS AS TOPIC;

EID: 84930820391     PISSN: 10699384     EISSN: 15315320     Source Type: Journal    
DOI: 10.3758/s13423-014-0601-x     Document Type: Review

References (84)

1. Alter, A. L., & Oppenheimer, D. M. (2009). Suppressing secrecy through metacognitive ease: Cognitive fluency encourages self-disclosure. Psychological Science, 20, 1414–1420.
2. Anderson, C., Kraus, M. W., Galinsky, A. D., & Keltner, D. (2012). The local-ladder effect: Social status and subjective well-being. Psychological Science, 23, 764–771.
3. Ashton-James, C. E., Maddux, W. W., Galinsky, A. D., & Chartrand, T. L. (2009). Who I am depends on how I feel: The role of affect in the expression of culture. Psychological Science, 20, 340–346.
4. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554. doi:10.1177/1745691612459060
5. Balcetis, E., & Dunning, D. (2010). Wishful seeing: More desired objects are seen as closer. Psychological Science, 21, 147–152.
6. Bauer, M. A., Wilkie, J. E. B., Kim, J. K., & Bodenhausen, G. V. (2012). Cuing consumerism: Situational materialism undermines personal and social well-being. Psychological Science, 23, 517–523.
7. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101.
8. Birtel, M. D., & Crisp, R. J. (2012). “Treating” prejudice: An exposure-therapy approach to reducing negative reactions toward stigmatized groups. Psychological Science, 23, 1379–1386.
9. Bowles, H. R., & Gelfand, M. (2010). Status and the evaluation of workplace deviance. Psychological Science, 21, 49–54.
10. Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49, 609–610.
11. Converse, B. A., & Fishbach, A. (2012). Instrumentality boosts appreciation: Helpers are more appreciated while they are useful. Psychological Science, 23, 560–566.
12. Converse, B. A., Risen, J. L., & Carter, T. J. (2012). Investing in karma: When wanting promotes helping. Psychological Science, 23, 923–930.
13. Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29. doi:10.1177/0956797613504966
14. Damisch, L., Stoberock, B., & Mussweiler, T. (2010). Keep your fingers crossed! How superstition improves performance. Psychological Science, 21, 1014–1020.
15. de Hevia, M. D., & Spelke, E. S. (2010). Number-space mapping in human infants. Psychological Science, 21, 653–660.
16. Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in a meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 639–634.
17. Eich, E. (2014). Business not as usual [Editorial]. Psychological Science, 25, 3–6. doi:10.1177/0956797613512465
18. Ersner-Hershfield, H., Galinsky, A. D., Kray, L. J., & King, B. G. (2010). Company, country, connections: Counterfactual origins increase organizational commitment, patriotism, and social investment. Psychological Science, 21, 1479–1486.
19. Evans, K. K., Horowitz, T. S., & Wolfe, J. M. (2011). When categories collide: Accumulation of information about multiple categories in rapid scene perception. Psychological Science, 22, 739–746. doi:10.1177/0956797611407930
20. Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PLoS ONE, 5, e10068. doi:10.1371/journal.pone.0010068
21. Fast, N. J., & Chen, S. (2009). When the boss feels inadequate: Power, incompetence, and aggression. Psychological Science, 20, 1406–1413.
22. Fast, N. J., Gruenfeld, D. H., Sivanathan, N., & Galinsky, A. D. (2009). Illusory control: A generative force behind power’s far-reaching effects. Psychological Science, 20, 502–508.
23. Francis, G. (2012a). Evidence that publication bias contaminated studies relating social class and unethical behavior. Proceedings of the National Academy of Sciences, 109, E1587.
24. Francis, G. (2012b). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7, 585–594. doi:10.1177/1745691612459520
25. Francis, G. (2012c). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19, 975–991. doi:10.3758/s13423-012-0322-y
26. Francis, G. (2012d). The same old New Look: Publication bias in a study of wishful seeing. i-Perception, 3, 176–178.
27. Francis, G. (2012e). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19, 151–156. doi:10.3758/s13423-012-0227-9
28. Francis, G. (2013a). Publication bias in “Red, rank, and romance in women viewing men” by Elliot et al. (2010). Journal of Experimental Psychology: General, 142, 292–296. doi:10.1037/a0027923
29. Francis, G. (2013b). Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology, 57, 153–169. doi:10.1016/j.jmp.2013.02.003
30. Francis, G. (2013c). We should focus on the biases that matter: A reply to commentaries. Journal of Mathematical Psychology, 57, 190–195. doi:10.1016/j.jmp.2013.06.001
31. Gao, T., McCarthy, G., & Scholl, B. J. (2010). The wolfpack effect: Perception of animacy irresistibly influences interactive behavior. Psychological Science, 21, 1845–1853.
32. Garcia, S. M., & Tor, A. (2009). The N-effect: More competitors, less competition. Psychological Science, 20, 871–877.
33. Gelman, A. (2013). Is it possible to be an ethicist without being mean to people? Chance, 26, 51–53.
34. Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. In Technical report, Department of Statistics, Columbia University. Retrieved January 30, 2014, from www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
35. González, J., & McLennan, C. T. (2009). Hemispheric differences in the recognition of environmental sounds. Psychological Science, 20, 887–894.
36. Hahn, U., Close, J., & Graf, M. (2009). Transformation direction influences shape-similarity judgments. Psychological Science, 20, 447–454.
37. Hart, W., & Albarracín, D. (2009). What I was doing versus what I did: Verb aspect influences memory and future actions. Psychological Science, 20, 238–244.
38. Inesi, M. E., Botti, S., Dubois, D., Rucker, D. D., & Galinsky, A. D. (2011). Power and choice: Their dynamic interplay in quenching the thirst for personal control. Psychological Science, 22, 1042–1048.
39. Ioannidis, J. P. A., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245–253.
40. Janssen, N., & Caramazza, A. (2009). Grammatical and phonological influences on word order. Psychological Science, 20, 1262–1268.
41. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524–532. doi:10.1177/0956797611430953
42. Johnson, V. E. (2013). On biases in assessing replicability, statistical consistency and publication bias. Journal of Mathematical Psychology, 57, 177–179. doi:10.1016/j.jmp.2013.04.003
43. Johnson, V., & Yuan, Y. (2007). Comments on “An exploratory test for an excess of significant findings” by JPA loannidis and TA Trikalinos. Clinical Trials, 4, 254–255. doi:10.1177/1740774507079437. disc. 256–257.
44. Jostmann, N. B., Lakens, D., & Schubert, T. W. (2009). Weight as an embodiment of importance. Psychological Science, 20, 1169–1174.
45. Keysar, B., Hayakawa, S. L., & An, S. G. (2012). The foreign-language effect: Thinking in a foreign tongue reduces decision biases. Psychological Science, 23, 661–668.
46. Koole, S. L., & Lakens, D. (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7, 608–614. doi:10.1177/1745691612462586
47. Kruschke, J. K. (2010). Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science, 1, 658–676. doi:10.1002/wcs.72
48. Labroo, A. A., Lambotte, S., & Zhang, Y. (2009). The “name-ease” effect and its dual impact on importance judgments. Psychological Science, 20, 1516–1522.
49. Lammers, J., Stapel, D. A., & Galinsky, A. D. (2010). Power increases hypocrisy: Moralizing in reasoning, immorality in behavior. Psychological Science, 21, 737–744.
50. Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107–112.
51. LeBel, E. P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., & Smith, C. T. (2013). PsychDisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8, 424–432.
52. Leung, A. K., Kim, S., Polman, E., Ong, L. S., Qiu, L., Goncalo, J. A., & Sanchez-Burks, J. (2012). Embodied metaphors and creative “acts”. Psychological Science, 23, 502–509.
53. Li, X., Wei, L., & Soman, D. (2010). Sealing the emotions genie: The effects of physical enclosure on psychological closure. Psychological Science, 21, 1047–1050.
54. Maddux, W. W., Yang, H., Falk, C., Adam, H, Adair, W., Endo, Y., … Heine, S. J. (2010). For whom is parting with possessions more painful? Cultural differences in the endowment effect. Psychological Science, 21, 1910–1917.
55. McGraw, A. P., & Warren, C. (2010). Benign violations: Making immoral behavior funny. Psychological Science, 21, 1141–1149. doi:10.1177/0956797610376073
56. Morey, R. D. (2013). The consistency test does not—and cannot—deliver what is advertised: A comment on Francis (2013). Journal of Mathematical Psychology, 57, 180–183. doi:10.1016/j.jmp.2013.03.004
57. Nordgren, L. F., Morris McDonnell, M.-H., & Loewenstein, G. (2011). What constitutes torture? Psychological impediments to an objective evaluation of enhanced interrogation tactics. Psychological Science, 22, 689–694.
58. Nordgren, L. F., van Harreveld, F., & van der Pligt, J. (2009). The restraint bias: How the illusion of self-restraint promotes impulsive behavior. Psychological Science, 20, 1523–1528.
59. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia II: Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631. doi:10.1177/1745691612459058
60. Renkewitz, F., Fuchs, H. M., & Fiedler, S. (2011). Is there evidence of publication biases in JDM research? Judgment and Decision Making, 6, 870–881.
61. Roediger, H. L., III. (2012, February). Psychology’s woes and a partial cure: The value of replication. APS Observer, 25(2). Retrieved from www.psychologicalscience.org/index.php/publications/observer/2012/february-11-2012-observer-publications/psychology’s-woes-and-a-partial-cure-the-value-of-replication.html
62. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237. doi:10.3758/PBR.16.2.225
63. Rounding, K., Lee, A., Jacobson, J. A., & Ji, L.-J. (2012). Religion replenishes self-control. Psychological Science, 23, 635–642.
64. Sackett, A. M., Meyvis, T., Nelson, L. D., Converse, B. A., & Sackett, A. L. (2010). You’re having fun when time flies: The hedonic consequences of subjective time progression. Psychological Science, 21, 111–117.
65. Savani, K., Markus, H. R., Naidu, N. V. R., Kumar, S., & Berlia, N. (2010). What counts as a choice? U.S. Americans are more likely than Indians to construe actions as choices. Psychological Science, 21, 391–398.
66. Savani, K., & Rattan, A. (2012). A choice mind-set increases the acceptance and maintenance of wealth inequality. Psychological Science, 23, 796–804.
67. Savani, K., Stephens, N. M., & Markus, H. R. (2011). The unanticipated interpersonal and societal consequences of choice: Victim blaming and reduced support for the public good. Psychological Science, 22, 795–802.
68. Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17, 551–566. doi:10.1037/a0029487
69. Senay, I., Albarracín, D., & Noguchi, K. (2010). Motivating goal-directed behavior through introspective self-talk: The role of the interrogative form of simple future tense. Psychological Science, 21, 499–504.
70. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632
71. Simonsohn, U. (2012). It does not follow: Evaluating the one-off publication bias critiques by Francis (2012a, 2012b, 2012c, 2012d, 2012e, in press). Perspectives on Psychological Science, 7, 597–599. doi:10.1177/1745691612463399
72. Simonsohn, U. (2013). It really just does not follow, comments on Francis (2013). Journal of Mathematical Psychology, 57, 174–176. doi:10.1016/j.jmp.2013.03.006
73. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (in press). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General.
74. Sterling, T. D. (1959). Publication decisions and the possible effects on inferences drawn from test of significance—or vice versa. Journal of the American Statistical Association, 54, 30–34.
75. Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician, 49, 108–112.
76. Todd, A. R., Hanko, K., Galinsky, A. D., & Mussweiler, T. (2011). When focusing on differences leads to similar perspectives. Psychological Science, 22, 134–141.
77. Tuk, M. A., Trampe, D., & Warlop, L. (2011). Inhibitory spillover: Increased urination urgency facilitates impulse control in unrelated domains. Psychological Science, 22, 627–633.
78. van Boxtel, J. J. A., & Koch, C. (2012). Visual rivalry without spatial conflict. Psychological Science, 23, 410–418.
79. Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804. doi:10.3758/BF03194105
80. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. doi:10.1177/1745691612463078
81. Wakslak, C., & Trope, Y. (2009). The effect of construal level on subjective probability estimates. Psychological Science, 20, 52–58.
82. West, G. L., Anderson, A. K., Bedwell, J. S., & Pratt, J. (2010). Red diffuse light suppresses the accelerated perception of fear. Psychological Science, 21, 992–999. doi:10.1177/0956797610371966
83. Wolfe, J. M. (2013). Registered reports and replications in Attention, Perception, & Psychophysics [Editorial]. Attention, Perception, & Psychophysics, 75, 781–783. doi:10.3758/s13414-013-0502-5
84. Zhou, X., Vohs, K. D., & Baumeister, R. F. (2009). The symbolic power of money: Reminders of money alter social distress and physical pain. Psychological Science, 20, 700–706. doi:10.1111/j.1467-9280.2009.02353.x