Does the conclusion follow from the evidence? Recommendations for improving research

Journal of Experimental Social Psychology

Volumn 66, Issue , 2016, Pages 39-46

  Hales, Andrew H ^a  

^a PURDUE UNIVERSITY   (United States)

Author keywords

Publication bias; Replication; Type I error; Type II error; Type III error; Type IV error

Indexed keywords

CONSENSUS DEVELOPMENT; DECISION MAKING; DRAWING; ERROR; HUMAN; HUMAN EXPERIMENT; PSYCHOLOGIST; PSYCHOLOGY; PUBLISHING; SCIENTIST; STATISTICAL MODEL;

EID: 84977138103     PISSN: 00221031     EISSN: 10960465     Source Type: Journal    
DOI: 10.1016/j.jesp.2015.09.011     Document Type: Article

References (54)

1. Abelson, R.P., Statistics as principled argument. 1995, Lawrence Erlbaum Associates, Mahwah, NJ.
2. Bem, D.J., Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology 100 (2011), 407–425, 10.1037/a0021524.
3. Burke, C.J., A brief note on one-tailed tests. Psychological Bulletin 50 (1953), 384–387.
4. Cesario, J., Priming, replication, and the hardest science. Perspectives on Psychological Science 9 (2014), 40–48, 10.1177/1745691613513470.
5. Cohen, J., Statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology 65 (1962), 145–153, 10.1037/h0045186.
6. Cohen, J., The earth is round (p < .05). American Psychologist 49 (1994), 997–1003, 10.1037/0003-066X.49.12.997.
7. Cozby, P., Methods in behavioral research. 10th ed., 2009, McGraw-Hill, New York.
8. Cumming, G., The new statistics: Why and how. Psychological Science 25 (2014), 7–29, 10.1177/0956797613504966.
9. Doyen, S., Klein, O., Pichon, C.L., Cleeremans, A., Behavioral priming: It's all in the mind but whose mind?. PloS One 7 (2012), 1–7, 10.1371/journal.pone.0029081.
10. Eich, E., Business not as usual. Psychological Science 25 (2014), 3–6, 10.1177/0956797613512465.
11. Elms, A.C., The crisis of confidence in social psychology. American Psychologist 30 (1975), 967–976, 10.1037/0003-066X.30.10.967.
12. Fabrigar, L.R., Wegener, D.T., Conceptualizing and evaluating the replication of research results. Journal of Experimental Social Psychology 66 (2016), 68–80.
13. Fiedler, K., Kutzner, F., Krueger, J.I., The long way from α-error control to validity proper: Problems with a short-sighted false-positive debate. Perspectives on Psychological Science 7 (2012), 661–669, 10.1177/1745691612462587.
14. Field, A., Discovering statistics using IBM SPSS statistics. 4th ed., 2013, Sage Publications, London.
15. Francis, G., Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review 19 (2012), 151–156, 10.3758/s13423-012-0227-9.
16. Funder, D.C., Levine, J.M., Mackie, D.M., Morf, C.C., Sansone, C., Vazire, S., West, S.G., Improving the dependability of research in personality and social psychology: Recommendations for research and educational practice. Personality and Social Psychology Review 18 (2014), 3–12, 10.1177/1088868313507536.
17. Galak, J., LeBoeuf, R.A., Nelson, L.D., Simmons, J.P., Correcting the past: Failures to replicate psi. Journal of Personality and Social Psychology 103 (2012), 933–948, 10.1037/a0029709.
18. Gergen, K.J., Social psychology as history. Journal of Personality and Social Psychology 26 (1973), 309–320.
19. Giner-Sorolla, R., Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science 7 (2012), 562–571, 10.1177/1745691612457576.
20. Goldacre, Bad science. 2009, Fourth Estate, London England.
21. Greenwald, A.G., Consequences of prejudice against the null hypothesis. Psychological Bulletin 82 (1975), 1–20, 10.1037/h0076157.
22. Hansson, S.O., Defining pseudoscience and science. Pigliucci, M., Boudry, M., (eds.) Philosophy of pseudoscience, 2013, The University of Chicago Press, Chicaco, IL, 61–77.
23. How science goes wrong, The Economist, 2013, October 19.
24. Jost, J.T., McGuire, W.J., An additional future for psychological science. Perspectives on Psychological Science 8 (2013), 414–423.
25. Kerr, N.L., HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review 2 (1998), 196–217, 10.1207/s15327957pspr0203_4.
26. Kimmel, H.D., Three criteria for the use of one-tailed tests. Psychological Bulletin 54 (1957), 351–353, 10.1037/h0046737.
27. Kruschke, J.K., Doing Bayesian data analysis: A tutorial with R and BUGS. 2010, Academic Press, Burlington, MA.
28. Lakens, D., Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology 44 (2014), 701–710, 10.1002/ejsp.2023.
29. Levitt, E.E., The one-tailed test: A statistical editorial. International Journal of Clinical and Experimental Hypnosis 42 (1994), 4–6, 10.1080/00207149408409336.
30. Maner, J.K., Into the wild: Field research can increase both replicability and real-world impact. Journal of Experimental Social Psychology 66 (2016), 100–106.
31. Marascuilo, L.A., Levin, J.R., Appropriate post Hoc comparisons for interaction and nested hypotheses in analysis of variance designs: The elimination of type IV errors. American Educational Research Journal 7 (1970), 397–421.
32. Maxwell, S.E., The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods 9 (2004), 147–163, 10.1037/1082-989X.9.2.147.
33. McGuire, W.J., Some impending reorientations in social psychology: Some thoughts provoked by Kenneth Ring. Journal of Experimental Social Psychology 3 (1967), 124–139, 10.1016/0022-1031(67)90017-0.
34. McGuire, W.J., The yin and yang of progress in social psychology: Seven koan. Journal of Personality and Social Psychology 26 (1973), 446–456, 10.1037/h0034345.
35. Mosteller, F., A k-sample slippage test for an extreme population. Ann. Math. Stat. 19 (1948), 58–65.
36. Nosek, B.A., Bar-Anan, Y., Scientific utopia: I. Opening scientific communication. Psychological Inquiry 23 (2012), 217–243, 10.1080/1047840X.2012.692215.
37. Nosek, B.A., Spies, J.R., Motyl, M., Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science 7 (2012), 615–631, 10.1177/1745691612459058.
38. Perrino, T., Howe, G., Sperling, A., Beardslee, W., Sandler, I., Shern, D. Brown, C., Advancing science through collaborative data sharing and synthesis. Perspectives on Psychological Science 8 (2013), 433–444, 10.1177/1745691613491579.
39. Ring, K., Experimental social psychology: some sober questions about some frivolous values. Journal of Experimental Social Psychology 3 (1967), 113–123, 10.1016/0022-1031(67)90016-9.
40. Ritchie, S.J., Wiseman, R., French, C.C., Failing the future: Three unsuccessful attempts to replicate Bem's 'retroactive facilitation of recall' effect. PloS One, 7, 2012, 10.1371/journal.pone.0033423.
41. Roberts, B., Minimal and incomplete methodological literacy in personality psychology. Retrieved from http://pigee.wordpress.com/minimal-and-incomplete-methodological-literacy-in-personality-psychology/, 2012, May 9.
42. Sagarin, B.J., Ambler, J.K., Lee, E.M., An ethical approach to peeking at data. Perspectives on Psychological Science, 2014, 10.1177/1745691614528214.
43. Schimmack, U., The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods 17 (2012), 551–566, 10.1037/a0029487.
44. Schwartz, S., Carpenter, K.M., The right answer for the wrong question: Consequences of type III error for public health research. American Journal of Public Health 89 (1999), 1175–1180.
45. Shadish, W.R., Cook, T.D., Campbell, D.T., Experimental and quasi-experimental designs for generalized causal inference. 2002, Houghton Mifflin, Boston.
46. Shaffer, J.P., Multiplicity, directional (type III) errors, and the null hypothesis. Psychological Methods 7 (2002), 356–369.
47. Simmons, J.P., Nelson, L.D., Simonsohn, U., False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22 (2011), 1359–1366, 10.1177/0956797611417632.
48. Simmons, J.P., Nelson, L.D., Simonsohn, U., A 21 word solution. Dialogue 26 (2012), 4–7.
49. Simons, D.J., The value of direct replication. Perspectives on Psychological Science 9 (2014), 76–80, 10.1177/1745691613514755.
50. Simonsohn, U., It does not follow: Evaluating the one-off publication bias critiques by Francis (2012a, 2012b, 2012c, 2012d, 2012e, in press). Perspectives on Psychological Science 7 (2012), 597–599, 10.1177/1745691612463399.
51. Simonsohn, U., Just post it: The lesson from two cases of fabricated data detected by statistics alone. Psychological Science 24 (2013), 1875–1888, 10.1177/0956797613480366.
52. Simonsohn, U., Nelson, L.D., Simmons, J.P., P-curve: A key to the file-drawer. Journal of Experimental Psychology: General 143 (2014), 534–547, 10.1037/a0033242.
53. Wagenmakers, E., Wetzels, R., Borsboom, D., van der Maas, H.J., Kievit, R.A., An agenda for purely confirmatory research. Perspectives on Psychological Science 7 (2012), 632–638, 10.1177/1745691612463078.
54. Wike, E.L., Data analysis: A statistical primer for psychology students. 1971, Aldine- Atherton, Chicago.