A concise set of core recommendations to improve the dependability of psychological research

Review of General Psychology

Volumn 19, Issue 1, 2015, Pages 52-68

  Lishner, David A ^a  

^a UNIVERSITY OF WISCONSIN OSHKOSH   (United States)

Author keywords

Dependability; False negative; False positive; Replication; Research

Indexed keywords

EID: 84925671912     PISSN: 10892680     EISSN: 19391552     Source Type: Journal    
DOI: 10.1037/gpr0000028     Document Type: Article

References (75)

1. Abelson, R. P. (1985). A variance explanation paradox: When a little is a lot. Psychological Bulletin, 97, 129-133. http://dx.doi.org/10.1037/0033-2909.97.1.129
2. *Anicich, E. M. (2014). What lies within: Superscripting references to reveal research trends. Perspectives on Psychological Science, 9, 682-691. http://dx.doi.org/10.1177/1745691614549772
3. *Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K.,... Wicherts, J. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108-119. http://dx.doi.org/10.1002/per.1919
4. *Bakker, M., Van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554. http://dx.doi.org/10.1177/1745691612459060
5. *Beghetto, R. A. (2014). Toward avoiding and empirical march to nowhere. Psychology of Aesthetics, Creativity, and the Arts, 8, 18-20. http://dx.doi.org/10.1037/a0035848
6. *Braver, S. L., Thoemmes, F. J., & Rosenthal, R. (2014). Continuously cumulating meta-analysis and replicability. Perspectives on Psychological Science, 9, 333-342. http://dx.doi.org/10.1177/1745691614529796
7. *Brown, S. D., Furrow, D., Hill, D. F., Gable, J. C., Porter, L. P., & Jacobs, W. J. (2014). A duty to describe: Better the devil you know than the devil you don't. Perspectives on Psychological Science, 9, 626-640. http://dx.doi.org/10.1177/1745691614551749
8. *Cesario, J. (2014). Priming, replication, and the hardest science. Perspectives on Psychological Science, 9, 40-48. http://dx.doi.org/10.1177/1745691613513470
9. *Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. The Journal of Abnormal and Social Psychology, 65, 145-153. http://dx.doi.org/10.1037/h0045186
10. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159. http://dx.doi.org/10.1037/0033-2909.112.1.155
11. *Cohen, J. (1994). The earth is round (p <.05). American Psychologist, 49, 997-1003. http://dx.doi.org/10.1037/0003-066X.49.12.997
12. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302. http://dx.doi.org/10.1037/h0040957
13. *Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7-29. http://dx.doi.org/10.1177/0956797613504966
14. *Dijksterhuis, A. (2014). Welcome back theory! Perspectives on Psychological Science, 9, 72-75. http://dx.doi.org/10.1177/1745691613513472
15. Festinger, L. (1957). A theory of cognitive dissonance. Evanston, IL: Row Peterson.
16. *Fidler, F., & Cumming, G. (2014). Yes, but don't underestimate estimation:Reply to Morey, Rouder, Verhagen, and Wagenmakers (2014). Psychological Science, 25, 1291-1292. http://dx.doi.org/10.1177/0956797614532658
17. Fiedler, K., Kutzner, F., & Krueger, J. I. (2012). The long way from _-error control to validity proper: Problems with a short-sighted false-positive debate. Perspectives on Psychological Science, 7, 661-669. http://dx.doi .org/10.1177/1745691612462587
18. Francis, G. (2012). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7, 585-594. http://dx .doi.org/10.1177/1745691612459520
19. *Frank, M. C., & Saxe, R. (2012). Teaching replication. Perspectives on Psychological Science, 7, 600-604.
20. *Funder, D. C., Levine, J. M., Mackie, D. M., Morf, C. C., Sansone, C., Vazire, S., & West, S. G. (2014). Improving the dependability of research in personality and social psychology: Recommendations for research and educational practice. Personality and Social Psychology Review, 18, 3-12. http://dx.doi.org/10.1177/1088868313507536
21. Galak, J., & Meyvis, T. (2012). You could have just asked: Reply to Francis (2012). Perspectives on Psychological Science, 7, 595-596. http://dx.doi.org/10.1177/1745691612463079
22. *Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science, 9, 641-651. http://dx.doi.org/10.1177/1745691 614551642
23. *Giger-Sorolla, R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562-571. http://dx.doi.org/10.1177/1745691612457576
24. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20. http://dx.doi.org/10.1037/h0076157
25. *Hunter, J. H. (1997). Needed: A ban on the significance test. Psychological Science, 8, 3-7. http://dx.doi.org/10.1111/j.1467-9280.1997.tb00534.x
26. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2, 0696-0701.
27. Ioannidis, J. P. A. (2012). Why science is not necessarily self-correcting. Perspectives on Psychological Science, 7, 645-654.
28. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524-532. http://dx.doi.org/10.1177/0956797611430953
29. *Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196-217. http://dx.doi .org/10.1207/s15327957pspr0203_4
30. *Klein, O., Doyen, S., Leys, C., Magalhaes de Saldanha da Gama, P. A., Miller, S., Questienne, L., & Cleeremans, A. (2012). Low hopes, high expectations: Expectancy effects and the replicability of behavioral experiments. Perspectives on Psychological Science, 7, 572-584. http://dx.doi.org/10.1177/1745691612463704
31. *Koole, S. L., & Lakens, D. (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7, 608-614. http://dx.doi.org/10.1177/1745691 612462586
32. *Lakens, D., & Evers, E. R. K. (2014). Sailing from the seas of chaos into the corridor of stability: Practical recommendations to increase the informational value of studies. Perspectives on Psychological Science, 9, 278-292. http://dx.doi.org/10.1177/1745691614528520
33. Ledgerwood, A. (2014a). Introduction to the special section on advancing out methods and practices. Perspectives on Psychological Science, 9, 275-277. http://dx.doi.org/10.1177/1745691614529448
34. Ledgerwood, A. (2014b). Introduction to the special section on moving toward a cumulative science: Maximizing what our research can tell us. Perspectives on Psychological Science, 9, 610-611. http://dx.doi.org/10.1177/1745691614553989
35. Lewin, K. (1997). Resolving social conflicts and field theory in social science. Washington, DC: American Psychological Association. http://dx.doi.org/10.1037/10269-000
36. *Makel, M. C. (2014). The empirical march: Making science better at self-correction. Psychology of Aesthetics, Creativity, and the Arts, 8, 2-7. http://dx.doi.org/10.1037/a0035803
37. *Makel, M. C., & Plucker, J. A. (2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43, 304-316. http://dx.doi.org/10.3102/0013189X14545513
38. *Maner, J. K. (2014). Let's put our money where our mouth is: If authors are to change their ways, reviewers (and Eds.) must change with them. Perspectives on Psychological Science, 9, 343-351. http://dx.doi.org/10.1177/1745691614528215
39. Marszalek, J. M., Barber, C., Kohlhart, J., & Holmes, C. B. (2011). Sample size in psychological research over the past 30 years. Perceptual and Motor Skills, 112, 331-348. http://dx.doi.org/10.2466/03.11.PMS.112.2.331-348
40. *McBee, M. T., & Matthews, M. S. (2014). Welcoming quality in nonsignificance and replication work, but moving beyond the p-value:Announcing new editorial policies for quantitative research in JOAA. Journal of Advanced Academics, 25, 73-87. http://dx.doi.org/10.1177/1932202X14532177
41. *McShane, B. B., & Böckenholt, U. (2014). You cannot step into the same river twice: When power analyses are optimistic. Perspectives on Psychological Science, 9, 612-625. http://dx.doi.org/10.1177/1745691614548513
42. Meehl, P. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1, 108-141. http://dx.doi.org/10.1207/s15327965pli0102_1
43. Mook, D. G. (1983). In defense of external validity. American Psychologist, 38, 379-387. http://dx.doi.org/10.1037/0003-066X.38.4.379
44. Morey, R. D., Rouder, J. N., Verhagen, J., & Wagenmakers, E. J. (2014). Why hypothesis tests are essential for psychological science: A comment on Cumming (2014). Psychological Science, 25, 1289-1290. http://dx.doi.org/10.1177/0956797614525969
45. *Murayama, K., Pekrun, R., & Fiedler, K. (2014). Research practices that can prevent an inflation of false-positive rates. Personality and Social Psychology Review, 18, 107-118. http://dx.doi.org/10.1177/1088868313496330
46. Neulip, J. W., & Crandall, R. (1990). Editorial bias against replication research. Journal of Social Behavior & Personality, 5, 85-90.
47. Neulip, J. W., & Crandall, R. (1993a). Everyone was wrong: There are lots of replications out there. Journal of Social Behavior & Personality, 8, 1-8.
48. Neulip, J. W., & Crandall, R. (1993b). Reviewer bias against replication research. Journal of Social Behavior & Personality, 8, 21-29.
49. *Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631. http://dx.doi .org/10.1177/1745691612459058
50. Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657-660. http://dx.doi.org/10.1177/1745691612462588
51. *Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531-536. http://dx.doi.org/10.1177/1745691612463401
52. Pashler, H., & Wangenmakers, E. (2012). Editor's introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528-530. http://dx.doi.org/10.1177/1745691612465253
53. Prentice, D. A., & Miller, D. T. (1992). When small effects are impressive. Psychological Bulletin, 112, 160-164. http://dx.doi.org/10.1037/0033-2909.112.1.160
54. *Reiter-Palmon, R., & Tinio, P. P. L. (2014). Pipeline revisions: A call to change. Psychology of Aesthetics, Creativity, and the Arts, 8, 13-14. http://dx.doi.org/10.1037/a0035845
55. Rosenthal, R., & Rubin, D. B. A. (1979). A note on percentage of variance explained as a measure of the importance of effects. Journal of Applied Psychology, 9, 395-396.
56. Rozin, P. (2009). What kind of empirical research should we publish, fund, and reward?: A different perspective. Perspectives on Psychological Science, 4, 435-439. http://dx.doi.org/10.1111/j.1745-6924.2009.01151.x
57. Russ, S. W. (2014). Demonstrating effects with children: When do we know we know? Psychology of Aesthetics, Creativity, and the Arts, 8, 24-26. http://dx.doi.org/10.1037/a0035847
58. *Sagarin, B. J., Ambler, J. K., & Lee, E. M. (2014). An ethical approach to peeking at data. Perspectives on Psychological Science, 9, 293-304. http://dx.doi.org/10.1177/1745691614528214
59. *Sakaluk, J., Williams, A., & Biernat, M. (2014). Analytic review as a solution to the misreporting of statistical results in psychological science. Perspectives on Psychological Science, 9, 652-660. http://dx.doi .org/10.1177/1745691614549257
60. Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13, 90-100. http://dx.doi.org/10.1037/a0015108
61. Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies. Psychological Bulletin, 105, 309-316. http://dx.doi.org/10.1037/0033-2909.105.2.309
62. *Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. http://dx.doi.org/10.1177/0956797611417632
63. *Simons, D. J. (2014). The value of direct replication. Perspectives on Psychological Science, 9, 76-80. http://dx.doi.org/10.1177/1745691613514755
64. Simonsohn, U. (2012). It does not follow: Evaluating the one-off publication bias critiques by Francis (2012a, 2012b, 2012c, 2012d, 2012e, in press). Perspectives on Psychological Science, 7, 597-599. http://dx.doi .org/10.1177/1745691612463399
65. *Simonsohn, U. (2013). Just post it: The lesson from two cases of fabricated data detected by statistics alone. Psychological Science, 24, 1875-1888. http://dx.doi.org/10.1177/0956797613480366
66. *Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve and effect size: Correcting for publication bias using only significant results. Perspectives on Psychological Science, 9, 666-681. http://dx.doi.org/10.1177/1745691614553988
67. Smith, N. C. (1970). Replication studies: A neglected aspect of psychological research. American Psychologist, 25, 970-975. http://dx.doi.org/10.1037/h0029774
68. *Smith, J. K., & Smith, L. F. (2014). Replicability? Not that again! Psychology of Aesthetics, Creativity, and the Arts, 8, 21-23. http://dx .doi.org/10.1037/a0035821
69. Spellman, B. A. (2012). Introduction to the special section on research practices. Perspectives on Psychological Science, 7, 655-656. http://dx .doi.org/10.1177/1745691612465075
70. *Stanley, D. J., & Spence, J. R. (2014). Expectations for replications: Are yours realistic? Perspectives on Psychological Science, 9, 305-318. http://dx.doi.org/10.1177/1745691614528518
71. *Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9, 59-71. http://dx.doi.org/10.1177/1745691613514450
72. *Tsuji, S., Bergmann, C., & Cristia, A. (2014). Community-augmented meta-analyses: Toward cumulative data assessment. Perspectives on Psychological Science, 9, 661-665. http://dx.doi.org/10.1177/1745691614552498
73. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E. (2011). Statistical evidence in experimental psychology:An empirical comparison using 855 t tests. Perspectives on Psychological Science, 6, 291-298. http://dx.doi.org/10.1177/1745691611406923
74. Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726-728. http://dx.doi.org/10.1037/0003-066X.61.7.726
75. Yeaton, W. H., & Sechrest, L. (1981). Meaningful measures of effect. Journal of Consulting and Clinical Psychology, 49, 766-767. http://dx .doi.org/10.1037/0022-006X.49.5.766