The state of social and personality science: Rotten to the core, not so bad, getting better, or getting worse?

Journal of Personality and Social Psychology

Volumn 113, Issue 1, 2017, Pages 34-58

  Motyl, Matt ^a   Demos, Alexander P ^a   Carsel, Timothy S ^a   Hanson, Brittany E ^a   Melton, Zachary J ^a   Mueller, Allison B ^a   Prims, J P ^a   Sun, Jiaqing ^a   Washburn, Anthony N ^a   Wong, Kendal M ^a   Yantis, Caitlyn ^a   Skitka, Linda J ^a  

^a UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

Author keywords

Meta science; QRPs; Questionable research practices; Replicability; Scientific quality

Indexed keywords

HUMAN; HUMAN EXPERIMENT; PERCEPTION; PERSONALITY; PSYCHOLOGIST; PSYCHOLOGY; SCIENTIST; STATISTICS; FEMALE; HEALTH PERSONNEL ATTITUDE; MALE; METHODOLOGY; QUESTIONNAIRE; RESEARCH; RESEARCH ETHICS; SOCIAL PSYCHOLOGY; STANDARDS;

ATTITUDE OF HEALTH PERSONNEL; ETHICS, RESEARCH; FEMALE; HUMANS; MALE; PERSONALITY; PSYCHOLOGY; PSYCHOLOGY, SOCIAL; RESEARCH; RESEARCH DESIGN; SURVEYS AND QUESTIONNAIRES;

EID: 85020500843     PISSN: 00223514     EISSN: None     Source Type: Journal    
DOI: 10.1037/pspa0000084     Document Type: Article

References (89)

1. Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J., Fiedler, K. Wicherts, J. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108- 119. http://dx.doi.org/10.1002/per.1919.
2. Bakker, M., & Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666- 678. http://dx.doi.org/10.3758/s13428-011-0089-5.
3. Barrett, L. F. (2015, September 1). Psychology is not in crisis. Retrieved from http://www.nytimes.com/2015/09/01/opinion/psychology-is-notin- crisis.html?_r=0.
4. Baumeister, R. F. (2016). Charting the future of social psychology on stormy seas: Winners, losers, and recommendations. Journal of Experimental Social Psychology, 66, 153-158. http://dx.doi.org/10.1016/j.jesp.2016.02.003.
5. Begley, C. G., & Ioannidis, J. P. (2015). Reproducibility in science: Improving the standard for basic and preclinical research. Circulation Research, 116, 116-126. http://dx.doi.org/10.1161/CIRCRESAHA.114.303819.
6. Bem, D. J. (2003). Writing the empirical journal article. In J. M. Darley, M. P. Zanna, & H. L. Roediger, III (Eds.), The complete academic: A career guide (pp. 171-201). Washington, DC: American Psychological Association.
7. Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407-425. http://dx.doi.org/10.1037/a0021524.
8. Bezerianos, A., Tong, S., & Thakor, N. (2003). Time-dependent entropy estimation of EEG rhythm changes following brain ischemia. Annals of Biomedical Engineering, 31, 221-232. http://dx.doi.org/10.1114/1.1541013.
9. Carter, E. C., Kofler, L. M., Forster, D. E., & McCullough, M. E. (2015). A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. Journal of Experimental Psychology: General, 144, 796-815. http://dx.doi.org/10.1037/xge0000083.
10. Cialdini, R. B., Reno, R. R., & Kallgren, C. A. (1990). A focus theory of normative conduct: Recycling the concept of norms to reduce littering in public places. Journal of Personality and Social Psychology, 58, 1015- 1026. http://dx.doi.org/10.1037/0022-3514.58.6.1015.
11. Cohen, B. (2008). Explaining psychological statistics. Hoboken, NJ: Wiley.
12. Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. The Journal of Abnormal and Social Psychology, 65, 145-153. http://dx.doi.org/10.1037/h0045186.
13. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
14. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159. http://dx.doi.org/10.1037/0033-2909.112.1.155.
15. Couper, M. (2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64, 464-494. http://dx.doi.org/10.1086/318641.
16. Curtin, R., Presser, S., & Singer, E. (2000). The effects of response rate changes on the index of consumer sentiment. Public Opinion Quarterly, 64, 413-428. http://dx.doi.org/10.1086/318638.
17. Davison, A. C., & Hinkley, D. V. (1997). Bootstrapping methods and their application. New York, NY: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511802843.
18. de Winter, J. C., & Dodou, D. (2015). A surge of p-values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too). PeerJ, 3, e733. http://dx.doi.org/10.7717/peerj.733.
19. Dickens, C. (1859). A tale of two cities. London, UK: Champman & Hall. Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B. Nosek, B. A. (in press). Many labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology.
20. Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B. Nosek, B. A. (in press). Many labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology.
21. Efron, B., & Tibshirani, R. J. (1994). An introduction to the bootstrap. Boca Raton, FL: CRC Press.
22. Eich, E. (2014). Business not as usual. Psychological Science, 25, 3-6. http://dx.doi.org/10.1177/0956797613512465.
23. Engber, D. (2016, March, 6). Everything is crumbling. An influential psychological theory, borne out in hundreds of experiments, may have just been debunked. How can so many scientists have been so wrong? Retrieved from http://www.slate.com/articles/health_and_science/ cover_story/2016/03/ego_depletion_an_influential_theory_in_psycho logy_may_have_just_been_debunked.html.
24. Enserink, M. (2012). Final report: Stapel affair points to bigger problems in social psychology. Science Insider. Retrieved from http://news .sciencemag.org/people-events/2012/11/final-report-stapel-affair-pointsbigger- problems-social-psychology.
25. Fiedler, K., & Schwarz, N. (2016). Questionable Research Practices revisited. Social Psychological & Personality Science, 7, 45-52. http://dx.doi .org/10.1177/1948550615612150.
26. Finkel, E., Eastwick, P. W., & Reis, H. T. (in press). Replicability and other features of a high-quality science: Toward a balanced and empirical approach. Journal of Personality and Social Psychology.
27. Fraley, R. C., & Vazire, S. (2014). The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS ONE, 9, e109019. http://dx.doi.org/10.1371/journal.pone.0109019.
28. Funder, D. C. (2016, March, 6). What if Gilbert was right? Retrieved from https://funderstorms.wordpress.com/2016/03/06/what-if-gilbert-is-right/.
29. Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on "Estimating the reproducibility of psychological science". Science, 351, 1037. http://dx.doi.org/10.1126/science.aad7243.
30. Giner-Sorolla, R. (2016). Approaching a fair deal for significance and other concerns. Journal of Experimental Social Psychology, 65, 1-6. http:// dx.doi.org/10.1016/j.jesp.2016.01.010.
31. Greenwald, A. G. (1976). Within-subjects designs: To use or not to use? Psychological Bulletin, 83, 314-320. http://dx.doi.org/10.1037/0033- 2909.83.2.314.
32. Hayfield, T., & Racine, J. S. (2008). Nonparametric econometrics: The np package. Journal of Statistical Software, 27, 1-32. http://dx.doi.org/10 .18637/jss.v027.i05.
33. Hedges, L. V. (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational and Behavioral Statistics, 9, 61-85. http://dx.doi.org/10.3102/10769986009001061.
34. Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55, 19-24. http://dx.doi.org/10.1198/000313001300339897.
35. Holbrook, A., Krosnick, J. A., & Pfent, A. (2007). The causes and consequences of response rates in surveys by the news media and government contractor survey research firms. In J. M. Lepkowski, C. Tucker, J. M. Brick, E. D. De Leeuw, L. Japec, P. J. Lavrakas,. R. L. Sangster (Eds.), Advances in telephone survey methodology (pp. 499-528). New York, NY: Wiley.
36. Inbar, Y. (2016). Association between contextual dependence and replicability in psychology may be spurious. Proceedings of the National Academy of Sciences of the United States of America, 113, E4933-4934.
37. Inbar, Y., & Lammers, J. (2012). Political diversity in social and personality psychology. Perspectives on Psychological Science, 7, 496-503. http://dx.doi.org/10.1177/1745691612448792.
38. Inzlicht, M. (2016, February, 29). Reckoning with the past. Retrieved from http://michaelinzlicht.com/getting-better/2016/2/29/reckoning-with-thepast.
39. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2, e124. http://dx.doi.org/10.1371/journal.pmed .0020124.
40. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of Questionable Research Practices with incentives for truth telling. Psychological Science, 23, 524-532. http://dx.doi.org/10.1177/0956797611430953.
41. Keeter, S., Miller, C., Kohut, A., Groves, R. M., & Presser, S. (2000). Consequences of reducing nonresponse in a national telephone survey. Public Opinion Quarterly, 64, 125-148.
42. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196-217. http://dx.doi .org/10.1207/s15327957pspr0203_4.
43. Kish-Gephart, J. J., Harrison, D. A., & Treviño, L. K. (2010). Bad apples, bad cases, and bad barrels: Meta-analytic evidence about sources of unethical decisions at work. Journal of Applied Psychology, 95, 1-31. http://dx.doi.org/10.1037/a0017103.
44. Klein, R., Ratliff, K., Vianello, M., Adams, R., Jr., Bahník, S., Bernstein, M. Cemalcilar, Z. (2014). Data from investigating variation in replicability: A "Many Labs" Replication Project. Journal of Open Psychology Data, 2, e4.
45. Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107-112.
46. LeBel, E. P., & Paunonen, S. V. (2011). Sexy but often unreliable: The impact of unreliability on the replicability of experimental findings with implicit measures. Personality and Social Psychology Bulletin, 37, 570- 583. http://dx.doi.org/10.1177/0146167211400619.
47. LeBel, E. P., & Peters, K. R. (2011). Fearing the future of empirical psychology: Bem's (2011) evidence of psi as a case study of deficiencies in modal research practice. Review of General Psychology, 15, 371-379. http://dx.doi.org/10.1037/a0025172.
48. Ledgerwood, A., & Sherman, J. W. (2012). Short, sweet, and problematic? The rise of the short report in psychological science. Perspectives on Psychological Science, 7, 60-66. http://dx.doi.org/10.1177/174569 1611427304.
49. Lehmann, E. L. (1993). The Fisher, Neyman-Pearson theories of testing hypotheses: One theory or two? Journal of the American Statistical Association, 88, 1242-1249.
50. Lehrer, J. (2010, December 13). The truth wears off: Is there something wrong with the scientific method? The New Yorker. Retrieved from http://jackmccallum.com/The%20Truth%20Wears%20Off-Scientific%20.Method%20Error.pdf.
51. Maasoumi, E., & Racine, J. (2002). Entropy and predictability of stock market returns. Journal of Econometrics, 107, 291-312. http://dx.doi.org/10.1016/S0304-4076(01)00125-7.
52. Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195-244. http://dx.doi.org/10.2466/pr0.1990.66.1.195.
53. Meehl, P. E. (1997). The problem is epistemology, not statistics: Replace significance tests by confidence intervals and quantify accuracy of risky numerical predictions. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 393-425). Mahwah, NJ: LEA.
54. Merkle, D., & Edelman, M. (2002). Non-response in exit polls: A comprehensive analysis. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey non-response (pp. 243-258). New York, NY: Wiley.
55. Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2, 109-138.
56. Nosek, B. A., & Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23, 217-243. http://dx.doi.org/10.1080/1047840X.2012.692215.
57. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631. http://dx.doi.org/10.1177/1745691612459058.
58. Open Science Collaboration. (2015). Psychology. Estimating the reproducibility of psychological science. Science, 349, aac4716. http://dx.doi.org/10.1126/science.aac4716.
59. Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531-536. http://dx.doi.org/10.1177/1745691612463401.
60. Patil, P., Peng, R. D., & Leek, J. T. (2016). What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspectives on Psychological Science, 11, 539- 544. http://dx.doi.org/10.1177/1745691616646366.
61. Resnick, B. (2016, March 25). What psychology's crisis means for the future of science. Retrieved from http://www.vox.com/2016/3/14/1121 9446/psychology-replication-crisis.
62. Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331-363. http://dx.doi.org/10.1037/1089-2680.7.4.331.
63. Richman, W. L., Kiesler, S., Weisband, S., & Drasgow, F. (1999). A meta-analytic study of social desirability distortion in computeradministered questionnaires, traditional questionnaires, and interviews. Journal of Applied Psychology, 84, 754-775.
64. Robert, B. W. (2015, September 17). The new rules of research. Retrieved from http://pigee.wordpress.com/2015/09/17/the-new-rules-of-research/.
65. Rosenthal, R. (1978). Combining results of independent studies. Psychological Bulletin, 85, 185-193.
66. Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17, 551- 566. http://dx.doi.org/10.1037/a0029487.
67. Schimmack, U. (2014a). The test of insufficient variance (TIVA): A new tool for the detection of questionable research practices. Retrieved from https://replicationindex.wordpress.com/2014/12/30/the-test-ofin sufficient-variance-tiva-a-new-tool-for-the-detection-of-questionableresearch-practices.
68. Schimmack, U. (2014b). Quantifying statistical research integrity: The replicability index. Retrieved from http://www.r-index.org/uploads/3/5/ 6/7/3567479/introduction_to_the_r-index__14-12-01.pdf.
69. Schimmack, U. (2015). Distinguishing QRPs from publication bias. Retrieved from https://replicationindex.wordpress.com/2015/12/08/distinguishing-questionable-research-practices-from-publication-bias/
70. Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W. Uhlmann, E. L. (2016). The pipeline project: Prepublication independent replications of a single laboratory's research pipeline. Journal of Experimental Social Psychology, 66, 55-67. http:// dx.doi.org/10.1016/j.jesp.2015.10.001.
71. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366.
72. Simonsohn, U. (2015). Falsely reassuring: Analyses of ALL p-values. Retrieved from http://datacolada.org/41.
73. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534-547. http://dx.doi.org/10.1037/a0033242.
74. Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2015). Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a reply to Ulrich and Miller (2015). Journal of Experimental Psychology: General, 144, 1146-1152. http://dx.doi.org/10.1037/xge0000104.
75. Sohn, D. (1998). Statistical significance and replicability: Why the former does not presage the latter. Theory & Psychology, 8, 291-311. http://dx .doi.org/10.1177/0959354398083001.
76. Spellman, B. A. (2015). A short (personal) future history of Revolution 2.0. Perspectives on Psychological Science, 10, 886-899. http://dx.doi.org/ 10.1177/1745691615609918.
77. Stergiou, N., & Decker, L. M. (2011). Human movement variability, nonlinear dynamics, and pathology: Is there a connection? Human Movement Science, 30, 869-888. http://dx.doi.org/10.1016/j.humov .2011.06.002.
78. Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9, 59-71. http://dx.doi.org/10.1177/1745691613514450.
79. Tourangeau, R., Conrad, F. G., & Couper, M. P. (2013). The science of web surveys. New York, NY: Oxford University Press. http://dx.doi.org/10 .1093/acprof:oso/9780199747047.001.0001.
80. Treviño, L. K. (1990). A cultural perspective on changing and developing organizational ethics. In R. Woodman & W. Passmore (Eds.), Research in organizational change and development (Vol. 4, pp. 195-230). Greenwich, CT: JAI Press.
81. van Aert, R. C., Wicherts, J. M., & van Assen, M. A. (2016). Conducting meta-analyses based on p-values: Reservations and recommendations for applying p-uniform and p-curve. Perspectives on Psychological Science, 11, 713-729. http://dx.doi.org/10.1177/1745691616650874.
82. Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016). Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences of the United States of America, 113, 6454-6459. http://dx.doi.org/10.1073/pnas.1521897113.
83. Vazire, S. (2016). Editorial. Social Psychological & Personality Science, 7, 3-7.
84. Venables, W. N., & Ripley, B. D. (2002). Random and mixed effects. In Modern applied statistics with S (pp. 271-300). New York, NY: Springer.
85. Victor, B., & Cullen, J. B. (1988). The organizational bases of ethical work climates. Administrative Science Quarterly, 33, 101-124. http://dx.doi.org/10.2307/2392857.
86. Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4, 274-290.
87. Wagenmakers, E. J., Wetzels, R., Borsboom, D., & van der Maas, H. L. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426-432.
88. Wickham, H. (2009). ggplot2: Elegant graphics for data analysis. Berlin, Germany: Springer Science & Business Media.
89. Wilson, T. D., & Dunn, E. W. (2004). Self knowledge: Its limit, value, and potential for improvement. Annual review of Psychology, 55, 493-518.