Increasing Transparency Through a Multiverse Analysis

Perspectives on Psychological Science

Volumn 11, Issue 5, 2016, Pages 702-712

  Steegen, Sara ^a   Tuerlinckx, Francis ^a   Gelman, Andrew ^b   Vanpaemel, Wolf ^a  

^a UNIVERSITY OF LEUVEN   (Belgium)

^b Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

arbitrary choices; data processing; good research practices; multiverse analysis; selective reporting; transparency

Indexed keywords

ATTITUDE; DECISION MAKING; FEMALE; FERTILITY; HUMAN; INFORMATION SYSTEM; MARRIAGE; METHODOLOGY; POLITICS; PROCEDURES; RELIGION; STATISTICAL ANALYSIS;

ATTITUDE; CHOICE BEHAVIOR; DATA INTERPRETATION, STATISTICAL; FEMALE; FERTILITY; HUMANS; INFORMATION MANAGEMENT; MARITAL STATUS; POLITICS; RELIGION; RESEARCH DESIGN;

EID: 84989266760     PISSN: 17456916     EISSN: 17456924     Source Type: Journal    
DOI: 10.1177/1745691616658637     Document Type: Article

References (40)

1. Asendorpf J. B., Conner M., De Fruyt F., De Houwer J., Denissen J. J., Fiedler K... Wicherts J. M., (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119.
2. Bakker M., van Dijk A., Wicherts J. M., (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554.
3. Chambers C. D., (2013). Registered reports: A new publishing initiative at cortex. Cortex, 49, 609–610.
4. Cumming G., (2013). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York, NY: Routledge.
5. Durante K. M., Arsena A. R., (2015). Playing the field: The effect of fertility on women’s desire for variety. Journal of Consumer Research, 41, 1372–1391.
6. Durante K. M., Arsena A. R., Griskevicius V., (2014). Fertility can have different effects on single and nonsingle women: Reply to Harris and Mickes (2014). Psychological Science, 25, 1150–1152.
7. Durante K. M., Griskevicius V., Cantú S. M., Simpson J. A., (2014). Money, status, and the ovulatory cycle. Journal of Marketing Research, 51, 27–39.
8. Durante K. M., Griskevicius V., Hill S. E., Perilloux C., Li N. P., (2011). Ovulation, female competition, and product choice: Hormonal influences on consumer behavior. Journal of Consumer Research, 37, 921–934.
9. Durante K. M., Griskevicius V., Simpson J. A., Cantú S. M., Li N. P., (2012). Ovulation leads women to perceive sexy cads as good dads. Journal of Personality and Social Psychology, 103, 292–305.
10. Durante K. M., Rae A., Griskevicius V., (2013). The fluctuating female vote: Politics, religion, and the ovulatory cycle. Psychological Science, 24, 1007–1016.
11. Gangestad S. W., Haselton M. G., Welling L. L., Gildersleeve K., Pillsworth E. G., Burriss R. P... Puts D. A., (2016). How valid are assessments of conception probability in ovulatory cycle research? Evaluations, recommendations, and theoretical implications. Evolution & Human Behavior, 37, 85–96.
12. Geisser S., (1993). Predictive inference: An introduction. New York, NY: Chapman & Hall.
13. Gelman A., Hill J., Yajima M., (2012). Why we (usually) don’t have to worry about multiple comparisons. Journal of Research on Educational Effectiveness, 5, 189–211.
14. Gelman A., Loken E., (2014). The statistical crisis in science. American Scientist, 102, 460–465.
15. Harris C. R., (2013). Shifts in masculinity preferences across the menstrual cycle: Still not there. Sex Roles, 69, 507–515.
16. Harris C. R., Pashler H., Mickes L., (2014). Elastic analysis procedures: An incurable (but preventable) problem in the fertility effect literature. Comment on Gildersleeve, Haselton, and Fales (2014). Psychological Bulletin, 140, 1260–1264.
17. Haselton M. G., Miller G. F., (2006). Women’s fertility across the cycle increases the short-term attractiveness of creative intelligence. Human Nature, 17, 50–73.
18. John L. K., Loewenstein G., Prelec D., (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532.
19. Johnson V. E., (2013). Revised standards for statistical evidence. Proceedings of the National Academy of Sciences, USA, 110, 19313–19317.
20. Kass R. E., Raftery A. E., (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795.
21. Kruschke J. K., (2010). What to believe: Bayesian methods for data analysis. Trends in Cognitive Sciences, 14, 293–300.
22. Leamer E. E., (1983). Let’s take the con out of econometrics. The American Economic Review, 73, 31–43.
23. Leamer E. E., (1985). Sensitivity analyses would help. The American Economic Review, 75, 308–313.
24. LeBel E. P., Campbell L., Loving T. J., (in press). Benefits of open and high-powered research outweigh costs. Journal of Personality and Social Psychology.
25. MacCallum R. C., Zhang S., Preacher K. J., Rucker D. D., (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19–40.
26. MacCoun R., Perlmutter S., (2015). Blind analysis: Hide results to seek the truth. Nature, 526, 187–189.
27. Morey R. D., Chambers C. D., Etchells P. J., Harris C. R., Hoekstra R., Lakens D... Zwaan R. A., (2016). The peer reviewers’ openness initiative: Incentivizing open research practices through peer review. Royal Society Open Science, 3(1), 150547. doi:10.1098/rsos.150547
28. Morey R. D., Rouder J. N., (2011). Bayes factor approaches for testing interval null hypotheses. Psychological Methods, 16, 406–419.
29. Nosek B. A., Alter G., Banks G., Borsboom D., Bowman S., Breckler S... Yarkoni T., (2015). Promoting an open research culture: Author guidelines for journals could help to promote transparency, openness, and reproducibility. Science, 348, 1422–1425.
30. Nosek B. A., Bar-Anan Y., (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23, 217–243.
31. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716.
32. Patel C. J., Burford B., Ioannidis J. P., (2015). Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. Journal of Clinical Epidemiology, 68, 1046–1058.
33. Ramsey F., Schafer D., (2012). The statistical sleuth: A course in methods of data analysis (3rd ed.). Stamford, CT: Cengage Learning.
34. Simmons J. P., Nelson L. D., Simonsohn U., (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
35. Simmons J. P., Nelson L. D., Simonsohn U., (2012). A 21 word solution. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2160588. doi:10.2139/ssrn.2160588
36. Simonsohn U., Nelson L. D., Simmons J. P., (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534–547.
37. Thornhill R., Gangestad S. W., (1999). The scent of symmetry: A human sex pheromone that signals fitness? Evolution & Human Behavior, 20, 175–201.
38. Wagenmakers E.-J., Wetzels R., Borsboom D., van der Maas H. L., (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426–432.
39. Wagenmakers E.-J., Wetzels R., Borsboom D., van der Maas H. L., Kievit R. A., (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638.
40. Yong E., (2012). In the wake of high profile controversies, psychologists are facing up to problems with replication. Nature, 483, 298–300.