A Unified Framework to Quantify the Credibility of Scientific Findings

Advances in Methods and Practices in Psychological Science

Volumn 1, Issue 3, 2018, Pages 389-402

  LeBel, Etienne P ^a   McCarthy, Randy J ^b   Earp, Brian D ^c   Elson, Malte ^d   Vanpaemel, Wolf ^e  

^a UNIVERSITY OF WESTERN ONTARIO   (Canada)

^b NORTHERN ILLINOIS UNIVERSITY   (United States)

^c YALE UNIVERSITY   (United States)

^d RUHR UNIVERSITY BOCHUM   (Germany)

^e UNIVERSITY OF LEUVEN   (Belgium)

Author keywords

analytic reproducibility; analytic robustness; open science; replicability; research credibility; transparency

Indexed keywords

EID: 85129741943     PISSN: 25152459     EISSN: 25152467     Source Type: Journal    
DOI: 10.1177/2515245918787489     Document Type: Article

References (37)

1. Begg C. Cho M. Eastwood S. Horton R. Moher D. Olkin I... Stroup D. F. (1996). Improving the quality of reporting of randomized controlled trials: The CONSORT statement. Journal of the American Medical Association, 276, 637–639.
2. Buss D. M. Shackelford T. K. Kirkpatrick L. A. Choe J. C. Lim H. K. Hasegawa M... Bennett K. (1999). Jealousy and the nature of beliefs about infidelity: Tests of competing hypotheses about sex differences in the United States, Korea, and Japan. Personal Relationships, 6, 125–150.
3. Chambers C. D. (2013). Registered Reports: A new publishing initiative at Cortex. Cortex, 49, 609–610.
4. Cooper H. Hedges L. V. Valentine J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis. New York, NY: Russell Sage Foundation.
5. Doyen S. Klein O. Simons D. J. Cleeremans A. (2014). On the other side of the mirror: Priming in cognitive and social psychology. Social Cognition, 32, 12–32.
6. Earp B. D. (in press). Falsification: How does it relate to reproducibility? In Morin J.-F. Olsson C. Atikcan E. O. (Eds.), Key concepts in research methods. Abingdon, England: Routledge.
7. Earp B. D. Trafimow D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology, 6, Article 621. doi:10.3389/fpsyg.2015.00621
8. Eich E. (2014). Business not as usual. Psychological Science, 25, 3–6.
9. Feynman R. P. (1974). Cargo cult science. Engineering and Science, 37(7), 10–13.
10. Hardwicke T. E. Mathur M. B. MacDonald K. Nilsonne G. Banks G. C. Kidwell M. C... Frank M. C. (2018). Data availability, reusability, and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal Cognition. Retrieved from https://osf.io/preprints/bitss/39cfb/
11. Hendrick C. (1991). Replication, strict replications, and conceptual replications: Are they important? In Neuliep J. W. (Ed.), Replication research in the social sciences (pp. 41–49). Newbury Park, CA: Sage.
12. Higgins J. P. T. Lasserson T. Chandler J. Tovey D. Churchill R. (2018). Methodological Expectations of Cochrane Intervention Reviews (MECIR): Standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates. Retrieved from http://community.cochrane.org/sites/default/files/uploads/MECIR%20PRINTED%20BOOKLET%20FINAL%20v1.02.pdf
13. IJzerman H. Blanken I. Brandt M. J. Oerlemans J. M. Van den Hoogenhof M. M. W. Franken S. J. M. Oerlemans M. W. G. (2014). Sex differences in distress from infidelity in early adulthood and in later life: A replication and meta-analysis of Shackelford et al. (2004). Social Psychology, 45, 202–208.
14. Kashy D. A. Donnellan M. B. Ackerman R. A. Russell D. W. (2009). Reporting and interpreting research in PSPB: Practices, principles, and pragmatics. Personality and Social Psychology Bulletin, 35, 1131–1142.
15. Kidwell M. C. Lazarevic’ L. B. Baranski E. Hardwicke T. E. Piechowski S. Falkenberg L.-S... Nosek B. A. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLOS Biology, 14(5), Article e1002456. doi:10.1371/journal.pbio.1002456
16. Lakatos I. (1970). Falsification and the methodology of scientific research programmes. In Lakatos I. Musgrave A. (Eds.), Criticism and the growth of knowledge (pp. 91–196). Cambridge, England: Cambridge University Press.
17. LeBel E. P. Berker D. Campbell L. Loving T. J. (2017). Falsifiability is not optional. Journal of Personality and Social Psychology, 113, 254–261.
18. LeBel E. P. Borsboom D. Giner-Sorolla R. Hasselman F. Peters K. R. Ratliff K. A. Smith C. T. (2013). PsychDisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8, 424–432.
19. LeBel E. P. John L. (2017). Psychological and institutional obstacles toward more transparent reporting of psychological science. In Lilienfeld S. O. Waldman I. D. (Eds.), Psychological science under scrutiny: Recent challenges and proposed solutions (pp. 73–84). New York, NY: John Wiley & Sons.
20. LeBel E. P. Peters K. R. (2011). Fearing the future of empirical psychology: Bem’s (2011) evidence of psi as a case study of deficiencies in modal research practice. Review of General Psychology, 15, 371–379.
21. LeBel E. P. Vanpaemel W. Cheung I. Campbell L. (2018). Evaluating replications with nuance. Retrieved from https://osf.io/paxyn/
22. Meehl P. E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103–115. doi:10.1086/288135
23. Meehl P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834. doi:10.1037/0022-006X.46.4.806
24. Moery E. Calin-Jageman R. J. (2016). Direct and conceptual replications of Eskine (2013): Organic food exposure has little to no effect on moral judgments and prosocial behavior. Social Psychological & Personality Science, 7, 312–319.
25. Nosek B. A. Ebersole C. R. DeHaven A. C. Mellor D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, USA, 115, 2600–2606.
26. Pashler H. Harris C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531–536.
27. Popper K. R. (1959). The logic of scientific discovery. London, England: Hutchinson.
28. Rosenthal R. (1991). Replication in behavioral research. In Neuliep J. W. (Ed.), Replication research in the social sciences (pp. 1–39). Newbury Park, CA: Sage.
29. Schmidt S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13, 90–100.
30. Schulz K. F. Altman D. G. Moher D. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. BMC Medicine, 8, Article 18. doi:10.1186/1741-7015-8-18
31. Shackelford T. K. Voracek M. Schmitt D. P. Buss D. M. Weekes-Shackelford V. A. Michalski R. L. (2004). Romantic jealousy in early adulthood and in later life. Human Nature, 15, 283–300.
32. Simmons J. Nelson L. Simonsohn U. (2012). A 21 word solution. Dialogue: The Official Newsletter of the Society for Personality and Social Psychology, 26(2), 4–7.
33. Simmons J. P. Nelson L. D. Simonsohn U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
34. Simonsohn U. Simmons J. P. Nelson L. D. (2015). Specification curve: Descriptive and inferential statistics on all reasonable specifications. Retrieved from http://ssrn.com/abstract=2694998
35. Steegen S. Tuerlinckx F. Gelman A. Vanpaemel W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11, 702–712.
36. Vandenbroucke J. P. von Elm E. Altman D. G. Gøtzsche P. C. Mulrow C. D. Pocock S. J... STROBE Initiative. (2014). Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and elaboration. International Journal of Surgery, 12, 1500–1524.
37. Zwaan R. A. Etz A. Lucas R. E. Donnellan M. B. (2017). Making replication mainstream. Behavioral & Brain Sci-ences. Advance online publication. doi:10.1017/S0140525X17001972