Research Practices That Can Prevent an Inflation of False-Positive Rates

Personality and Social Psychology Review

Volumn 18, Issue 2, 2014, Pages 107-118

  Murayama, Kou ^a,b   Pekrun, Reinhard ^c   Fiedler, Klaus ^d  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b JAPAN SOCIETY FOR THE PROMOTION OF SCIENCE   (Japan)

^c UNIVERSITY OF MUNICH   (Germany)

^d UNIVERSITY OF HEIDELBERG   (Germany)

Author keywords

false positives; optimal stopping; replication; Type 1 error

Indexed keywords

BEHAVIORAL RESEARCH; HUMAN; INFORMATION PROCESSING; LABORATORY DIAGNOSIS; PROCEDURES; PSYCHOLOGICAL MODEL; PUBLISHING; REPRODUCIBILITY; SOCIAL PSYCHOLOGY; STANDARDS; STATISTICAL ANALYSIS;

BEHAVIORAL RESEARCH; DATA COLLECTION; DATA INTERPRETATION, STATISTICAL; FALSE POSITIVE REACTIONS; HUMANS; MODELS, PSYCHOLOGICAL; PSYCHOLOGY, SOCIAL; PUBLICATION BIAS; REPRODUCIBILITY OF RESULTS;

EID: 84898688933     PISSN: 10888683     EISSN: None     Source Type: Journal    
DOI: 10.1177/1088868313496330     Document Type: Article

References (65)

1. Abelson R. P.,Prentice D. A.Contrast tests of interaction hypothesis.Psychological Methods. 1997;2:315-328.
2. Bakker M.,van Dijk A.,Wicherts J. M.The rules of the game called psychological science.Perspectives on Psychological Science. 2012;7:543-554.
3. Bakker M.,Wicherts J. M.The (mis)reporting of statistical results in psychology journals.Behavior Research Methods. 2011;43:666-678.
4. Bertamini M.,Munafo M. R.Bite-size science and its undesired side effects.Perspectives on Psychological Science. 2012;7:67-71.
5. Bones A. K.,Johnson N. R.Measuring the immeasurable, or, "could Abraham Lincoln take the implicit association test?".Perspectives on Psychological Science. 2007;2:406-411.
6. Bradley J. V.Robustness?.British Journal of Mathematical & Statistical Psychology. 1978;31:144-152.
7. Button K. S.,Ionnidis P. A.,Mokrysz C.,Nosek B. A.,Flint J.,Robinson E. S. J.,Munafo M. R.Power failure: Why small sample size undermines the reliability of neuroscience.Nature Reviews Neuroscience. 2013;14:365-376.
8. Carey B.Fraud case seen as a red flag for psychology research.The New York Times. 2011;:A3.
9. Carver C. S.Editorial.Journal of Personality and Social Psychology. 2004;86:95.
10. Cohen J.Statistical power analysis for the behavioral sciences. 1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.Hillsdale, NJ: Lawrence Erlbaum; 1988:.
11. Cook T. D.P value adjustment in sequential clinical trials.Biometrics. 2002;58:1005-1011.
12. Diener E.Editorial.Journal of Personality and Social Psychology. 1998;74:5-6.
13. Dunnett C. W.A multiple comparison procedure for comparing several treatments with a control.Journal of the American Statistical Association. 1955;50:1096-1121.
14. Fanelli D.Negative results are disappearing from most disciplines and countries.Scientometrics. 2012;90:891-904.
15. Fiedler K.Voodoo correlations are everywhere-Not only in neuroscience.Perspectives on Psychological Science. 2011;6:163-171.
16. Fiedler K.,Kutzner F.,Krueger J. I.The long way from alpha-control to validity proper: Problems with a shortsighted false-positive debate.Perspectives on Psychological Science. 2012;7:661-669.
17. Francis G.Evidence that publication bias contaminated studies relating social class and unethical behavior.Proceedings of the National Academy of Sciences of the United States of America. 2012a;:.
18. Francis G.Publication bias in "Red, rank, and romance in women viewing men" by Elliot et al.Journal of Experimental Psychology: General. 2012b;142:292-296.
19. Francis G.The same old new look: Publication bias in a study of wishful seeing.I-Perception. 2012c;3:176-178.
20. Francis G.Too good to be true: Publication bias in two prominent studies from experimental psychology.Psychonomic Bulletin & Review. 2012d;19:151-156.
21. Garcia-Perez M. A.Statistical conclusion validity: Some common threats and simple remedies.Frontiers in Psychology. 2012;3:325.
22. Garner W. R.,Hake H. W.,Eriksen C. W.Operationalism and the concept of perception.Psychological Review. 1956;63:149-159.
23. Higgins E. T.Value from regulatory fit.Current Directions in Psychological Science. 2005;14:209-213.
24. Hunt D. E.Person-environment interaction: A challenge found wanting before it was tried.Review of Educational Research. 1975;45:209-230.
25. Ioannidis J. P. A.Why most published research findings are false.PLoS Medicine. 2005;2:e124.
26. Ioannidis J. P. A.,Trikalinos T. A.An exploratory test for an excess of significant findings.Clinical Trials. 2007;4:245-253.
27. Jeffreys H.Theory of probability. 1961). Theory of probability (3rd ed.). Oxford, UK: Oxford University Press.Oxford, UK: Oxford University Press; 1961:.
28. Jennison C.,Turnbull B. W.Statistical approaches to interim monitoring of medical trials: A review and commentary.Statistical Science. 1990;5:299-317.
29. John L. K.,Loewenstein G.,Prelec D.Measuring the prevalence of questionable research practices with incentives for truth telling.Psychological Science. 2012;:.
30. Kane M. T.An argument-based approach to validity.Psychological Bulletin. 1992;112:527-535.
31. Kerr N. L.HARKing: Hypothesizing after the results are known.Personality and Social Psychology Review. 1998;2:196-217.
32. Lai T. L.,Lavori P. W.,Shih M. C.Annual review of pharmacology and toxicology. Insel P. A.Amara S. G.Blaschke T. F., ed. Palo Alto, CA: Annual Reviews; 2012:101-110.
33. Ledgerwood A.,Sherman J. W.Short, sweet, and problematic? The rise of the short report in psychological science.Perspectives on Psychological Science. 2012;7:60-66.
34. Lee P. M.Bayesian statistics. New York, NY: John Wiley; 1997:.
35. Legault L.,Inzlicht M.Self-determination, self-regulation, and the brain: Autonomy improves performance by enhancing neuroaffective responsiveness to self-regulation failure.Journal of Personality and Social Psychology. 2013;105:123-138.
36. Lieberman M. D.,Cunningham W. A.Type I and Type II error concerns in fMRI research: Re-balancing the scale.Social Cognitive and Affective Neuroscience. 2009;4:423-428.
37. Lilienfeld S. O.Public skepticism of psychology: Why many people perceive the study of human behavior as unscientific.American Psychologist. 2012;67:111-129.
38. Maxwell S. E.The persistence of underpowered studies in psychological research: Causes, consequences, and remedies.Psychological Methods. 2004;9:147-163.
39. McCarroll D.,Crays N.,Dunlap W. P.Sequential ANOVAs and Type I error rates.Educational and Psychological Measurement. 1992;52:387-393.
40. Nosek B. A.,Spies J. R.,Motyl M.Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability.Perspectives on Psychological Science. 2012;7:615-631.
41. Okada T.,Shimokido T.Designing for science: Implications from everyday, classroom, and professional settings. Crowley K.Schunn C. D.Okada T., ed. Mahwah, NJ: Lawrence Erlbaum; 2001:445-464.
42. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science.Perspectives on Psychological Science. 2012;7:657-660.
43. Pashler H.,Harris C. R.Is the replicability crisis overblown? Three arguments examined.Perspectives on Psychological Science. 2012;7:531-536.
44. Pashler H.,Wagenmakers E. J.Editors' introduction to the special section on replicability in psychological science: A crisis of confidence?.Perspectives on Psychological Science. 2012;7:528-530.
45. Pawitan Y.In all likelihood: Statistical modelling and inference using likelihood. Oxford, UK: Clarendon Press; 2001:.
46. Platt J. R.Strong inference.Science. 1964;146:347-353.
47. R Development Core Team. (2012). R: A language and environment for statistical computing (ISBN 3-900051-07-0). Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/.
48. Roediger H. L.Psychology's woes and a practical cure.APS Observer. 2012;25 (2): 27-29.
49. Rosenthal R.,Rosnow R. L.Contrast analysis: Focused comparisons in the analysis of variance. New York, NY: Cambridge University Press; 1985:.
50. Schimmack U.The ironic effect of significant results on the credibility of multiple study articles.Psychological Methods. 2012;17:551-566.
51. Schmidt S.Shall we really do it again? The powerful concept of replication is neglected in the social sciences.Review of General Psychology. 2009;13:90-100.
52. Sedlmeier P.,Gigerenzer G.Do studies of statistical power have an effect on the power of studies?.Psychological Bulletin. 1989;105:309-316.
53. Sherman R. C.,Buddie A. M.,Dragan K. L.,End C. M.,Finney L. J.Twenty years of PSPB: Trends in content, design, and analysis.Personality and Social Psychology Bulletin. 1999;25:177-187.
54. Simmons J. P.,Nelson L. D.,Simonsohn U.False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.Psychological Science. 2011;22:1359-1366.
55. SimmonsJ. P.NelsonL. D.SimonsohnU. (2012). A 21 word solution. SSRN eLibrary. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2160588.
56. Spellman B. A.Introduction to the special section: Data, data, everywhere. Especially in my file drawer.Perspectives on Psychological Science. 2012a;7:58-59.
57. Spellman B. A.Introduction to the special section on research practices.Perspectives on Psychological Science. 2012b;7:655-666.
58. Spencer S. J.,Steele C. M.,Quinn D. M.Stereotype threat and women's math performance.Journal of Experimental Social Psychology. 1999;35:4-28.
59. Steele C. M.,Aronson J.Stereotype threat and the intellectual test performance of African Americans.Journal of Personality and Social Psychology. 1995;69:797-811.
60. Strube M. J.SNOOP: A program for demonstrating the consequences of premature and repeated null hypothesis testing.Behavior Research Methods. 2006;38:24-27.
61. Vul E.,Harris C.,Winkielman P.,Pashler H.Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition.Perspectives on Psychological Science. 2009;4:274-290.
62. Wagenmakers E.-J.A practical solution to the pervasive problems of p values.Psychonomic Bulletin & Review. 2007;14:779-804.
63. Wagenmakers E.-J.,Wetzels R.,Borsboom D.,van der Maas H. L. J.,Kievit R. A.An agenda for purely confirmatory research.Perspectives on Psychological Science. 2012;7:632-638.
64. Young E.Bad copy.Nature. 2012;485:298-300.
65. Young N. S.,Ioannidis J. P. A.,Al-Ubaydli O.Why current publication practices may distort science.PLoS Medicine. 2008;5:e201.