Researchers’ Intuitions About Power in Psychological Research

Psychological Science

Volumn 27, Issue 8, 2016, Pages 1069-1077

  Bakker, Marjan ^a   Hartgerink, Chris H J ^a   Wicherts, Jelte M ^a   van der Maas, Han L J ^b  

^a TILBURG UNIVERSITY   (Netherlands)

^b UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

effect size; methodology; open data; open materials; power; sample size; survey

Indexed keywords

HUMAN; INTUITION; KNOWLEDGE; METHODOLOGY; PERSONNEL; PHYSIOLOGY; PSYCHOLOGY; QUESTIONNAIRE; RESEARCH; SAMPLE SIZE; SELF REPORT;

HUMANS; INTUITION; KNOWLEDGE; PSYCHOLOGY; RESEARCH; RESEARCH DESIGN; RESEARCH PERSONNEL; SAMPLE SIZE; SELF REPORT; SURVEYS AND QUESTIONNAIRES;

EID: 84981294421     PISSN: 09567976     EISSN: 14679280     Source Type: Journal    
DOI: 10.1177/0956797616647519     Document Type: Article

References (41)

1. C.A.AndersonJ.J.LindsayB.J.Bushman (1999). Research in the psychological laboratory. Current Directions in Psychological Science, 8, 3–9. doi:10.1111/1467-8721.00002
2. J.B.AsendorpfM.ConnerF.D.FruytJ.D.HouwerJ.J.A.DenissenK.FiedlerJ.M.. Wicherts (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119. doi:10.1002/per.1919
3. M.BakkerA.van DijkJ.M.Wicherts (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554. doi:10.1177/1745691612459060
4. M.BakkerJ.M.Wicherts (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666–678. doi:10.3758/s13428-011-0089-5
5. M.BakkerJ.M.Wicherts (2014). Outlier removal, sum scores, and the inflation of the Type I error rate in independent samples t tests: The power of alternatives and recommendations. Psychological Methods, 19, 409–427. doi:10.1037/met0000014
6. K.S.ButtonJ.P.A.IoannidisC.MokryszB.A.NosekJ.FlintE.S.J.RobinsonM.R.Munafò (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 1–12. doi:10.1038/nrn3475
7. S.Champely (2009). pwr: Basic functions for power analysis. Version 1.1.1 [Computer software]. Available from https://cran.r-project.org/web/packages/pwr/pwr.pdf
8. J.Cohen (1965). Some statistical issues in psychological research. In (Ed.), Handbook of clinical psychology (pp. 95–121). New York, NY: McGraw-Hill.
9. J.Cohen (1990). Things I have learned (thus far). American Psychologist, 45, 1304–1312. doi:10.1037/0003-066X.45.12.1304
10. N.EpleyT.Gilovich (2006). The anchoring-and-adjustment heuristic: Why the adjustments are insufficient. Psychological Science, 17, 311–318. doi:10.1111/j.1467-9280.2006.01704.x
11. D.Fanelli (2010). “Positive” results increase down the hierarchy of the sciences. PLoS ONE, 5(4), Article e10068. doi:10.1371/journal.pone.0010068
12. K.FiedlerF.KutznerJ.I.Krueger (2012). The long way from α-error control to validity proper: Problems with a short-sighted false-positive debate. Perspectives on Psychological Science, 7, 661–669. doi:10.1177/1745691612462587
13. G.Francis (2014). The frequency of excess success for articles in Psychological Science. Psychonomic Bulletin & Review, 21, 1180–1187. doi:10.3758/s13423-014-0601-x
14. G.Gigerenzer (2004). Mindless statistics. The Journal of Socio-Economics, 33, 587–606. doi:10.1016/j.socec.2004.09.033
15. D.T.GilbertG.KingS.PettigrewT.D.Wilson (2016). Comment on “Estimating the reproducibility of psychological science.”Science, 351, 1037. doi:10.1126/science.aad7243
16. A.G.Greenwald (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20. doi:10.1037/h0076157
17. C.H.J.HartgerinkJ.M.WichertsM.A.L.M.van Assen. (2015). Too good to be false: Non-significant results revisited [Project files]. Retrieved from osf.io/qpfnw
18. L.K.JohnG.LoewensteinD.Prelec (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532. doi:10.1177/0956797611430953
19. M.W.LipseyD.B.Wilson (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181–1209. doi:10.1037/0003-066X.48.12.1181
20. J.M.MarszalekC.BarberJ.KohlhartC.B.Holmes (2011). Sample size in psychological research over the past 30 years. Perceptual and Motor Skills, 112, 331–348. doi:10.2466/03.11.pms.112.2.331-348
21. S.E.Maxwell (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147–163. doi:10.1037/1082-989X.9.2.147
22. S.E.MaxwellM.Y.LauG.S.Howard (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?American Psychologist, 70, 487–498. doi:10.1037/a0039400
23. G.J.MeyerS.E.FinnL.D.EydeG.G.KayK.L.MorelandR.R.DiesG.. Reed (2001). Psychological testing and psychological assessment: A review of evidence and issues. American Psychologist, 56, 128–156. doi:10.1037/0003-066X.56.2.128
24. R.S.Nickerson (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301. doi:10.1037//1082-989x.5.2.241
25. M.B.NuijtenC.H.J.HartgerinkM.A.L.M.van AssenS.EpskampJ.M.Wicherts (2015). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods. Advance online publication. doi:10.3758/s13428-015-0664-2
26. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, 943. doi:10.1126/science.aac4716
27. M.PeruginiM.GallucciG.Costantini (2014). Safeguard power as a protection against imprecise power estimates. Perspectives on Psychological Science, 9, 319–332. doi:10.1177/1745691614528519
28. F.RichardC.F.BondJr.J.J.Stokes-Zoota (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331–363. doi:10.1037/1089-2680.7.4.331
29. R.Rosenthal (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641. doi:10.1037/0033-2909.86.3.638
30. J.N.RouderP.L.SpeckmanD.SunR.D.Morey (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237. doi:10.3758/PBR.16.2.225
31. P.SedlmeierG.Gigerenzer (1989). Do studies of statistical power have an effect on the power of studies?Psychological Bulletin, 105, 309–316. doi:10.1037//0033-2909.105.2.309
32. J.P.SimmonsL.D.NelsonU.Simonsohn (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632
33. T.D.SterlingW.L.RosenbaumJ.J.Weinkam (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108–112. doi:10.1080/00031305.1995.10476125
34. R.P.TettJ.P.MeyerN.J.Roese (1994). Applications of meta-analysis: 1987-1992. In (Eds.), International review of industrial and organizational psychology (Vol. 9, pp. 71–112). Chichester, England: John Wiley & Sons.
35. A.TverskyD.Kahneman (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105–110. doi:10.1037/h0031322
36. E.-J.WagenmakersR.WetzelsD.BorsboomH.L.J.van der Maas (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426–432. doi:10.1037/a0022790
37. E.-J.WagenmakersR.WetzelsD.BorsboomH.L.J.van der MaasR.A.Kievit (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. doi:10.1177/1745691612463078
38. B.L.Welch (1938). The significance of the difference between two means when the population variances are unequal. Biometrika, 29, 350–362.
39. R.WetzelsD.MatzkeM.D.LeeJ.N.RouderG.J.IversonE.-J.Wagenmakers (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science, 6, 291–298. doi:10.1177/1745691611406923
40. R.R.Wilcox (2012). Modern statistics for the social and behavioral sciences: A practical introduction. Boca Raton, FL: CRC Press.
41. K.K.Yuen (1974). The two-sample trimmed t for unequal population variances. Biometrika, 61, 165–170. doi:10.1093/biomet/61.1.165