Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty

Psychological Science

Volumn 28, Issue 11, 2017, Pages 1547-1562

  Anderson, Samantha F ^a   Kelley, Ken ^a   Maxwell, Scott E ^a  

^a UNIVERSITY OF NOTRE DAME   (United States)

Author keywords

effect size; methodology; publication bias; sample size; statistical power

Indexed keywords

HUMAN; PUBLISHING; SAMPLE SIZE; STATISTICAL ANALYSIS; UNCERTAINTY;

DATA INTERPRETATION, STATISTICAL; HUMANS; PUBLICATION BIAS; SAMPLE SIZE; UNCERTAINTY;

EID: 85033228473     PISSN: 09567976     EISSN: 14679280     Source Type: Journal    
DOI: 10.1177/0956797617723724     Document Type: Article

References (63)

1. Ackerman J. M., Maner J. K., Carpenter S. K., (2016). Going all in: Unfavorable sex ratios attenuate choice diversification. Psychological Science, 27, 799–809. doi:10.1177/0956797616636631
2. Anderson S. F., Kelley K., (2017). BUCSS: Bias and Uncertainty Corrected Sample Size (Version 0.0.2) [R package]. Retrieved from https://cran.r-project.org/web/packages/BUCSS/index.html
3. Anderson S. F., Maxwell S. E., (2016). There’s more than one way to conduct a replication study: Beyond statistical significance. Psychological Methods, 21, 1–12. doi:10.1037/met0000051
4. Anderson S. F., Maxwell S. E., (2017). Addressing the “replication crisis”: Using original studies to design replication studies with appropriate statistical power. Multivariate Behavioral Research, 52, 305–324. doi:10.1080/00273171.2017.1289361
5. Asendorpf J. B., Conner M., de Fruyt F., de Houwer J., Denissen J. J. A., Fiedler K.,.. Wicherts J. M., (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–199. doi:10.1002/per1919
6. Association for Psychological Science. (2016). Submission guidelines. Retrieved from http://www.psychologicalscience.org/publications/psychological_science/ps-submissions#PG
7. Bakker M., Hartgerink C. H. J., Wicherts J. M., van der Maas H. L. J., (2016). Researchers’ intuitions about power in psychological research. Psychological Science, 27, 1069–1077. doi:10.1177/0956797616647519
8. Bakker M., van Dijk A., Wicherts J. M., (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554. doi:10.1177/1745691612459060
9. Baror S., Bar M., (2016). Associative activation and its relation to exploration and exploitation in the brain. Psychological Science, 27, 775–789. doi:10.1177/0956797616634487
10. Brand A., Bradley M. T., Best L. A., Stoica G., (2008). Accuracy of effect size estimates from published psychological research. Perceptual and Motor Skills, 106, 645–649. doi:10.2466/pms.106.2.645-649
11. Button K. S., Ioannidis J. P. A., Mokrysz C., Nosek B. A., Flint J., Robinson E. S. J., Munaf M. R., (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 1–12. doi:10.1038/nrn3475
12. Chattopadhyay B., Kelley K., (2017). Estimating the standardized mean difference with minimum risk: Maximizing accuracy and minimizing cost with sequential estimation. Psychological Methods, 22, 94–113.
13. Chow S.-C., Shao J., Wang H., (2003). Sample size calculations in clinical research. Boca Raton, FL: Taylor & Francis.
14. Cohen J., (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145–153.
15. Cohen J., (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
16. Cohen J., (1994). The earth is round (p <.05). American Psychologist, 49, 997–1003.
17. Cook T. D., DeMets D. L., (2008). Introduction to statistical methods for clinical trials. Boca Raton, FL: Chapman & Hall/CRC.
18. Dallow N., Fina P., (2011). The perils with the misuse of predictive power. Pharmaceutical Statistics, 10, 311–317. doi:10.1002/pst.467
19. Faul F., Erdfelder E., Lang A. G., Buchner A., (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.
20. Fraley R. C., Vazire S., (2014). The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS ONE, 9(10), Article e109019. doi:10.1371/journal.pone.0109019
21. Gelman A., Carlin J., (2014). Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science, 9, 641–651. doi:10.1177/1745691614551642
22. Green S. B., (1991). How many subjects does it take to do a regression analysis? Multivariate Behavioral Research, 26, 499–510.
23. Hedges L., (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational and Behavioral Statistics, 9, 61–85.
24. Ioannidis J. P. A., (2005). Why most published research findings are false. PLoS Medicine, 2(8), Article e124. doi:10.1371/journal.pmed.0020124
25. John L. K., Loewenstein G., Prelec D., (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532. doi:10.1177/0956797611430953
26. Kelley K., Darku F. B., Chattopadhyay B., (2017). Accuracy in parameter estimation for a general class of effect sizes: A sequential approach. Psychological Methods. Advance online publication. doi:10.1037/met0000127
27. Kelley K., Maxwell S. E., (2003). Sample size for multiple regression: Obtaining regression coefficients that are accurate, not simply significant. Psychological Methods, 8, 305–321. doi:10.1037/1082-989X.8.3.305
28. Kelley K., Preacher K. J., (2012). On effect size. Psychological Methods, 17, 137–152. doi:10.1037/a0028086
29. Kelley K., Rausch J., (2006). Sample size planning for the standardized mean difference: Accuracy in parameter estimation via narrow confidence intervals. Psychological Methods, 11, 363–385. doi:10.1037/1082-989X.11.4.363
30. Kraemer H. C., Blasey C. M., (2016). How many subjects? Statistical power analysis in research (2nd ed.). Thousand Oaks, CA: Sage.
31. Lai T. L., Lavori P. W., Shih M. C., (2012). Adaptive trial designs. Annual Review of Pharmacology and Toxicology, 52, 101–110. doi:10.1146/annurev-pharmtox-010611-134504
32. Lakens D., (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, Article 863. doi:10.3389/fpsyg.2013.00863
33. Lakens D., (2014). Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44, 701–710. doi:10.1002/ejsp.2023
34. Lakens D., (2015a). On the challenges of drawing conclusions from p-values just below 0.05. PeerJ, 3, e1142. doi:10.7717/peerj.1142
35. Lakens D., (2015b). What p-hacking really looks like: A comment on Masicampo and Lalande (2012). The Quarterly Journal of Experimental Psychology, 68, 829–832. doi:10.1080/17470218.2014.982664
36. Lakens D., (in press). Equivalence tests: A practical primer for t-tests, correlations, and meta-analyses. Social Psychological & Personality Science.
37. Lane D. M., Dunlap W. P., (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107–112.
38. Leon A. C., Davis L. L., Kraemer H. C., (2011). The role and interpretation of pilot studies in clinical research. Journal of Psychiatric Research, 45, 626–629. doi:10.1016/j.jpsychires.2010.10.008
39. Lipsey M. W., (1990). Design sensitivity: Statistical power for experimental research. Newbury Park, CA: Sage.
40. Marszalek J. M., Barber C., Kohlhart J., Holmes C. B., (2011). Sample size in psychological research over the past 30 years. Perceptual and Motor Skills, 112, 331–348. doi:10.2466/03.11.pms112.2.331-348
41. Maxwell S. E., (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147–163. doi:10.1037/1082-989X.9.2.147
42. Maxwell S. E., Delaney H. D., (2004). Designing experiments and analyzing data: A model comparison perspective (2nd ed.). Mahwah, NJ: Erlbaum.
43. Maxwell S. E., Delaney H. D., Kelley K., (2018). Designing experiments and analyzing data: A model comparison perspective (3rd ed.). New York, NY: Taylor & Francis.
44. Maxwell S. E., Kelley K., (2011). Ethics and sample size planning. In Panter A., Sterba S., (Eds.), Handbook of ethics in quantitative methodology (pp. 159–184). New York, NY: Taylor & Francis.
45. Maxwell S. E., Kelley K., Rausch J., (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537–563. doi:10.1146/annurev.psych.59.103006.093735
46. Maxwell S. E., Lau M. Y., Howard G. S., (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70, 487–498. doi:10.1037/a0039400
47. Muller K. E., Fetterman B. A., (2002). Regression and ANOVA: An integrated approach using SAS software. Cary, NC: SAS Institute.
48. O’Brien R. G., Muller K. E., (1993). Unified power analysis for t-tests through multivariate hypotheses. In Edwards L. K., (Ed.), Applied analysis of variance in behavioral science (pp. 297–344). New York, NY: Marcel Dekker.
49. Olejnik S., Algina A., (2000). Measures of effect sizes for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241–286. doi:10.1006/ceps.2000.1040
50. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, 943. doi:10.1126/science.aac4716
51. Pashler H., Wagenmakers E.-J., (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528–530. doi:10.1177/1745691612465253
52. Peace K. E., Chen D.-G., (2011). Clinical trial methodology. Boca Raton, FL: Taylor & Francis.
53. Perugini M., Gallucci M., Constantini G., (2014). Safeguard power as a protection against imprecise power estimates. Perspectives on Psychological Science, 9, 319–332. doi:10.1177/1745691614528519
54. Reinhart R. M. G., McClenahan L. J., Woodman G. F., (2016). Attention’s accelerator. Psychological Science, 27, 790–798. doi:10.1177/0956797616636416
55. Schönbrodt F. D., Wagenmakers E.-J., Zehetleitner M., Perugini M., (2015). Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences. Psychological Methods. Advance online publication. doi:10.1037/met0000061
56. Sedlmeier P., Gigerenzer G., (1989). Do studies on statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309–316.
57. Simmons J. P., Nelson L. D., Simonsohn U., (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632
58. Simonsohn U., (2015). Small telescopes: Detectability and the evaluation of replication results. Psychological Science, 26, 559–569. doi:10.1177/0956797614567341
59. Skidmore S. T., Thompson B., (2013). Bias and precision of some classical ANOVA effect sizes when assumptions are violated. Behavior Research Methods, 45, 536–546.
60. Smaldino P. E., McElreath R., (2016). The natural selection of bad science. Royal Society Open Science, 3, 160384. doi:10.1098/rsos.160384
61. Taylor D. J., Muller K. E., (1996). Bias in linear model power and sample size calculations due to estimating noncentrality. Communications in Statistics: Theory and Methods, 25, 1–14. doi:10.1080/03610929608831787
62. Wetzels R., Matzke D., Lee M. D., Rouder J. N., Iverson G. J., Wagenmakers E.-J., (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science, 6, 291–298. doi:10.1177/1745691611406923
63. Yuan K. H., Maxwell S. E., (2005). On the post-hoc power in testing mean differences. Journal of Educational and Behavioral Statistics, 30, 141–167.