Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli

Journal of Experimental Psychology: General

Volumn 143, Issue 5, 2014, Pages 2020-2045

  Westfall, Jacob ^a   Kenny, David A ^b   Judd, Charles M ^a  

^a UNIVERSITY OF COLORADO   (United States)

^b UNIVERSITY OF CONNECTICUT   (United States)

Author keywords

Experimental design; Mixed models; Optimal design; Statistical power; Stimulus sampling

Indexed keywords

HUMAN; METHODOLOGY; SAMPLE SIZE; STATISTICAL MODEL; STATISTICS;

HUMANS; MODELS, STATISTICAL; RESEARCH DESIGN; SAMPLE SIZE; STATISTICS AS TOPIC;

EID: 84925660761     PISSN: 00963445     EISSN: None     Source Type: Journal    
DOI: 10.1037/xge0000014     Document Type: Article

References (43)

1. Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., . . . Wicherts, J. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108-119. doi:10.1002/per.1919
2. Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59, 390-412. doi:10.1016/j.jml.2007.12.005
3. Baguley, T. (2004). Understanding statistical power in the context of applied research. Applied Ergonomics, 35, 73-80. doi:10.1016/j.apergo .2004.01.002
4. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554. doi:10.1177/1745691612459060
5. Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68, 255-278. doi:10.1016/j.jml.2012.11.001
6. Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., . . . van 't Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217-224. doi:10.1016/j.jesp.2013.10.005
7. Brunswik, E. (1955). Representative design and probabilistic theory in a functional psychology. Psychological Review, 62, 193-217. doi: 10.1037/h0047470
8. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365-376. doi:10.1038/nrn3475
9. Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12, 335-359. doi:10.1016/S0022-5371(73)80014-3
10. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
11. Coleman, B. E. (1964). Generalizing to a language population. Psychological Reports, 14, 219-226. doi:10.2466/pr0.1964.14.1.219
12. Coleman, E. B. (1979). Generalization effects vs random effects: Is TL2 a source of Type 1 or Type 2 error? Journal of Verbal Learning & Verbal Behavior, 18, 243-256. doi:10.1016/S0022-5371(79)90145-2
13. Cornfield, J., & Tukey, J. W. (1956). Average values of mean squares in factorials. Annals of Mathematical Statistics, 27, 907-949. doi:10.1214/ aoms/1177728067
14. Dixon, W. J., & Massey, F. J. (1957). Introduction to statistical analysis. New York, NY: McGraw-Hill.
15. Francis, G. (2012). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19, 975- 991. doi:10.3758/s13423-012-0322-y
16. Goldstein, H., Browne, W., & Rasbash, J. (2002). Partitioning variation in multilevel models. Understanding Statistics, 1, 223-231. doi:10.1207/ S15328031US0104_02
17. Graham, J. W., Taylor, B. J., Olchowski, A. E., & Cumsille, P. E. (2006). Planned missing data designs in psychological research. Psychological Methods, 11, 323-343. doi:10.1037/1082-989X.11.4.323
18. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33, 61-83. doi:10.1017/ S0140525X0999152X
19. Hönekopp, J. (2006). Once more: Is beauty in the eye of the beholder? Relative contributions of private and shared taste to judgments of facial attractiveness. Journal of Experimental Psychology: Human Perception and Performance, 32, 199-209. doi:10.1037/0096-1523.32.2.199
20. Ioannidis, J. P. A. (2008). Why most discovered true associations are inflated. Epidemiology, 19, 640-648. doi:10.1097/EDE.0b013e31818131e7
21. Ioannidis, J. P. A. (2012). Why science is not necessarily self-correcting. Perspectives on Psychological Science, 7, 645- 654. doi:10.1177/ 1745691612464056
22. Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology, 103, 54-69. doi:10.1037/a0028347
23. Kenny, D. A. (1994). Interpersonal perception: A social relations analysis. New York, NY: Guilford Press.
24. Koole, S. L., & Lakens, D. (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7, 608-614. doi:10.1177/1745691612462586
25. Li, X., Sudarsanam, N., & Frey, D. D. (2006). Regularities in data from factorial experiments. Complexity, 11, 32-45. doi:10.1002/cplx.20123
26. McClelland, G. H. (1997). Optimal design in psychological research. Psychological Methods, 2, 3-19. doi:10.1037/1082-989X.2.1.3
27. McClelland, G. H. (2000). Increasing statistical power without increasing sample size. American Psychologist, 55, 963-964. doi:10.1037/0003-066X.55.8.963
28. Montgomery, D. C. (2013). Design and analysis of experiments (Vol. 8). New York, NY: Wiley.
29. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631. doi:10.1177/ 1745691612459058
30. Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657-660. doi:10.1177/1745691612462588
31. Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173-185. doi: 10.1037/1082-989X.2.2.173
32. Raudenbush, S. W., & Liu, X. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5, 199-213. doi:10.1037/1082-989X.5.2.199
33. Rosenthal, R., & Rubin, D. B. (1982). Comparing effect sizes of independent studies. Psychological Bulletin, 92, 500-504. doi:10.1037/0033-2909.92.2.500
34. Santa, J. L., Miller, J. J., & Shaw, M. L. (1979). Using quasi F to prevent alpha inflation due to stimulus variation. Psychological Bulletin, 86, 37-46. doi:10.1037/0033-2909.86.1.37
35. Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17, 551- 566. doi:10.1037/a0029487
36. Shoemaker, D. M. (1973). Principles and procedures of multiple matrix sampling. Cambridge, MA: Ballinger.
37. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. doi:10.1177/0956797611417632
38. Snijders, T. A. B. (2001). Sampling. In H. Goldstein & A. Leyland (Eds.), Multilevel modeling of health statistics (pp. 159-174). New York, NY: Wiley.
39. Snijders, T. A. B., & Bosker, R. J. (1993). Standard errors and sample sizes for two-level research. Journal of Educational Statistics, 18, 237-259. doi:10.3102/10769986018003237
40. Wells, G. L., & Windschitl, P. D. (1999). Stimulus sampling and social psychological experimentation. Personality and Social Psychology Bulletin, 25, 1115-1125. doi:10.1177/01461672992512005
41. Westfall, J., Judd, C. M., & Kenny, D. A. (2014). Replicating studies in which samples of participants respond to samples of stimuli.
42. Winer, B. J. (1971). Statistical principles in experimental design. New York, NY: McGraw-Hill.
43. Wu, C. J., & Hamada, M. S. (2000). Experiments: Planning, analysis, and optimization. New York, NY: Wiley.