Thinking About Data, Research Methods, and Statistical Analyses: Commentary on Sijtsma’s (2014) “Playing with Data”

Psychometrika

Volumn 81, Issue 1, 2016, Pages 16-26

  Waldman, Irwin D ^a   Lilienfeld, Scott O ^a  

^a EMORY UNIVERSITY   (United States)

Author keywords

minimizing false positives; questionable research practices; replicability

Indexed keywords

HUMAN; METHODOLOGY; PSYCHOMETRY;

HUMANS; PSYCHOMETRICS; RESEARCH DESIGN;

EID: 84959253970     PISSN: 00333123     EISSN: None     Source Type: Journal    
DOI: 10.1007/s11336-015-9447-z     Document Type: Article

References (49)

1. Abelson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Erlbaum.
2. Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J., Fiedler, K., et al. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119.
3. Bertamini, M., & Munafò, M. R. (2012). Bite-size science and its undesired side effects. Perspectives on Psychological Science, 7, 67–71.
4. Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376.
5. Cohen, J. (1962). The statistical power of abnormal-social psycological research: A review. Journal of Abnormal and Social Psycology, 65, 145–153.
6. Cohen, J. (1969). Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Erlbaum.
7. Cohen, J. (1994). The earth is round (p<.05). American Psychologist, 49, 997–1003.
8. Collins, A. L., & Sullivan, P. F. (2013). Genome-wide association studies in psychiatry: What have we learned. British Journal of Psychiatry, 202, 1–4.
9. Cumming, G. (2014). The new statistics why and how. Psychological Science, 25, 7–29.
10. Fidler, F., Thomason, N., Cumming, G., Finch, S., & Leeman, J. (2004). Editors can lead researchers to confidenceintervals, but can’t make them think: Statistical reform lessons from medicine. Psychological Science, 15, 119–126.
11. Fisher, R. A. (1925). Statistical methods for research workers. London: Oliver & Boyd.
12. Frank, M. C., & Saxe, R. (2012). Teaching replication. Perspectives on Psychological Science, 7, 600–604.
13. Gigerenzer, G. (1998). We need statistical thinking, not statistical rituals. Behavioral and Brain Sciences, 21, 199–200.
14. Giner-Sorolla, R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562–571.
15. Gray, K., & Wegner, D. M. (2013). Six guidelines for interesting research. Perspectives on Psychological Science, 8, 549–553.
16. Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests? Hillsdale, NJ: Erlbaum.
17. Harzing, A. W. (2011). The publish or perish book: A guide to the software. London: Tarma Software Research.
18. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.
19. Hersh, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 16569–16572.
20. Huff, D. (1954). How to Lie with Statistics. New York: W.W. Norton & Company.
21. Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
22. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi:10.1371/journal.pmed.0020124.
23. Ioannidis, J., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends in Cognitive Sciences, 18, 235–241.
24. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532.
25. Kahneman, D. (2011). Thinking: Fast and slow. New York: Farrar, Straus and Giroux.
26. Kuhn, T. S. (1970). The Structure of Scientific Revolutions. Chicago, IL: University of Chicago Press.
27. Lehrer, J. (2010). The truth wears off. The New Yorker, 52–57.
28. Lykken, D. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 1151–1159.
29. Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834.
30. Neyman, J., & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference. Biomettika, 20A, 175–240, 263–294.
31. Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301.
32. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.
33. Nosek, B. A., & Lakens, D. (2014). Registered reports. Social Psychology, 45, 137–141.
34. Oakes, M. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York: Wiley.
35. Pashler, H., & Wagenmakers, E. J. (2012). Editors’ Introduction to the Special Section on replicability in psychological science. A crisis of confidence? Perspectives on Psychological Science, 7, 528–530.
36. Ripke, S., et al. (2013). Genome-wide association analysis identifies 13 new risk loci for schizophrenia. Nature Genetics, 45, 150–159.
37. Rosenthal, R. (2001). Meta-analytic procedures for social research (2nd ed.). Beverly Hills, CA: Sage Publications.
38. Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284.
39. Rozeboom, W. W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57, 416–428.
40. Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173–1181.
41. Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies. Psychological Bulletin, 105(2), 309–316.
42. Simons, D. J. (2014). The value of direct replication. Perspectives on Psychological Science, 9, 76–80.
43. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
44. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2013). P-Curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534–537.
45. Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits of psychotherapy. Baltimore, MD: Johns Hopkins University Press.
46. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124–1131.
47. Vrij, A., Granhag, P. A., & Porter, S. (2010). Pitfalls and opportunities in nonverbal and verbal lie detection. Psychological Science in the Public Interest, 11, 89–121.
48. Wicherts, J.M. (2014, May 24). The power paradox and the myth of the failed study. Paper presented at the Annual Convention of the Association for Psychological Science, San Francisco, California.
49. Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PloS One, 6, e26828.