Exploring Small, Confirming Big: An alternative system to The New Statistics for advancing cumulative and replicable psychological research

Journal of Experimental Social Psychology

Volumn 66, Issue , 2016, Pages 47-54

  Sakaluk, John Kitchener ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

Big data; Data analysis; Meta analysis; Open science; Replicability

Indexed keywords

DATA ANALYSIS; EXPLORATORY RESEARCH; HUMAN; META ANALYSIS; NULL HYPOTHESIS; PSYCHOLOGICAL MODEL; PSYCHOLOGIST; STATISTICS; VISION;

EID: 84978096837     PISSN: 00221031     EISSN: 10960465     Source Type: Journal    
DOI: 10.1016/j.jesp.2015.09.013     Document Type: Article

References (51)

1. Bakker, M., Wicherts, J.M., The (misreporting) of statistical results in psychology journals. Behavior Research Methods 43 (2011), 666–678.
2. Baumeister, R.F., Vohs, K.D., Funder, D.C., Psychology as the science of self-reports and finger movements: Whatever happened to actual behavior?. Perspectives on Psychological Science 2 (2007), 396–403.
3. Belia, S., Fidler, F., Williams, J., Cumming, G., Researchers misunderstand confidence intervals and standard error bars. Psychological Methods 10 (2005), 389–396.
4. Brown, T.A., Confirmatory factor analysis for applied research. 2nd edition, 2015, Guilford Press, New York, NY.
5. Buhrmester, M., Kwang, T., Gosling, S.D., Amazon's mechanical turk: A new source of inexpensive, yet high-quality, data?. Perspectives on Psychological Science 6 (2011), 3–5.
6. Burke, B.L., Martens, A., Faucher, E.H., Two decades of terror management theory: A meta-analysis of mortality salience research. Personality and Social Psychology Review 14 (2010), 155–195.
7. Carey, B., Fraud case seen as a red flag for psychology research. 2011, November (Retrieved from http://www.nytimes.com).
8. Chan, M.E., Avery, R.D., Meta-analysis and the development of knowledge. Perspectives on Psychological Science 7 (2012), 79–92.
9. Crocker, J., Cooper, M.L., Addressing scientific fraud. Science, 334, 2011, 1182.
10. Cumming, G., The new statistics: Why and how. Psychological Science 25 (2014), 7–29.
11. Danneman, N., Heimann, R., Social media mining with R. 2014, Packt Publishing, Birmingham, UK.
12. Eich, E., Business not as usual. Psychological Science 27 (2014), 3–6.
13. Eichstaedt, J.C., Schwartz, H.A., Kern, M.L., Park, G., Labarthe, D.R., Merchant, R.M. Seligman, M.E.P., Psychological language on twitter predicts country-level heart disease mortality. Psychological Science, 2015, 10.1177/0956797614557867.
14. Fabrigar, L.R., Wegener, D.T., Conceptualizing and evaluating the replication of research results. Journal of Experimental and Social Psychology 66 (2016), 68–80 (in this issue).
15. Fidler, F., From statistical significance to effect estimation: Statistical reform in psychology, medicine, and ecology (doctoral dissertation). 2005 (Retrieved from tiny.cc/fionasphd).
16. Fraley, R.C., Attachment stability from infancy to adulthood: Meta-analysis and dynamic modeling of developmental mechanisms. Personality and Social Psychology Review 6 (2002), 123–151.
17. Gentry, J., Package ‘twitteR’. Retrieved from, 2015 http://cran.r-project.org/web/packages/twitteR.
18. Gøtzsche, P.C., Hróbjartsson, A., Marić, K., Tendal, B., Data extraction errors in meta-analyses that use standardized mean differences. Journal of the American Medical Association 298 (2007), 430–437.
19. Hoyle, R.H., Handbook of structural equation modeling. 2012, Guilford Press, New York, NY.
20. Ionnidis, J.P.A., Why most published research findings are false. PLoS Medicine, 2, 2005, e124.
21. John, L.K., Loewenstein, G., Prelec, D., Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science 23 (2012), 524–532.
22. Kirk, R.E., The importance of effect magnitude. Davis, S.F., (eds.) Handbook of research methods in experimental psychology, 2003, Blackwell, Malden, MA, 83–105.
23. Klein, R.A., Ratliff, K.A., Vianello, M., Adams, R.B. Jr., Bahník, S., Berstein, M.J. Nosek, B.A., Investigating variation in replicability: A “many labs” replication project. Social Psychology 45 (2014), 142–152.
24. Klein, R.B., Beyond significance testing: Reforming data analysis methods in behavioral research. 2004, APA Books, Washington, DC.
25. Kruschke, J., Doing Bayesian data analysis, second edition: A tutorial with R, JAGS, and Stan. 2014, Academic Press, New York, NY.
26. Makel, M.C., Plucker, J.A., Hegarty, B., Replications in psychology research: How often do they really occur?. Perspectives on Psychological Science 7 (2012), 537–542.
27. Maner, J.K., Into the wild: Field research can increase both replicability and real-world impact. Journal of Experimental Social Psychology 66 (2015), 100–106 (in this issue).
28. Maxwell, S.E., The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods 9 (2004), 147–163.
29. Mehl, M.R., Conner, T.S., Handbook of research methods for studying daily life. 2013, Guilford Press, New York, NY.
30. Nelson, L.D., Simmons, J.P., Simonsohn, U., Let's publish fewer papers. Psychological Inquiry 23 (2012), 291–293.
31. Neville, L., Do economic equality and generalized trust inhibit academic dishonesty? Evidence from state-level search-engine queries. Psychological Science 23 (2012), 339–345.
32. Open Science Collaboration. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science 7 (2012), 657–660.
33. Rosenthal, R., The file drawer problem and tolerance for null results. Psychological Bulletin 86 (1979), 638–641.
34. Russell, M.A., Mining the social web. 2nd edition, 2013, O'Reilly, Sebastopol, CA.
35. Sakaluk, J.K., Williams, A.J., Biernat, M., Analytic review as a solution to the misreporting of statistical results in psychological science. Perspectives on Psychological Science 9 (2014), 652–660.
36. Schmidt, F.L., What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist 47 (1992), 1173–1181.
37. Schmidt, F.L., Hunter, J.E., Eight common but false objections to the discontinuation of significance testing in the analysis of research data. Harlow, L.L., Mulaik, S.A., Steiger, J.H., (eds.) What if there were no significance tests?, 1997, Erlbaum, Mahwah, NJ, 37–64.
38. Schwartz, H.A., Eichstaedt, J.C., Kern, M.L., Dziurzynski, L., Ramones, S.M., Argawal, M. Ungar, L.H., Personality, gender, and age in the language of social media: The open-vocabulary approach. PloS One, 8, 2013, e73791.
39. Sharpe, D., Why the resistance to statistical innovations? Bridging the communication gap. Psychological Methods 18 (2013), 572–582.
40. Sibley, C.G., Duckitt, J., Personality and prejudice: A meta-analysis and theoretical review. Personality and Social Psychology Review 12 (2008), 248–279.
41. Simmons, J.P., Nelson, L.D., Simonsohn, U., False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22 (2011), 1359–1366.
42. Simonsohn, U., Just post it: The lesson from two cases of fabricated data detected by statistics alone. Psychological Science 24 (2013), 1875–1888.
43. Simonsohn, U., Nelson, L.D., Simmons, J.P., P-curve: A key to the file-drawer. Journal of Experimental Psychology: General 143 (2014), 534–547.
44. Snijders, T., Bosker, R., Multilevel analysis: An introduction to basic and advanced multilevel modeling. 1999, Sage, Thousand Oaks, CA.
45. Stanley, T.D., Doucouliagos, H., Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods 5 (2014), 60–78.
46. Trafimow, D., Marks, M., Editorial. Basic and Applied Social Psychology 37 (2015), 1–2.
47. Tsuji, S., Bergmann, C., Cristia, A., Community-augmented meta-analyses: Toward cumulative data assessment. Perspectives on Psychological Science 9 (2014), 661–665.
48. Uhlmann, E.L., et al. The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline. Journal of Experimental Social Psychology 66 (2016), 55–67 (in this issue).
49. Wagenmakers, E., Wetzels, R., van der Maas, H.L.J., Kievit, R.A., An agenda for purely confirmatory research. Perspectives on Psychological Science 7 (2012), 632–638.
50. Walasek, L., Brown, G.D.A., Income inequality and status seeking: Searching for positional goods in unequal U.S. states. Psychological Science 26 (2015), 527–533.
51. Wicherts, J.M., Bakker, M., Molenaar, D., Willingness to share research data is related to the strength of the evidence and quality of reporting of statistical results. PloS One, 6, 2011, e26828.