Small Telescopes: Detectability and the Evaluation of Replication Results

Psychological Science

Volumn 26, Issue 5, 2015, Pages 559-569

  Simonsohn, Uri ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

hypothesis testing; open materials; replication; statistical power

Indexed keywords

ASTRONOMY; METHODOLOGY; PROCEDURES; REPRODUCIBILITY; SAMPLE SIZE; STANDARDS; STATISTICAL ANALYSIS; STATISTICAL MODEL; TELESCOPE;

ASTRONOMY; DATA INTERPRETATION, STATISTICAL; MODELS, STATISTICAL; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SAMPLE SIZE; TELESCOPES;

EID: 84930587816     PISSN: 09567976     EISSN: 14679280     Source Type: Journal    
DOI: 10.1177/0956797614567341     Document Type: Article

References (45)

1. Alogna V. K.,Attaya M. K.,Aucoin P.,Bahník Š.,Birch S.,Birt A. R.,.. Zwaan R. A.Registered Replication Report: Schooler and Engstler-Schooler (1990).Perspectives on Psychological Science. 2014;9:556-578
2. Asendorpf J. B.,Conner M.,De Fruyt F.,De Houwer J.,Denissen J. J.,Fiedler K.,.. Wicherts J. M.Recommendations for increasing replicability in psychology.European Journal of Personality. 2013;27:108-119
3. Association for Psychological Science. (2014). Registered Replication Reports. Retrieved from http://web.archive.org/web/20140623042346/http://www.psychologicalscience.org/index.php/replication
4. Bond R. M.,Fariss C. J.,Jones J. J.,Kramer A. D.,Marlow C.,Settle J. E.,Fowler J. H.A 61-million-person experiment in social influence and political mobilization.Nature. 2012;489:295-298
5. Button K. S.,Ioannidis J. P.,Mokrysz C.,Nosek B. A.,Flint J.,Robinson E. S.,Munafò M. R.Power failure: Why small sample size undermines the reliability of neuroscience.Nature Reviews Neuroscience. 2013;14:365-376
6. Cohen J.The statistical power of abnormal-social psychological research: A review.Journal of Abnormal and Social Psychology. 1962;65:145-153
7. Cohen J.Statistical power analysis for the behavioral sciences. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.Hillsdale, NJ: Erlbaum; 1988:
8. Cohen J.The earth is round (p <.05).American Psychologist. 1994;49:997-1003
9. Coursey D.,Hovis J.,Schulze W.The disparity between willingness to accept and willingness to pay measures of value.Quarterly Journal of Economics. 1987;102:679-690
10. Cumming G.Replication and p intervals: p values predict the future only vaguely, but confidence intervals do much better.Perspectives on Psychological Science. 2008;3:286-300
11. Cumming G.The new statistics: Why and how.Psychological Science. 2014;25:7-29
12. Dahl M. S.,Dezsā C. L.,Ross D. G.Fatherhood and managerial style: How a male CEO’s children affect the wages of his employees.Administrative Science Quarterly. 2012;57:669-693
13. FeddersenJ.MetcalfeR.WoodenM. (2012). Subjective well-being: Weather matters; climate doesn’t (Melbourne Institute Working Paper Series, 25/2012). Melbourne, Victoria, Australia: University of Melbourne. Retrieved from http://web.archive.org/web/20150107020727/http://melbourneinstitute.com/downloads/working_paper_series/wp2012n25.pdf
14. Fisher R. A.The arrangement of field experiments.Journal of the Ministry of Agriculture. 1926;33:503-513
15. Gámez E.,Díaz J. M.,Marrero H.The uncertain universality of the Macbeth effect with a Spanish sample.The Spanish Journal of Psychology. 2011;14:156-162
16. Greenwald A. G.Consequences of prejudice against the null hypothesis.Psychological Bulletin. 1975;82:1-20
17. Hedges L. V.Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences.Journal of Educational and Behavioral Statistics. 1984;9:61-85
18. Hodges J.,Lehmann E.Testing the approximate validity of statistical hypotheses.Journal of the Royal Statistical Society: Series B (Statistical Methodology). 1954;16:261-268
19. Ioannidis J. P. A.Why most discovered true associations are inflated.Epidemiology. 2008;19:640-646
20. Jeffreys H.Theory of probability. (1961). Theory of probability (3rd ed.). Oxford, England: Oxford University Press.Oxford, England: Oxford University Press; 1961:
21. Kahneman D.,Knetsch J. L.,Thaler R. H.Experimental tests of the endowment effect and the Coase theorem.Journal of Political Economy. 1990;98:1325-1348
22. Knetsch J. L.,Sinden J. A.Willingness to pay and compensation demanded: Experimental evidence of an unexpected disparity in measures of value.Quarterly Journal of Economics. 1984;99:507-521
23. Kraemer H. C.Theory of estimation and testing of effect sizes: Use in meta-analysis.Journal of Educational and Behavioral Statistics. 1983;8:93-101
24. Lane D. M.,Dunlap W. P.Estimating effect size: Bias resulting from the significance criterion in editorial decisions.British Journal of Mathematical and Statistical Psychology. 1978;31:107-112
25. Lindley D. V.Introduction to probability and statistics from a Bayesian viewpoint. Cambridge, England: Cambridge University Press; 1965:
26. Lucas R. E.,Lawless N. M.Does life seem better on a sunny day? Examining the association between daily weather conditions and life satisfaction judgments.Journal of Personality and Social Psychology. 2013;104:872-884
27. Nosek B. A.,Banaji M.,Greenwald A. G.Harvesting implicit group attitudes and beliefs from a demonstration web site.Group Dynamics: Theory, Research, and Practice. 2002;6:101-115
28. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science.Perspectives on Psychological Science. 2012;7:657-660
29. PopperK. R. (2005). The logic of scientific discovery [Taylor & Francis e-Library edition]. Retrieved from http://web.archive.org/web/20150218163435/http://strangebeautiful.com/other-texts/popper-logic-scientific-discovery.pdf (Original work published 1935)
30. Rindskopf D. M.What if there were no significance tests. Harlow L. L.Mulaik S. A.Steiger J. H., ed. Mahwah, NJ: Erlbaum; 1997:319-332.
31. Rossi J. S.Statistical power of psychological research: What have we gained in 20 years?.Journal of Consulting and Clinical Psychology. 1990;58:646-656
32. Rouanet H.Bayesian methods for assessing importance of effects.Psychological Bulletin. 1996;119:149-158
33. Rouder J. N.,Speckman P. L.,Sun D.,Morey R. D.,Iverson G.Bayesian t tests for accepting and rejecting the null hypothesis.Psychonomic Bulletin & Review. 2009;16:225-237
34. Schooler J. W.,Engstler-Schooler T. Y.Verbal overshadowing of visual memories: Some things are better left unsaid.Cognitive Psychology. 1990;22:36-71
35. Schwarz N.,Clore G.Mood, misattribution, and judgments of well-being: Informative and directive functions of affective states.Journal of Personality and Social Psychology. 1983;45:513-523
36. Sedlmeier P.,Gigerenzer G.Do studies of statistical power have an effect on the power of studies?.Psychological Bulletin. 1989;105:309-316
37. Serlin R. C.,Lapsley D. K.Rationality in psychological research: The good-enough principle.American Psychologist. 1985;40:73-83
38. Serlin R. C.,Lapsley D. K.A handbook for data analysis in the behavioral sciences: Methodological issues. Keren G.Lewis C., ed. Mahwah, NJ: Erlbaum; 1993:199-228.
39. SievJ. (2012). [Attempt to replicate Zhong & Liljenquist]. Unpublished raw data.
40. SimonsohnU. (2014). [17] No-way interactions. Retrieved from http://web.archive.org/web/20150206205257/http://datacolada.org/2014/03/12/17-no-way-interactions-2/
41. Simonsohn U.,Nelson L. D.,Simmons J. P.p-curve and effect size: Correcting for publication bias using only significant results.Perspectives on Psychological Science. 2014a;9:666-681
42. Simonsohn U.,Nelson L. D.,Simmons J. P.P-curve: A key to the file-drawer.Journal of Experimental Psychology: General. 2014b;143:534-547
43. Valentine J. C.,Biglan A.,Boruch R. F.,Castro F. G.,Collins L. M.,Flay B. R.,.. Schinke S. P.Replication in prevention science.Prevention Science. 2011;12:103-117
44. Verhagen J.,Wagenmakers E.-J.A Bayesian test to quantify the success or failure of a replication attempt.Journal of Experimental Psychology: General. 2014;143:1457-1475
45. Zhong C.-B.,Liljenquist K.Washing away your sins: Threatened morality and physical cleansing.Science. 2006;313:1451-1452