Metastudies for robust tests of theory

Proceedings of the National Academy of Sciences of the United States of America

Volumn 115, Issue 11, 2018, Pages 2607-2612

  Baribault, Beth ^a   Donkin, Chris ^b   Little, Daniel R ^c   Trueblood, Jennifer S ^d   Oravecz, Zita ^e   Van Ravenzwaaij, Don ^f   White, Corey N ^g   De Boeck, Paul ^h   Vandekerckhove, Joachim ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF NEW SOUTH WALES   (Australia)

^c UNIVERSITY OF MELBOURNE   (Australia)

^d VANDERBILT UNIVERSITY   (United States)

^e PENNSYLVANIA STATE UNIVERSITY   (United States)

^f UNIVERSITY OF GRONINGEN   (Netherlands)

^g MISSOURI WESTERN STATE UNIVERSITY   (United States)

^h OHIO STATE UNIVERSITY   (United States)

Author keywords

Generalizability; Many labs; Metastudy; Radical randomization; Robustness

Indexed keywords

ARTICLE; ASSOCIATION; BAYES THEOREM; EXECUTIVE FUNCTION; EXPLORATORY RESEARCH; INFORMATION PROCESSING; MATHEMATICAL ANALYSIS; MATHEMATICAL COMPUTING; PRIORITY JOURNAL; PROTOCOL COMPLIANCE; PSYCHOLOGY; PUBLISHING; RANDOMIZATION; REPLICATION STUDY; STIMULUS; HUMAN; MEDICAL RESEARCH; METHODOLOGY; STANDARDS; STATISTICAL ANALYSIS;

BAYES THEOREM; BIOMEDICAL RESEARCH; DATA INTERPRETATION, STATISTICAL; HUMANS; RANDOM ALLOCATION; RESEARCH DESIGN;

EID: 85043762113     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.1708285114     Document Type: Article

References (19)

1. Pashler H, Wagenmakers EJ (2012) Editor’s introduction to the special section on replicability in psychological science: A crisis of confidence? Perspect Psychol Sci 7: 528–530.
2. Francis G (2012) Publication bias and the failure of replication in experimental psychology. Psychon Bull Rev 19:975–991.
3. Guan M, Vandekerckhove J (2016) A Bayesian approach to mitigation of publication bias. Psychon Bull Rev 23:74–86.
4. Rosenthal R (1979) The file drawer problem and tolerance for null results. Psychol Bull 86:638–641.
5. Vasishth S, Gelman A (2017) The illusion of power: How the statistical significance filter leads to overconfident expectations of replicability. arXiv:1702.00556.
6. Sterling TD (1959) Publication decisions and their possible effects on inferences drawn from tests of significance–or vice versa. J Am Stat Assoc 54:30–34.
7. Etz A, Vandekerckhove J (2016) A Bayesian perspective on the reproducibility project: Psychology. PLoS One 11:e0149794.
8. Gelman A, Loken E (2014) The statistical crisis in science data-dependent analysis—a “garden of forking paths”—explains why many statistically significant comparisons don’t hold up. Am Scientist 102:460–465.
9. Fisher RA (1935) The Design of Experiments (Oliver and Boyd, Edinburgh).
10. Popper K (1963) Conjectures and Refutations: The Growth of Scientific Knowledge (Routledge & Kegan Paul, Abingdon, UK).
11. Simons DJ, Shoda Y, Lindsay DS (in press) Constraints on generality (COG): A proposed addition to all empirical papers. Perspect Psychol Sci, 10.1177/1745691617708630.
12. Kenett RS, Rubinstein AA (2017) A generalization approach to reproducibility claims. Available at https://papers.ssrn.com/sol3/papers.cfm?abstractid=3035070.
13. Cronbach LJ, Rajaratnam N, Gleser GC (1963) Theory of generalizability: A liberalization of reliability theory. Br J Stat Psychol 16:137–163.
14. Clark HH (1973) The language–as–fixed–effect fallacy: A critique of language statistics in psychological research. J Verbal Learn Verbal Behav 12:335–359.
15. Worrall J (2007) Why there’s no cause to randomize. Br J Philos Sci 58:451–488.
16. Gelman A, Hill J (2006) Data Analysis Using Regression and Multilevel/Hierarchical Models (Cambridge Univ Press, Cambridge, UK).
17. Sutton AJ, Abrams KR (2001) Bayesian methods in meta-analysis and evidence synthesis. Stat Methods Med Res 10:277–303.
18. Reuss H, Kiesel A, Kunde W (2015) Adjustments of response speed and accuracy to unconscious cues. Cognition 134:57–62.
19. Stan Development Team (2014) Stan: A c++ library for probability and sampling. Available at mc-stan.org/.