Aggregate and Individual Replication Probability Within an Explicit Model of the Research Process

Psychological Methods

Volumn 16, Issue 3, 2011, Pages 337-360

  Miller, Jeff ^a   Schwarz, Wolf ^b  

^a UNIVERSITY OF OTAGO   (New Zealand)

^b UNIVERSITY OF POTSDAM   (Germany)

Author keywords

Bayesian inference; Posterior statistical power; Probability of replication; Random effects model; Statistical estimation

Indexed keywords

ARTICLE; BAYES THEOREM; BEHAVIORAL RESEARCH; HUMAN; METHODOLOGY; PROBABILITY; PSYCHOLOGICAL MODEL; REPRODUCIBILITY; STATISTICAL ANALYSIS; STATISTICAL MODEL;

BAYES THEOREM; BEHAVIORAL RESEARCH; DATA INTERPRETATION, STATISTICAL; HUMANS; MODELS, PSYCHOLOGICAL; MODELS, STATISTICAL; PROBABILITY; REPRODUCIBILITY OF RESULTS;

EID: 80052823469     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0023347     Document Type: Article

References (35)

1. Cohen J. The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology 1962, 65:145-153. 10.1037/h0045186.
2. Cumming G. Understanding the average probability of replication: Comment on Killeen (2005). Psychological Science 2005, 16:1002-1004. 10.1111/j.1467-9280.2005.01650.x.
3. rep, and confidence intervals: Comment prompted by Iverson, Wagenmakers, and Lee (2010); Lecoutre, Lecoutre, and Poitevineau (2010); and Maraun and Gabriel (2010). Psychological Methods 2010, 15:192-198. 10.1037/a0019521.
4. Doros G., Geier A.B. Probability of replication revisited: Comment on " An alternative to null-hypothesis significance tests." Psychological Science 2005, 16:1005-1006. 10.1111/j.1467-9280.2005.01651.x.
5. Duncan J., Phillips L., McLeod P. Measuring the mind: Speed, control, and age 2005, Oxford University Press, Oxford, England.
6. Froman T., Shneyderman A. Replicability reconsidered: An excessive range of possibilities. Understanding Statistics 2004, 3:365-373. 10.1207/s15328031us0304_9.
7. Gorroochurn P., Hodge S.E., Heiman G.A., Durner M., Greenberg D.A. Non-replication of association studies: " Pseudo-failures" to replicate?. Genetics in Medicine 2007, 9:325-331. 10.1097/GIM.0b013e3180676d79.
8. Greenwald A.G., Gonzalez R., Harris R.J., Guthrie D. Effect sizes and p values: What should be reported and what should be replicated?. Psychophysiology 1996, 33:175-183. 10.1111/j.1469-8986.1996.tb02121.x.
9. Hoenig J.M., Heisey D.M. The abuse of power: The pervasive fallacy of power calculations for data analysis. American Statistician 2001, 55:19-24.
10. Holender D. Semantic activation without conscious identification in dichotic listening, parafoveal vision, and visual masking: A survey and appraisal. Behavioral and Brain Sciences 1986, 9:1-23. 10.1017/S0140525X00021269.
11. rep misestimates the probability of replication. Psychonomic Bulletin & Review 2009, 16:424-429. 10.3758/PBR.16.2.424.
12. rep continues to mispredict the probability of replication. Psychonomic Bulletin & Review 2010, 17:270-272. 10.3758/PBR.17.2.270.
13. rep: An agony in five fits. Journal of Mathematical Psychology 2009, 53:195-202. 10.1016/j.jmp.2008.09.004.
14. rep. Psychological Methods 2010, 15:172-181. 10.1037/a0017182.
15. Killeen P.R. An alternative to null-hypothesis significance tests. Psychological Science 2005, 16:345-353. 10.1111/j.0956-7976.2005.01538.x.
16. Killeen P.R. Replicability, confidence, and priors. Psychological Science 2005, 16:1009-1012. 10.1111/j.1467-9280.2005.01653.x.
17. Killeen P.R. Replication statistics. Best practices in quantitative methods 2007, 103-124. Sage, Thousand Oaks, CA. J.W. Osborne (Ed.).
18. Lecoutre B., Killeen P.R. Replication is not coincidence: Reply to Iverson, Lee, and Wagenmakers (2009). Psychonomic Bulletin & Review 2010, 17:263-269. 10.3758/PBR.17.2.263.
19. Lecoutre B., Lecoutre M.-P., Poitevineau J. Killeen's probability of replication and predictive probabilities: How to compute, use, and interpret them. Psychological Methods 2010, 15:158-171. 10.1037/a0015915.
20. Lien M.-C., Allen P.A., Ruthruff E., Grabbe J., McCann R.S., Remington R.W. Visual word recognition without central attention: Evidence for greater automaticity with advancing age. Psychology and Aging 2006, 21:431-447. 10.1037/0882-7974.21.3.431.
21. Macdonald R.R. On determining replication probabilities: Comments on Posavac (2002). Understanding Statistics 2003, 2:69-70. 10.1207/S15328031US0201_04.
22. Macdonald R.R. Why replication probabilities depend on prior probability distributions: A rejoinder to Killeen (2005). Psychological Science 2005, 16:1007-1008. 10.1111/j.1467-9280.2005.01652.x.
23. Maher P. The concept of inductive probability. Erkenntnis 2006, 65:185-206. 10.1007/s10670-005-5087-5.
24. rep coefficient: Logical and mathematical problems. Psychological Methods 2010, 15:182-191. 10.1037/a0016955.
25. Miller J. What is the probability of replicating a statistically significant effect?. Psychonomic Bulletin & Review 2009, 16:617-640. 10.3758/PBR.16.4.617.
26. Onwuegbuzie A.J., Leech N.L. Post hoc power: A concept whose time has come. Understanding Statistics 2004, 3:201-230. 10.1207/s15328031us0304_1.
27. Posavac E.J. Using p values to estimate the probability of a statistically significant replication. Understanding Statistics 2002, 1:101-112. 10.1207/S15328031US0102_03.
28. Richard F.D., Bond C.F., Stokes-Zoota J.J. One hundred years of social psychology quantitatively described. Review of General Psychology 2003, 7:331-363. 10.1037/1089-2680.7.4.331.
29. Sanabria F., Killeen P.R. Better statistics for better decisions: Rejecting null hypotheses statistical tests in favor of replication statistics. Psychology in the Schools 2007, 44:471-481. 10.1002/pits.20239.
30. Sohn D. Statistical significance and replicability: Why the former does not presage the latter. Theory & Psychology 1998, 8:291-311. 10.1177/0959354398083001.
31. SPSS [Computer software] SPSS 14.0 for Windows 2006, Author, Chicago, IL.
32. Stroop J.R. Studies of interference in serial verbal reactions. Journal of Experimental Psychology 1935, 18:643-662. 10.1037/h0054651.
33. Wagenmakers E.-J., Grünwald P. A Bayesian perspective on hypothesis testing: A comment on Killeen (2005). Psychological Science 2006, 17:641-642. 10.1111/j.1467-9280.2006.01757.x.
34. Wilson D.B., Lipsey M.W. The role of method in treatment effectiveness research: Evidence from meta-analysis. Psychological Methods 2001, 6:413-429. 10.1037/1082-989X.6.4.413.
35. Yuan K.-H., Maxwell S. On the post hoc power in testing mean differences. Journal of Educational and Behavioral Statistics 2005, 30:141-167. 10.3102/10769986030002141.