Comparing the accuracy of experimental estimates to guessing: A new perspective on replication and the "Crisis of Confidence" in psychology

Behavior Research Methods

Volumn 46, Issue 1, 2014, Pages 1-14

  Davis Stober, Clintin P ^a   Dana, Jason ^b  

^a UNIVERSITY OF MISSOURI   (United States)

^b UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

Estimation accuracy; Improper linear models; Replicability; Statistical power

Indexed keywords

ANALYSIS OF VARIANCE; ARTICLE; COMPARATIVE STUDY; COMPUTER PROGRAM; HUMAN; META ANALYSIS (TOPIC); METHODOLOGY; PSYCHOLOGY; QUALITY CONTROL; REGRESSION ANALYSIS; REPRODUCIBILITY; SAMPLE SIZE; STANDARD; STATISTICAL ANALYSIS;

ANALYSIS OF VARIANCE; BENCHMARKING; DATA INTERPRETATION, STATISTICAL; HUMANS; LEAST-SQUARES ANALYSIS; META-ANALYSIS AS TOPIC; PSYCHOLOGY, EXPERIMENTAL; REGRESSION ANALYSIS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SAMPLE SIZE; SOFTWARE; SOFTWARE DESIGN;

EID: 84894654665     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-013-0342-1     Document Type: Article

References (42)

1. Amemiya, T. (1985). Advanced econometrics. Harvard University Press.
2. Bem, D. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407-425.
3. Bickel, P. J., & Doksum, K. A. (2001). Mathematical statistics: Basic ideas and selected topics (Vol. 1) (2nd ed.). Upper Saddle River, NJ: Prentice Hall.
4. Carey, B. (2011). Fraud case seen as a red flag for psychology research. The New York Times, published November 2.
5. Chipman, J. S., & Rao, M. M. (1964). The treatment of linear restrictions in regression analysis. Econometrica, 32, 198-209.
6. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
7. Dana, J., & Dawes, R. (2004). The superiority of simple alternatives to regression for social science predictions. Journal of Educational and Behavioral Statistics, 29, 317-331.
8. Dana, J. (2008). What makes improper linear models tick? In J. Krueger (Ed.), Rationality and social responsibility: Essays in honor of Robyn Mason Dawes. Mahwah, NJ: Lawrence Erlbaum Associates.
9. Davis-Stober, C. P. (2011). A geometric analysis of when fixed weighting schemes will outperform ordinary least squares. Psychometrika, 76, 650-669.
10. Davis-Stober, C. P., Dana, J., & Budescu, D. (2010a). A constrained linear estimator for multiple regression. Psychometrika, 75, 521-541.
11. Davis-Stober, C. P., Dana, J., & Budescu, D. (2010b). Why recognition is rational: Optimality results on single-variable decision rules. Judgment and Decision Making, 5, 216-229.
12. Dawes, R. M. (1979). The robust beauty of improper linear models. American Psychologist, 34, 571-582.
13. Dawes, R. M., & Corrigan, B. (1974). Linear models in decision making. Psychological Bulletin, 81, 95-106.
14. Eldar, Y. C., Ben-Tal, A., & Nemirovski, A. (2005). Robust mean-squared error estimation in the presence of bounded data uncertainties. IEEE Transactions on Signal Processing, 53, 168-181.
15. Francis, G. (2012a). Replication initiative: Beware misinterpretation. Science, 336, 802.
16. Francis, G. (2012b). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin and Review, 19, 151-156.
17. Gelman, A., & Weakliem, D. (2009). Of beauty, sex, and power: Statistical challenges in estimating small effects. American Scientist, 97, 310-316.
18. Hartung, J., Knapp, G., & Sinha, B. K. (2008). Statistical meta-analysis with applications. Hoboken, NJ: John Wiley & Sons.
19. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2, 696-701.
20. Ioannidis, J. P. A. (2008). Why most discovered true associations are inflated. Epidemiology, 19, 640-648.
21. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524-532.
22. Kelley, K., & Maxwell, S. E. (2003). Sample size for multiple regression: Obtaining regression coefficients that are accurate, not simply significant. Psychological Methods, 8, 305-321.
23. Kutner, M., Nachtsheim, C., Neter, J., & Li, W. (2004). Applied linear statistical models. McGraw-Hill.
24. Lai, K., & Kelley, K. (2012). Accuracy in parameter estimation for ANCOVA and ANOVA contrasts: Sample size planning via narrow confidence intervals. British Journal of Mathematical and Statistical Psychology, 65, 350-370.
25. Lehmann, E. L., & Casella, G. (1998). Theory of point estimation (2nd ed.). New York: Springer.
26. Lehrer, J. (2010). The truth wears off. The New Yorker, published December 13.
27. Maxwell, S. E. (2000). Sample size and multiple regression analysis. Psychological Methods, 5, 434-458.
28. Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147-163.
29. Maxwell, S. E., & Delaney, H. D. (2003). Designing experiments and analyzing data: A model comparison perspective (2nd ed.). Mahwah, NJ: Erlbaum.
30. Miller, J. (2009). What is the probability of replicating a statistically significant effect? Psychonomic Bulletin and Review, 16, 617-640.
31. Miller, J., & Schwarz, W. (2011). Aggregate and individual replication probability within an explicit model of the research process. Psychological Methods, 16, 337-360.
32. Muller, K. E., & Fetterman, B. A. (2003). Regression and ANOVA: An integrated approach using SAS software. Cary, NC: John WIley & Sons Inc., SAS Institute Inc.
33. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631.
34. Pashler, H., & Wagenmakers, E.-J. (2012). Editors' introduction to the special section on replicability in psychological science: A crisis of confidence. Perspectives on Psychological Science, 7, 528-530.
35. R Core Team. (2012). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.
36. Ritchie, S. J., Wiseman, R., French, C. C. (2012). Failing the future: Three unsuccessful attempts to replicate Bem's 'retroactive facilitation of recall' effect. PLoS ONE, 7, doi: 10. 1371/journal. pone. 0033423.
37. Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316.
38. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366.
39. Teräsvirta, T. (1983). Restricted superiority of linear homogeneous estimators over ordinary least squares. Scandinavian Journal of Statistics, 10, 27-33.
40. Toro-Vizcarrondo, C., & Wallace, T. D. (1968). A test of the mean square error criterion for restrictions in linear regression. Journal of the American Statistical Association, 63, 558-572.
41. Tressoldi, P. E. (2012). Replication unreliability in psychology: Elusive phenomena or "elusive" statistical power? Frontiers in Psychology, 3, 218. doi: 10. 3389/fpsyg. 2012. 00218.
42. Wainer, H. (1976). Estimating coefficients in linear models: It don't make no nevermind. Psychological Bulletin, 83, 213-217.