Tracking replicability as a method of post-publication open evaluation

Frontiers in Computational Neuroscience

Volumn , Issue JANUARY 2012, 2012, Pages

  Hartshorne, Joshua ^a   Schachner, Adena ^a  

^a HARVARD UNIVERSITY   (United States)

Author keywords

Open evaluation; Post publication evaluation; Replicability; Replication

Indexed keywords

CURRENT SYSTEM; DATA QUALITY; INCENTIVE STRUCTURE; LAY-OUT; OPEN ACCESS; OPEN EVALUATION; PEER REVIEW; POST-PUBLICATION EVALUATION; QUALITY OF WORK; REPLICABILITY; REPLICATION; RESEARCH COMMUNITIES;

DATABASE SYSTEMS; PUBLISHING;

QUALITY CONTROL;

ACCESS TO INFORMATION; ACCURACY; ALGORITHM; ANALYTIC METHOD; ARTICLE; CITATION ANALYSIS; DATA BASE; FEASIBILITY STUDY; INFORMATION PROCESSING; INFORMATION SYSTEM; PUBLICATION; QUALITY CONTROL; RELIABILITY; REPRODUCIBILITY; SAMPLE SIZE; SCIENTIFIC LITERATURE; SCIENTIST; STATISTICAL ANALYSIS; TRACKING REPLICABILITY;

EID: 84856770812     PISSN: 16625188     EISSN: None     Source Type: Journal    
DOI: 10.3389/fncom.2012.00008     Document Type: Article

References (75)

1. Adamic, L. A., Zhang, J., Bakshy, E., & Ackerman, M. S. (2008). Knowledge sharing and yahoo answers: Everyone knows something. Proceedings of the 17th International Conference on World Wide Web.
2. Albert, P. S., & Dodd, L. E. (2004). A cautionary note on the robustness of latent class models for estimating diagnostic error without a gold standard. Biometrics, 60, 427-435.
3. Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated Significance Tests on Accumulating Data. Journal of the Royal Statistical Society: Series A, 132, 235-244.
4. Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Langauge, 59, 390-412.
5. Bederson, B. B., Hu, C., & Resnik, P. (2010). Translation by interactive collaboration between monolingual users. Proceedings of Graphics Interface, 39-46.
6. Bezeau, S., & Graves, R. (2001). Statistical power and effect sizes of clinical neuropsychology research. Journal of Clinical and Experimental Neuropsychology, 2001(23), 3.
7. Boffetta, P., Mclaughlin, J. K., Vecchia, C. L., Tarone, R. E., Lipworth, L., & Blot, W. J. (2008). False-Positive Results in Cancer Epidemiology: A Plea for Epistemological Modesty. Journal of hte National Cancer Institute, 100, 988-995.
8. Callaham, M. L., Wears, R. L., Weber, E. J., Barton, C., & Young, G. (1998). Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. JAMA: the journal of the American Medical Association, 280, 254-257.
9. Callison-Burch, C. (2009). Fast, cheap, and creative: Evaluating translation quality using Amazon's Mechanical Turk. Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, 286-295.
10. Chase, L. J., & Chase, R. B. (1976). A statistical power analysis of applied psychological research. Journal of Applied Psychology, 61(2), 234-237.
11. Cohen, J. (1962). The statistical power of abnormal-social psychological research. Journal of Abnormal and Social Psychology, 65, 145-153.
12. Cole Jr., S., Cole, J. R., & Simon, G. A. (1981). Chance and consensus in peer review. Science, 214, 881-886.
13. Coursol, A., & Wagner, E. E. (1986). Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias. Professional Psychology: Research and Practice, 17, 136-137.
14. Cuijpers, P., Smit, F., Bohlmeijer, E., Hollon, S. D., & Andersson, G. (2010). Efficacy of cognitive-behavioral therapy and other psychological treatments for adult depression: Meta-analytic study of publication bias. The British Journal of Psychiatry, 196, 173-178.
15. Dewald, W. G., Thursby, J. G., & Anderson, R. G. (1986). Replication in Empirical Economics: The Journal of Money, Credit and Banking Project. The American Economic Review, 76, 587-603.
16. Dickersin, K., Chan, S., Chalmers, T. C., Sacks, H. S., & Smith, H. (1987). Publication bias and clinical trials. Controlled clinical trials, 8, 343-353.
17. Dickersin, K., Min, Y.-I., & Meinert, C. L. (1992). Factors influencing publication of research results: Follow-up of applictions submitted to two institutional review boards. Journal of the American Mecial Assocaition, 267, 374-378.
18. Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011). Crowdsourcing systems on the WorldWide Web. Communications of the ACM, 54(4), 86-96.
19. Dwan, K., Altman, D. G., Arnaiz, J. a., Bloom, J., Chan, A.-W., Cronin, E., et al. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PloS one, 3, e3081.
20. Easterbrook, P. J., Berlin, J. A., Gopalan, R., & Matthews, D. R. (1991). Publication bias in clinical research. The Lancet, 337, 867-872.
21. Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70, 193-242.
22. Ernst, C., & Angst, J. (1983). Birth Orer: Its Influence on Personality. New York: Springer- Verlag.
23. Eysenck, H. J., & Eysenck, S. B. (1992). Peer review: Advice to referees and contributors. Personality and Individual Differences, 13(4), 393-399.
24. Feller, W. (1940). Statistical aspects of ESP. Journal of Parapsychology, 4, 271-298.
25. Ferguson, C. J., & Kilburn, J. (2010). Much Ado About Nothing: The Misestimation and Overinterpretation of Violent Video Game Effects in Eastern and Western Nations: Comment on Anderson et al. (2010). Psychological Bulletin, 5-9.
26. Field, M., Munafo, M. R., & Franken, I. H. A. (2009). A meta-analytic inverstigation of the relationship between attentional bias and subjective craving in substance abuse. Psychological Bulletin, 135(4), 589-607.
27. Flint, J., & Munafo, M. R. (2009). Replication and heterogeneity in gene x environment interaction studies. International Journal of Neuropsychopharmacology, 727-729.
28. Franklin, M., Kossmann, D., Kraska, T., Ramesh, S., & Xin, R. (2011). CrowdDB: Answering queries with crowdsourcing. Paper presented at the SIGMOD 2011.
29. Gehr, B. T., Weiss, C., & Porzsolt, F. (2006). The fading of reported effectiveness. A metaanalysis of randomised controlled trials. BMC medical research methodology, 6, 25.
30. Gerberg, A. S., Green, D. P., & Nickerson, D. (2001). Testing for publication bias in political science. Political Analysis, 9(4), 385-392.
31. Giles (2005). Internet encyclopedias go head to head. Nature, 438, 900-901.
32. Hardin, G. (1968). The Tragedy of the Commons. Science, 162(3859), 1243-1248.
33. Harris, J. R. (1998). The Nurture Assumption: Why Children Turn out the Way that They Do. New York: Free Press.
34. Hartshorne, J. K., Salem-Hartshorne, N., & Hartshorne, T. S. (2009). Birth order effects in the formation of long-term relationships. Journal of Individual Psychology, 65(2), 156-176.
35. Hirschhorn, J. N., Lohmueller, K. E., Byrne, E., & Hirschhorn, K. (2002). A comprehensive review of genetic association studies. Genetics in Medicine, 4, 45-61.
36. Hubbard, R., Vetter, D. E., & Little, E. L. (1998). Replication in strategic management: scientific testing for validity, generalizability, and usefulness. Strategic Management Journal, 19, 243-254.
37. Ioannidis, J. P. A. (2005a). Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association, 294(2), 218-228.
38. Ioannidis, J. P. A. (2005b). Why most published research findings are false. PLoS Medicine, 2(8), e124.
39. Ioannidis, J. P. A., Ntzani, E. E., Trikalinos, T. a., & Contopoulos-Ioannidis, D. G. (2001). Replication validity of genetic association studies. Nature genetics, 29, 306-309.
40. Ioannidis, J. P. A., Trikalinos, T. A., Ntzani, E. E., & Contopoulos-Ioannidis, D. G. (2003). Genetic associations in large versus small studies: An empirical assessment. The Lancet, 361(9357), 567-571.
41. Jaeger, T. F. (2008). Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models. Journal of Memory and Language, 59, 434-446.
42. Jennions, M. D., & Moller, A. P. (2002a). Publication bias in ecology and evolution: an empirical assessment using the 'trim and fill' method. Biological reviews of the Cambridge Philosophical Society, 77, 211-222.
43. Jennions, M. D., & M0ller, A. P. (2002b). Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution. Proceedings. Biological sciences/The Royal Society, 269, 43-48.
44. Kosciulek, J. F., & Szymanski, E. M. (1993). Statistical power analysis of rehabilitation counseling research. Rehabilitation Counseling Bulletin, 36(4), 212-219.
45. Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009). Circular analysis in systems neuroscience: The dangers of double dipping. Nature Neuroscience, 12(5), 535-540.
46. Kristensen, P., & Bjerkedal, T. (2007). Explaining the relation between birth order and intelligence. Science, 316, 1717.
47. Law, E., von Ahn, L., Dannenberg, R., & Crawford, M. (2007). TagATune: A game for sound and music annotaiton. Paper presented at the iSMIR.
48. Lohmueller, K. E., Pearce, C. L., Pike, M., Lander, E. S., & Hirschhorn, J. N. (2003). Metaanalysis of genetic association studies supports a contribution of common variants to susceptibility to common disease. Nature genetics, 33, 177-182.
49. Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognative Therapy and Research, 1(2), 161-175.
50. Misakian, A. L., & Bero, L. A. (1998). On passive smoking: Comparison of published and unpublished studies. Journal of the American Medical Association, 280, 250-253.
51. Mone, M. A., Mueller, G. C., & Mauland, W. (1996). The perceptions and usage of statistical power in applied psychology and management research. Personnel Psychology, 49, 103-120.
52. Moonesinghe, R., Khoury, M. J., & Janssens, C. J. W. (2007). Most published research findings are false-but a little replication goes a long way. PLoS medicine, 4, e28.
53. Munafo, M. R., Stothart, G., & Flint, J. (2009). Bias in genetic association studies and impact factor. Molecular psychiatry, 14, 119-120.
54. Newton, D. P. (2010). Quality and peer review of research: an adjudicating role for editors. Accountability in research, 17, 130-145.
55. Olson, C. M., Rennie, D., Cook, D., Dickersin, K., Flanagin, A., Hogan, J. W., et al. (2002). Publication bias in editorial decision making. JAMA: the journal of the American Medical Association, 287, 2825-2828.
56. Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5, 187-195.
57. Richard, F. D., Bond, C. F., Jr., & Stokes-Zoota, J. J. (2003). One Hundred Years of Social Psychology Quantitatively Described. Review of General Psychology, 7, 331-363.
58. Saxe, R., Brett, M., & Kanwisher, N. (2006). Divide and conquer: A defense of functional localizers. NeuroImage, 30, 1088-1096.
59. Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effects on the power of studies? Psychological Bulletin, 105(2), 309-316.
60. Sena, E. S., Worp, H. B. V. D., Bath, P. M. W., Howells, D. W., & Macleod, M. R. (2010). Publication Bias in Reports of Animal Stroke Studies Leads to Major Overstatement of Efficacy. Review Literature And Arts Of The Americas, 8.
61. Shen, W., Kiger, T. B., Davies, S. E., Rasch, R. L., Simon, K. M., & Ones, D. S. (in press). Samples in applied psychology: Over a decade of research in a review. Journal of Applied Psychology.
62. Snow, R., O'Conner, B., Jurafsky, D., & Ng, A. Y. (2008). Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, 254-263.
63. Taubes, G., & Mann, C. C. (1995). Epidemiology faces its limits. Science, 269(5221), 164-169.
64. Trikalinos, T. A., Ntzani, E. E., Contopoulos-Ioannidis, D. G., & loannidis, J. P. A. (2004). Establishment of genetic associations for complex diseases is independent of early study findings. European Journal of Human Genetics, 12, 762-769.
65. Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76(2), 105-110.
66. von Ahn, L., & Dabbish, L. (2008). General techniques for designing games with a purpose. Communications of the ACM, 51(8), 58-67.
67. Von Ahn, L., Maurer, B., McMillen, C., Abraham, D., & Blum, M. (2008). Recaptcha: Human- based character recognition via web security measures. Science, 321(5895), 1465-1468.
68. Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4, 274-290.
69. Wagenmakers, E. J., Wetzels, R., Borsboom, D., & Van, H. (in press). Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi. Psychology, 1-14.
70. Wager, T. D., Lindquist, M., & Kaplan, L. (2007). Meta-analysis of functional neuroimaging data: Current and future directions. Social Cognitive and Affective Neuroscience, 2(2), 150-158.
71. Wager, T. D., Lindquist, M. A., Nichols, T. E., Kober, H., & van Snellenberg, J. X. (2009). Evaluating the consistency and specificity of neuroimaging data using meta-analysis. NeuroImage, 45, S210-S221.
72. Welinder, P., Branson, S., Belongie, S., & Perona, P. (2010). The multidimensional wisdom of the crowds. Paper presented at the Advances in Neural Information Processing Systems 2010.
73. Yan, T., Kumar, V., & Ganesan, D. (2010). CrowdSearch: Exploiting crowds for accurate realtime image search on mobile phones. Proceedings of the 8th International Conference on Mobile Systems, Applications, and Services.
74. Yarkoni, T. (2009). Big Correlations in Little Studies: Inflated fMRI Correlations Reflect Low Statistical Power-Commentary on Vul et al. (2009). Perspectives on Psychological Science, 4, 294-298.
75. Yarkoni, T., & Braver, T. S. (2010). Cognitive neuroscience approaches to individual differences in working memory and executive control: Conceptual and methodological issues. In A. Gruzka, G. Matthews & B. Szymura (Eds.), Handbook of Individual Differences in Cognition: Attention, Memory, and Executive Control (pp. 87-108). New York: Springer.