Tracking replicability as a method of post-publication open evaluation

Frontiers in Computational Neuroscience

Volumn , Issue MARCH 2012, 2012, Pages 1-14

  Hartshorne, Joshua K ^a   Schachner, Adena ^a  

^a HARVARD UNIVERSITY   (United States)

Author keywords

Open evaluation; Post publication evaluation; Replicability; Replication

Indexed keywords

CURRENT SYSTEM; DATA QUALITY; INCENTIVE STRUCTURE; LAY-OUT; OPEN ACCESS; OPEN EVALUATION; PEER REVIEW; POST-PUBLICATION EVALUATION; QUALITY OF WORK; REPLICABILITY; REPLICATION; RESEARCH COMMUNITIES;

DATABASE SYSTEMS; PUBLISHING;

QUALITY CONTROL;

ANALYTICAL ERROR; ARTICLE; CONTENT ANALYSIS; DATA ANALYSIS; DATA COLLECTION METHOD; DATA EXTRACTION; DOCUMENTATION; EVALUATION RESEARCH; HUMAN; INFORMATION RETRIEVAL; INFORMATION STORAGE; INFORMATION SYSTEM; MACHINE LEARNING; MATHEMATICAL ANALYSIS; MEDICAL LITERATURE; MEDICAL RESEARCH; PRACTICE GUIDELINE; PUBLISHING; QUALITATIVE RESEARCH; QUALITY CONTROL; QUANTITATIVE ANALYSIS; REPRODUCIBILITY; SAMPLE SIZE; SYSTEMATIC ERROR;

EID: 84857987676     PISSN: 16625188     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (81)

1. Adamic, L. A., Zhang, J., Bakshy, E., and Ackerman, M. S. (2008). "Knowledge sharing and yahoo answers: everyone knows something," in Proceedings of the 17th International Conference on World Wide Web, Beijing.
2. Albert, P. S., and Dodd, L. E. (2004). A cautionary note on the robustness of latent class models for estimating diagnostic error without a gold standard. Biometrics 60, 427-435.
3. Armitage, P., McPherson, C. K., and Rowe, B. C. (1969). Repeated significance tests on accumulating data. J. R. Stat. Soc. Ser. A Stat. Soc. 132, 235-244.
4. Baayen, R. H., Davidson, D. J., and Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. J. Mem. Lang. 59, 390-412.
5. Bederson, B. B., Hu, C., and Resnik, P. (2010). "Translation by interactive collaboration between monolingual users," in Proceedings of Graphics Interface, Ottawa, 39-46.
6. Bezeau, S., and Graves, R. (2001). Statistical power and effect sizes of clinical neuropsychology research. J. Clin. Exp. Neuropsychol. 23, 399-406.
7. Boffetta, P., Mclaughlin, J. K., Vecchia, C. L., Tarone, R. E., Lipworth, L., and Blot, W. J. (2008). False-positive results in cancer epidemiology: a plea for epistemological modesty. J. Natl. Cancer Inst. 100, 988-995.
8. Callaham, M. L., Wears, R. L., Weber, E. J., Barton, C., and Young, G. (1998). Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. JAMA 280, 254-257.
9. Callison-Burch, C. (2009). "Fast, cheap, and creative: evaluating translation quality using Amazon's mechanical turk," in Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, Singapore, 286-295.
10. Chase, L. J., and Chase, R. B. (1976). A statistical power analysis of applied psychological research. J. Appl. Psychol. 61, 234-237.
11. Cohen, J. (1962). The statistical power of abnormal-social psychological research. J. Abnorm. Soc. Psychol. 65, 145-153.
12. Cole, S. Jr., Cole, J. R., and Simon, G. A. (1981). Chance and consensus in peer review. Science 214, 881-886.
13. Coursol, A., and Wagner, E. E. (1986). Effect of positive findings on submission and acceptance rates: a note on meta-analysis bias. Prof. Psychol. Res. Pr. 17, 136-137.
14. Cuijpers, P., Smit, F., Bohlmeijer, E., Hollon, S. D., and Andersson, G. (2010). Efficacy of cognitive-behavioral therapy and other psychological treatments for adult depression: meta-analytic study of publication bias. Br. J. Psychiatry 196, 173-178.
15. Dewald, W. G., Thursby, J. G., and Anderson, R. G. (1986). Replication in empirical economics: the journal of money, credit and banking project. Am. Econ. Rev. 76, 587-603.
16. Dickersin, K., Chan, S., Chalmers, T. C., Sacks, H. S., and Smith, H. (1987). Publication bias and clinical trials. Control. Clin. Trials 8, 343-353.
17. Dickersin, K., Min, Y.-I., and Meinert, C. L. (1992). Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. J. Am. Med. Assoc. 267, 374-378.
18. Doan, A., Ramakrishnan, R., and Halevy, A. Y. (2011). Crowdsourcing systems on the world-wide web. Commun. ACM 54, 86-96.
19. Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., Ghersi, D., Ioannidis, J. P., Simes, J., and Williamson, P. R. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3, e3081. doi:10.1371/journal.pone.0003081
20. Easterbrook, P. J., Berlin, J. A., Gopalan, R., and Matthews, D. R. (1991). Publication bias in clinical research. Lancet 337, 867-872.
21. Edwards, W., Lindman, H., and Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychol. Rev. 70, 193-242.
22. Ernst, C., and Angst, J. (1983). Birth Order: Its Influence on Personality. New York: Springer-Verlag.
23. Eysenck, H. J., and Eysenck, S. B. (1992). Peer review: advice to referees and contributors. Pers. Individ. Dif. 13, 393-399.
24. Feller, W. (1940). Statistical aspects of ESP. J. Parapsychol. 4, 271-298.
25. Ferguson, C. J., and Kilburn, J. (2010). Much ado about nothing: the misestimation and overinterpretation of violent video game effects in eastern and western nations: comment on Anderson et al. (2010). Psychol. Bull. 136, 174-178.
26. Field, M., Munafo, M. R., and Franken, I. H. A. (2009). A meta-analytic investigation of the relationship between attentional bias and subjective craving in substance abuse. Psychol. Bull. 135, 589-607.
27. Flint, J., and Munafo, M. R. (2009). Replication and heterogeneity in gene x environment interaction studies. Int. J. Neuropsychopharmacol. 12, 727-729.
28. Franklin, M., Kossmann, D., Kraska, T., Ramesh, S., and Xin, R. (2011). CrowdDB: answering queries with crowdsourcing. Paper Presented at the SIGMOD 2011, Athens.
29. Gehr, B. T., Weiss, C., and Porzsolt, F. (2006). The fading of reported effectiveness. A meta-analysis of randomised controlled trials. BMC Med. Res. Methodol. 6, 25. doi:10.1186/1471-2288-6-25
30. Gerberg, A. S., Green, D. P., and Nickerson, D. (2001). Testing for publication bias in political science. Polit. Anal. 9, 385-392.
31. Giles, J. (2005). Internet encyclopedias go head to head. Nature 438, 900-901.
32. Hardin, G. (1968). The tragedy of the commons. Science 162, 1243-1248.
33. Harris, J. R. (1998). The Nurture Assumption: Why Children Turn out the Way That They Do. New York: Free Press.
34. Hartshorne, J. K., Salem-Hartshorne, N., and Hartshorne, T. S. (2009). Birth order effects in the formation of long-term relationships. J. Individ. Psychol. 65, 156-176.
35. Healy, D. G., Abou-Sleiman, P. M., Casas, J. P., Ahmadi, K. R., Lynch, T., Gandhi, S., Muqit, M. M., Foltynie, T., Barker, T., Bhatia, K. P., Quinn, N. P., Lees, A. J., Gibson, J. M., Holton, J. L., Revesz, T., Goldstein, D. B., and Wood, N. W. (2006). UCHL1 is not a Parksinon's disease susceptibility gene. Ann. Neurol. 59, 627-633.
36. Hirschhorn, J. N., Lohmueller, K. E., Byrne, E., and Hirschhorn, K. (2002). A comprehensive review of genetic association studies. Genet. Med. 4, 45-61.
37. Hubbard, R., Vetter, D. E., and Little, E. L. (1998). Replication in strategic management: scientific testing for validity, generalizability, and usefulness. Strateg. Manage. J. 19, 243-254.
38. Ioannidis, J. P. A. (2005a). Contradicted and initially stronger effects in highly cited clinical research. J. Am. Med. Assoc. 294, 218-228.
39. Ioannidis, J. P. A. (2005b). Why most published research findings are false. PLoS Med. 2, e124. doi:10.1371/journal.pmed.0020124
40. Ioannidis, J. P. A. (2011). Meta-research: the art of getting it wrong. Res. Syn. Methods 1, 169-184.
41. Ioannidis, J. P. A., Ntzani, E. E., Trikalinos, T. A., and Contopoulos-Ioannidis, D. G. (2001). Replication validity of genetic association studies. Nat. Genet. 29, 306-309.
42. Ioannidis, J. P. A., Trikalinos, T. A., Ntzani, E. E., and Contopoulos-Ioannidis, D. G. (2003). Genetic associations in large versus small studies: an empirical assessment. Lancet 361, 567-571.
43. Jaeger, T. F. (2008). Categorical data analysis: away from ANOVAs (transformation or not) and towards logit mixed models. J. Mem. Lang. 59, 434-446.
44. Jennions, M. D., and Møller, A. P. (2002a). Publication bias in ecology and evolution: an empirical assessment using the "trim and fill" method. Biol. Rev. Camb. Philos. Soc. 77, 211-222.
45. Jennions, M. D., and Møller, A. P. (2002b). Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution. Proc. Biol. Sci. 269, 43-48.
46. Kosciulek, J. F., and Szymanski, E. M. (1993). Statistical power analysis of rehabilitation counseling research. Rehabil. Couns. Bull. 36, 212-219.
47. Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., and Baker, C. I. (2009). Circular analysis in systems neuroscience: the dangers of double dipping. Nat. Neurosci. 12, 535-540.
48. Kristensen, P., and Bjerkedal, T. (2007). Explaining the relation between birth order and intelligence. Science 316, 1717.
49. Law, E., von Ahn, L., Dannenberg, R., and Crawford, M. (2007). TagATune: a game for sound and music annotation. Paper Presented at the ISMIR, Vienna.
50. Lerner, J. S., and Gonzalez, R. M. (2005). Forecasting one's future based on fleeting subjective experiences. Pers. Soc. Psychol. B. 31, 454-466.
51. Lohmueller, K. E., Pearce, C. L., Pike, M., Lander, E. S., and Hirschhorn, J. N. (2003). Meta-analysis of genetic association studies supports a contribution of common variants to susceptibility to common disease. Nat. Genet. 33, 177-182.
52. Mahoney, M. J. (1977). Publication prejudices: an experimental study of confirmatory bias in the peer review system. Cognit. Ther. Res. 1, 161-175.
53. Maraganore, D. M., Lesnick, T. G., Elbaz, A., Chartier-Harlin, M.-C., Gasser, T., Kruger, R., Hattori, N., Mellick, G. K., Quattrone, A., Satoh, J.-I., Toda, T., Wang, J., Ioannidis, J. P. A., de Andrade, M., Rocca, W. A., ad the UCHL1 Global Genetics Consortium. (2004). UCHL1 is a Parkinson's disease susceptibility gene. Ann. Neurol. 55, 512-521.
54. Misakian, A. L., and Bero, L. A. (1998). On passive smoking: comparison of published and unpublished studies. J. Am. Med. Assoc. 280, 250-253.
55. Mone, M. A., Mueller, G. C., and Mauland, W. (1996). The perceptions and usage of statistical power in applied psychology and management research. Pers. Psychol. 49, 103-120.
56. Moonesinghe, R., Khoury, M. J., and Janssens, C. J. W. (2007). Most published research findings are false-but a little replication goes a long way. PLoS Med. 4, e28. doi:10.1371/journal.pmed.0040028
57. Munafò, M. R., Stothart, G., and Flint, J. (2009). Bias in genetic association studies and impact factor. Mol. Psychiatry 14, 119-120.
58. Newton, D. P. (2010). Quality and peer review of research: an adjudicating role for editors. Account. Res. 17, 130-145.
59. Olson, C. M., Rennie, D., Cook, D., Dickersin, K., Flanagin, A., Hogan, J. W., Zhu, Q., Reiling, J., and Pace, B. (2002). Publication bias in editorial decision making. JAMA 287, 2825-2828.
60. Peters, D. P., and Ceci, S. J. (1982). Peer-review practices of psychological journals: the fate of published articles, submitted again. Behav. Brain Sci. 5, 187-195.
61. Richard, F. D., Bond, C. F. Jr., and Stokes-Zoota, J. J. (2003). One hundred years of social psychology quantitatively described. Rev. Gen. Psychol. 7, 331-363.
62. Ruggiero, K. M., and Marx, D. M. (1999). Less pain and more to gain: why high-status group members blame their failure on discrimination. J. Pers. Soc. Psychol. 77, 774-784.
63. Ruggiero, K. M., Steele, J., Hwang, A., and Marx, D. M. (2000). Why did I get a 'D'? The effects of social comparisons on women's attributions to discrimination. Pers. Soc. Psychol. B. 26, 1271-1283.
64. Saxe, R., Brett, M., and Kanwisher, N. (2006). Divide and conquer: a defense of functional localizers. Neuroimage 30, 1088-1096.
65. Sedlmeier, P., and Gigerenzer, G. (1989). Do studies of statistical power have an effects on the power of studies? Psychol. Bull. 105, 309-316.
66. Sena, E. S., Worp, H. B. V. D., Bath, P. M. W., Howells, D. W., and Macleod, M. R. (2010). Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol. 8, e1000344. doi:10.1371/journal.pbio.1000344
67. Shen, W., Kiger, T. B., Davies, S. E., Rasch, R. L., Simon, K. M., and Ones, D. S. (2011). Samples in applied psychology: over a decade of research in a review. J. Appl. Psychol. 96, 1055-1064.
68. Snow, R., O'Conner, B., Jurafsky, D., and Ng, A. Y. (2008). "Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks," in Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing Edinburgh, 254-263.
69. Taubes, G., and Mann, C. C. (1995). Epidemiology faces its limits. Science 269, 164-169.
70. Trikalinos, T. A., Ntzani, E. E., Contopoulos-Ioannidis, D. G., and Ioannidis, J. P. A. (2004). Establishment of genetic associations for complex diseases is independent of early study findings. Eur. J. Hum. Genet. 12, 762-769.
71. Tversky, A., and Kahneman, D. (1971). Belief in the law of small numbers. Psychol. Bull. 76, 105-110.
72. von Ahn, L., and Dabbish, L. (2008). General techniques for designing games with a purpose. Commun. ACM 51, 58-67.
73. von Ahn, L., Maurer, B., McMillen, C., Abraham, D., and Blum, M. (2008). reCAPTCHA: human-based character recognition via web security measures. Science 321, 1465-1468.
74. Vul, E., Harris, C., Winkielman, P., and Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspect. Psychol. Sci. 4, 274-290.
75. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., and van der Maas, H. L. (2011). Why psychologists must change the way they analyze their data: the case of psi: Comment on Bem, 2011. J. Pers. Soc. Psychol. 100, 426-432.
76. Wager, T. D., Lindquist, M., and Kaplan, L. (2007). Meta-analysis of functional neuroimaging data: current and future directions. Soc. Cogn. Affect. Neurosci. 2, 150-158.
77. Wager, T. D., Lindquist, M. A., Nichols, T. E., Kober, H., and van Snellenberg, J. X. (2009). Evaluating the consistency and specificity of neuroimaging data using meta-analysis. Neuroimage 45, S210-S221.
78. Welinder, P., Branson, S., Belongie, S., and Perona, P. (2010). The multidimensional wisdom of the crowds. Paper Presented at the Advances in Neural Information Processing Systems 2010, Vancouver.
79. Yan, T., Kumar, V., and Ganesan, D. (2010). "Crowdsearch: exploiting crowds for accurate real-time image search on mobile phones," in Proceedings of the 8th International Conference on Mobile Systems, Applications, and Services, San Francisco.
80. Yarkoni, T. (2009). Big correlations in little studies: inflated fmri correlations reflect low statistical power-commentary on Vul et al. (2009). Perspect. Psychol. Sci. 4, 294-298.
81. Yarkoni, T., and Braver, T. S. (2010). "Cognitive neuroscience approaches to individual differences in working memory and executive control: conceptual and methodological issues," in Handbook of Individual Differences in Cognition: Attention, Memory, and Executive Control, eds A. Gruzka, G. Matthews, and B. Szymura (New York: Springer), 87-108.