Evidence of experimental bias in the life sciences: Why we need blind data recording

PLoS Biology

Volumn 13, Issue 7, 2015, Pages

  Holman, Luke ^a   Head, Megan L ^a   Lanfear, Robert ^a,b   Jennions, Michael D ^a  

^a AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

^b MACQUARIE UNIVERSITY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; BIOMEDICINE; CLINICAL PROTOCOL; COMPARATIVE STUDY; DATA ANALYSIS; DATA MINING; EFFECT SIZE; EVIDENCE BASED PRACTICE; HUMAN; INFORMATION PROCESSING; MEDICAL RESEARCH; NONHUMAN; OBSERVER BIAS; PREDICTION; BIOLOGY; STANDARDS; STATISTICS AND NUMERICAL DATA;

BIOLOGY; DATA COLLECTION; DATA MINING;

EID: 84938709675     PISSN: 15449173     EISSN: 15457885     Source Type: Journal    
DOI: 10.1371/journal.pbio.1002190     Document Type: Article

References (37)

1. Nickerson RS (1998) Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology 2: 175–220.
2. Rosenthal R (1966) Experimenter Effects in Behavioral Research. East Norwalk, CT: Appleton-Century- Crofts.
3. Rosenthal R (2009) Artifacts in Behavioral Research. Oxford: Oxford University Press.
4. Rosenthal R (1994) Interpersonal expectancy effects: A 30-year perspective. Current Directions in Psychological Science 3: 176–179.
5. Schulz KF, Grimes DA (2002) Blinding in randomised trials: hiding who got what. Lancet 359: 696–700. PMID: 11879884
6. Kilkenny C, Parsons N, Kadyszewski E, Festing MFW, Cuthill IC, et al. (2009) Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS ONE 4: e7824. doi: 10.1371/journal.pone.0007824 PMID: 19956596
7. Rosenthal R, Fode KL (1963) The effect of experimenter bias on the performance of the albino-rat. Behavioral Science 8: 183–189.
8. Hróbjartsson A, Skou Thomsen AS, Emanuelsson F, Tendal B, Hilden J, et al. (2012) Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and nonblinded outcome assessors. BMJ 344: e1119. doi: 10.1136/bmj.e1119 PMID: 22371859
9. Hróbjartsson A, Thomsen ASS, Emanuelsson F, Tendal B, Hilden J, et al. (2013) Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors. Canadian Medical Association Journal 185: E201–E211. doi: 10.1503/cmaj.120744 PMID: 23359047
10. Hróbjartsson A, Thomsen ASS, Emanuelsson F, Tendal B, Rasmussen JV, et al. (2014) Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. Int J Epidemiol 43: 937–948. doi: 10.1093/ije/dyt270 PMID: 24448109
11. Savović J, Jones HE, Altman DG, Harris RJ, Jüni P, et al. (2012) Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 157: 429–438. PMID: 22945832
12. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, et al. (2008) Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 336: 601–605. doi: 10.1136/bmj.39465.451748.AD PMID: 18316340
13. Schulz KF, Chalmers I, Hayes RJ, Altman DG (1995) Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273: 408–412. PMID: 7823387
14. Colditz GA, Miller JN, Mosteller F (1989) How study design affects outcomes in comparisons of therapy. I: Medical. Stat Med 8: 441–454. PMID: 2727468
15. Bello S, Krogsbøll LT, Gruber J, Zhao ZJ, Fischer D, et al. (2014) Lack of blinding of outcome assessors in animal model experiments implies risk of observer bias. J Clin Epidemiol 67: 973–983. doi: 10.1016/j.jclinepi.2014.04.008 PMID: 24972762
16. Hirst JA, Howick J, Aronson JK, Roberts N, Perera R, et al. (2014) The need for randomization in animal trials: an overview of systematic reviews. PLoS ONE 9: e98856. doi: 10.1371/journal.pone.0098856 PMID: 24906117
17. Crossley NA, Sena E, Goehler J, Horn J, van der Worp B, et al. (2008) Empirical evidence of bias in the design of experimental stroke studies: a metaepidemiologic approach. Stroke 39: 929–934. doi: 10.1161/STROKEAHA.107.498725 PMID: 18239164
18. Macleod MR, van der Worp HB, Sena ES, Howells DW, Dirnagl U, et al. (2008) Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke 39: 2824–2829. doi: 10.1161/STROKEAHA.108.515957 PMID: 18635842
19. Burghardt GM, Bartmess-LeVasseur JN, Browning SA, Morrison KE, Stec CL, et al. (2012) Minimizing observer bias in behavioral studies: a review and recommendations. Ethology 118: 511–517.
20. van Luijk J, Bakker B, Rovers MM, Ritskes-Hoitinga M, de Vries RBM, et al. (2014) Systematic reviews of animal studies; missing link in translational research? PLoS ONE 9: e89981. doi: 10.1371/journal. pone.0089981 PMID: 24670965
21. Bebarta V, Luyten D, Heard K (2003) Emergency medicine animal research: does use of randomization and blinding affect the results? Acad Emerg Med 10: 684–687. PMID: 12782533
22. van Wilgenburg E, Elgar MA (2013) Confirmation bias in studies of nestmate recognition: a cautionary note for research into the behaviour of animals. PLoS ONE 8: e53548. doi: 10.1371/journal.pone. 0053548 PMID: 23372659
23. Benignus VA (1993) Importance of experimenter-blind procedure in neurotoxicology. Neurotoxicol Teratol 15: 45–49. PMID: 8459788
24. Tuyttens F, de Graaf S, Heerkens J, Jacobs L (2014) Observer bias in animal behaviour research: can we believe what we score, if we score what we believe? Anim Behav 90: 273–280.
25. Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2: e124. PMID: 16060722
26. Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD (2015) The extent and consequences of phacking in science. PLoS Biol 13: e1002106. doi: 10.1371/journal.pbio.1002106 PMID: 25768323
27. John LK, Loewenstein G, Prelec D (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 23: 524–532. doi: 10.1177/0956797611430953 PMID: 22508865
28. Simonsohn U, Nelson LD, Simmons JP (2014) p-curve and effect size: Correcting for publicaiton bias using only significant results. Perspectives on Psychological Science 9: 666–681.
29. Cohen J (1988) Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Erlbaum.
30. Slatyer RA, Mautz BS, Backwell PRY, Jennions MD (2012) Estimating genetic benefits of polyandry from experimental studies: a meta-analysis. Biol Rev 87: 1–33. doi: 10.1111/j.1469-185X.2011.00182.x PMID: 21545390
31. Uller T, Nakagawa S, English S (2013) Weak evidence for anticipatory parental effects in plants and animals. J Evol Biol 26: 2161–2170. doi: 10.1111/jeb.12212 PMID: 23937440
32. Prokop ZM, Michalczyk Ł, Drobniak SM, Herdegen M, Radwan J (2012) Meta-analysis suggests choosy females get sexy sons more than "good genes". Evolution 66: 2665–2673. doi: 10.1111/j.1558-5646.2012.01654.x PMID: 22946794
33. Trikalinos TA, Ioannidis J (2005) Assessing the evolution of effect sizes over time. Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. Chichester, UK: Wiley. pp. 241–259.
34. Jennions MD, Møller AP (2002) Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution. Proc Biol Sci 269: 43–48. PMID: 11788035
35. Jennions MD, Lortie CJ, Rosenberg MS, Rothstein HR (2013) Publication and related biases. Handbook of Meta-analysis in Ecology and Evolution. Princeton: Princeton University Press. pp. 207–236.
36. Low-Décarie E, Chivers C, Granados M (2014) Rising complexity and falling explanatory power in ecology. Frontiers in Ecology and the Environment 12: 412–418.
37. Gelman A (2008) Scaling regression inputs by dividing by two standard deviations. Stat Med 27: 2865–2873. PMID: 17960576