Instrumentation bias in the use and evaluation of scientific software: Recommendations for reproducible practices in the computational sciences

Frontiers in Neuroscience

Volumn , Issue 7 SEP, 2013, Pages

  Tustison, Nicholas J ^a   Johnson, Hans J ^b   Rohlfing, Torsten ^c   Klein, Arno ^d   Ghosh, Satrajit S ^e   Ibanez, Luis ^f   Avants, Brian B ^g  

^a UNIVERSITY OF VIRGINIA   (United States)

^b UNIVERSITY OF IOWA   (United States)

^c SRI INTERNATIONAL   (United States)

^d STONY BROOK UNIVERSITY   (United States)

^e MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^f KITWARE INC   (United States)

^g UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

Best practices; Comparative evaluations; Confirmation bias; Open science; Reproducibility

Indexed keywords

ALGORITHM; ARTICLE; COMPARATIVE STUDY; DATA PROCESSING; EVALUATION STUDY; HUMAN; IMAGING SOFTWARE; NEUROIMAGING; PRACTICE GUIDELINE; REPRODUCIBILITY;

EID: 84891080632     PISSN: 16624548     EISSN: 1662453X     Source Type: Journal    
DOI: 10.3389/fnins.2013.00162     Document Type: Article

References (32)

1. Anonymous. (2007). Who is accountable? Nature 450:1. doi: 10.1038/450001a
2. Ashburner, J. (2012). SPM: a history. Neuroimage 62, 791-800. doi: 10.1016/j.neuroimage.2011.10.025
3. Avants, B. B., Tustison, N. J., Song, G., Cook, P. A., Klein, A., and Gee, J. C. (2011). A reproducible evaluation of ANTs similarity metric performance in brain image registration. Neuroimage 54, 2033-2044. doi: 10.1016/j.neuroimage.2010.09.025
4. Avants, B. B., Yushkevich, P., Pluta, J., Minkoff, D., Korczykowski, M., Detre, J., et al. (2010). The optimal template effect in hippocampus studies of diseased populations. Neuroimage 49, 2457-2466. doi: 10.1016/j.neuroimage.2009.09.062
5. Clarkson, M. J., Cardoso, M. J., Ridgway, G. R., Modat, M., Leung, K. K., Rohrer, J. D., et al. (2011). A comparison of voxel and surface based cortical thickness estimation methods. Neuroimage 57, 856-865. doi: 10.1016/j.neuroimage.2011.05.053
6. Cox, R. W. (2012). AFNI: what a long strange trip it's been. Neuroimage 62, 743-747. doi: 10.1016/j.neuroimage.2011.08.056
7. Eskildsen, S. F., Coupé, P., Fonov, V., Manjó, J. V., Leung, K. K., Guizard, N., et al. (2012). BEaST: brain extraction based on nonlocal segmentation technique. Neuroimage 59, 2362-2373. doi: 10.1016/j.neuroimage.2011.09.012
8. Fedorov, A., Beichel, R., Kalpathy-Cramer, J., Finet, J., Fillion-Robin, J.-C., Pujol, S., et al. (2012). 3D slicer as an image computing platform for the quantitative imaging network. Magn. Reson. Imaging 30, 1323-1341. doi: 10.1016/j.mri.2012.05.001
9. Fischl, B. (2012). FreeSurfer. Neuroimage 62, 774-781. doi: 10.1016/j.neuroimage.2012.01.021
10. Gorgolewski, K., Burns, C. D., Madison, C., Clark, D., Halchenko, Y. O., Waskom, M. L., et al. (2011). Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in python. Front. Neuroinform. 5:13. doi: 10.3389/fninf.2011.00013
11. Gronenschild, E. H. B. M., Habets, P., Jacobs, H. I. L., Mengelers, R., Rozendaal, N., van Os, J., et al. (2012). The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements. PLoS ONE 7:e38234. doi: 10.1371/journal.pone.0038234
12. Haegelen, C., Coupé, P., Fonov, V., Guizard, N., Jannin, P., Morandi, X., et al. (2013). Automated segmentation of basal ganglia and deep brain structures in MRI of Parkinson's disease. Int. J. Comput. Assist. Radiol. Surg. 8, 99-110. doi: 10.1007/s11548-012-0675-8
13. Ibanez, L., Audette, M., Yeo, B. T., and Goland, P. (2009). Spherical demons registration of spherical surfaces. Insight J. Available online at: http://hdl.handle.net/10380/3117
14. Jenkinson, M., Beckmann, C. F., Behrens, T. E. J., Woolrich, M. W., and Smith, S. M. (2012). FSL. Neuroimage 62, 782-790. doi: 10.1016/j.neuroimage.2011.09.015
15. Johnson, H. J., Harris, G., and Williams, K. (2007). BRAINSFit: mutual information registrations of whole-brain 3D images, using the insight toolkit. Insight J. Available online at: http://hdl.handle.net/1926/1291
16. Klein, A., Andersson, J., Ardekani, B. A., Ashburner, J., Avants, B., Chiang, M.-C., et al. (2009). Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. Neuroimage 46, 786-802. doi: 10.1016/j.neuroimage.2008.12.037
17. Klein, S., Staring, M., Murphy, K., Viergever, M. A., and Pluim, J. P. W. (2010). elastix: a toolbox for intensity-based medical image registration. IEEE Trans. Med. Imaging 29, 196-205. doi: 10.1109/TMI.2009.2035616
18. Kovacevic, J. (2006). From the editor in chief. IEEE Trans. Image Process. 15, 12.
19. Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., and Baker, C. I. (2009). Circular analysis in systems neuroscience: the dangers of double dipping. Nat. Neurosci. 12, 535-540. doi: 10.1038/nn.2303
20. Merali, Z. (2010). Computational science: error, why scientific programming does not compute. Nature 467, 775-777. doi: 10.1038/467775a
21. Prlic, A., and Procter, J. B. (2012). Ten simple rules for the open development of scientific software. PLoS Comput. Biol. 8:e1002802. doi: 10.1371/journal.pcbi.1002802
22. Rohlfing, T. (2012). Image similarity and tissue overlaps as surrogates for image registration accuracy: widely used but unreliable. IEEE Trans. Med. Imaging 31, 153-163. doi: 10.1109/TMI.2011.2163944
23. Sackett, D. L. (1979). Bias in analytic research. J. Chronic. Dis. 32, 51-63. doi: 10.1016/0021-9681(79)90012-2
24. Shattuck, D. W., Prasad, G., Mirza, M., Narr, K. L., and Toga, A. W. (2009). Online resource for validation of brain segmentation methods. Neuroimage 45, 431-439. doi: 10.1016/j.neuroimage.2008.10.066
25. Tustison, N. J., and Avants, B. B. (2012). The TVDMFFDVR algorithm. Insight J. Available online at: http://hdl.handle.net/10380/3334
26. Tustison, N. J., Avants, B. B., Cook, P. A., Kim, J., Whyte, J., Gee, J. C., et al. (2012). Logical circularity in voxel-based analysis: normalization strategy may induce statistical bias. Hum. Brain Mapp. doi: 10.1002/hbm.22211. [Epub ahead of print].
27. Tustison, N. J., Avants, B. B., Cook, P. A., Zheng, Y., Egan, A., Yushkevich, P. A., et al. (2010). N4ITK: improved N3 bias correction. IEEE Trans. Med. Imaging 29, 1310-1320. doi: 10.1109/TMI.2010.2046908
28. Tustison, N. J., and Gee, J. C. (2009). N4ITK: Nick's N3 ITK implementation for MRI bias field correction. Technical report, University of Pennsylvania. Available online at: http://www.insight-journal.org/browse/publication/640
29. Vul, E., and Pashler, H. (2012). Voodoo and circularity errors. Neuroimage 62, 945-948. doi: 10.1016/j.neuroimage.2012.01.027
30. Wolpert, D., and Macready, W. G. (1997). No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1, 67-82. doi: 10.1109/4235.585893
31. Wu, G., Kim, M., Wang, Q., and Shen, D. (2013). S-HAMMER: hierarchical attribute-guided, symmetric diffeomorphic registration for MR brain images. Hum. Brain Mapp. doi: 10.1002/hbm.22233. [Epub ahead of print].
32. Yushkevich, P. A., Avants, B. B., Das, S. R., Pluta, J., Altinay, M., Craige, C., et al. (2010). Bias in estimation of hippocampal atrophy using deformation-based morphometry arises from asymmetric global normalization: an illustration in ADNI 3 T MRI data. Neuroimage 50, 434-445. doi: 10.1016/j.neuroimage.2009.12.007