Combining citizen science and deep learning to amplify expertise in neuroimaging

Frontiers in Neuroinformatics

Volumn 13, Issue , 2019, Pages

  Keshavan, Anisha ^a   Yeatman, Jason D ^a   Rokem, Ariel ^a  

^a UNIVERSITY OF WASHINGTON   (United States)

Author keywords

Brain development; Brain imaging; Citizen science (CS); Deep learning (DL); MRI magnetic resonance imaging

Indexed keywords

ALGORITHM; AREA UNDER THE CURVE; ARTICLE; BIG DATA; BRAIN DEVELOPMENT; CITIZEN SCIENTIST; COGNITION; COMPUTER ASSISTED TOMOGRAPHY; CONTRAST TO NOISE RATIO; DATA COLLECTION METHOD; DECISION MAKING; DEEP LEARNING; FUNCTIONAL NEUROIMAGING; HUMAN; IMAGE QUALITY; IMAGE SEGMENTATION; LEARNING; LEARNING ALGORITHM; LIKERT SCALE; MACHINE LEARNING; MEASUREMENT ACCURACY; NERVE CELL NETWORK; NEUROIMAGING; NUCLEAR MAGNETIC RESONANCE IMAGING; QUALITY CONTROL; RATING SCALE; RECEIVER OPERATING CHARACTERISTIC; SCIENTIST; TRAINING;

EID: 85067385565     PISSN: None     EISSN: 16625196     Source Type: Journal    
DOI: 10.3389/fninf.2019.00029     Document Type: Article

References (59)

1. Ahmed, A., Yu, K., Xu, W., Gong, Y., Xing, E., (2008). Training hierarchical feed-forward visual recognition models using transfer learning from pseudo-tasks, in European Conference on Computer Vision (Springer),69–82. 10.1007/978-3-540-88690-76.
2. Alexander, L. M., Escalera, J., Ai, L., Andreotti, C., Febre, K., Mangone, A.,. (2017). An open resource for transdiagnostic research in pediatric mental health and learning disorders. Sci. Data, 4:170181. 10.1038/sdata.2017.181, 29257126.
3. Ashburner, J., Friston, K. J., (2005). Unified segmentation. NeuroImage, 26, 839–851. 10.1016/j.neuroimage.2005.02.018, 15955494.
4. Avants, B. B., Tustison, N. J., Song, G., Cook, P. A., Klein, A., Gee, J. C., (2011). A reproducible evaluation of ANTs similarity metric performance in brain image registration. NeuroImage, 54, 2033–2044. 10.1016/j.neuroimage.2010.09.025, 20851191.
5. Benson, N., Butt, O., Brainard, D., Aguirre, G., (2014). Correction of distortion in flattened representations of the cortical surface allows prediction of V1-V3 functional organization from anatomy. PLoS Comput. Biol., 10:e1003538. 10.1016/504j.neuroimage.2010.09.025, 24676149.
6. Benson, N., Butt, O., Datta, R., Radoeva, P., Brainard, D., Aguirre, G., (2012). The retinotopic organization of striate cortex is well predicted by surface topology. Curr. Biol., 22, 2081–2085. 10.1016/j.cub.2012.09.014, 23041195.
7. Brett, M., Hanke, M., Markiewicz, C., Côté, M.-A., McCarthy, P., Ghosh, S.,. (2018). nipy/nibabel: 2.3.0. Available online at: https://github.com/nipy/nibabel.
8. Casey, B., Cannonier, T., Conley, M. I., Cohen, A. O., Barch, D. M., Heitzeg, M. M.,. (2018). The adolescent brain cognitive development (ABCD) study: imaging acquisition across 21 sites. Dev. Cogn. Neurosci., 32, 43–54. 10.1016/j.dcn.2018.03.001, 29567376.
9. Catani, M., Thiebautdeschotten, M., (2008). A diffusion tensor imaging tractography atlas for virtual in vivo dissections. Cortex, 44, 1105–1132. 10.1016/j.cortex.2008.05.004, 18619589.
10. Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F.,. (2017). Interpretability of deep learning models: a survey of results, in IEEE Smart World Congress 2017 Workshop: DAIS 2017 - Workshop on Distributed Analytics InfraStructure and Algorithms for Multi-Organization Federations (San Francisco, CA). 10.1109/UIC-ATC.2017.8397411.
11. Chen, T., Guestrin, C., (2016). Xgboost: a scalable tree boosting system, in Proceedings of the 22nd Acm Sigkdd International Conference on Knowledge Discovery and Data Mining (San Francisco, CA: ACM), 785–794.
12. Cheng, J., Teevan, J., Iqbal, S. T., Bernstein, M. S., (2015). Break it down: a comparison of macro-and microtasks, in Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul: ACM), 4061–4064.
13. Chollet, F., (2015). Keras. Available online at: https://github.com/keras-team/keras.
14. Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., Ronneberger, O., (2016). 3D U-Net: learning dense volumetric segmentation from sparse annotation, in International Conference on Medical Image Computing and Computer-Assisted Intervention (Athens: Springer), 424–432.
15. Dalmış, M. U., Litjens, G., Holland, K., Setio, A., Mann, R., Karssemeijer, N.,. (2017). Using deep learning to segment breast and fibroglandular tissue in MRI volumes. Med. Phys., 44, 533–546. 10.1002/mp.12079, 28035663.
16. Das, S., Zijdenbos, A. P., Vins, D., Harlap, J., Evans, A. C., (2012). LORIS: a web-based data management system for multi-center studies. Front. Neuroinform., 5:37. 10.3389/fninf.2011.00037, 22319489.
17. Ducharme, S., Albaugh, M. D., Nguyen, T.-V., Hudziak, J. J., Mateos-Pérez, J., Labbe, A.,. (2016). Trajectories of cortical thickness maturation in normal brain development—The importance of quality control procedures. Neuroimage, 125, 267–279. 10.1016/j.neuroimage.2015.10.010, 26463175.
18. Esteban, O., Birman, D., Schaer, M., Koyejo, O. O., Poldrack, R. A., Gorgolewski, K. J., (2017). MRIQC: advancing the automatic prediction of image quality in MRI from unseen sites. PLoS ONE, 12:e0184661. 10.1371/journal.pone.0184661, 28945803.
19. Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M.,. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542:115. 10.1038/nature21056, 28117445.
20. Fan, J., Han, F., Liu, H., (2014). Challenges of big data analysis. Nat. Sci. Rev., 1, 293–314. 10.1093/nsr/nwt032, 25419469.
21. Ferguson, A. R., Nielson, J. L., Cragin, M. H., Bandrowski, A. E., Martone, M. E., (2014). Big data from small data: data-sharing in the ‘long tail’of neuroscience. Nat. Neurosci., 17, 1442–1447. 10.1038/nn.3838, 25349910.
22. Fischl, B., (2012). FreeSurfer. Neuroimage, 62, 774–781. 10.1016/j.neuroimage.2012.01.021, 22248573.
23. Girshick, R., Donahue, J., Darrell, T., Malik, J., (2014). Rich feature hierarchies for accurate object detection and semantic segmentation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Washington, DC), 580–587.
24. Glasser, M. F., Smith, S. M., Marcus, D. S., Andersson, J. L., Auerbach, E. J., Behrens, T. E.,. (2016). The human connectome project's neuroimaging approach. Nat. Neurosci., 19:1175–1187. 10.1038/nn.4361, 27571196.
25. Gorgolewski, K., Esteban, O., Schaefer, G., Wandell, B., Poldrack, R., (2017a). OpenNeuro—a Free Online Platform for Sharing and Analysis of Neuroimaging Data., Vancouver, BC: Organization for Human Brain Mapping.
26. Gorgolewski, K. J., Alfaro-Almagro, F., Auer, T., Bellec, P., Capotă, M., Chakravarty, M. M.,. (2017b). BIDS apps: improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods. PLoS Comput. Biol., 13:e1005209. 10.1371/journal.pcbi.1005209, 28278228.
27. Gorgolewski, K. J., Auer, T., Calhoun, V. D., Craddock, R. C., Das, S., Duff, E. P.,. (2016). The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Sci. Data, 3:160044. 10.1038/sdata.2016.44.27326542.
28. Griffanti, L., Douaud, G., Bijsterbosch, J., Evangelisti, S., Alfaro-Almagro, F., Glasser, M. F.,. (2017). Hand classification of fMRI ICA noise components. Neuroimage, 154, 188–205. 10.1016/j.neuroimage.2016.12.036, 27989777.
29. Heuer, K., Ghosh, S., Sterling, A. R., Toro, R., (2016). Open neuroimaging laboratory. Res. Ideas Outcomes, 2:e9113. 10.3897/rio.2.e9113.
30. Jordan, K. M., Amirbekian, B., Keshavan, A., Henry, R. G., (2017a). Cluster confidence index: a streamline-wise pathway reproducibility metric for diffusion-weighted MRI tractography. J. Neuroimaging, 28, 64–69. 10.1111/jon.12467, 28940825.
31. Jordan, K. M., Keshavan, A., Caverzasi, E., Osorio, J., Papinutto, N., Amirbekian, B.,. (2017b). Investigating the functional consequence Of white matter damage: an automatic pipeline to create longitudinal disconnection. biorxiv., 10.1101/140137.
32. Keshavan, A., Datta, E., McDonough, I. M., Madan, C. R., Jordan, K., Henry, R. G., (2017). Mindcontrol: a web application for brain segmentation quality control. NeuroImage, 170, 365–372. 10.1016/j.neuroimage.2017.03.055, 28365419.
33. Keshavan, A., Yeatman, J., Rokem, A., (2018). Combining citizen science and deep learning to amplify expertise in neuroimaging. bioRxiv 363382. 10.1101/363382.
34. Kim, J. S., Greene, M. J., Zlateski, A., Lee, K., Richardson, M., Turaga, S. C.,. (2014). Space–time wiring specificity supports direction selectivity in the retina. Nature, 509:331. 10.1038/nature13240, 24805243.
35. Klein, A., Ghosh, S. S., Bao, F. S., Giard, J., Häme, Y., Stavsky, E.,. (2017). Mindboggling morphometry of human brains. PLoS Comput. Biol., 13:e1005350. 10.1371/journal.pcbi.1005350, 28231282.
36. Krizhevsky, A., Sutskever, I., Hinton, G. E., (2012). Imagenet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems (New York, NY), 1097–1105. 10.1145/3065386.
37. Lebel, C., Beaulieu, C., (2011). Longitudinal development of human brain wiring continues from childhood into adulthood. J. Neurosci., 31, 10937–10947. 10.1523/JNEUROSCI.5302-10.2011, 21795544.
38. Lee, C. S., Baughman, D. M., Lee, A. Y., (2017a). Deep learning is effective for classifying normal versus age-related macular degeneration OCT images. Ophthalmol. Retina, 1, 322–327. 10.1016/j.oret.2016.12.009.
39. Lee, C. S., Tyring, A. J., Deruyter, N. P., Wu, Y., Rokem, A., Lee, A. Y., (2017b). Deep-learning based automated segmentation of macular edema in optical coherence tomography. Biomed. Opt. Express, 8, 3440–3448. 10.1364/BOE.8.003440, 28717579.
40. Long, J., Shelhamer, E., Darrell, T., (2015). Fully convolutional networks for semantic segmentation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Boston, MA), 3431–3440.
41. Marx, V., (2013). Neuroscience waves to the crowd. Nat. Methods, 10, 1069–1074. 10.1038/nmeth.2695, 24173382.
42. Mortamet, B., Bernstein, M. A., Jack, C. R., Gunter, J. L., Ward, C., Britson, P. J.,. (2009). Automatic quality assessment in structural brain magnetic resonance imaging. Mag. Res. Med., 62, 365–372. 10.1002/mrm.21992, 19526493.
43. Pan, S. J., Yang, Q., (2010). A survey on transfer learning. IEEE Trans. Knowledge Data Eng., 22, 1345–1359. 10.1109/TKDE.2009.191.
44. Patenaude, B., Smith, S. M., Kennedy, D. N., Jenkinson, M., (2011). A Bayesian model of shape and appearance for subcortical brain segmentation. NeuroImage, 56, 907–922. 10.1016/j.neuroimage.2011.02.046, 21352927.
45. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O.,. (2011). Scikit-learn: machine learning in Python. J. Mach. Learn. Res., 12, 2825–2830. Available online at: https://scikit-learn.org/stable/about.html#citing-scikit-learn.
46. Poldrack, R. A., Gorgolewski, K. J., (2014). Making big data open: data sharing in neuroimaging. Nat. Neurosci., 17, 1510–1517. 10.1038/nn.3818, 25349916.
47. Ronneberger, O., Fischer, P., Brox, T., (2015). U-net: convolutional networks for biomedical image segmentation, in International Conference on Medical Image Computing and Computer-Assisted Intervention (Munich: Springer), 234–241. 10.1007/978-3-319-24574-428.
48. Roskams, J., Popović, Z., (2016). Power to the people: addressing big data challenges in neuroscience by creating a new cadre of citizen neuroscientists. Neuron, 92, 658–664. 10.1016/j.neuron.2016.10.045, 27810012.
49. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S.,. (2015). ImageNet large scale visual recognition challenge. Intl. J. Comput. Vision, 115, 211–252. 10.1007/s11263-015-0816-y.
50. Sahiner, B., Chan, H.-P., Petrick, N., Wei, D., Helvie, M. A., Adler, D. D.,. (1996). Classification of mass and normal breast tissue: a convolution neural network classifier with spatial domain and texture images. IEEE Trans. Med. Imaging, 15, 598–610. 10.1109/42.538937, 18215941.
51. Saito, T., Rehmsmeier, M., (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE, 10:e0118432. 10.1371/journal.pone.0118432, 25738806.
52. Shehzad, Z., Giavasis, S., Li, Q., Benhajali, Y., Yan, C., Yang, Z.,. (2015). The preprocessed connectomes project quality assessment protocol: a resource for measuring the quality of MRI data. Front. Neurosci., 9:47. 10.3389/conf.fnins.2015.91.00047.
53. Simonyan, K., Zisserman, A., (2014). Very deep convolutional networks for large-scale image recognition. arXiv [Preprint]. arXiv:1409.1556.
54. Simpson, R., Page, K. R., De Roure, D., (2014). Zooniverse: observing the world's largest citizen science platform, in Proceedings of the 23rd International Conference on World Wide Web (Seoul: ACM), 1049–1054. 10.1145/2567948.2579215.
55. Van Horn, J. D., Toga, A. W., (2013). Human neuroimaging as a “Big Data” science. Brain Imaging Behav., 8, 323–331. 10.1007/s11682-013-9255-y, 24113873.
56. Van Opbroek, A., Ikram, M. A., Vernooij, M. W., De Bruijne, M., (2015). Transfer learning improves supervised image segmentation across imaging protocols. IEEE Trans. Med. Imaging, 34, 1018–1030. 10.1109/TMI.2014.2366792, 25376036.
57. Wandell, B., Winawer, J., (2011). Imaging retinotopic maps in the human brain. Vision Res., 51, 718–737. 10.1016/j.visres.2010.08.004, 20692278.
58. Winawer, J., Witthoft, N., (2017). Identification of the ventral occipital visual field maps in the human brain. F1000Res, 6:1526. 10.12688/f1000research.12364.1, 29188017.
59. Yeatman, J. D., Dougherty, R. F., Myall, N. J., Wandell, B. A., Feldman, H. M., (2012). Tract profiles of white matter properties: automating fiber-tract quantification. PLoS ONE, 7:e49790. 10.1371/journal.pone.0049790, 23166771.