Unsupervised decoding of long-term, naturalistic human neural recordings with automated video and audio annotations

Frontiers in Human Neuroscience

Volumn 10, Issue APR2016, 2016, Pages 1-13

  Wang, Nancy X R ^a   Olson, Jared D ^a   Ojemann, Jeffrey G ^a   Rao, Rajesh P N ^a   Brunton, Bingni W ^a  

^a UNIVERSITY OF WASHINGTON   (United States)

Author keywords

Automation; Computer vision; Electrocorticography (ecog); Functional brain mapping; Long term recording; Neural decoding; Speech processing; Unsupervised machine learning

Indexed keywords

BRAIN MAPPING; ELECTROCORTICOGRAPHY; HOSPITAL; HUMAN; NERVOUS SYSTEM; SPEECH; VIDEORECORDING; VISION;

EID: 84964763172     PISSN: None     EISSN: 16625161     Source Type: Journal    
DOI: 10.3389/fnhum.2016.00165     Document Type: Article

References (66)

1. Arya, R., Wilson, J. A., Vannest, J., Byars, A. W., Greiner, H. M., Buroker, J., et al. (2015). Electrocorticographic language mapping in children by highgamma synchronization during spontaneous conversation: comparison with conventional electrical cortical stimulation. Epilepsy Res. 110, 78-87. doi: 10.1016/j.eplepsyres.2014.11.013
2. Ball, T., Kern, M., Mutschler, I., Aertsen, A., and Schulze-Bonhage, A. (2009). Signal quality of simultaneously recorded invasive and non-invasive EEG. Neuroimage 46, 708-716. doi: 10.1016/j.neuroimage.2009.02.028
3. Bauer, P. R., Vansteensel, M. J., Bleichner, M. G., Hermes, D., Ferrier, C. H., Aarnoutse, E. J., et al. (2013). Mismatch between electrocortical stimulation and electrocorticography frequency mapping of language. Brain Stimul. 6, 524-531. doi: 10.1016/j.brs.2013.01.001
4. Blakely, T., Miller, K. J., Rao, R. P., Holmes, M. D., and Ojemann, J. G. (2008). Localization and classification of phonemes using high spatial resolution electrocorticography (ECoG) grids. Conf. Proc. IEEE Eng. Med. Biol. Soc. 4964-4967. doi: 10.1109/IEMBS.2008.4650328
5. Breshears, J. D., Gaona, C. M., Roland, J. L., Sharma, M., Bundy, D. T., Shimony, J. S., et al. (2012). Mapping sensorimotor cortex using slow cortical potential resting-state networks while awake and under anesthesia. Neurosurgery 71, 305. doi: 10.1227/NEU.0b013e318258e5d1
6. Brunton, B. W., Johnson, L. A., Ojemann, J. G., and Kutz, J. N. (2015). Extracting spatial-temporal coherent patterns in large-scale neural recordings using dynamic mode decomposition. J. Neurosci. Methods 258, 1-15. doi: 10.1016/j.jneumeth.2015.10.010
7. Cajochen, C., Foy, R., and Dijk, D.-J. (1999). Frontal predominance of a relative increase in sleep delta and theta eeg activity after sleep loss in humans. Sleep Res. Online 2, 65-69.
8. Chang, E. F., Rieger, J. W., Johnson, K., Berger, M. S., Barbaro, N. M., and Knight, R. T. (2010). Categorical speech representation in human superior temporal gyrus. Nat. Neurosci. 13, 1428-1432. doi: 10.1038/nn.2641
9. Dastjerdi, M., Ozker, M., Foster, B. L., Rangarajan, V., and Parvizi, J. (2013). Numerical processing in the human parietal cortex during experimental and natural conditions. Nat. Commun. 4, 2528. doi: 10.1038/ ncomms3528
10. Derix, J., Iljina, O., Schulze-Bonhage, A., Aertsen, A., and Ball, T. (2012). "Doctor" or "darling"? Decoding the communication partner from ECoG of the anterior temporal lobe during non-experimental, real-life social interaction. Front. Hum. Neurosci. 6: 251. doi: 10.3389/fnhum.2012.00251
11. Derix, J., Iljina, O., Weiske, J., Schulze-Bonhage, A., Aertsen, A., and Ball, T. (2014). From speech to thought: the neuronal basis of cognitive units in non-experimental, real-life communication investigated using ECoG. Front. Hum. Neurosci. 8: 383. doi: 10.3389/fnhum.2014.00383
12. DeWitt, I., and Rauschecker, J. P. (2013). Wernicke’s area revisited: parallel streams and word processing. Brain Lang. 127, 181-191. doi: 10.1016/j.bandl.2013.09.014
13. Erhan, D., Szegedy, C., Toshev, A., and Anguelov, D. (2014). "Scalable object detection using deep neural networks," in 2014 IEEE Conference on Computer Vision and Pattern Recognition (Columbus: IEEE), 2155-2162.
14. Felsen, G., and Dan, Y. (2005). A natural approach to studying vision. Nat. Neurosci. 8, 1643-1646. doi: 10.1038/nn1608
15. Fifer, M. S., Hotson, G., Wester, B. A., McMullen, D. P., Wang, Y., Johannes, M. S., et al. (2014). Simultaneous neural control of simple reaching and grasping with the modular prosthetic limb using intracranial EEG. IEEE Trans. Neural Syst. Rehabil. Eng. 22, 695-705. doi: 10.1109/TNSRE.2013.22 86955
16. Girshick, R., Donahue, J., Darrell, T., andMalik, J. (2014). "Rich feature hierarchies for accurate object detection and semantic segmentation," in 2014 IEEE Conference on Computer Vision and Pattern Recognition (Columbus: IEEE), 580-587.
17. He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. IEEE Int. Conf. Comput. Vis. 1026-1034. arXiv: 1502.01852v1.
18. Herff, C.,Heger,D., de Pesters, A., Telaar, D., Brunner, P., Schalk,G., and Schultz, T. (2015). Brain-to-text: decoding spoken phrases from phone representations in the brain. Front. Neurosci. 9: 217. doi: 10.3389/fnins.2015.00217
19. Hill, N. J., Gupta, D., and Brunner, P. (2012). Recording human electrocorticographic (ECoG) signals for neuroscientific research and real-time functional cortical mapping. J. Vis. Exp. 64: 3993. doi: 10.3791/3993
20. Huang, X., Baker, J., and Reddy, R. (2014). A historical perspective of speech recognition. Commun. ACM 57, 94-103. doi: 10.1145/2500887
21. Jackson, A., Mavoori, J., and Fetz, E. (2007). Correlations between the same motor cortex cells and arm muscles during a trained task, free behavior, and natural sleep in the macaque monkey. J. Neurophysiol. 97, 360-374. doi: 10.1152/jn.00710.2006
22. Jordan, M. I., and Mitchell, T. M. (2015). Machine learning: trends, perspectives, and prospects. Science 349, 255-260. doi: 10.1126/science.aaa8415
23. Kanas, V. G., Mporas, I., Benz, H. L., Sgarbas, K. N., Bezerianos, A., and Crone, N. E. (2014a). Joint spatial-spectral feature space clustering for speech activity detection from ECoG signals. IEEE Trans. Biomed. Eng. 61, 1241-1250. doi: 10.1109/TBME.2014.2298897
24. Kanas, V. G., Mporas, I., Benz, H. L., Sgarbas, K. N., Bezerianos, A., and Crone, N. E. (2014b). "Real-time voice activity detection for ECoG-based speech brain machine interfaces," in 2014 19th International Conference on Digital Signal Processing (Hong Kong: IEEE), 862-865.
25. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Fei-Fei, L. (2014). "Large-scale video classification with convolutional neural networks," in 2014 IEEE Conference on Computer Vision and Pattern Recognition (Columbus: IEEE), 1725-1732.
26. Kellis, S., Miller, K., Thomson, K., Brown, R., House, P., and Greger, B. (2010). Decoding spoken words using local field potentials recorded from the cortical surface. J. Neural Eng. 7: 056007. doi: 10.1088/1741-2560/7/5/ 056007
27. Kipp, M. (2012). "Anvil: a universal video research tool," in Handbook of Corpus Phonology, eds J. Durand, U. Gut, and G. Kristoffersen (Oxford, UK: Oxford University Press).
28. Lal, T., Hinterberger, T., Widman, G., Michael, S., Hill, J., Rosenstiel, W., et al. (2004). "Methods towards invasive human brain computer interfaces," in Eighteenth Annual Conference on Neural Information Processing Systems (NIPS 2004) (Vancouver, BC: MIT Press), 737-744.
29. Lamrous, S., and Taileb, M. (2006). Divisive hierarchical k-means. in International Conference on Computational Intelligence for Modelling, Control and Automation, 2006 and International Conference on Intelligent Agents, Web Technologies and Internet Commerce (Sydney: IEEE), 18-18.
30. Längkvist, M., Karlsson, L., and Loutfi, A. (2012). Sleep stage classification using unsupervised feature learning. Adv. Artif. Neural Syst. 2012: 107046. doi: 10.1155/2012/107046
31. Leuthardt, E. C., Miller, K. J., Schalk, G., Rao, R. P., and Ojemann, J. G. (2006). Electrocorticography-based brain computer interface-the Seattle experience. IEEE Trans. Neural Syst. Rehabil. Eng. 14, 194-198. doi: 10.1109/TNSRE.2006.875536
32. Leuthardt, E. C., Gaona, C., Sharma, M., Szrama, N., Roland, J., Freudenberg, Z., et al. (2011). Using the electrocorticographic speech network to control a braincomputer interface in humans. J. Neural Eng. 8: 036004. doi: 10.1088/1741-2560/8/3/036004
33. McMullen, D. P.,Hotson, G., Katyal, K. D.,Wester, B. A., Fifer,M. S.,McGee, T. G., et al. (2014). Demonstration of a semi-autonomous hybrid brain-machine interface using human intracranial EEG, eye tracking, and computer vision to control a robotic upper limb prosthetic. IEEE Trans. Neural Syst. Rehabil. Eng. 22, 784-796. doi: 10.1109/TNSRE.2013.2294685
34. Mercier, M. R., Molholm, S., Fiebelkorn, I. C., Butler, J. S., Schwartz, T. H., and Foxe, J. J. (2015). Neuro-oscillatory phase alignment drives speeded multisensory response times: an electro-corticographic investigation. J. Neurosci. 35, 8546-8557. doi: 10.1523/JNEUROSCI.4527-14.2015
35. Miller, K. J., Schalk, G., Fetz, E. E., den Nijs, M., Ojemann, J. G., and Rao, R. P. N. (2010). Cortical activity during motor execution, motor imagery, and imagery-based online feedback. Proc. Natl. Acad. Sci. U.S.A. 107, 4430-4435. doi: 10.1073/pnas.0913697107
36. Miller, K. J., denNijs, M., Shenoy, P., Miller, J. W., Rao, R. P., and Ojemann, J. G. (2007). Real-time functional brain mapping using electrocorticography. Neuroimage 37, 504-507. doi: 10.1016/j.neuroimage.2007.05.029
37. Miller, K. J., Zanos, S., Fetz, E. E., den Nijs, M., and Ojemann, J. G. (2009). Decoupling the cortical power spectrum reveals real-time representation of individual finger movements in humans. J. Neurosci. 29, 3132-3137. doi: 10.1523/JNEUROSCI.5506-08.2009
38. Moran, D. (2010). Evolution of brain-computer interface: action potentials, local field potentials and electrocorticograms. Curr. Opin. Neurobiol. 20, 741-745. doi: 10.1016/j.conb.2010.09.010
39. Mugler, E. M., Patton, J. L., Flint, R. D., Wright, Z. A., Schuele, S. U., Rosenow, J., et al. (2014). Direct classification of all American English phonemes using signals from functional speech motor cortex. J. Neural Eng. 11: 035015. doi: 10.1088/1741-2560/11/3/035015
40. Muller-Putz, G., Leeb, R., Tangermann, M., Hohne, J. H., Kubler, A. K., Cincotti, F., et al. (2015). Towards noninvasive hybrid brain-computer interfaces: framework, practice, clinical application, and beyond. Proc. IEEE 103, 926-943. doi: 10.1109/JPROC.2015.2411333
41. Nakanishi, Y., Yanagisawa, T., Shin, D., Fukuma, R., Chen, C., Kambara, H., et al. (2013). Prediction of three-dimensional arm trajectories based on ECoG signals recorded from human sensorimotor cortex. PLoS ONE 8: e72085. doi: 10.1371/journal.pone.0072085
42. Pei, X., Barbour, D. L., Leuthardt, E. C., and Schalk, G. (2011). Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans. J. Neural Eng. 8: 046028. doi: 10.1088/1741-2560/8/4/ 046028
43. Pistohl, T., Schulze-Bonhage, A., and Aertsen, A. (2012). Decoding natural grasp types from human ECoG. Neuroimage 59, 248-260. doi: 10.1016/j.neuroimage.2011.06.084
44. Pluta, T., Bernardo, R., Shin, H. W., and Bernardo, D. R. (2014). "Unsupervised learning of electrocorticography motifs with binary descriptors of wavelet features and hierarchical clustering," in Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Chicago, IL), 2657-2660.
45. Poppe, R. (2007). Vision-based human motion analysis: an overview. Comput. Vis. Image Underst. 108, 4-18. doi: 10.1016/j.cviu.2006.10.016
46. Potes, C., Brunner, P., Gunduz, A., Knight, R. T., and Schalk, G. (2014). Spatial and temporal relationships of electrocorticographic alpha and gamma activity during auditory processing. Neuroimage 97, 188-195. doi: 10.1016/j.neuroimage.2014.04.045
47. Ramirez, J., Segura, J. C., Benitez, C., de la Torre, A., and Rubio, A. (2004). Efficient voice activity detection algorithms using long-term speech information. Speech Commun. 42, 271-287. doi: 10.1016/j.specom.2003.10.002
48. Rao, R. P. N. (2013). Brain-Computer Interfacing: An Introduction. New York, NY: Cambridge University Press.
49. Ruescher, J., Iljina, O., Altenmüller, D.-M., Aertsen, A., Schulze-Bonhage, A., and Ball, T. (2013). Somatotopic mapping of natural upper- and lower-extremity movements and speech production with high gamma electrocorticography. Neuroimage 81, 164-177. doi: 10.1016/j.neuroimage.2013. 04.102
50. Ryoo, M. S., and Matthies, L. (2013). "First-person activity recognition: what are they doing to me? in 2013 IEEE Conference on Computer Vision and Pattern Recognition (Portland: IEEE), 2730-2737.
51. Schalk, G., Miller, K. J., Anderson, N. R., Wilson, J. A., Smyth, M. D., Ojemann, J. G., et al. (2008). Two-dimensional movement control using electrocorticographic signals in humans. J. Neural Eng. 5, 75-84. doi: 10.1088/1741-2560/5/1/008
52. Toshev, A., and Szegedy, C. (2014). "DeepPose: human pose estimation via deep neural networks," in 2014 IEEE Conference on Computer Vision and Pattern Recognition (Columbus: IEEE), 1653-1660.
53. Towle, V. L., Yoon, H.-A., Castelle, M., Edgar, J. C., Biassou, N. M., Frim, D. M., et al. (2008). ECOG gamma activity during a language task: differentiating expressive and receptive speech areas. Brain 131, 2013-2027. doi: 10.1093/brain/awn147
54. Vansteensel, M. J., Bleichner, M. G., Dintzner, L., Aarnoutse, E., Leijten, F., Hermes, D., et al. (2013). Task-free electrocorticography frequency mapping of the motor cortex. Clin. Neurophysiol. 124, 1169-1174. doi: 10.1016/j.clinph.2012.08.048
55. Vansteensel, M. J., Hermes, D., Aarnoutse, E. J., Bleichner, M. G., Schalk, G., van Rijen, P. C., et al. (2010). Brain-computer interfacing based on cognitive control. Ann. Neurol. 67, 809-816. doi: 10.1002/ana.21985
56. Vinje, W. E., and Gallant, J. (2000). Sparse coding and decorrelation in primary visual cortex during natural vision. Science 287, 1273-1276. doi: 10.1126/science.287.5456.1273
57. Wang, N., Cullis-Suzuki, S., and Albu, A. (2015). "Automated analysis of wild fish behavior in a natural habitat," in Environmental Multimedia Retrieval at ICMR (Shanghai).
58. Wang, P. T., Puttock, E. J., King, C. E., Schombs, A., Lin, J. J., Sazgar, M., et al. (2013a). "State and trajectory decoding of upper extremity movements from electrocorticogram," in 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER) (San Diego, CA: IEEE) 969-972.
59. Wang, W., Collinger, J. L., Degenhart, A. D., Tyler-Kabara, E. C., Schwartz, A. B., Moran, D. W., et al. (2013b). An electrocorticographic brain interface in an individual with tetraplegia. PLoS ONE 8: e55344. doi: 10.1371/journal.pone.0055344
60. Wang, Z., Gunduz, A., Brunner, P., Ritaccio, A. L., Ji, Q., and Schalk, G. (2012). Decoding onset and direction of movements using Electrocorticographic (ECoG) signals in humans. Front. Neuroeng. 5: 15. doi: 10.3389/fneng.2012.00015
61. Wang, Z., Ji, Q., Miller, K. J., and Schalk, G. (2010). "Decoding finger flexion from electrocorticographic signals using a sparse Gaussian process," in 2010 20th International Conference on Pattern Recognition (Istanbul: IEEE), 3756-3759.
62. Williams, J. C., Hippensteel, J. A., Dilgen, J., Shain, W., and Kipke, D. R. (2007). Complex impedance spectroscopy for monitoring tissue responses to inserted neural implants. J. Neural Eng. 4, 410-423. doi: 10.1088/1741-2560/4/4/007
63. Wolpaw, J., and Wolpaw, E. W. (2012). Brain-computer interfaces: principles and practice. London: Oxford University Press.
64. Yanagisawa, T., Hirata, M., Saitoh, Y., Goto, T., Kishima, H., Fukuma, R., et al. (2011). Real-time control of a prosthetic hand using human electrocorticography signals. J. Neurosurg. 114, 1715-1722. doi: 10.3171/2011.1.JNS101421
65. Zander, T. O., and Kothe, C. (2011). Towards passive brain-computer interfaces: applying brain-computer interface technology to human-machine systems in general. J. Neural Eng. 8: 025005. doi: 10.1088/1741-2560/8/2/025005
66. Zhang, D., Gong, E., Wu, W., Lin, J., Zhou, W., and Hong, B. (2012). "Spoken sentences decoding based on intracranial high gamma response using dynamic time warping," in Annual International Conference of the IEEE Engineering in Medicine and Biology Society (San Diego, CA), 3292-3295.