Encoding and decoding models in cognitive electrophysiology

Frontiers in Systems Neuroscience

Volumn 11, Issue , 2017, Pages

  Holdgraf, Christopher R ^a   Rieger, Jochem W ^b   Micheli, Cristiano ^b,c   Martin, Stephanie ^a,d   Knight, Robert T ^a   Theunissen, Frederic E ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF OLDENBURG   (Germany)

^c INSTITUT DES SCIENCES COGNITIVES   (France)

^d EPFL   (Switzerland)

Author keywords

Decoding models; Electrocorticography (ECoG); Electrophysiology evoked potentials; Encoding models; Machine learning applied to neuroscience; Natural stimuli; Predictive modeling; Tutorials

Indexed keywords

BRAIN FUNCTION; ELECTROCORTICOGRAPHY; EVOKED RESPONSE; HEARING; HUMAN; HUMAN EXPERIMENT; LANGUAGE; MACHINE LEARNING; MOTOR SYSTEM; NEUROSCIENCE; RECEPTIVE FIELD; SOUND; SPEECH; STATISTICS; STIMULUS;

EID: 85032941467     PISSN: 16625137     EISSN: None     Source Type: Journal    
DOI: 10.3389/fnsys.2017.00061     Document Type: Review

References (128)

1. Aertsen, A. M. H. J., and Johannesma, P. I. M. (1981). The spectro-temporal receptive field. Biol Cybern. 42, 133-143. doi: 10.1007/BF00336731
2. Ahrens, M. B., Paninski, L., and Sahani, M. (2008). Inferring input nonlinearities in neural encoding models. Network 19, 35-67. doi: 10.1080/095489807018 13936
3. Andoni, S., and Pollak, G. D. (2011). Selectivity for spectral motion as a neural computation for encoding natural communication signals in bat inferior colliculus. J. Neurosci. 31, 16529-16540. doi: 10.1523/JNEUROSCI.1306- 11.2011
4. Bell, A. J., and Sejnowski, T. J. (1995). An information-maximization approach to blind separation and blind deconvolution. Neural Comput. 7, 1129-1159. doi: 10.1162/neco.1995.7.6.1129
5. Bennett, C. M., Baird, A. A., Miller, M. B., and Wolfrod, G. L. (2009). Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: an argument for multiple comparisons correction. Neuroimage 47, S125. doi: 10.1016/S1053-8119(09)71202-9
6. Blakely, T., Miller, K. J., Rao, R. P. N., Holmes, M. D., and Ojemann, J. G. (2008). Localization and classification of phonemes using high spatial resolution electrocorticography (ECoG) grids. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2008, 4964-4967. doi: 10.1109/IEMBS.2008.4650328
7. Bouchard, K. E., and Chang, E. F. (2014). Neural decoding of spoken vowels from human sensory-motor cortex with high-density electrocorticography. Conf. Proc. IEEE Engi. Med. Biol. Soc. 2014, 6782-6785. doi: 10.1109/EMBC.2014.6945185
8. Bressler, S. L., and Seth, A. K. (2011). Wiener-Granger Causality: a well established methodology. Neuroimage 58, 323-329. doi: 10.1016/j.neuroimage.2010.02.059
9. Brumberg, J. S., Nieto-Castanon, A., Kennedy, P. R., and Guenther, F. H. (2010). Brain-Computer Interfaces for Speech Communication. Speech Commun. 52, 367-379. doi: 10.1016/j.specom.2010.01.001
10. Brumberg, J. S., Wright, E. J., Andreasen, D. S., Guenther, F. S. and Kennedy, P. R. (2011). Classification of intended phoneme production from chronic intracortical microelectrode recordings in speech-motor cortex. Front. Neurosci. 5:65. doi: 10.3389/fnins.2011.00065
11. Campbell, D. T., and Stanley, J. C. (2015). Experimental and Quasi-Experimental Designs for Research. Ravenio Books. Available online at: https://www.amazon.com/Experimental-Quasi-Experimental-Designs-Research-Campbell-ebook/dp/B014WQS2SY
12. Chakrabarti, S., Krusienski, D. J., Schalk, G., and Brumberg, J. S. (2013). “Predicting mel-frequency cepstral coefficients from electrocorticographic signals during continuous speech production,” in 6th International IEEE/EMBS Conference on Neural Engineering (NER) (San Diego, CA).
13. Chang, E. F., Edwards, E., Nagarajan, S. S., Fogelson, N., Dalal, S. S., Canolty, R. T., et al. (2011). Cortical spatio-temporal dynamics underlying phonological target detection in humans. J. Cogn. Neurosci. 23, 1437-1446. doi: 10.1162/jocn.2010.21466
14. Chang, E. F., Rieger, J.W., Johnson, K., Berger, M. S., Barbaro, N. M., and Knight, R. T. (2010). Categorical speech representation in human superior temporal gyrus. Nat. Neurosci. 13, 1428-1432. doi: 10.1038/nn.2641
15. Chi, T., Ru, P., and Shamma, S. A. (2005). Multiresolution spectrotemporal analysis of complex sounds. J. Acous. Soc. Am. 118:887. doi: 10.1121/1.1945807
16. Christianson, G. B., Sahani, M., and Linden, J. F. (2008). The consequences of response nonlinearities for interpretation of spectrotemporal receptive fields. J. Neurosci. 28, 446-455. doi: 10.1523/JNEUROSCI.1775- 07.2007
17. Crosse, M. J., Di Liberto, G. M., Bednar, A., and Lalor, E. C. (2016). The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli. Front. Hum. Neurosci. 10:604. doi: 10.3389/fnhum.2016.00604
18. Çukur, T., Nishimoto, S., Huth, A. G., and Gallant, J. L. (2013). Attention during natural vision warps semantic representation across the human brain. Nat. Neurosci. 16, 763-770. doi: 10.1038/nn.3381
19. Curran-Everett, D. (2000). Multiple comparisons: philosophies and illustrations. Am. J. Physiol. Regul. Integr. Comp. Physiol. 279, R1-R8. Available online at: http://ajpregu.physiology.org/content/279/1/R1.article-info
20. David, S. V. (2004). Natural stimulus statistics alter the receptive field structure of V1 neurons. J. Neurosci. 24, 6991-7006. doi: 10.1523/JNEUROSCI.1422-04.2004
21. David, S. V., and Gallant, J. L. (2005). Predicting neuronal responses during natural vision. Netw. Comput. Neural Syst. 16, 239-260. doi: 10.1080/09548980500464030
22. David, S. V., and Shamma, S. A. (2013). Integration over multiple timescales in primary auditory cortex. J. Neurosci. 33, 19154-19166. doi: 10.1523/JNEUROSCI.2270-13.2013
23. De Heer, W. A., Huth, A. G., Griffiths, T. L., Gallant, J. L., and Theunissen, F. E. (2017). The hierarchical cortical organization of human speech processing. J. Neurosci. 3267-16. doi: 10.1523/JNEUROSCI.3267-16.2017
24. Degenhart, A. D., Sudre, G. P., Pomerleau, D. A., and Tyler-Kabara, E. C. (2011). Decoding semantic information from human electrocorticographic (ECoG) signals. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2011, 6294-6298. doi: 10.1109/IEMBS.2011.6091553
25. Depireux, D. A., Simon, J. Z., Klein, D. J., and Shamma, S. A. (2001). Spectrotemporal response field characterization with dynamic ripples in ferret primary auditory cortex. J. Neurophysiol. 85, 1220-1234. Available online at: http://jn.physiology.org/content/85/3/1220
26. DeWitt, I., and Rauschecker, J. P. (2012). Phoneme and word recognition in the auditory ventral stream. Proc. Natl. Acad. Sci. U.S.A. 109, E505-E514. doi: 10.1073/pnas.1113427109
27. Di Liberto, G. M., O’Sullivan, J. A., and Lalor, E. C. (2015). Low-frequency cortical entrainment to speech reflects phoneme-level processing. Curr. Biol. 25, 2457-2465. doi: 10.1016/j.cub.2015.08.030
28. Donders, F. C. (1969). On the speed of mental processes. Acta Psychol. 30, 412-431. doi: 10.1016/0001-6918(69)90065-1
29. Eggermont, J. J. (1993). Wiener and Volterra analyses applied to the auditory system. Hear. Res. 66, 177-201. doi: 10.1016/0378-5955(93)90139-R
30. Eggermont, J. J. (2001). Between sound and perception: reviewing the search for a neural code. Hear. Res. 157, 1-42. doi: 10.1016/S0378-5955(01) 00259-3
31. Eggermont, J. J., Johannesma, P. I. M., and Aertsen, A. M. H. J. (1983). Reverse-correlation methods in auditory research. Q. Rev. Biophys. 16:341. doi: 10.1017/S0033583500005126
32. Elie, J. E., and Theunissen, F. E. (2016). The vocal repertoire of the domesticated zebra finch: a data-driven approach to decipher the information-bearing acoustic features of communication signals. Anim. Cogn. 19, 285-315. doi: 10.1007/s10071-015-0933-6
33. Elliott, T. M., and Theunissen, F. E. (2009). The modulation transfer function for speech intelligibility. PLoS Comput. Biol. 5:e1000302. doi: 10.1371/journal.pcbi.1000302
34. Escabí, M. A., and Schreiner, C. E. (2002). Nonlinear spectrotemporal sound analysis by neurons in the auditory midbrain. J. Neurosci. 22, 4114-4131. Available online at: http://www.jneurosci.org/content/22/10/4114/
35. Felleman, D. J., and Van Essen, D. C. (1991). Distributed hierachical processing in the primate cerebral cortex. Cereb. Cortex 1, 1-47. doi: 10.1093/cercor/1.1.1
36. Felsen, G., and Dan, Y. (2005). A natural approach to studying vision. Nat. Neurosci. 8, 1643-1646. doi: 10.1038/nn1608
37. Fitzgerald, J. D., Sincich, L. C., and Sharpee, T. O. (2011). Minimal models of multidimensional computations. PLoS Comput. Biol. 7:e1001111. doi: 10.1371/journal.pcbi.1001111
38. Friston, K. J. (2003). “Introduction: experimental design and statistical parametric mapping,” in SPM Introduction, eds A. Toga and J. Mazziotta (San Diego, CA: Imprint Publishing), 605-631.
39. Fritz, J. B., Elhilali, M., and Shamma, S. A. (2005). Differential dynamic plasticity of A1 receptive fields during multiple spectral tasks. J. Neurosci. 25, 7623-7635. doi: 10.1523/JNEUROSCI.1318-05.2005
40. Fritz, J. B., Shamma, S. A., Elhilali, M., and Klein, D. J. (2003). Rapid task-related plasticity of spectrotemporal receptive fields in primary auditory cortex. Nat. Neurosci. 6, 1216-1223. doi: 10.1038/nn1141
41. Frye, M., Micheli, C., Schepers, I. M., Schalk, G., Rieger, J. W., and Meyer, B. T. (2016). Neural responses to speech-specific modulations derived from a spectro-temporal filter bank. In Proc. Interspeech. 1368-1372. doi: 10.21437/Interspeech.2016-1327
42. Green, D. M., and Swets, J. A. (1988). Signal Detection Theory and Psychophysics. Los Altos Hills, CA: Peninsula Publishing.
43. Güçlü, U., and van Gerven, M. A. J. (2014). Unsupervised feature learning improves prediction of human brain activity in response to natural Images. PLoS Comput. Biol. 10:e1003724. doi: 10.1371/journal.pcbi.1003724
44. Hanke, M., Halchenko, Y. O., Sederberg, P. B., Hanson, S. J., Haxby, J. V., and Pollmann, S. (2009). PyMVPA: a python toolbox for multivariate pattern analysis of fMRI data. Neuroinformatics, 7, 37-53. doi: 10.1007/s12021-008-9041-y
45. Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning. New York, NY: Springer.
46. Haufe, S., Meinecke, F., Görgen, K., Döhne, S., Haynes, J. D., Blankertz, B., et al. (2014). On the interpretation of weight vectors of linear models in multivariate neuroimaging. Neuroimage 87, 96-110. doi: 10.1016/j.neuroimage.2013.10.067
47. Henniges, M., and Puertas, G. (2010). “Binary sparse coding,” in International Conference on Latent Variable Analysis and Signal Separation, LVA/ICA 2010. (St. Malo), 450-457.
48. Hermansky, H., and Morgan, N. (1994). RASTA processing of speech. IEEE Trans. Speech Audio Process. 2, 578-589. doi: 10.1109/89.326616
49. Hickok, G., and Small, S. L. (2015). Neurobiology of Language. Amsterdam: Academic Press.
50. Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., et al. (2012). Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485, 372-375. doi: 10.1038/nature11076
51. Holdgraf, C. R., de Heer, W., Pasley, B. N., Rieger, J. W., Crone, N., Lin, J. J., et al. (2016). Rapid tuning shifts in human auditory cortex enhance speech intelligibility. Nat. Commun. 7:13654. doi: 10.1038/ncomms13654
52. Hollmann, M., Rieger, J. W., Baecke, S., Lützkendorf, R., Müller, C., Adolf, D., et al. (2011). Predicting decisions in human social interactions using real-time fMRI and pattern classification. PLoS ONE 6:e25304. doi: 10.1371/journal.pone.0025304
53. Hsu, A., Borst, A., and Theunissen, F. E. (2004). Quantifying variability in neural responses and its application for the validation of model predictions. Netw. Comput. Neural Syst. 15, 91-109. doi: 10.1088/0954-898X_15_2_002
54. Hullett, P. W., Hamilton, L. S., Mesgarani, N., Schreiner, C. E., and Chang, E. F. (2016). Human superior temporal gyrus organization of spectrotemporal modulation tuning derived from speech stimuli. J. Neurosci. 36, 2014-2026. doi: 10.1523/JNEUROSCI.1779-1 5.2016
55. Huth, A. G., De Heer, W. A., Griffiths, T. L., Theunissen, F. E., and Gallant, J. L. (2016). Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532, 453-458. doi: 10.1038/nature17637
56. Huth, A. G., Nishimoto, S., Vu, A. T., and Gallant, J. L. (2012). A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron 76, 1210-1224. doi: 10.1016/j.neuron.2012.10.014
57. Kay, K. N., and Gallant, J. L. (2009). I can see what you see. Nat. Neurosci. 12:245. doi: 10.1038/nn0309-245
58. Kay, K. N., Naselaris, T., Prenger, R. J., and Gallant, J. L. (2008). Identifying natural images from human brain activity. Nature 452, 352-355. doi: 10.1038/nature06713
59. Kellis, S., Miller, K. J., Thomson, K., Brown, R., House, P., and Greger, B. (2010). Decoding spoken words using local field potentials recorded from the cortical surface. J. Neural Eng. 7:56007. doi: 10.1088/1741-2560/7/5/056007
60. Khalighinejad, B., Cruzatto da Silva, G., and Mesgarani, N. (2017). Dynamic encoding of acoustic features in neural responses to continuous speech. J. Neurosci. 37, 2176-2185. doi: 10.1523/JNEUROSCI.2383-16.2017
61. Kiang, N. (1984). Peripheral neural processing of auditory information. Compr. Physiol. 639-674.
62. Kubanek, J., Brunner, P., Gunduz, A., Poeppel, D., and Schalk, G. (2013). The tracking of speech envelope in the human cortex. PLoS ONE 8:e53398. doi: 10.1371/journal.pone.0053398
63. Leonard, M. K., Bouchard, K. E., Tang, C., and Chang, E. F. (2015). Dynamic encoding of speech sequence probability in human temporal cortex. J. Neurosci. 35, 7203-7214. doi: 10.1523/JNEUROSCI.4100-14.2015
64. Lescroart, M. D., Kanwisher, N., and Golomb, J. D. (2016). No evidence for automatic remapping of stimulus features or location found with fMRI. Front. Syst. Neurosci. 10:53. doi: 10.3389/fnsys.2016.00053
65. Lescroart, M. D., Stansbury, D. E., and Gallant, J. L. (2015). Fourier power, subjective distance, and object categories all provide plausible models of BOLD responses in scene-selective visual areas. Front. Comput. Neurosci. 9:135. doi: 10.3389/fncom.2015.00135
66. Lewicki,M. S. (2002). Efficient coding of natural sounds. Nat. Neurosci. 5, 356-363. doi: 10.1038/nn831
67. Lotte, F., Brumberg, J. S., Brunner, P., Gunduz, A., Ritaccio, A. L., Guan, C., et al. (2015). Electrocorticographic representations of segmental features in continuous speech. Front. Hum. Neurosci. 9:97. doi: 10.3389/fnhum.2015.00097
68. Maris, E., and Oostenveld, R. (2007). Nonparametric statistical testing of EEG- and MEG-data. J. Neurosci. Methods 164, 177-190. doi: 10.1016/j.jneumeth.2007.03.024
69. Marmarelis, P. Z., and Marmarelis, V. Z. (1978). Analysis of Physiological Systems. Boston, MA: Springer.
70. Martin, S., Brunner, P., Holdgraf, C. R., Heinze, H.-J., Crone, N. E., Rieger, J. W., et al. (2014). Decoding spectrotemporal features of overt and covert speech from the human cortex. Front. Neuroeng. 7:14. doi: 10.3389/fneng.2014.00014
71. Martin, S., Brunner, P., Iturrate, I., Millán, J. R., Schalk, G., Knight, R. T. (2016). Word pair classification during imagined speech using direct brain recordings. Sci. Rep. 6:25803. doi: 10.1038/srep25803
72. McFarland, J. M., Cui, Y., and Butts, D. A. (2013). Inferring nonlinear neuronal computation based on physiologically plausible inputs. PLoS Comput. Biol. 9: e1003143. doi: 10.1371/journal.pcbi.1003143
73. Mesgarani, N., and Chang, E. F. (2012). Selective cortical representation of attended speaker in multi-talker speech perception. Nature 485, 233-236. doi: 10.1038/nature11020
74. Mesgarani, N., Cheung, C., Johnson, K., and Chang, E. F. (2014). Phonetic feature encoding in human superior temporal gyrus. Science 343, 1006-1010. doi: 10.1126/science.1245994
75. Mesgarani, N., David, S. V., Fritz, J. B., and Shamma, S. A. (2009). Influence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex. J. Neurophysiol. 102, 3329-3339. doi: 10.1152/jn.91128.2008
76. Mesgarani, N., Slaney, M., and Shamma, S. A. (2006). Discrimination of speech from nonspeech based on multiscale spectro-temporal modulations. IEEE Trans. Audio Speech Lang. Process. 14, 920-930. doi: 10.1109/TSA.2005.858055
77. Meyer, A. F., Diepenbrock, J.-P., Ohl, F. W., and Anemüller, J. (2014). Temporal variability of spectro-temporal receptive fields in the anesthetized auditory cortex. Front. Comput. Neurosci. 8:165. doi: 10.3389/fncom.2014.00165
78. Meyer, A. F., Williamson, R. S., Linden, J. F., and Sahani, M. (2017). Models of neuronal stimulus-response functions: elaboration, estimation, and evaluation. Front. Syst. Neurosci. 10:109. doi: 10.3389/fnsys.2016.00109
79. Miller, L. M., Escabí, M. A., Read, H. L., and Schreiner, C. E. (2002). Spectrotemporal receptive fields in the lemniscal auditory thalamus and cortex. J. Neurophysiol. 87, 516-527. doi: 10.1152/jn.00395.2001
80. Moerel, M., De Martino, F., Santoro, R., Ugurbil, K., Goebel, R., Yacoub, E., et al. (2013). Processing of natural sounds: characterization of multipeak spectral tuning in human auditory cortex. J. Neurosci. 33, 11888-11898. doi: 10.1523/JNEUROSCI.5306-12.2013
81. Moreno-Bote, R., Beck, J. M., Kanitscheider, I., Pitkow, X., Latham, P. E., and Pouget, A. (2014). Information-limiting correlations. Nat. Neurosci. 17, 1410-1417. doi: 10.1038/nn.3807
82. Mugler, E. M., Patton, J. L., Flint, R. D., Wright, Z. A., Schuele, S. U., Rosenow, J., et al. (2014). Direct classification of all American English phonemes using signals from functional speech motor cortex. J. Neural Eng. 11:35015. doi: 10.1088/1741-2560/11/3/035015
83. Naselaris, T., Kay, K. N., Nishimoto, S., and Gallant, J. L. (2011). Encoding and decoding in fMRI. Neuroimage 56, 400-410. doi: 10.1016/j.neuroimage.2010.07.073
84. Naselaris, T., Prenger, R. J., Kay, K. N., Oliver, M., and Gallant, J. L. (2009). Bayesian reconstruction of natural images from human brain activity. Neuron 63, 902-915. doi: 10.1016/j.neuron.2009.09.006
85. Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., and Gallant, J. L. (2011). Reconstructing visual experiences frombrain activity evoked by natural movies. Curr. Biol. 21, 1641-1646. doi: 10.1016/j.cub.2011.08.031
86. Norman, K., a, Polyn, S. M., Detre, G. J., and Haxby, J. V. (2006). Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci. 10, 424-430. doi: 10.1016/j.tics.2006.07.005
87. Olshausen, B. A., and Field, D. J. (1997). Sparse coding with an incomplete basis set: a strategy employed by V1. Vision. Res. 37, 3311-3325. doi: 10.1016/S0042-6989(97)00169-7
88. Olshausen, B. A., and Field, D. J. (2004). Sparse coding of sensory inputs. Curr. Opin. Neurobiol. 14, 481-487. doi: 10.1016/j.conb.2004.07.007
89. Paninski, L. (2003). Convergence properties of three spike-triggered analysis techniques. Netw. Comput. Neural Syst. 14, 437-464. doi: 10.1088/0954-898X_14_3_304
90. Paninski, L. (2004). Maximum likelihood estimation of cascade pointprocess neural encoding models. Netw. Comput. Neural Syst. 15, 243-262. doi: 10.1088/0954-898X_15_4_002
91. Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Nathan, E., et al. (2012). Reconstructing speech from human auditory cortex. PLoS Biol. 10:e1001251. doi: 10.1371/journal.pbio.1001251
92. Pasley, B. N., and Knight, R. T. (2012). Decoding speech for understanding and treating aphasia. Prog. Brain Res. 207, 435-456. doi: 10.1016/B978 -0-444-63327-9.00018-7
93. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., et al. (2011). Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825-2830.
94. Pei, X., Barbour, D. L., Leuthardt, E. C., and Schalk, G. (2011). Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans. J. Neural Eng. 8:46028. doi: 10.1088/1741-2560/8/4/046028
95. Pillow, J. W., Ahmadian, Y., and Paninski, L. (2011). Model-based decoding, information estimation, and change-point detection techniques for multineuron spike trains. Neural Comput. 23, 1-45. doi: 10.1162/NECO_a_00058
96. Poeppel, D., Emmorey, K., Hickok, G., and Pylkkänen, L. (2012). Towards a new neurobiology of language. J. Neurosci. 32, 14125-14131. doi: 10.1523/JNEUROSCI.3244-12.2012
97. Pulvermüller, F., Lutzenberger,W., and Preissl, H. (1999). Nouns and verbs in the intact brain: evidence fromevent-related potentials and high-frequency cortical responses. Cereb. Cortex 9, 497-506. doi: 10.1093/cercor/9.5.497
98. Qiu, A., Schreiner, C. E., and Escabí, M. A. (2003). Gabor analysis of auditory midbrain receptive fields: spectro-temporal and binaural composition. J. Neurophysiol. 90, 456-476. doi: 10.1152/jn.00851.2002
99. Quandt, F., Reichert, C., Hinrichs, H., Heinze, H.-J., Knight, R. T., and Rieger, J. W. (2012). Single trial discrimination of individual finger movements on one hand: a combined MEG and EEG study. Neuroimage 59, 3316-3324. doi: 10.1016/j.neuroimage.2011.11.053
100. Ray, S., and Maunsell, J. H. R. (2011). Different origins of gamma rhythm and high-gamma activity in macaque visual cortex. PLoS Biol. 9:e1000610. doi: 10.1371/journal.pbio.1000610
101. Reichert, C., Fendrich, R., Bernarding, J., Tempelmann, C., Hinrichs, H., and Rieger, J. W. (2014). Online tracking of the contents of conscious perception using real-time fMRI. Front. Neurosci. 8:116. doi: 10.3389/fnins.2014. 00116
102. Rieger, J. W., Reichert, C., Gegenfurtner, K. R., Noesselt, T., Braun, C., Heinze, H.-J., et al. (2008). Predicting the recognition of natural scenes from single trial MEG recordings of brain activity. Neuroimage 42, 1056-1068. doi: 10.1016/j.neuroimage.2008.06.014
103. Sahani, M., and Linden, J. F. (2003). How linear are auditory cortical responses? Adv. Neural Information Process. Syst. 15, 109-116. Available online at: https://papers.nips.cc/paper/2335-how-linear-are-auditory-cortical-responses
104. Santoro, R., Moerel, M., De Martino, F., Goebel, R., Ugurbil, K., Yacoub, E., et al. (2014). Encoding of natural sounds at multiple spectral and temporal resolutions in the human auditory cortex. PLoS Comput. Biol. 10:e1003412. doi: 10.1371/journal.pcbi.1003412
105. Schwartz,O., Pillow, J.W., Rust,N. C., and Simoncelli, E. P. (2006). Spike-triggered neural characterization. J. Vis. 6, 484-507. doi: 10.1167/6.4.13
106. Sen, K., Theunissen, F. E., and Doupe, A. J. (2001). Feature analysis of natural sounds in the songbird auditory forebrain. J. Neurophysiol. 86, 1445-1458. Available online at: http://jn.physiology.org/content/86/3/1445.long
107. Shamma, S. (2013). “Spectro-temporal receptive fields,” in Encyclopedia of Computational Neuroscience (New York, NY: Springer), 1-6. doi: 10.1007/978-1-4614-7320-6_437-1
108. Sharpee, T. O. (2016). How invariant feature selectivity is achieved in cortex. Front. Synaptic Neurosci. 8:26. doi: 10.3389/fnsyn.2016.00026
109. Sharpee, T. O., Atencio, C. A., and Schreiner, C. E. (2011). Hierarchical representations in the auditory cortex. Curr. Opin. Neurobiol. 21, 761-767. doi: 10.1016/j.conb.2011.05.027
110. Sharpee, T. O., Rust, N. C., and Bialek, W. (2004). Analyzing neural responses to natural signals: maximally informative dimensions. Neural Comput. 16, 223-250. doi: 10.1162/089976604322742010
111. Shelton, J. A., Sheikh, A. S., Bornschein, J., Sterne, P., and Lücke, J. (2015). Nonlinear spike-And-Slab sparse coding for interpretable image encoding. PLoS ONE 10:e0124088. doi: 10.1371/journal.pone.0124088
112. Slee, S. J., and David, S. V. (2015). Rapid task-related plasticity of spectrotemporal receptive fields in the auditory midbrain. J. Neurosci. 35, 13090-13102. doi: 10.1523/JNEUROSCI.1671-15.2015
113. Theunissen, F. E., and Elie, J. E. (2014). Neural processing of natural sounds. Nat. Rev. Neurosci. 15, 355-366. doi: 10.1038/nrn3731
114. Theunissen, F. E., David, S. V., Singh, N. C., Hsu, A., Vinje, W. E., and Gallant, J. L. (2001). Estimating spatio-temporal receptive fields of auditory and visual neurons from their responses to natural stimuli. Network 12, 289-316. doi: 10.1080/net.12.3.289.316
115. Theunissen, F. E., Sen, K., and Doupe, A. J. (2000). Spectral-temporal receptive fields of nonlinear auditory neurons obtained using natural sounds. J. Neurosci. 20, 2315-2331. Available online at: http://www.jneurosci.org/content/20/6/2315
116. Thirion, B., Duchesnay, E., Hubbard, E., Dubois, J., Poline, J.-B., Lebihan, D., et al. (2006). Inverse retinotopy: inferring the visual content of images from brain activation patterns. Neuroimage 33, 1104-1116. doi: 10.1016/j.neuroimage.2006.06.062
117. Thorson, I. L., Liénard, J., and David, S. V. (2015). The essential complexity of auditory receptive fields. PLoS Comput. Biol. 11:e1004628. doi: 10.1371/journal.pcbi.1004628
118. Touryan, J., Felsen, G., and Dan, Y. (2005). Spatial structure of complex cell receptive fields measured with natural images. Neuron 45, 781-791. doi: 10.1016/j.neuron.2005.01.029
119. Varoquaux, G., Raamana, P., Engemann, D., Hoyos-Idrobo, A., Schwartz, Y., and Thirion, B. (2016). Assessing and tuning brain decoders: crossvalidation, caveats, and guidelines. 145, 166-179. doi: 10.1016/j.neuroimage. 2016
120. Weichwald, S., Meyer, T., Özdenizci, O., Schölkopf, B., Ball, T., and Grosse-Wentrup, M. (2015). Causal interpretation rules for encoding and decoding models in neuroimaging. Neuroimage 110, 48-59. doi: 10.1016/j.neuroimage.2015.01.036
121. Willmore, B., and Smyth, D. (2003). Methods for first-order kernel estimation: simple-cell receptive fields from responses to natural scenes. Network 14, 553-577. doi: 10.1088/0954-898X_14_3_309
122. Woolley, S. M. N., Gill, P. R., Fremouw, T., and Theunissen, F. E. (2009). Functional groups in the avian auditory system. J. Neurosci. 29, 2780-2793. doi: 10.1523/JNEUROSCI.2042-08.2009
123. Wu, M. C.-K., David, S. V., and Gallant, J. L. (2006). Complete functional characterization of sensory neurons by system identification. Annu. Rev. Neurosci. 29, 477-505. doi: 10.1146/annurev.neuro.29.051605. 113024
124. Yamins, D. L. K., and DiCarlo, J. J. (2016). Eight open questions in the computational modeling of higher sensory cortex. Curr. Opin. Neurobiol. 37, 114-120. doi: 10.1016/j.conb.2016.02.001
125. Yin, P., Fritz, J. B., and Shamma, S. A. (2014). Rapid spectrotemporal plasticity in primary auditory cortex during behavior. J. Neurosci. 34, 4396-4408. doi: 10.1523/JNEUROSCI.2799-13.2014
126. Zhang, D., Gong, E., Wu, W., Lin, J., Zhou, W., and Hong B. (2012). “Spoken sentences decoding based on intracranial high gamma response using dynamic time warping,” in 2012 Annual International Conference of the IEEE Engineering inMedicine and Biology Society (San Diego, CA: IEEE), 3292-3295. doi: 10.1109/EMBC.2012.6346668
127. Zion Golumbic, E. M., Ding, N., Bickel, S., Lakatos, P., Schevon, C. A., McKhann, G. M., et al. (2013). Mechanisms underlying selective neuronal tracking of attended speech at a “cocktail party.” Neuron 77, 980-991. doi: 10.1016/j.neuron.2012.12.037
128. Zue, V., Seneff, S., and Glass, J. (1990). Speech database development at MIT: TIMIT and beyond. Speech Commun. 9, 351-356. doi: 10.1016/0167-6393 (90)90010-7