Decoding mental states from brain activity in humans

Nature Reviews Neuroscience

Volumn 7, Issue 7, 2006, Pages 523-534

  Haynes, John Dylan ^a,b,c   Rees, Geraint ^b,c  

^a MAX PLANCK INSTITUTE FOR HUMAN COGNITIVE AND BRAIN SCIENCES   (Germany)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

BLOOD OXYGENATION; BRAIN COMPUTER INTERFACE; COGNITION; MENTAL HEALTH; NEUROIMAGING; PRIORITY JOURNAL; REVIEW; VISION; VISUAL CORTEX;

BRAIN; BRAIN MAPPING; CONSCIOUSNESS; HUMANS; MAGNETIC RESONANCE IMAGING; MENTAL PROCESSES; PHOTIC STIMULATION; VISUAL PERCEPTION;

EID: 33745587777     PISSN: 1471003X     EISSN: 14710048     Source Type: Journal    
DOI: 10.1038/nrn1931     Document Type: Review

References (109)

1. Farah, M. J. Emerging ethical issues in neuroscience. Nature Neurosci. 5, 1123-1129 (2002).
2. Cox, D. D. & Savoy, R. L. Functional magnetic resonance imaging (fMRI) 'brain reading': detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage 19, 261-270 (2003). This study compares various classification techniques and outlines important principles of decoding-based fMRI research.
3. O'Craven, K. M. & Kanwisher, N. Mental imagery of faces and places activates corresponding stimulus-specific brain regions. J. Cogn. Neurosci. 12, 1013-1023 (2000).
4. Haxby, J. V. et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425-2430 (2001). One of the first studies using pattern-based analysis to investigate the nature of object representations in the human ventral visual cortex.
5. Carlson, T. A., Schrater, P. & He, S. Patterns of activity in the categorical representation of objects. J. Cogn. Neurosci. 15, 704-717 (2003).
6. Mitchell, T. M. et al. Classifying instantaneous cognitive states from FMRI data. AMIA Annu. Symp. Proc. 465-469 (2003).
7. Mitchell, T. M., Hutchinson, R., Niculescu, R. S., Pereira, F. & Wang, X. Learning to decode cognitive states from brain images. Machine Learning 57, 145-175 (2004).
8. Kamitani, Y. & Tong, F. Decoding the visual and subjective contents of the human brain. Nature Neurosci. 8, 679-685 (2005). The first application of multivariate classification to reveal processing of features in the primary visual cortex represented below the resolution of fMRI.
9. Haynes, J. D. & Rees, G. Predicting the orientation of invisible stimuli from activity in primary visual cortex. Nature Neurosci. 8, 686-691 (2005). This study directly compares perceptual performance with the performance of a decoder trained on fMRI-signals from the early visual cortex.
10. Haynes, J. D. & Rees, G. Predicting the stream of consciousness from activity in human visual cortex. Curr. Biol. 15, 1301-1307 (2005). This work reveals the potential power of multivariate decoding to track perception quasi-online on a second-to-second basis.
11. Kamitani, Y. & Tong, F. Decoding motion direction from activity in human visual cortex. J. Vision 5, 152a (2005).
12. Kriegeskorte, N., Goebel, R. & Bandettini, P. Information-based functional brain mapping. Proc. Natl Acad. Sci. USA 103, 3863-3868 (2006). This study introduces the 'searchlight' approach that searches across the entire brain for specific local patterns that encode information about a cognitive or perceptual state.
13. LaConte, S., Strother, S., Cherkassky, V., Anderson, J. & Hu, X. Support vector machines for temporal classification of block design fMRI data. Neuroimage 26, 317-329 (2005).
14. Mourao-Miranda, J., Bokde, A. L., Born, C., Hampel, H. & Stetter, M. Classifying brain states and determining the discriminating activation patterns: support vector machine on functional MRI data. Neuroimage 28, 980-995 (2005).
15. O'Toole, A., Jiang, F., Abdi, H. & Haxby, J. V. Partially distributed representation of objects and faces in ventral temporal cortex. J. Cogn. Neurosci. 17, 580-590 (2005).
16. Polyn, S. M., Natu, V. S., Cohen, J. D. & Norman, K. A. Category-specific cortical activity precedes retrieval during memory search. Science 310, 1963-1966 (2005).
17. Sidtis, J. J., Strother, S. C. & Rottenberg, D. A. Predicting performance from functional imaging data: methods matter. Neuroimage 20, 615-624 (2003).
18. Logothetis, N. K. & Pfeuffer, J. On the nature of the BOLD fMRI contrast mechanism. Magn. Reson. Imaging 22, 1517-1531 (2004).
19. Allison, T. et al. Face recognition in human extrastriate cortex. J. Neurophysiol. 71, 821-825 (1994).
20. Kanwisher, N., McDermott, J. & Chun, M. M. The fusiform face area: a module in human extrastriate cortex specialized for face perception. J. Neurosci. 17, 4302-4311 (1997).
21. Epstein, R. & Kanwisher, N. A cortical representation of the local visual environment. Nature 392, 598-601 (1999).
22. Engel, S. A. et al. FMRI of human visual cortex. Nature 369, 525 (1994).
23. Sereno, M. I. et al. Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. Science 268, 889-893 (1995).
24. Dehaene, S. et al. Inferring behavior from functional brain images. Nature Neurosci. 1, 549-550 (1998).
25. Tsao, D. Y., Freiwald, W. A., Tootell, R. B. & Livingstone, M. S. A cortical region consisting entirely of face-selective cells. Science 311, 670-674 (2006).
26. Downing, P. E., Jiang, Y., Shuman, M. & Kanwisher, N. A cortical area selective for visual processing of the human body. Science 293, 2470-2473 (2001).
27. Cohen, L. et al. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients, Brain 123, 291-307 (2000).
28. Downing, P. E., Chan, A. W. Y., Peelen, M. V., Dodds, C. M. & Kanwisher, N. Domain specificity in visual cortex. Cereb. Cortex Dec 7 2005 (doi: 10.1093/cercor/bhj086).
29. Ishai, A., Schmidt, C. F. & Boesiger, P. Face perception is mediated by a distributed cortical network. Brain Res. Bull. 67, 87-93 (2005).
30. Edelman, S., Grill-Spector, K., Kushnir, T. & Malach, R. Towards direct visualization of the internal shape space by fMRI. Psychobiology 26, 309-321 (1998). This early work is the first application of multivariate techniques to the study of object representation. Of special interest is the demonstration that shape space is reflected in the similarity space between evoked cortical responses.
31. Spiridon, M. & Kanwisher, N. How distributed is visual category information in human occipitotemporal cortex? An fMRI study. Neuron 35, 1157-1165 (2002).
32. Tanaka, K. Mechanisms of visual object recognition: monkey and human studies. Curr. Opin. Neurobiol. 7, 523-529 (1997).
33. Wang, C., Tanaka, K. & Tanifuji, M. Optical imaging of functional organization in the monkey inferotemporal cortex. Science 272, 1665-1668 (1996).
34. Obermayer, K. & Blasdel, G. G. Geometry of orientation and ocular dominance columns in monkey striate cortex. J. Neurosci. 13, 4114-4129 (1993).
35. James, W. The Principles of Psychology (Henry Holt, New York, 1890).
36. Blake, R. & Logothetis, N. K. Visual competition. Nature Rev. Neurosci. 3, 13-21 (2002).
37. Haynes, J. D., Deichmann, R. & Rees, G. Eye-specific effects of binocular rivalry in the human lateral geniculate nucleus. Nature 438, 496-499 (2005).
38. Brown, R. J. & Norcia, A. M. A method for investigating binocular rivalry in real-time with the steady-state VEP. Vision Res. 37, 2401-2408 (1997).
39. Hung, C. P., Kreiman, G., Poggio, T. & DiCarlo, J. J. Fast readout of object identity from macaque inferior temporal cortex. Science 310, 863-866 (2005).
40. Hasson, U., Nir, Y., Levy, I., Fuhrmann, G. & Malach, R. Intersubject synchronization of cortical activity during natural vision. Science 303, 1634-1640 (2004).
41. Bartels, A. & Zeki, S. Functional brain mapping during free viewing of natural scenes. Hum. Brain Mapp. 21, 75-85 (2004),
42. Yarbus, A. L. Eye Movements and Vision (Plenum, New York, 1967).
43. Duhamel, J. R., Colby, C. L. & Goldberg, M. E. The updating of the representation of visual space in parietal cortex by intended eye movements. Science 255, 90-92 (1992).
44. Davatzikos, C. et al. Classifying spatial patterns of brain activity with machine learning methods: application to lie detection. Neuroimage 28, 663-668 (2005). This study is the first demonstration that multivariate decoding can be applied to lie detection.
45. Langleben, D. D. et al. Telling truth from lie in individual subjects with fast event-related fMRI. Hum. Brain Mapp. 26, 262-272 (2005).
46. Kozel, F. A. et al. Detecting deception using functional magnetic resonance imaging. Biol. Psychiatry 58, 605-613 (2005).
47. Marcel, A. J. Conscious and unconscious perception: experiments on visual masking and word recognition. Cognit. Psychol. 15, 197-237 (1983).
48. Reingold, E. M. & Merikle, P. M. Using direct and indirect measures to study perception without awareness. Percept. Psychophys. 44, 563-575 (1988).
49. Dehaene, S. et al. Imaging unconscious semantic priming. Nature 395, 597-600 (1998)
50. Crick, F., & Koch, C. Are we aware of neural activity in primary visual cortex? Nature 375, 121-123 (1995)
51. Phelps, E. A. et al. Performance on indirect measures of race evaluation predicts amygdala activation. J. Cogn. Neurosci. 12, 729-738 (2000).
52. Libet, B., Gleason, C. A., Wright, E. W., Pearl, D. K. Time of conscious intention to act in relation to onset of cerebral activity (readiness-potential) . The unconscious initiation of a freely voluntary act. Brain 106, 623-642 (1983).
53. Haggard, P. & Eimer, M. On the relation between brain potentials and the awareness of voluntary movements. Exp. Brain. Res. 126, 128-133 (1999).
54. Tesla, N. Tremendous New Power to be Unleashed. Kansas City Journal-Post (10 Sep 1933).
55. Obrig, H. & Villringer, A. Beyond the visible - imaging the human brain with light. J. Cereb. Blood Flow Metab. 23, 1-18 (2003).
56. Suppes, P., Han, B., Epelboim, J. & Lu, Z. L. Invariance between subjects of brain wave representations of language. Proc. Natl Acad. Sci. USA 96, 12953-12958 (1999).
57. Friston, K. J. et al. Spatial registration and normalisation of images. Hum. Brain Mapp. 2, 165-189 (1995).
58. Fischl, B., Sereno, M. I., Tootell, R. B. & Dale, A. M. High resolution intersubject averaging and a coordinate system for the cortical surface. Hum. Brain Mapp. 8, 272-284 (1999).
59. Grill-Spector, K., Henson, R. & Martin, A. Repetition and the brain: neural models of stimulus-specific effects. Trends Cogn. Sci. 10, 14-23 (2006).
60. Mountcastle, V. B. The columnar organization of the neocortex. Brain 120, 701-722 (1997).
61. Horton, J. C. & Adams, D. L. The cortical column: a structure without a function. Philos. Trans. R. Soc. Lond. B Biol. Sci. 360, 837-862 (2005).
62. Poldrack, R. A. Can cognitive processes be inferred from neuroimaging data? Trends Cogn. Sci. 10, 59-63 (2006).
63. Cunningham, W. A. et al. Separable neural components in the processing of black and white faces. Psychol. Sci. 15, 806-813 (2004).
64. Richeson, J. A. et al. An fMRI investigation of the impact of interracial contact on executive function. Nature Neurosci. 6, 1323-1328 (2003).
65. Beauregard, M., Levesque, J. & Bourgouin, P. Neural correlates of conscious self-regulation of emotion. J. Neurosci. 21, RC165 (2001).
66. Phan, K. L., Wager, T., Taylor, S. F. & Liberzon, I. Functional neuroanatomy of emotion: a meta-analysis of emotion activation studies in PET and fMRI. Neuroimage 16, 331-348 (2002).
67. Canli, T. & Amin, Z. Neuroimaging of emotion and personality: scientific evidence and ethical considerations. Brain Cogn. 50, 414-431 (2002).
68. McCloskey, M. S., Phan, K. L. & Coccaro, E. F. Neuroimaging and personality disorders. Curr. Psychiatry Rep. 7, 65-72 (2005).
69. Pridmore, S., Chambers, A. & McArthur, M. Neuroimaging in psychopathy. Aust. N. Z. J. Psychiatry 39, 856-865 (2005).
70. Raine, A. et al. Reduced prefrontal and increased subcortical brain functioning assessed using positron emission tomography in predatory and affective murderers. Behav. Sci. Law 16, 319-332 (1998).
71. Childress, A. R. et al. Limbic activation during cue-induced cocaine craving. Am. J. Psychiatry 156, 11-18 (1999).
72. McClure, S. M. et al. Neural correlates of behavioral preference for culturally familiar drinks. Neuron 44, 379-387 (2004).
73. Heekeren, H. R., Marrett, S., Bandettini, P. A. & Ungerleider, L. G. A general mechanism for perceptual decision-making in the human brain. Nature 431, 859-862 (2004).
74. Levy, D. E. et al. Differences in cerebral blood flow and glucose utilization in vegetative versus locked-in patients. Ann. Neurol. 22, 673-682 (1987).
75. Laureys, S., Perrin, F., Schnakers, C., Boly, M. & Majerus, S. Residual cognitive function in comatose, vegetative and minimally conscious states. Curr. Opin. Neurol. 18, 726-733 (2005).
76. Farah, M. J. Neuroethics: the practical and the philosophical. Trends Cogn. Sci. 9, 34-40 (2005).
77. Brammer, M. Brain scam? Nature Neurosci. 7, 1015 (2004).
78. Dickson, K. & McMahon, M. Will the law come running? The potential role of 'brain fingerprinting' in crime investigation and adjudication in Australia. J. Law Med. 13, 204-222 (2005).
79. Dewan, E. M. Occipital alpha rhythm, eye position and lens accommodation. Nature 214, 975-977 (1967).
80. Nicolelis, M. A. Actions from thoughts. Nature 409, 403-407 (2001).
81. Andersen, R. A., Burdick, J. W., Musallam, S., Pesaran, B. & Cham, J. C. Cognitive neural prosthetics. Trends Cogn. Sci. 8, 486-493 (2004).
82. Blankertz, B. et al. Boosting bit rates and error detection for the classification of fast-paced motor commands based on single-trial EEG analysis. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 127-131 (2003).
83. Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G. & Vaughan, T. M. Brain-computer interfaces for communication and control. Clin. Neurophysiol. 113, 767-791 (2002).
84. Wolpaw, J. R. & McFarland, D. J. Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc. Natl Acad. Sci. USA 101, 17849-17854 (2004).
85. Kreiman, G., Koch, C. & Fried, I. Category-specific visual responses of single neurons in the human median temporal lobe. Nature Neurosci. 3, 946-953 (2000).
86. Quiroga, R. Q., Reddy, L., Kreiman, G., Koch, C. & Fried, I. Invariant visual representation by single neurons in the human brain. Nature 435, 1102-1107 (2005). Here, multivariate decoding is applied to simultaneous recordings of spike trains from the human medial temporal lobe.
87. Kennedy, P. R. & Bakay, R. A. Restoration of neural output from a paralyzed patient by a direct brain connection. Neuroreport 9, 1707-1711 (1998).
88. Birbaumer, N. et al. A spelling device for the paralysed. Nature 398, 297-298 (1999).
89. Weiskopf, N. et al. Self-regulation of local brain activity using real-time functional magnetic resonance imaging (fMRI). J. Physiol. (Paris) 98, 357-373 (2004).
90. Stanley, G. B., Li, F. F. & Dan, Y. Reconstruction of natural scenes from ensemble responses in the lateral geniculate nucleus. J. Neurosci. 19, 8036-8042 (1999).
91. Ekman, P. & O'Sullivan, M. Who can catch a liar? Am. Psychol. 46, 913-920 (1991).
92. Marston, W. M. The systolic blood pressure symptoms of deception. J. Exp. Psy. 2, 117-163 (1917).
93. Geddes, L. A. History of the polygraph, an instrument for the detection of deception. Biomed. Eng. 8, 154-156 (1973).
94. Burtt, H. E. The inspiration-expiration ratio during truth and falsehood. J. Exp. Psy. 4, 1-23 (1921).
95. Thackeray, R. J. & Orne, M. T. A comparison of physiological indices in detection of deception. Psychophysiol. 4, 329-339 (1968).
96. Horvath, F. Detecting deception: the promise and the reality of voice stress analysis. J. Forensic Sci. 27, 340-351 (1982).
97. Pavlidis, I., Eberhardt, N. L. & Levine, J. A. Seeing through the face of deception. Nature 415, 35 (2002).
98. Pollina, D. A., Dollins, A. B., Senter, S. M., Krapohl, D. J. & Ryan, A. H. Comparison of polygraph data obtained from individuals involved in mock crimes and actual crime investigations. J. Appl. Psychol. 89, 1099-1105 (2004).
99. Lykken, D. T. Tremor in the Blood: Uses and Abuses of the Lie Detector (McGraw-Hill, New York, 1981).
100. Honts, C. R., Raskin, D. C. & Kircher, J. C. Mental and physical countermeasures reduce the accuracy of polygraph tests. J. App. Psy. 79, 252-259 (1994).
101. Farwell, L. A. & Smith, S. S. Using brain MERMER testing to detect knowledge despite efforts to conceal. J. Forensic Sci. 46, 135-143 (2001).
102. Spence, S. A. et al. Behavioral and functional anatomical correlates of deception in humans. Neuroreport 12, 2849-2853 (2001).
103. Phan, K. L. et al. Neural correlates of telling lies: a functional magnetic resonance imaging study at 4 Tesla. Acad. Radiol. 12, 164-172 (2005).
104. Ganis, G., Kosslyn, S. M., Stose, S., Thompson, W. L. & Yurgelun, T. Neural correlates of different types of deception: an fMRI investigation. Cereb. Cortex 13, 830-836 (2003).
105. Lee, T. M. et al. Lie detection by functional magnetic resonance imaging. Hum. Brain Mapp. 15, 157-164 (2002).
106. Boynton, G. Imaging orientation selectivity: decoding conscious perception in V1. Nature Neurosci. 8, 541-542 (2005).
107. Nevado, Y., Young, M. P. & Panzeri, S. Functional imaging and neural information coding. Neuroimage 21, 1083-1095 (2004).
108. Turner, R. How much cortex can a vein drain? Downstream dilution of activation-related cerebral blood oxygenation changes. Neuroimage 16, 1062-1067 (2002).
109. Duvernoy, H. The Human Brain (Springer, New York, 1999).