Towards noninvasive hybrid brain-computer interfaces: Framework, practice, clinical application, and beyond

Proceedings of the IEEE

Volumn 103, Issue 6, 2015, Pages 926-943

  Müller Putz, Gernot ^a   Leeb, Robert ^b   Tangermann, Michael ^c   Höhne, Johannes ^d   Kübler, Andrea ^e   Cincotti, Febo ^f   Mattia, Donatella ^g   Rupp, Rüdiger ^h   Müller, Klaus Robert ^d,i   Millán, José Del R ^b  

^a GRAZ UNIVERSITY OF TECHNOLOGY   (Austria)

^b EPFL   (Switzerland)

^c UNIVERSITY OF FREIBURG   (Germany)

^d TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

^e UNIVERSITY OF WÜRZBURG   (Germany)

^f SAPIENZA UNIVERSITY OF ROME   (Italy)

^g IRCCS FONDAZIONE SANTA LUCIA   (Italy)

^h UNIVERSITY OF HEIDELBERG   (Germany)

ⁱ Korea University   (South Korea)

Author keywords

Assistive technology; communication; electroencephalogram; hybrid brain computer interface (hBCI); neuroprosthesis

Indexed keywords

COMMUNICATION; ELECTROENCEPHALOGRAPHY; INTERFACE STATES; NEUROSURGERY;

ASSISTIVE TECHNOLOGY; BRAIN COMPUTER INTERFACES (BCIS); CLINICAL APPLICATION; EUROPEAN COMMISSION; HYBRID BRAIN-COMPUTER INTERFACE (HBCI); INTEGRATED PROJECT; NEURO-PROSTHESIS; REAL TIME INTERACTIONS;

BRAIN COMPUTER INTERFACE;

EID: 84930948309     PISSN: 00189219     EISSN: 15582256     Source Type: Journal    
DOI: 10.1109/JPROC.2015.2411333     Document Type: Article

References (91)

1. J. J. Vidal, "Toward direct brain-computer communication," Annu. Rev. Biophys. Bioeng., vol. 2, pp. 157-180, 1973.
2. S. Amiri, R. Fazel-Rezai, and V. Asadpour, "A review of hybrid brain-computer interface systems," Adv. Human-Comput. Interact., vol. 2013, pp. 1-8, 2013.
3. J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, "Brain-computer interfaces for communication and control," Clin. Neurophysiol., vol. 113, no. 6, pp. 767-791, 2002.
4. T. Zeyl and T. Chau, "A case study of linear classifiers adapted using imperfect labels derived from human event-related potentials," Pattern Recogn. Lett., vol. 37, pp. 54-62, 2014.
5. G. Dornhege, J. d. R. Millán, T. Hinterberger, D. J. Mcfarland, and K.-R. Müller, Eds., Toward Brain-Computing Interfacing. Cambridge, MA, USA: MIT Press, 2007.
6. J. d. R. Millán et al., "Combining brain-computer interfaces and assistive technologies: State-of-the-art and challenges," Front Neurosci., vol. 4, p. 161, 2010.
7. B. Allison, S. Dunne, R. Leeb, J. d. R. Millán, and A. Nijholt, Towards Practical Brain-Computer Interfaces: Bridging the Gap from Research to Real-World Applications. Berlin, Germany: Springer-Verlag, 2012.
8. N. Birbaumer et al., "A spelling device for the paralysed," Nature, vol. 398, no. 6725, pp. 297-298, 1999.
9. G. Pfurtscheller, G. R. Müller, J. Pfurtscheller, H. J. Gerner, and R. Rupp, "'Thought'-control of functional electrical stimulation to restore hand grasp in a patient with tetraplegia," Neurosci. Lett., vol. 351, no. 1, pp. 33-36, 2003.
10. G. Müller-Putz, R. Scherer, G. Pfurtscheller, and R. Rupp, "EEG-based neuroprosthesis control: A step towards clinical practice," Neurosci. Lett., vol. 382, pp. 169-174, 2005.
11. F. Galán et al., "A brain-actuated wheelchair: Asynchronous and non-invasive brain-computer interfaces for continuous control of robots," Clin. Neurophysiol., vol. 119, no. 9, pp. 2159-2169, 2008.
12. J. d. R. Millán et al., "Asynchronous non-invasive brain-actuated control of an intelligent wheelchair," in Proc. 31st Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., 2009, pp. 3361-3364.
13. R. Leeb et al., "Brain-computer communication: Motivation, aim and impact of exploring a virtual apartment," IEEE Trans. Neural Syst. Rehab. Eng., vol. 15, pp. 473-482, 2007.
14. F. Lotte et al., "Exploring large virtual environments by thoughts using a brain-computer interface based on motor imagery and high-level commands," Presence-Teleop. Virt., vol. 19, no. 1, pp. 54-70, 2010.
15. S. Haufe et al., "EEG potentials predict upcoming emergency brakings during simulated driving," J. Neural Eng., vol. 8, no. 5, p. 056001, 2011.
16. J. d. R. Millán, "Adaptive brain interfaces," Commun. ACM, vol. 46, no. 3, pp. 74-80, 2003.
17. K.-R. Müller et al., "Machine learning for real-time single-trial EEG-analysis: From brain-computer interfacing to mental state monitoring," J. Neurosci. Meth., vol. 167, no. 1, pp. 82-90, 2008.
18. B. Blankertz et al., "The Berlin brain-computer interface:Non-medical uses of BCI technology," Front Neurosci., vol. 4, no. 198, 2010.
19. Z. Khaliliardali, R. Chavarriaga, L. A. Gheorghe, and J. d. R. Millán, "Detection of anticipatory brain potentials during car driving," in Proc. 34st Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., 2012, pp. 3829-3832.
20. S. Scholler et al., "Towards a direct measure of video quality perception using EEG," IEEE Trans. Image Process., vol. 21, no. 5, pp. 2619-2629, 2012.
21. A. K. Porbadnigk et al., "Single-trial analysis of the neural correlates of speech quality perception," J. Neural Eng., vol. 10, no. 5, p. 056003, 2013.
22. J. d. R. Millán, P. W. Ferrez, and T. Seidl, "Validation of brain-machine interfaces during parabolic flight," Int. Rev. Neurobiol., vol. 86, pp. 189-197, 2009.
23. D. deNegueruela, M. Broschart, C.Menon, and J. d. R. Millán, "Brain-computer interfaces for space applications," Pers. Ubiquitous Comput., vol. 15, no. 5, pp. 527-537, 2011.
24. L. A. Gheorghe, R. Chavarriaga, and J. d. R. Millán, "Steering timing prediction in a driving simulator task," in Proc. 35th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., 2013.
25. B. Blankertz, G. Curio, and K. R. Muller, "Classifying single trial EEG: Towards brain computer interfacing," NIPS, vol. 1, pp. 157-164, 2002.
26. J. d. R. Millán et al., "A local neural classifier for the recognition of EEG patterns associated to mental tasks," IEEE Trans. Neural Netw., vol. 13, no. 3, pp. 678-686, 2002.
27. J. d. R. Millán, F. Renkens, J. Mouriño, and W. Gerstner, "Brain-actuated interaction," Artif. Intell., vol. 159, no. 1-2, pp. 241-259, 2004.
28. B. Blankertz, G. Dornhege, M. Krauledat, K. R. Müller, and G. Curio, "The non-invasive Berlin brain-computer interface: Fast acquisition of effective performance in untrained subjects," NeuroImage, vol. 37, no. 2, pp. 539-550, 2007.
29. B. Blankertz, R. Tomioka, S. Lemm, M. Kawanabe, and K. R. Müller, "Optimizing Spatial Filters for Robust EEG Single-Trial Analysis," IEEE Signal Process. Mag., vol. 25, no. 1, pp. 41-56, 2008.
30. S. Lemm, T. Dickhaus, B. Blankertz, and K.-R. Müller, "Introduction to machine learning for brain imaging," Neuroimage, vol. 56, no. 2, pp. 387-399, 2011.
31. R. Tomioka and K. R. Müller, "A regularized discriminative framework for EEG analysis with application to brain-computer interface," Neuroimage, vol. 49, no. 1, pp. 415-432, 2010.
32. J. d. R. Millán, P. W. Ferrez, F. Galán, E. Lew, and R. Chavarriaga, "Non-invasive brain-machine interaction," Int. J. Pattern Recogn. Artif. Intell., vol. 22, no. 5, pp. 959-972, 2008.
33. J. d. R. Millán et al., "Adaptive Brain Interfaces for Physically-Disabled People," in Proc. 20th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc.Hong Kong, 1998, pp. 2008-2011.
34. J. d. R.Millán et al., "Local Neural Classifier for EEG-based Recognition of Mental Tasks," in Proc. IEEE-INNS-ENNS Int. Joint Conf. Neural Netw., Como, Italy, 2000, vol. III, pp. 632-636.
35. J. d. R. Millán and J. Mouriño, "Asynchronous BCI and Local Neural Classifiers: An Overview of the Adaptive Brain Interface Project," IEEE Trans. Neural Syst. Rehab. Eng., vol. 11, no. 2, pp. 159-161, 2003.
36. A. Buttfield, P. W. Ferrez, and J. d. R. Millán, "Towards a Robust BCI: Error recognition and online learning," IEEE Trans. Neural Syst. Rehab. Eng., vol. 14, no. 2, pp. 164-168, 2006.
37. T. Kaufmann, S. M. Schulz, C. Grünzinger, and A. Kübler, "Flashing characters with famous faces improves ERP-based brain-computer interface performance," J. Neural Eng., vol. 8, no. 5, p. 056016, 2011.
38. S. Debener, F. Minow, R. Emkes, K. Gandras, and M. de Vos, "How about taking a low-cost, small, and wireless EEG for a walk?" Psychophysiol., 2012, 10.1111/j.1469-8986. 2012.01471.x.
39. F. Popescu, S. Fazli, Y. Badower, B. Blankertz, and K. R. Müller, "Single trial classification of motor imagination using 6 dry EEG electrodes," PLoS One, vol. 2, no. 7, 2007.
40. C. Guger, G. Krausz, B. Z. Allison, and G. Edlinger, "Comparison of dry and gel based electrodes for P300 brain-computer interfaces," Front. Neurosci., vol. 6, no. 60, 2012.
41. C. Grozea, C. D. Voinescu, and S. Fazli, "Bristle-sensors-low-cost flexible passive dry EEG electrodes for neurofeedback and BCI applications," J. Neural Eng., vol. 8, no. 2, p. 025008, 2011.
42. F. Aloise et al., "Asynchronous P300-based brain-computer interface to control a virtual environment: Initial tests on end users," Clin. EEG Neurosci., vol. 42, no. 4, pp. 219-224, Oct. 2011.
43. C. Zickler et al., "A brain-computer interface as input channel for a standard assistive technology software," Clin. EEG Neurosci., vol. 42, no. 4, pp. 236-44, 2011.
44. E. W. Sellers, T.M. Vaughan, and J. R. Wolpaw, "A brain-computer interface for long-term independent home use," Amyotroph. Lateral Scler., vol. 11, no. 5, pp. 449-55, 2010.
45. R. Leeb et al., "Are we ready? Issues in transferring BCI technology from experts to users," in Proc. 5th Int. Brain-Computer Interface Conf., 2011.
46. A. Riccio, F. Leotta, S. Tiripicchio, D. Mattia, and F. Cincotti, "Can Severe Acquired Brain Injury Users Control a Communication Application Operated through a P300-Based Brain Computer Interface?" in Proc. 5th Int. Brain-Computer Interface Conf. 2011, Graz, Austria, 2011.
47. D. Mattia, F. Pichiorri, M. Molinari, and R. Rupp, "Brain Computer Interface for Hand Motor Function Restoration and Rehabilitation Towards Practical Brain-Computer Interfaces. New York, NY, USA: Springer, 2013, pt. 2, pp. 131-15310.1007978-3-642-29746-5-7. [Online]. Available: http://www.springerlink. com/content/j1847t5654245675/).
48. A. Kübler, B. Kotchoubey, J. Kaiser, J. R. Wolpaw, and N. Birbaumer, "Brain-computer communication: Unlocking the locked in," Psychol. Bull., vol. 127, no. 3, pp. 358-375, 2001.
49. G. Pfurtscheller et al., "The hybrid BCI," Front. Neurosci., vol. 4, no. 30, 2010, 10.3389/fnpro.2010.00003.
50. G. Müller-Putz et al., "Tools for brain-computer interaction: A general concept for a hybrid BCI," Front. Neuroinf., vol. 5, no. 30, pp. 1-10, 2011.
51. C. Breitwieser, I. Daly, C. Neuper, and G. Müller-Putz, "Proposing a standardized protocol for raw biosignal transmission," IEEE Trans. Biomed. Eng., vol. 59, no. 3, pp. 852-859, 2012.
52. S. G. Mason and G. E. Birch, "A general framework for brain-computer interface design," IEEE Trans. Neural Syst. Rehab. Eng., vol. 11, no. 1, pp. 70-85, 2003.
53. O. Flemisch et al., "The H-Metaphor as a guideline for vehicle automation and interaction," in NASA, NASA/TM-2003-212672, 2003
54. D. Vanhooydonck, E. Demeester, M. Nuttin, and H. Van Brussel, "Shared control for intelligent wheelchairs: An implicit estimation of the user intention," in Proc. 1st Int. Workshop Adv. Service Robot., 2003, pp. 176-182.
55. J. R. Wolpaw, "Brain-computer interfaces as new brain output pathways," J. Physiol., vol. 579, pt. 3, pp. 614-619, 2007.
56. K. D. Anderson, "Targeting recovery: Priorities of the spinal cord-injured population," J. Neurotrauma, vol. 21, pp. 1371-83, 2004.
57. G. J. Snoek, M. J. Ijzerman, H. J. Hermens, D. Maxwell, and F. Biering-Sorensen, "Survey of the needs of patients with spinal cord injury: Impact and priority for improvement in hand function in tetraplegics," Spinal Cord, vol. 42, pp. 526-532, 2004.
58. M. B. Popovic, D. B. Popovic, T. Sinkjaer, A. Stefanovic, and L. Schwirtlich, "Clinical evaluation of functional electrical therapy in acute hemiplegic subjects," J. Rehab. Res. Dev., vol. 40, pp. 443-453, 2003.
59. D. Popovic et al., "Clinical evaluation of the bionic glove," Arch. Phys. Med. Rehab., vol. 80, pp. 299-304, 1999.
60. G. Alon and K. McBride, "Persons with C5 or C6 tetraplegia achieve selected functional gains using a neuroprosthesis," Arch. Phys. Med. Rehab., vol. 84, pp. 119-124, 2003.
61. M. Rohm et al., "Hybrid brain-computer interfaces and hybrid neuroprostheses for restoration of upper limb functions in individuals with high-level spinal cord injury," Artif. Intel. Med., vol. 59, no. 2, pp. 133-142, 2013, 10.1016/j.artmed.2013.07.004PMID: 24064256.
62. M. Rohm et al., "Modular FES-hybrid orthosis for individualized setup of BCI controlled motor substitution and recovery," Int. J. Bioelectromagn., vol. 13, pp. 127-128, 2011.
63. G. R.Müller-Putz, R. Scherer, G. Pfurtscheller, and R. Rupp, "Brain-computer interfaces for control of neuroprostheses: from synchronous to asynchronous mode of operation," Biomed. Tech., vol. 51, pp. 57-63, 2006.
64. Z. Wu et al., "The convergence of machine and biological intelligence," IEEE Intell. Syst., vol. 28, no. 5, pp. 28-43, 2013, 10.1109/MIS.2013.137.
65. R. Leeb, H. Sagha, R. Chavarriaga, and J. d. R. Millán, "A hybrid brain-computer interface based on the fusion of electroencephalographic and electromyographic activities," J. Neural Eng., vol. 8, no. 2, p. 025011, 2011.
66. R. Leeb et al., "Transferring brain-computer interfaces beyond the laboratory: Successful application control for motor-disabled users," Artif. Intell. Med., vol. 59, no. 2, pp. 121-132, 2013.
67. G. Rogova and V. Nimier, "Reliability in information fusion: Literature survey," in Proc. 7th Int. Conf. Inf. Fusion, 2004, pp. 1158-1165.
68. N. A. Dimitrova and G. V. Dimitrov, "Interpretation of EMG changes with fatigue: Facts, pitfalls, and fallacies," J. Electromyog. Kinesiol., vol. 13, no. 1, pp. 13-36, 2003.
69. G. Schalk, J. R. Wolpaw, D. J. McFarland, and G. Pfurtscheller, "EEG-based communication: Presence of an error potential," Clin. Neurophysiol., vol. 111, no. 12, pp. 2138-2144, 2000.
70. P. W. Ferrez and J. d. R. Millán, "Error-related EEG potentials generated during simulated brain-computer interaction," IEEE Trans. Biomed. Eng., vol. 55, pp. 923-929, 2008.
71. P. W. Ferrez and J. d. R. Millán, "Simultaneous real-time detection of motor imagery and error-related potentials for improved BCI accuracy," in Proc 4th Int. Brain-Computer Interface Workshop, Graz, Austria, 2008b, pp. 197-202.
72. R. Chavarriaga and J. d. R. Millán, "Learning from EEG Error-related Potentials in Noninvasive Brain-Computer Interfaces," IEEE Trans. Neural Syst. Rehab. Eng., vol. 18, no. 4, pp. 381-388, 2010.
73. B. Blankertz et al., "BCI bit rates and error detection for fast-pace motor commands based on single-trial EEG analysis," IEEE Trans. Neural Syst. Rehab. Eng., vol. 11, no. 2, pp. 127-131, 2003.
74. R. Leeb, M. Lancelle, V. Kaiser, D. Fellner, and G. Pfurtscheller, "Thinking Penguin: Multi-modal brain-computer interface control of a VR Game," IEEE Trans. Computat. Intell. AI Games, vol. 5, no. 2, pp. 117-128, 2013.
75. G. Pfurtscheller and T. Solis-Escalante, "Could the beta rebound in the EEG be suitable to realize a 'brain switch'?" in Clin. Neurophysiol., vol. 120, pp. 24-29, 2009.
76. S. Perdikis et al., "Clinical evaluation of BrainTree, a motor imagery hybrid BCI speller," J. Neural Eng., vol. 11, no. 3, p. 036003, 2014.
77. G. Dornhege, B. Blankertz, G. Curio, and K.-R. Müller, "Boosting Bit Rates in Noninvasive EEG single-trial classifications by feature combination and multi-class paradigms," IEEE Trans. Biomed. Eng., vol. 51, no. 6, pp. 993-1002, 2004.
78. A. Kübler, N. Neumann, B. Wilhelm, T. Hinterberger, and N. Birbaumer, "Predictability of brain-computer communication," J. Psychophysiol., vol. 18, no. 2-3, pp. 121-129, 2004.
79. B. Rebsamen et al., "A brain controlled wheelchair to navigate in familiar environments," IEEE Trans. Neural Syst. Rehab. Eng., vol. 18, no. 6, pp. 590-598, 2010.
80. G. Vanacker et al., "Context-based filtering for assisted brain-actuated wheelchair driving," Computat. Intell. Neurosci., 2007.
81. T. Carlson and J. d. R. Millán, "Brain-controlled wheelchairs: A robotic architecture," IEEE Robot. Autom. Mag., vol. 20, no. 1, pp. 65-73, 2013.
82. J. Höhne et al., "Motor imagery for severely motor-impaired patients: Evidence for brain-computer interfacing as superior control solution," PLoS One, vol. 9, no. 8, 2014, 10.1371/journal.pone.0104854.
83. A. Riccio et al., "Workload measurement in a communication application operated through a P300-based brain-computer interface," J. Neural Eng., vol. 8, no. 2, p. 025028, 2011.
84. F. I. Mahoney and D. W. Barthel, "Functional evaluation: The barthel index," Maryland State Med. J., vol. 14, pp. 61-65, 1965.
85. S. G. Hart, "Nasa-task load index (nasa-tlx); 20 years later," in Proc. Human Factors Ergonom. Soc. Annu. Meeting, 2006, vol. 50, no. 9, pp. 904-908.
86. S. Fazli et al., "Enhanced performance by a Hybrid NIRS-EEG Brain Computer Interface," NeuroImage, vol. 59, no. 1, pp. 519-529, 2012.
87. M. A. Dimyan and L. G. Cohen., "Neuroplasticity in the context of motor rehabilitation after stroke," Nature Rev. Neurol., vol. 7, no. 2, pp. 76-85, 2011.
88. P. Langhorne, F. Coupar, and A. Pollock, "Motor recovery after stroke: A systematic review," The Lancet. Neurol., vol. 8, no. 8, pp. 741-754, 2009.
89. J. J. Shih, D. J. Krusienski, and J. R. Wolpaw, "Brain-computer interfaces in medicine," Mayo Clin. Proc., vol. 87, no. 3, pp. 268-279, 2012.
90. S. R. Zeiler and J. W. Krakauer, "The interaction between train-ing and plasticity in the poststroke brain," Current Opinion Neurol., vol. 26, no. 6, pp. 609-616, 2013.
91. P. Bünau, F. C. Meinecke, F. Kiraly, and K.-R. Müller, "Finding stationary subspaces in multivariate time series," Phys. Rev. Lett., vol. 103, p. 214101, 2009.