Performance measurement for brain-computer or brain-machine interfaces: A tutorial

Journal of Neural Engineering

Volumn 11, Issue 3, 2014, Pages

  Thompson, David E ^a,j   Quitadamo, Lucia R ^b   Mainardi, Luca ^c   Laghari, Khalil Ur Rehman ^d   Gao, Shangkai ^e   Kindermans, Pieter Jan ^f   Simeral, John D ^g   Fazel Rezai, Reza ^h   Matteucci, Matteo ^c   Falk, Tiago H ^d   Bianchi, Luigi ^b   Chestek, Cynthia A ^a   Huggins, Jane E ^a,i  

^a UNIVERSITY OF MICHIGAN   (United States)

^b UNIVERSITY OF ROME TOR VERGATA   (Italy)

^c POLITECNICO DI MILANO   (Italy)

^d INRS EMT   (Canada)

^e Tsinghua University   (China)

^f GHENT UNIVERSITY   (Belgium)

^g Brown University   (United States)

^h University of North Dakota   (United States)

ⁱ UNIVERSITY OF MICHIGAN MEDICAL SCHOOL   (United States)

^j Kansas State University   (United States)

more..

Author keywords

Brain computer interface; Brain machine interface; Metrics; Performance measurement

Indexed keywords

INTERFACES (COMPUTER); RESEARCH; STUDENTS;

BRAIN COMPUTER INTERFACES (BCIS); BRAIN MACHINE INTERFACE; CLINICAL TOOLS; CONFERENCE CENTERS; GRADUATE STUDENTS; METRICS; PERFORMANCE MEASUREMENTS; PERFORMANCE REPORTING;

BRAIN COMPUTER INTERFACE;

ACCURACY; ALGORITHM; ARTICLE; BINOMIAL DISTRIBUTION; BRAIN COMPUTER INTERFACE; COGNITION; CONFIDENCE INTERVAL; CONSENSUS; CORRELATION COEFFICIENT; EVENT RELATED POTENTIAL; EVOKED VISUAL RESPONSE; GRADUATE STUDENT; HUMAN; LATENT PERIOD; MOTOR PERFORMANCE; PRIORITY JOURNAL; QUALITY OF LIFE; RANGE OF MOTION; STEADY STATE; WORKSHOP; DEVICE FAILURE ANALYSIS; DEVICES; ELECTROENCEPHALOGRAPHY; NEUROFEEDBACK; PRACTICE GUIDELINE; STANDARDS; UNITED STATES;

BRAIN-COMPUTER INTERFACES; ELECTROENCEPHALOGRAPHY; EQUIPMENT FAILURE ANALYSIS; GUIDELINE ADHERENCE; NEUROFEEDBACK; PRACTICE GUIDELINES AS TOPIC; UNITED STATES;

EID: 84901269742     PISSN: 17412560     EISSN: 17412552     Source Type: Journal    
DOI: 10.1088/1741-2560/11/3/035001     Document Type: Article

References (65)

1. Wolpaw J R, Birbaumer N, McFarland D J, Pfurtscheller G and Vaughan T M 2002 Brain-computer interfaces for communication and control Clin. Neurophysiol. 113 767-91 (Pubitemid 34621187)
2. Sellers E W, Vaughan T M and Wolpaw J R 2010 A brain-computer interface for long-term independent home use Amyotroph Lateral Scler. 11 449-55
3. Hochberg L R et al 2012 Reach and grasp by people with tetraplegia using a neurally controlled robotic arm Nature 485 372-5
4. Hochberg L R et al 2006 Neuronal ensemble control of prosthetic devices by a human with tetraplegia Nature 442 164-71 (Pubitemid 44086558)
5. Brunner P, Ritaccio A L, Emrich J F, Bischof H and Schalk G 2011 Rapid communication with a 'P300' matrix speller using electrocorticographic signals (ECoG) Front. Neurosci. 5 5 (www.ncbi.nlm.nih.gov/pmc/articles/PMC3037528/)
6. Leuthardt E C et al 2011 Using the electrocorticographic speech network to control a brain-computer interface in humans J. Neural Eng. 8 036004
7. Wang W et al 2013 An electrocorticographic brain interface in an individual with tetraplegia PloS One 8 e55344
8. Zhang D, Song H, Xu R, Zhou W, Ling Z and Hong B 2013 Toward a minimally invasive brain-computer interface using a single subdural channel: a visual speller study Neuroimage 71 30-41
9. Wolpaw J R and McFarland D J 2004 Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans Proc. Natl Acad. Sci. USA 101 17849-54 (Pubitemid 40051975)
10. Blankertz B, Dornhege G, Krauledat M, Müller K-R and Curio G 2007 The non-invasive Berlin brain-computer interface: fast acquisition of effective performance in untrained subjects Neuroimage 37 539-50 (Pubitemid 47444247)
11. Cincotti F, Quitadamo L R, Aloise F, Bianchi L, Babiloni F and Mattia D 2009 Interacting with the environment through non-invasive brain-computer interfaces Universal Access in Human-Computer Interaction. Intelligent and Ubiquitous Interaction Environments (Berlin: Springer) pp 483-92
12. Halder S et al 2010 An auditory oddball brain-computer interface for binary choices Clin. Neurophysiol. 121 516-23
13. Pfurtscheller G et al 2010 The hybrid BCI Front. Neurosci. 4 30 (www.ncbi.nlm.nih.gov/pmc/articles/PMC2891647/)
14. Coyle S M, Ward T E and Markham C M 2007 Brain-computer interface using a simplified functional near-infrared spectroscopy system J. Neural Eng. 4 219
15. Power S D, Kushki A and Chau T 2011 Towards a system-paced near-infrared spectroscopy brain-computer interface: differentiating prefrontal activity due to mental arithmetic and mental singing from the no-control state J. Neural Eng. 8 066004
16. Sorger B, Reithler J, Dahmen B and Goebel R 2012 A real-time fMRI-based spelling device immediately enabling robust motor-independent communication Curr. Biol. 22 1333-8
17. Andersson P, Pluim J P, Viergever M A and Ramsey N F 2013 Navigation of a telepresence robot via covert visuospatial attention and real-time fMRI Brain Topography 26 177-85
18. Sellers E W and Donchin E 2006 A P300-based brain-computer interface: Initial tests by ALS patients Clin. Neurophysiol. 117 538-48 (Pubitemid 43343986)
19. Treder M S and Blankertz B 2010 (C)overt attention and visual speller design in an ERP-based brain-computer interface Behav. Brain Funct. 6 28
20. Aloise F et al 2012 A covert attention P300-based brain-computer interface: geospell Ergonomics 55 538-51
21. Muller-Putz G R and Pfurtscheller G 2008 Control of an electrical prosthesis with an SSVEP-based BCI IEEE Trans. Biomed. Eng. 55 361-4 (Pubitemid 350308399)
22. Ramos-Murguialday A et al 2012 Proprioceptive feedback and brain computer interface (BCI) based neuroprostheses PloS One 7 e47048
23. Galán F et al 2008 A brain-actuated wheelchair: asynchronous and non-invasive brain-computer interfaces for continuous control of robots Clin. Neurophysiol. 119 2159-69
24. Li Y, Pan J, Wang F and Yu Z 2013 A hybrid BCI SYSTEM Combining P300 and SSVEP and its application to wheelchair control IEEE Trans. Biomed. Eng. 60 3156-66
25. Rebsamen B et al 2010 A brain controlled wheelchair to navigate in familiar environments IEEE Trans. Neural Syst. Rehabil. Eng. 18 590-8
26. Rebsamen B et al 2007 Controlling a wheelchair using a BCI with low information transfer rate IEEE 10th Int. Conf. on Rehabilitation Robotics (ICORR (2007)) pp 1003-8
27. Aloise F et al 2011 Asynchronous P300-based brain-computer interface to control a virtual environment: initial tests on end users Clin. EEG Neurosci. 42 219-24
28. Doud A J, Lucas J P, Pisansky M T and He B 2011 Continuous three-dimensional control of a virtual helicopter using a motor imagery based brain-computer interface PloS One 6 e26322
29. Yuan P, Gao X, Allison B, Wang Y, Bin G and Gao S 2013 A study of the existing problems of estimating the information transfer rate in online brain-computer interfaces J. Neural Eng. 10 026014
30. Thompson D E, Blain-Moraes S and Huggins J E 2013 Performance assessment in brain-computer interface-based augmentative and alternative communication Biomed. Eng. Online 12 43
31. Simeral J D, Kim S P, Black M J, Donoghue J P and Hochberg L R 2011 Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array J. Neural Eng. 8 025027
32. Gilja V et al 2012 A high-performance neural prosthesis enabled by control algorithm design Nature Neurosci. 15 1752-7
33. D'albis T, Blatt R, Tedesco R, Sbattella L and Matteucci M 2012 A predictive speller controlled by a brain-computer interface based on motor imagery ACM Trans. Comput.-Hum. Interact. (TOCHI) 19 20
34. Dal Seno B, Matteucci M and Mainardi L 2010 Online detection of P300 and error potentials in a BCI speller Comput. Intell. Neurosci. 2010 11
35. Mason S G and Birch G E 2003 A general framework for brain-computer interface design IEEE Trans. Neural Syst. Rehabil. Eng. 11 70-85 (Pubitemid 36613254)
36. Ryan D B et al 2011 Predictive spelling with a P300-based brain-computer interface: increasing the rate of communication Int. J. Hum.-Comput. Interact. 27 69-84
37. Felton E A, Lewis N L, Wills S A, Radwin R G and Williams J C 2007 Neural signal based control of the dasher writing system CNE'07: 3rd Int. IEEE/EMBS Conf. on Neural Engineering pp 366-70
38. Golland P and Fischl B 2003 Permutation tests for classification: towards statistical significance in image-based studies Information Processing in Medical Imaging (Berlin: Springer) pp 330-41
39. Zou G Y 2007 Toward using confidence intervals to compare correlations Psychol. Methods. 12 399-413
40. Wolpaw J R, Ramoser H, McFarland D J and Pfurtscheller G 1998 EEG-based communication: improved accuracy by response verification IEEE Trans. Rehabil. Eng. 6 326-33 (Pubitemid 28409341)
41. Shannon C E 2001 A mathematical theory of communication ACM SIGMOBILE Mobile Comput. Commun. Rev. 5 3-55
42. Höhne J, Schreuder M, Blankertz B and Tangermann M 2011 A novel 9-class auditory ERP paradigm driving a predictive text entry system Front. Neurosci. 5 99 (www.ncbi.nlm.nih.gov/pmc/articles/PMC3163907/)
43. Kindermans P-J, Verschore H and Schrauwen B 2013 A unified probabilistic approach to improve spelling in an event-related potential based brain-computer interface IEEE Trans. Biomed. Eng. 60 2696-705
44. Kindermans P-J, Verschore H, Verstraeten D and Schrauwen B 2012 A P300 BCI for the masses: prior information enables instant unsupervised spelling Adv. Neural Inf. Process. Syst. 25 719-27 (http://books.nips.cc/papers/files/nips25/ NIPS2012-0332.pdf)
45. Speier W, Arnold C, Lu J, Taira R K and Pouratian N 2012 Natural language processing with dynamic classification improves P300 speller accuracy and bit rate J. Neural Eng. 9 016004
46. Bianchi L, Quitadamo L, Garreffa G, Cardarilli G and Marciani M 2007 Performances evaluation and optimization of Brain Computer Interface systems in a copy spelling task IEEE Trans. Neural Syst. Rehabil. Eng. 15 207-16 (Pubitemid 46957028)
47. Quitadamo L R et al 2012 Evaluation of the performances of different P300 based brain-computer interfaces by means of the efficiency metric J. Neurosci. Methods 203 361-8
48. Dal Seno B, Matteucci M and Mainardi L T 2010 The utility metric: a novel method to assess the overall performance of discrete brain-computer interfaces IEEE Trans. Neural Syst. Rehabil. Eng. 18 20-8
49. Carmena J M et al 2003 Learning to control a brain-machine interface for reaching and grasping by primates PLoS Biol. 1 e42
50. Taylor D M, Tillery S I H and Schwartz A B 2002 Direct cortical control of 3D neuroprosthetic devices Science 296 1829-32 (Pubitemid 34596262)
51. Lebedev M A et al 2005 Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain-machine interface J. Neurosci. 25 4681-93 (Pubitemid 40664117)
52. Chestek C A et al 2007 Single-neuron stability during repeated reaching in macaque premotor cortex J. Neurosci. 27 10742-50 (Pubitemid 47535861)
53. Ganguly K and Carmena J M 2009 Emergence of a stable cortical map for neuroprosthetic control PLoS Biol. 7 e1000153
54. Chase S M, Schwartz A B and Kass R E 2009 Bias, optimal linear estimation, and the differences between open-loop simulation and closed-loop performance of spiking-based brain-computer interface algorithms Neural Netw. 22 1203-13
55. Vargas-Irwin C E, Shakhnarovich G, Yadollahpour P, Mislow J M, Black M J and Donoghue J P 2010 Decoding complete reach and grasp actions from local primary motor cortex populations J. Neurosci. 30 9659-69
56. Velliste M, Perel S, Spalding M C, Whitford A S and Schwartz A B 2008 Cortical control of a prosthetic arm for self-feeding Nature 453 1098-101
57. Collinger J L et al 2013 Collaborative approach in the development of high-performance brain-computer interfaces for a neuroprosthetic arm: translation from animal models to human control Clin. Transl. Sci. 7 52-59
58. Cunningham J P, Nuyujukian P, Gilja V, Chestek C A, Ryu S I and Shenoy K V 2011 A closed-loop human simulator for investigating the role of feedback control in brain-machine interfaces J. Neurophysiol. 105 1932-49
59. MacKenzie I S, Kauppinen T and Silfverberg M 2001 Accuracy measures for evaluating computer pointing devices Proc. SIGCHI Conf. on Human Factors in Computing Systems pp 9-16 (http://dl.acm.org/citation.cfm?id=365028)
60. ISO 1998 Ergonomic requirements for office work with visual display terminals (VDTs): Part 11: Guidance on usability ISO Standard 9241-11:1998
61. Laghari K U R and Connelly K 2012 Toward total quality of experience: A QoE model in a communication ecosystem IEEE. Commun. Mag. 50 58-65
62. Seffah A, Donyaee M, Kline R B and Padda H K 2006 Usability measurement and metrics: a consolidated model Softw. Qual. J. 14 159-78 (Pubitemid 44195278)
63. Morris J D 1995 Observations: SAM: the self-assessment manikin; an efficient cross-cultural measurement of emotional response J. Adv. Res. 35 63-68
64. Gupta R, Arndt S, Antons J-N, Schleicher R, Moller S and Falk T H 2013 Neurophysiological experimental facility for quality of experience (QoE) assessment IFIP/IEEE Int. Symp. on Integrated Network Management (IM 2013) pp 1300-5 (http://ieeexplore.ieee.org/xpls/abs-all.jsp?arnumber=6573181)
65. Coyne J T, Baldwin C, Cole A, Sibley C and Roberts D M 2009 Applying real time physiological measures of cognitive load to improve training Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience (Berlin: Springer) pp 469-78