Adapting the P300-based brain-computer interface for gaming: A review

IEEE Transactions on Computational Intelligence and AI in Games

Volumn 5, Issue 2, 2013, Pages 141-149

  Kaplan, Alexander Y ^a   Shishkin, Sergei L ^a,b   Ganin, Ilya P ^a   Basyul, Ivan A ^a,c   Zhigalov, Alexander Y ^a,d  

^a MOSCOW STATE UNIVERSITY   (Russian Federation)

^b KURCHATOV INSTITUTE   (Russian Federation)

^c INSTITUTE OF PSYCHOLOGY   (Russian Federation)

^d UNIVERSITY OF HELSINKI   (Finland)

Author keywords

Brain computer interfaces (BCIs); games; P300 BCI

Indexed keywords

ASSISTIVE TECHNOLOGY; BRAIN COMPUTER INTERFACES (BCIS); GAMES; GAMING APPLICATIONS; NEW APPROACHES; P300 BCI; USER TRAINING; VIDEO GAME;

CALIBRATION; HUMAN COMPUTER INTERACTION; INTERFACES (COMPUTER);

BRAIN COMPUTER INTERFACE;

EID: 84879379565     PISSN: 1943068X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCIAIG.2012.2237517     Document Type: Review

References (79)

1. J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, "Brain-computer interfaces for communication and control," Clin. Neurophysiol., vol. 113, pp. 767-791, Jun. 2002. (Pubitemid 34621187)
2. J. R. Wolpaw, "Brain-computer interfaces as new brain output pathways," J. Physiol., vol. 579, pt. 3, pp. 613-619, Mar. 2007. (Pubitemid 46404345)
3. S. C. Kleih, T. Kaufmann, C. Zickler, S. Halder, F. Leotta, F. Cincotti, F. Aloise, A. Riccio, C. Herbert, D. Mattia, and A. Kübler, "Out of the frying pan into the fire-The P300-based BCI faces real-world challenges," Progr. Brain Res., vol. 194, pp. 27-46, 2011.
4. The Future BNCI Consortium "Future BNCI: A roadmap for future directions in brain/neuronal computer interaction," 2012 [Online]. Available: http://future-bnci.org/
5. B. Blankertz, M. Tangermann, C. Vidaurre, S. Fazli, C. Sannelli, S. Haufe, C. Maeder, L. Ramsey, I. Sturm, G. Curio, and K.R. Müller, "The Berlin brain-computer interface: Non-medical uses of BCI technology," Front. Neurosci., vol. 4, Dec. 2010, article 198.
6. J. N. Mak, Y. Arbel, J. W. Minett, L. M. McCane, B. Yuksei, D. Ryan, D. Thompson, L. Bianchi, and D. Erdogmus, "Optimizing the P300-based brain-computer interface: Current status, limitations and future directions," J. Neural Eng., vol. 8, 2011, 025003.
7. L. A. Farwell and E. Donchin, "Talking off the top of your head: Toward a mental prosthesis utilizing event-related brain potentials," Elec-troencephalogr. Clin. Neurophysiol., vol. 70, pp. 510-523, 1988.
8. R. Fazel-Rezai, B. Z. Allison, C. Guger, E. W. Sellers, S. C. Kleih, and A. Kübler, "P300 brain computer interface: Current challenges and emerging trends," Front. Neuroeng., vol. 5, 2012, article 14.
9. C. Guger, S. Daban, E. Sellers, C. Holzner, G. Krausz, R. Carabalona, F. Gramatica, and G. Edlinger, "How many people are able to control a P300-based brain-computer interface (BCI)?," Neurosci. Lett., vol. 462, pp. 94-98, Oct. 2009.
10. E. W. Sellers and E. Donchin, "A P300-based brain-computer interface: Initial tests by ALS patients," Clin. Neurophysiol., vol. 117, pp. 538-548, 2006. (Pubitemid 43343986)
11. T. M. Vaughan, D. J. McFarland, G. Schalk, W. A. Sarnacki, D. J. Krusienski, E. W. Sellers, and J. R. Wolpaw, "The wadsworth BCI research and development program: At home with BCI," IEEE Trans. Neural Syst. Rehabil. Eng., vol. 14, no. 2, pp. 229-233, Jun. 2006. (Pubitemid 44150249)
12. E. W. Sellers, T. M. Vaughan, and J. R. Wolpaw, "A brain-computer interface for long-term independent home use," Amyotroph. Lateral Scler., vol. 11, pp. 449-455, 2010.
13. A. Finke, A. Lenhardt, and H. Ritter, "The MindGame: A P300-based brain-computer interface game," Neural Netw., vol. 22, pp. 1329-1333, Nov. 2009.
14. B. Blankertz, S. Lemm, M. Treder, S. Haufe, and K.-R. Müller, "Single-trial analysis and classification of ERP components-A tutorial," Neuroimage, vol. 56, pp. 814-825, May 2011.
15. A. Nijholt, B. Reuderink, and D. Plass-Oude Bos, "Turning shortcomings into challenges: Brain-computer interfaces for games," Entertain. Comput., vol. 1, pp. 85-94, 2009.
16. R. J. K. Jacob, "The use of eye movements in human-computer interaction techniques: What you look at is what you get," ACM Trans. Inf. Syst., vol. 9, pp. 152-169, 1991.
17. D. Zhu, J. Bieger, G. G. Molina, and R. M. Aarts, "A survey of stimulation methods used in SSVEP-based BCIs," Comput. Intell. Neurosci., vol. 2010, 2010, 702357.
18. R. S. Fisher, G. Harding, G. Erba, G. L. Barkley, and A. Wilkins, "Photic-and pattern-induced seizures: A review for the Epilepsy Foundation of America working group," Epilepsia, vol. 46, pp. 1426-1441, Sep. 2005.
19. D. Plass-Oude Bos, B. Reuderink, B. van de Laar, H. Gürkök, C. Mühl, M. Poel, A. Nijholt, and D. Heylen, "Brain-computer interfacing and games," in Brain-Computer Interfaces: Applying our Minds to Human-Computer Interaction, D. S. Tan and A. Nijholt, Eds. London, U.K.: Springer-Verlag, 2010, pp. 149-178.
20. P. Barr, J. Noble, and R. Biddle, "Video game values: Human-computer interaction and games," Interacting Comput., vol. 19, pp. 180-195, 2007. (Pubitemid 46162199)
21. H. Gürkök, A. Nijholt, and M. Poel, "Brain-computer interface games: Towards a framework," in ICEC 2012, ser. Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 2012, vol. 7522, pp. 373-380.
22. A. Y. Kaplan, J. J. Lim, K. S. Jin, B. W. Park, J. G. Byeon, and S. U. Tarasova, "Unconscious operant conditioning in the paradigm of brain-computer interface based on color perception," Int. J. Neurosci., vol. 115, pp. 781-802, 2005. (Pubitemid 40769750)
23. G. Hakvoort, "Immersion and affect in a brain-computer interface game" M.S. thesis, Faculty EEMCS, Univ. Twente, Enschede, The Netherlands, 2011.
24. G. Townsend, B. Graimann, and G. Pfurtscheller, "Continuous EEG classification during motor imagery-Simulation of an asynchronous BCI," IEEE Trans. Neural Syst. Rehabil. Eng., vol. 12, no. 2, pp. 258-265, Jun. 2004. (Pubitemid 38765755)
25. M. Tangermann, M. Krauledat, K. Grzeska, M. Sagebaum, B. Blankertz, C. Vidaurre, and K.-R. Müller, "Playing pinball with non-invasive BCI," in Advances in Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 2009, vol. 21, pp. 1641-1648.
26. S. C. Kleih, F. Nijboer, S. Halder, and A. Kübler, "Motivation modulates the P300 amplitude during brain-computer interface use," Clin. Neurophysiol., vol. 121, pp. 1023-1031, Jul. 2010.
27. T. Baranowski, R. Buday, D. I. Thompson, and J. Baranowski, "Playing for real: Video games and stories for health-related behavior change," Amer. J. Prev. Med., vol. 34, pp. 74-82, Jan. 2008.
28. S. L. Shishkin, I. A. Basyul, and A. Y. Kaplan, "Distractors in BCI: Can harmful commands be activated automatically?," Psychophysiology, vol. 46, no. 1, 2009, S121.
29. D. Boland, M. Quek, M. Tangermann, J. Williamson, and R. Murray-Smith, "Using simulated input into brain-computer interfaces for user-centred design," Int. J. Bioelectromagn., vol. 13, pp. 86-87, 2011.
30. G. Edlinger and C. Guger, "Social environments, mixed communication and goal-oriented control application using a brain-computer interface," in Universal Access in Human-Computer Interaction. Users Diversity, ser. Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 2011, vol. 6766, pp. 545-554.
31. U. Hoffmann, J. M. Vesin, T. Ebrahimi, and K. Diserens, "An efficient P300-based brain-computer interface for disabled subjects," J. Neu-rosci. Methods, vol. 167, pp. 115-125, 2008. (Pubitemid 350180084)
32. B. Z. Allison, "P3 or not P3: Toward a better P300 BCI," Ph.D. dissertation, Univ. California San Diego, La Jolla, CA, USA, 2003.
33. C. Guan, M. Thulasidas, and J. Wu, "High performance P300 Speller for brain-computer interface," in Proc. IEEE Int. Workshop Biomed. Circuits Syst., Singapore, 2004, DOI: 10.1109/BIOCAS. 2004.1454155.
34. G. Pires, U. Nunes, and M. Castelo-Branco, "Comparison of a row-column speller vs. a novel lateral single-character speller: Assessment of BCI for severe motor disabled patients," Clin. Neurophysiol., vol. 123, pp. 1168-1181, Jun. 2012.
35. C. Mühl, H. Gürkök, D. Plass-Oude Bos, M. E. Thurlings, L. Scherffig, M. Duvinage, A. A. Elbakyan, S. Kang, M. Poel, and D. Heylen, "Bacteria Hunt: A multimodal, multiparadigm BCI game," in Proc. 5th Int. Summer Workshop Multimodal Interfaces, Genua, Italy, 2009, pp. 41-62.
36. M. Congedo, M. Goyat, N. Tarrin, G. Ionescu, L. Varnet, B. Rivet, R. Phlypo, N. Jrad, M. Acquadro, and C. Jutten, "Brain invaders: A prototype of an open-source P300-based video game working with the Open-ViBE platform," in Proc. 5th Int. Brain-Computer Interface Conf., Graz, Austria, 2011, pp. 280-283.
37. B. Rivet, A. Souloumiac, V. Attina, and G. Gibert, "xDAWN algorithm to enhance evoked potentials: Application to brain-computer interface," IEEE Trans. Biomed. Eng., vol. 56, no. 8, pp. 2035-2043, Aug. 2009.
38. C. Angeloni, D. Salter, V. Corbit, T. Lorence, Y.-C. Yu, and L. A. Gabel, "P300-based brain-computer interface memory game to improve motivation and performance," in Proc. 38th Northeast Bioeng. Conf., Philadelphia, PA, USA, 2012, pp. 35-36.
39. J. D. Bayliss and D. H. Ballard, "A virtual reality testbed for brain-computer interface research," IEEE Trans. Rehabil. Eng., vol. 8, no. 2, pp. 188-190, Jun. 2000. (Pubitemid 30437009)
40. A. J. Kaplan and S. V. Logachev, "Game and method of its playing," Russian Patent 2 406 554, Dec. 20, 2010.
41. I. P. Ganin, S. L. Shishkin, and A. Y. Kaplan, "A P300 BCI with stimuli presented on moving objects," in Proc. 5th Int. Brain-Computer Interface Conf., Graz, Austria, 2011, pp. 308-311.
42. S. L. Shishkin, I. P. Ganin, and A. Y. Kaplan, "A P300-based brain-computer interface with stimuli at moving objects: Four-session single-trial and triple-trial tests with a game-like task design," 2013, to be published.
43. D. J. Krusienski, E. W. Sellers, F. Cabestaing, S. Bayoudh, D. J. Mc-Farland, T. M. Vaughan, and J. R. Wolpaw, "A comparison of classification techniques for the P300 Speller," J. Neural. Eng., vol. 3, pp. 299-305, Dec. 2006.
44. D. J. Krusienski, E. W. Sellers, D. J. McFarland, T. M. Vaughan, and J. R. Wolpaw, "Toward enhanced P300 Speller performance," J. Neu-rosci. Methods, vol. 167, pp. 15-21, 2008. (Pubitemid 350180088)
45. N. V. Manyakov, N. Chumerin, A. Combaz, and M. M. Van Hulle, "Comparison of classification methods for P300 brain-computer interface on disabled subjects," Comput. Intell. Neurosci., vol. 2011, 2011, 519868.
46. H. Zhang, C. Guan, and C. Wang, "Asynchronous P300-based brain-computer interfaces: A computational approach with statistical models," IEEE Trans. Biomed. Eng., vol. 55, no. 6, pp. 1754-1763, Jun. 2008. (Pubitemid 351728023)
47. R. C. Panicker, S. Puthusserypady, and Y. Sun, "An asynchronous P300 BCI with SSVEP-based control state detection," IEEE Trans. Biomed. Eng., vol. 58, no. 6, pp. 1781-1788, Jun. 2011.
48. B. Z. Allison and J. A. Pineda, "ERPs evoked by different matrix sizes: Implications for a brain computer interface (BCI) system," IEEE Trans. Neural Syst. Rehabil. Eng., vol. 11, no. 2, pp. 110-113, Jun. 2003. (Pubitemid 36935033)
49. B. Z. Allison and J. A. Pineda, "Effects of SOA and flash pattern manipulations on ERPs, performance, and preference: Implications for a BCI system," Int. J. Psychophysiol., vol. 59, pp. 127-140, 2006. (Pubitemid 43138730)
50. L. Bianchi, S. Sami, A. Hillebrand, I. P. Fawcett, L. R. Quitadamo, and S. Seri, "Which physiological components are more suitable for visual ERP based brain-computer interface? A preliminary MEG/EEG study," Brain Topography, vol. 23, pp. 180-185, 2010.
51. M. S. Treder and B. Blankertz, "(C)overt attention and visual speller design in an ERP-based brain-computer interface," Behav. Brain Funct., vol. 6, pp. 28-28, 2010.
52. S. Frenzel and E. Neubert, "Is the P300 Speller independent?" 2010, arXiv:1006.3688v1 [cs.HC].
53. S. L. Shishkin, I. P. Ganin, I. A. Basyul, A. Y. Zhigalov, and A. Y. Kaplan, "N1 wave in the P300 BCI is not sensitive to the physical characteristics of stimuli," J. Integr. Neurosci., vol. 8, pp. 471-485, 2009.
54. Y. Li, C. Guan, H. Li, and Z. Chin, "A self-training semi-supervised SVM algorithm and its application in an EEG-based brain computer interface speller system," Pattern Recognit. Lett., vol. 29, pp. 1285-1294, 2008.
55. R. Panicker, S. Puthusserypady, and Y. Sun, "Adaptation in P300 brain-computer interfaces: A two-classifier cotraining approach," IEEE Trans. Biomed. Eng., vol. 57, no. 12, pp. 2927-2935, Dec. 2010.
56. S. Dähne, J. Höhne, and M. Tangermann, "Adaptive classification improves control performance in ERP-based BCIs," in Proc. 5th Int. Brain-Computer Interface Conf., Graz, Austria, 2011, pp. 92-95.
57. S. Lu, C. Guan, and H. Zhang, "Unsupervised brain computer interface based on intersubject information and online adaptation," IEEE Trans. Neural Syst. Rehabil. Eng., vol. 17, no. 2, pp. 135-145, Apr. 2009.
58. P.-J. Kindermans, D. Verstraeten, and B. Schrauwen, "A Bayesian model for exploiting application constraints to enable unsupervised training of a P300-based BCI," PLoS ONE, vol. 7, 2012, e33758.
59. S. L. Shishkin, A. A. Nikolaev, Y. O. Nuzhdin, A. Y. Zhigalov, I. P. Ganin, and A. Y. Kaplan, "Calibration of the P300 BCI with the single-stimulus protocol," in Proc. 5th Int. Brain-Computer Interface Conf., Graz, Austria, 2011, pp. 256-259.
60. M. Salvaris and F. Sepulveda, "Visual modifications on the P300 Speller BCI paradigm," J. Neural Eng., vol. 6, 2009, 046011.
61. K. Takano, T. Komatsu, N. Hata, Y. Nakajima, and K. Kansaku, "Visual stimuli for the P300 brain-computer interface: A comparison of white/gray and green/blue flicker matrices," Clin. Neurophysiol., vol. 120, pp. 1562-1566, 2009.
62. M. Salvaris, C. Cinel, R. Poli, L. Citi, and F. Sepulveda, "Exploring multiple protocols for a brain-computer interface mouse," in Proc. 32nd IEEE Eng. Med. Biol. Soc. Conf., Buenos Aires, Argentina, 2010, pp. 4189-4192.
63. M. Tangermann, J. Höhne, M. Schreuder, M. Sagebaum, B. Blankertz, A. Ramsay, and R. Murray-Smith, "Data driven neuroergonomic optimization of BCI stimuli," in Proc. 5th Int. Brain-Computer Interface Conf., Graz, Austria, 2011, pp. 160-163.
64. G. Gibert, V. Attina, J. Mattout, E. Maby, and O. Bertrand, "Size enhancement coupled with intensification of symbols improves P300 Speller accuracy," in Proc. 4th Brain-Computer Interface Workshop Training Course, Graz, Austria, 2008, pp. 250-255.
65. S. M. M. Martens, N. J. Hill, J. Farquhar, and B. Schölkopf, "Overlap and refractory effects in a brain-computer interface speller based on the visual P300 event-related potential," J. Neural Eng., vol. 6, 2009, 026003.
66. F. Guo, B. Hong, X. Gao, and S. Gao, "A brain-computer interface using motion-onset visual evoked potential," J. Neural Eng., vol. 5, pp. 477-485, 2008.
67. B. Hong, F. Guo, T. Liu, X. Gao, and S. Gao, "N200-Speller using motion-onset visual response," Clin. Neurophysiol., vol. 120, pp. 1658-1666, 2009.
68. T. Liu, L. Goldberg, S. Gao, and B. Hong, "An online brain-computer interface using non-flashing visual evoked potentials," J. Neural Eng., vol. 7, 2010, 036003.
69. J. Jin, B. Z. Allison, X. Wang, and C. Neuper, "A combined brain-computer interface based on P300 potentials and motion-onset visual evoked potentials," J. Neurosci. Methods, vol. 205, pp. 265-276, 2012.
70. T. Kaufmann, S. M. Schulz, C. Grünzinger, and A. Kübler, "Flashing characters with famous faces improves ERP-based brain-computer interface performance," J. Neural Eng., vol. 8, 2011, 056016.
71. A. Onishi, Y. Zhang, Q. Zhao, and A. Cichocki, "Fast and reliable P300-based BCI with facial images," in Proc. 5th Int. Brain-Computer Interface Conf., Graz, Austria, 2011, pp. 192-195.
72. Y. Zhang, Q. Zhao, J. Jin, X. Wang, and A. Cichocki, "A novel BCI based on ERP components sensitive to configural processing of human faces," J. Neural. Eng., vol. 9, Apr. 2012, 026018.
73. S. L. Shishkin, I. P. Ganin, and A. Y. Kaplan, "Event-related potentials in a moving matrix modification of the P300 brain-computer interface paradigm," Neurosci. Lett., vol. 496, pp. 95-99, 2011.
74. A. Belitski, J. Farquhar, and P. Desain, "P300 audio-visual speller," J. Neural Eng., vol. 8, 2011, 025022.
75. M. E. Thurlings, A. M. Brouwer, J. B. Van Erp, B. Blankertz, and P. J. Werkhoven, "Does bimodal stimulus presentation increase ERP components usable in BCIs?," J. Neural Eng., vol. 9, 2012, 045005.
76. G. Pfurtscheller, B. Z. Allison, C. Brunner, G. Bauernfeind, T. Solis-Escalante, R. Scherer, T. O. Zander, G. Mueller-Putz, C. Neuper, and N. Birbaumer, "The hybrid BCI," Front. Neurosci., vol. 4, Apr. 2010, article 42.
77. B. Z. Allison, R. Leeb, C. Brunner, G. R. Müller-Putz, G. Bauernfeind, J. W. Kelly, and C. Neuper, "Toward smarter BCIs: Extending BCIs through hybridization and intelligent control," J. Neural Eng., vol. 9, Feb. 2012, 013001.
78. P. Brunner, L. Bianchi, C. Guger, F. Cincotti, and G. Schalk, "Current trends in hardware and software for brain-computer interfaces (BCIs)," J. Neural Eng., vol. 8, Apr. 2011, 025001.
79. I. Martinovic, D. Davies, M. Frank, D. Perito, T. Ros, and D. Song, "On the feasibility of side-channel attacks with brain-computer interfaces," in Proc. 21st USENIX Security Symp., Aug. 2012 [Online]. Available: https://www.usenix.org/conference/usenixsecurity12/feasi-bility-side- channel-attacks-brain-computer-interfaces