Real-Time State estimation in a flight simulator using fNIRS

PLoS ONE

Volumn 10, Issue 3, 2015, Pages

  Gateau, Thibault ^a   Durantin, Gautier ^a   Lancelot, Francois ^a   Scannella, Sebastien ^a   Dehais, Frederic ^a  

^a UNIVERSITÉ DE TOULOUSE   (France)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; AIRPLANE PILOT; ALGORITHM; ARTICLE; BRAIN COMPUTER INTERFACE; FEMALE; FLIGHT SIMULATOR; FUNCTIONAL NEAR INFRARED SPECTROSCOPY; FUZZY SYSTEM; HUMAN; MALE; MEASUREMENT ACCURACY; MENTAL HEALTH; NEAR INFRARED SPECTROSCOPY; ONLINE SYSTEM; SENSITIVITY AND SPECIFICITY; SIMULATOR; WORKING MEMORY; AEROSPACE MEDICINE; AIRCRAFT; AVIATION; COMPUTER SIMULATION; HEMODYNAMICS; PHYSIOLOGY; PREFRONTAL CORTEX; SHORT TERM MEMORY; WORKLOAD;

ADULT; AEROSPACE MEDICINE; AIRCRAFT; AVIATION; BRAIN-COMPUTER INTERFACES; COMPUTER SIMULATION; FEMALE; HEMODYNAMICS; HUMANS; MALE; MEMORY, SHORT-TERM; PREFRONTAL CORTEX; SPECTROSCOPY, NEAR-INFRARED; WORKLOAD;

EID: 84926322595     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0121279     Document Type: Article

References (78)

1. Causse M, Dehais F, Arexis M, Pastor J. Cognitive aging and flight performances in general aviation pilots. Aging, Neuropsychology, and Cognition. 2011; 18(5):544-561. doi: 10.1080/13825585.2011.586018
2. Causse M, Dehais F, Pastor J. Executive functions and pilot characteristics predict flight simulator performance in general aviation pilots. The International Journal of Aviation Psychology. 2011; 21(3):217-234. doi: 10.1080/10508414.2011.582441
3. Morrow D, Lee A, Rodvold M. Analysis of problems in routine controller-pilot communication. The International Journal of Aviation Psychology. 1993; 3(4):285-302. doi: 10.1207/s15327108ijap0304-3
4. Miller GA. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review. 1994; 101(2):343. doi: 10.1037/0033-295X.101.2.343 PMID: 8022966
5. Baddeley A. Working memory. Science. 1992; 255(5044):556-559. doi: 10.1126/science.1736359 PMID: 1736359
6. Risser MR, Scerbo MW, Baldwin CL, McNamara DS. Interference timing and acknowledgement response with voice and datalink ATC commands. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. vol. 50. Sage Publications; 2006. p. 11-15.
7. Taylor JL, Yesavage JA, Morrow DG, Dolhert N, Brooks JO, Poon LW. The effects of information load and speech rate on younger and older aircraft pilots' ability to execute simulated air-traffic controller instructions. Journal of Gerontology. 1994; 49(5):191-200. doi: 10.1093/geronj/49.5.P191
8. Scerbo MW, Risser MR, Baldwin CL. Implementing Speech and Simulated DataLink Commands: The Role of Task Interference and Message Length. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. vol. 47. SAGE Publications; 2003. p. 95-99.
9. Taylor JL, O 'Hara R, Mumenthaler MS, Rosen AC, Yesavage JA. Cognitive ability, expertise, and age differences in following air-traffic control instructions. Psychology and aging. 2005; 20(1):117-133. doi: 10.1037/0882-7974.20.1.117 PMID: 15769218
10. Billings CE, Cheaney ES. Information transfer problems in the aviation system. In: NASA Technical Report 1875; 1981. p. 89-90.
11. George L, Léecuyer A. An overview of research on 'passive'brain-computer interfaces for implicit human-computer interaction. In: International Conference on Applied Bionics and Biomechanics ICABB 2010-Workshop W1'Brain-Computer Interfacing and Virtual Reality'; 2010.
12. Buttfield A, Ferrez PW, Millan JR. Towards a robust BCI: error potentials and online learning. Neural Systems and Rehabilitation Engineering, IEEE Transactions on. 2006; 14(2):164-168. doi: 10.1109/ TNSRE.2006.875555
13. St John M, Kobus DA, Morrison JG, Schmorrow D. Overview of the DARPA augmented cognition technical integration experiment. International Journal of Human-Computer Interaction. 2004; 17(2):131-149. doi: 10.1207/s15327590ijhc1702-2
14. Gratton G, Fabiani M. Optical imaging of brain function. Oxford Series in Human-Techology Interaction. 2006;p. 65.
15. Strangman G, Culver JP, Thompson JH, Boas DA. A quantitative comparison of simultaneous BOLD fMRI and NIRS recordings during functional brain activation. Neuroimage. 2002;17(2):719-731. Available from: ftp://fhs-r:Thei7oow@depozit.isae.fr/papers/Strangman2002.pdf.
16. Villringer K, Minoshima S, Hock C, Obrig H, Ziegler S, Dirnagl U, et al. Assessment of local brain activation. In: Optical Imaging of Brain Function and Metabolism 2. Springer; 1997. p. 149-153.
17. Cui X, Bray S, Bryant DM, Glover GH, Reiss AL. A quantitative comparison of NIRS and fMRI across multiple cognitive tasks. Neuroimage. 2011; 54(4):2808-2821. doi: 10.1016/j.neuroimage.2010.10.069 PMID: 21047559
18. Ayaz H, Shewokis PA, Bunce S, Izzetoglu K, Willems B, Onaral B. Optical brain monitoring for operator training and mental workload assessment. Neuroimage. 2012; 59(1):36-47. doi: 10.1016/j. neuroimage.2011.06.023 PMID: 21722738
19. Takeuchi Y. Change in blood volume in the brain during a simulated aircraft landing task. Journal Of Occupational Health-English Edition. 2000; 42(2):60-65. doi: 10.1539/joh.42.60
20. Gagnon JF, Durantin G, Vachon F, Causse M, Tremblay S, Dehais F. Anticipating human error before it happens: Towards a psychophysiological model for online prediction of mental workload. In: Proceedings of the Human Factors and Ergonomics Society Chapter Europe; 2012.
21. Li C, Gong H, Zeng S, Luo Q. Verbal working memory load affects prefrontal cortices activation: Evidence from a functional NIRS study in humans. In: Biomedical Optics 2005. International Society for Optics and Photonics; 2005. p. 33-40.
22. Schreppel T, Egetemeir J, Schecklmann M, Plichta MM, Pauli P, Ellgring H, et al. Activation of the prefrontal cortex in working memory and interference resolution processes assessed with near-infrared spectroscopy. Neuropsychobiology. 2008; 57(4):188-193. doi: 10.1159/000147473 PMID: 18654088
23. Hirshfield LM, Gulotta R, Hirshfield S, Hincks S, Russell M, Ward R, et al. This is your brain on interfaces: enhancing usability testing with functional near-infrared spectroscopy. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM; 2011. p. 373-382.
24. Durantin G, Gagnon JF, Tremblay S, Dehais F. Using near infrared spectroscopy and heart rate variability to detect mental overload. Behavioural brain research. 2014; 259:16-23. doi: 10.1016/j.bbr. 2013.10.042 PMID: 24184083
25. Matsuyama H, Asama H, Otake M. Design of differential near-infrared spectroscopy based brain machine interface. In: Robot and Human Interactive Communication, 2009. RO-MAN 2009. The 18th IEEE International Symposium on. IEEE; 2009. p. 775-780.
26. Coyle S, Ward T, Markham C, McDarby G. On the suitability of near-infrared (NIR) systems for nextgeneration brain-computer interfaces. Physiological Measurement. 2004; 25(4):815. doi: 10.1088/0967-3334/25/4/003 PMID: 15382823
27. Coyle SM, Ward TE, Markham CM. Brain-computer interface using a simplified functional near-infrared spectroscopy system. Journal of neural engineering. 2007; 4(3):219. doi: 10.1088/1741-2560/4/3/007 PMID: 17873424
28. Power SD, Falk TH, Chau T. Classification of prefrontal activity due to mental arithmetic and music imagery using hidden Markov models and frequency domain near-infrared spectroscopy. Journal of neural engineering. 2010; 7(2):026002. doi: 10.1088/1741-2560/7/2/026002
29. Naseer N, Hong KS. Classification of functional near-infrared spectroscopy signals corresponding to the right-and left-wrist motor imagery for development of a brain-computer interface. Neuroscience letters. 2013; 553:84-89. doi: 10.1016/j.neulet.2013.08.021 PMID: 23973334
30. Hong KS, Naseer N, Kim YH. Classification of prefrontal and motor cortex signals for three-class fNIRS-BCI. Neuroscience letters. 2015; 587:87-92. doi: 10.1016/j.neulet.2014.12.029 PMID: 25529197
31. Girouard A, Solovey ET, Hirshfield LM, Chauncey K, Sassaroli A, Fantini S, et al. Distinguishing difficulty levels with non-invasive brain activity measurements. In: Human-Computer Interaction-INTERACT 2009. Springer; 2009. p. 440-452.
32. Solovey ET, Lalooses F, Chauncey K, Weaver D, Parasi M, Scheutz M, et al. Sensing cognitive multitasking for a brain-based adaptive user interface. In: Proceedings of the 2011 annual conference on Human factors in computing systems. ACM; 2011. p. 383-392.
33. Derosièere G, Dalhoumi S, Perrey S, Dray G, Ward T. Towards a Near Infrared Spectroscopy-Based Estimation of Operator Attentional State. PloS one. 2014; 9(3):e92045. doi: 10.1371/journal.pone.0092045
34. Sitaram R, Zhang H, Guan C, Thulasidas M, Hoshi Y, Ishikawa A, et al. Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain-computer interface. NeuroImage. 2007; 34(4):1416-1427. doi: 10.1016/j.neuroimage.2006.11.005 PMID: 17196832
35. Strait M, Canning C, Scheutz M. What we can and cannot do with near infrared spectroscopy. Frontiers in Neuroscience. 2014; 8(117).
36. Naseer N, Hong K. fNIRS-based brain-computer interfaces: a review. Name: Frontiers in Human Neuroscience. 2015; 9(3).
37. Abdelnour AF, Huppert T. Real-time imaging of human brain function by near-infrared spectroscopy using an adaptive general linear model. Neuroimage. 2009; 46(1):133-143. doi: 10.1016/j.neuroimage.2009.01.033 PMID: 19457389
38. Naseer N, Hong MJ, Hong KS. Online binary decision decoding using functional near-infrared spectroscopy for the development of brain- computer interface. Experimental brain research. 2014; 232(2):555-564. doi: 10.1007/s00221-013-3764-1 PMID: 24258529
39. Girouard A, Solovey ET, Jacob RJ. Designing a passive brain computer interface using real time classification of functional near-infrared spectroscopy. International Journal of Autonomous and Adaptive Communications Systems. 2013; 6(1):26-44. doi: 10.1504/IJAACS.2013.050689
40. Solovey E, Schermerhorn P, Scheutz M, Sassaroli A, Fantini S, Jacob R. Brainput: enhancing interactive systems with streaming fnirs brain input. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM; 2012. p. 2193-2202.
41. Afergan D, Peck EM, Solovey ET, Jenkins A, Hincks SW, Brown ET, et al. Dynamic difficulty using brain metrics of workload. In: Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM; 2014. p. 3797-3806.
42. Chan J, Power S, Chau T. Investigating the need for modelling temporal dependencies in a brain-computer interface with real-time feedback based on near infrared spectra. Journal of Near Infrared Spectroscopy. 2012; 20(1):107-116. doi: 10.1255/jnirs.971
43. Schudlo LC, Chau T. Dynamic topographical pattern classification of multichannel prefrontal NIRS signals: II. Online differentiation of mental arithmetic and rest. Journal of neural engineering. 2014; 11 (1):016003. doi: 10.1088/1741-2560/11/1/016003 PMID: 24311057
44. Strait M, Canning C, Scheutz M. Reliability of NIRS-Based BCIs: a placebocontrolled replication and reanalysis of Brainput. In: CHI'14 Extended Abstracts on Human Factors in Computing Systems. ACM; 2014. p. 619-630.
45. McFarland DJ, Anderson CW, Muller K, Schlogl A, Krusienski DJ. BCI meeting 2005-workshop on BCI signal processing: feature extraction and translation. IEEE transactions on neural systems and rehabilitation engineering. 2006; 14(2):135. doi: 10.1109/TNSRE.2006.875637 PMID: 16792278
46. Fazli S, Mehnert J, Steinbrink J, Curio G, Villringer A, Müller KR, et al. Enhanced performance by a hybrid NIRS-EEG brain computer interface. Neuroimage. 2012; 59(1):519-529. doi: 10.1016/j. neuroimage.2011.07.084 PMID: 21840399
47. Tomita Y, Mitsukura Y. Hemodynamic characteristics for improvement of EEGBCI performance. In: Human System Interaction (HSI), 2013 The 6th International Conference on. IEEE; 2013. p. 495-500.
48. Utsugi K, Obata A, Sato H, Katsura T, Sagara K, Maki A, et al. Development of an optical brain-machine interface. In: Engineering in Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE. IEEE; 2007. p. 5338-5341.
49. Cui X, Bray S, Reiss AL. Speeded near infrared spectroscopy (NIRS) response detection. PLoS One. 2010;5(11):e15474. Available from: ftp://fhs-r:Thei7oow@depozit.isae.fr/papers/Cui2010.pdf.
50. Durantin G, Scannella S, Gateau T, Delorme A, Dehais F. Moving Average Convergence Divergence filter preprocessing for real-time event-related peak activity onset detection: Application to fNIRS signals. In: Engineering in Medicine and Biology Society (EMBC), 2014 36th Annual International Conference of the IEEE. IEEE; 2014. p. 2107-2110.
51. Derosièere G, Mandrick K, Dray G, Ward TE, Perrey S. NIRS-measured prefrontal cortex activity in neuroergonomics: strengths and weaknesses. Frontiers in human neuroscience. 2013; 7.
52. Sato H, Fuchino Y, Kiguchi M, Katura T, Maki A, Yoro T, et al. Intersubject variability of near-infrared spectroscopy signals during sensorimotor cortex activation. Journal of biomedical optics. 2005; 10 (4):044001-1. doi: 10.1117/1.1960907
53. Jasdzewski G, Strangman G, Wagner J, Kwong K, Poldrack R, Boas D. Differences in the hemodynamic response to event-related motor and visual paradigms as measured by near-infrared spectroscopy. Neuroimage. 2003; 20(1):479-488. doi: 10.1016/S1053-8119(03)00311-2 PMID: 14527608
54. Tai K, Chau T. Single-trial classification of NIRS signals during emotional induction tasks: towards a corporeal machine interface. Journal of neuroengineering and rehabilitation. 2009;6:39. Available from: ftp://fhs-r:Thei7oow@depozit.isae.fr/papers/Tai2009.pdf.
55. Appel G. Technical analysis: power tools for active investors. FT Press; 2005.
56. Delpy DT, Cope M, Van der Zee P, Arridge S, Wray S, Wyatt J. Estimation of optical pathlength through tissue from direct time of flight measurement. Physics in medicine and biology. 1988; 33(12):1433. doi: 10.1088/0031-9155/33/12/008 PMID: 3237772
57. Ayaz H, Onaral B. Analytical software and stimulus-presentation platform to utilize, visualize and analyze near-infrared spectroscopy measures. Master's Degree Thesis, Drexel University; 2005.
58. Ayaz H, Shewokis PA, Curtin A, Izzetoglu M, Izzetoglu K, Onaral B. Using Maze-Suite and functional near infrared spectroscopy to study learning in spatial navigation. Journal of visualized experiments: JoVE. 2011;(56). doi: 10.3791/3443 PMID: 22005455
59. Yamamoto D, Kuroda S, Okada E. Modeling of influence of frontal sinus on NIRS signal of brain activation. In: European Conference on Biomedical Optics. International Society for Optics and Photonics; 2007. p. 66290S-66290S.
60. Cortes C, Vapnik V. Support-vector networks. Machine learning. 1995; 20(3):273-297. doi: 10.1023/A:1022627411411
61. Kuhn M. Building Predictive Models in R Using the caret Package. Journal of Statistical Software. 2008;28(5):1-26. Available from: ftp://fhs-r:Thei7oow@depozit.isae.fr/papers/Kuhn2008.pdf.
62. Parasuraman R, Wilson GF. Putting the brain to work: Neuroergonomics past, present, and future. Human Factors: The Journal of the Human Factors and Ergonomics Society. 2008; 50(3):468-474. doi: 10.1518/001872008X288349
63. Fletcher P, Henson RNA. Frontal lobes and human memory insights from functional neuroimaging. Brain. 2001; 124(5):849-881. doi: 10.1093/brain/124.5.849 PMID: 11335690
64. Miller EK, Cohen JD. An integrative theory of prefrontal cortex function. Annual review of neuroscience. 2001; 24(1):167-202. doi: 10.1146/annurev.neuro.24.1.167 PMID: 11283309
65. Kubler A, Mushahwar V, Hochberg LR, Donoghue JP. BCI meeting 2005-workshop on clinical issues and applications. Neural Systems and Rehabilitation Engineering, IEEE Transactions on. 2006; 14 (2):131-134. doi: 10.1109/TNSRE.2006.875585
66. Pfurtscheller G, Allison BZ, Bauernfeind G, Brunner C, Solis Escalante T, Scherer R, et al. The hybrid BCI. Frontiers in neuroscience. 2010; 4:3.
67. Khan MJ, Hong MJ, Hong KS. Decoding of four movement directions using hybrid NIRS-EEG braincomputer interface. Frontiers in human neuroscience. 2014; 8. doi: 10.3389/fnhum.2014.00244
68. Guirgis M, Falk T, Power S, Blain S, Chau T. Harnessing physiological responses to improve NIRS-based brain-computer interface performance. In: Proc. ISSNIP Biosignals Biorobotics Conf; 2010. p. 59-62.
69. Sitaram R, Zhang H, Guan C, Thulasidas M, Hoshi Y, Ishikawa A, et al. Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain-computer interface. NeuroImage. 2007;34(4):1416-1427. Available from: ftp://fhs-r:Thei7oow@depozit.isae.fr/papers/Sitaram2007.pdf.
70. Zhang JH, Peng XD, Liu H, Raisch J, Wang RB. Classifying human operator functional state based on electrophysiological and performance measures and fuzzy clustering method. Cognitive Neurodynamics. 2013;p. 1-18.
71. Réegis N, Dehais F, Rachelson E, Thooris C, Pizziol S, Causse M, et al. Formal Detection of Attentional Tunneling in Human Operator-Automation Interactions. IEEE Transaction on Human Machin System. 2014;.
72. Kaelbling LP, Littman ML, Moore AW. Reinforcement learning: a survey. Journal of Artificial Intelligence Research. 1996; 4:237-285.
73. Dijk DJ, Duffy JF, Czeisler CA. Circadian and sleep/wake dependent aspects of subjective alertness and cognitive performance. Journal of sleep research. 1992; 1(2):112-117. doi: 10.1111/j.1365-2869.1992.tb00021.x PMID: 10607036
74. Lin WH, Hauptmann A. Meta-classification: Combining multimodal classifiers. In: Mining Multimedia and Complex Data. Springer; 2003. p. 217-231.
75. Tsirogiannis G, Frossyniotis D, Nikita K, Stafylopatis A. A Meta-classifier Approach for Medical Diagnosis. In: Vouros G, Panayiotopoulos T, editors. Methods and Applications of Artificial Intelligence. vol. 3025 of Lecture Notes in Computer Science. Springer Berlin Heidelberg; 2004. p. 154-163. Available from: http://dx.doi.org/10.1007/978-3-540-24674-9-17.
76. Mackey JR, Harrivel AR, Adamovsky G, Lewandowski BE, Gotti DJ, Tin P, et al. Effects of Varying Gravity Levels on fNIRS Headgear Performance and Signal Recovery. The American Institute of Aeronautics and Astronautics. 2013;.
77. Power SD, Kushki A, Chau T. Intersession consistency of single-trial classification of the prefrontal response to mental arithmetic and the no-control state by nirs. PloS one. 2012; 7(7):e37791. doi: 10.1371/journal.pone.0037791 PMID: 22844390
78. Krauledat M, Tangermann M, Blankertz B, Müller KR. Towards zero training for brain-computer interfacing. PloS one. 2008; 3(8):e2967. doi: 10.1371/journal.pone.0002967 PMID: 18698427