Prediction of cognitive states during flight simulation using multimodal psychophysiological sensing

AIAA Information Systems-AIAA Infotech at Aerospace, 2017

Volumn , Issue , 2017, Pages

  Harrivel, Angela R ^a   Stephens, Chad L ^a   Milletich, Robert J ^a   Heinich, Christina M ^a   Last, Mary Carolyn ^a   Napoli, Nicholas J ^a   Abraham, Nijo A ^a   Prinzel, Lawrence J ^a   Motter, Mark A ^a   Pope, Alan T ^a  

^a NASA LANGLEY RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AIRCRAFT ACCIDENTS; BENCHMARKING; CIVIL AVIATION; DECISION TREES; ELECTROENCEPHALOGRAPHY; ELECTROPHYSIOLOGY; FEATURE EXTRACTION; FLIGHT SIMULATORS; FORECASTING; INFORMATION SYSTEMS; INFORMATION USE; PSYCHOPHYSIOLOGY; SUPPORT VECTOR MACHINES;

ATTENTION MANAGEMENT; CLASSIFIER COMBINATION; COMMERCIAL AVIATION SAFETY TEAMS; GALVANIC SKIN RESPONSE; HIGH-FIDELITY FLIGHT SIMULATION; MULTIPLE CLASSIFIERS; PHYSIOLOGICAL MARKERS; SUPPORT VECTOR MACHINE CLASSIFIERS;

CLASSIFICATION (OF INFORMATION);

EID: 85085849781     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.2514/6.2017-1135     Document Type: Conference Paper

References (42)

1. Commercial Aviation Safety Team, “Airplane State Awareness Joint Safety Analysis Team Interim Report,” URL: http://www.skybrary.aero/index.php/Commercial_Aviation_Safety_Team_%28.CAST%29_Reports [cited 3 March 2015].
2. Commercial Aviation Safety Team, “SE211: Airplane State Awareness - Training for Attention Management (R-D),” URL: http://www.skybrary.aero/index.php/SE211:_Airplane_State_Awareness_-_Training_for_Attention_Management_(R-D) [cited 3 March 2015].
3. Harrivel, A., Liles, C., Stephens, C., Ellis, K., Prinzel, L., Pope, A. Psychophysiological Sensing and State Classification for Attention Management in Commercial Aviation. in American Institute of Aeronautics and Astronautics, SciTech. 2016: San Diego, California.
4. Fairclough, S., and Gilleade, K., “Capturing user engagement via psychophysiology: measures and mechanisms for biocybernetic adaptation,” International Journal of Autonomous and Adaptive Communications Systems, Vol. 6, No. 1, 2013, pp. 63–79.
5. Thomas, L., Gast, C., Grube, R., Craig, K. Fatigue detection in commercial flight operations: Results using physiological measures. Proceedings of the 6th International Conference on Applied Human Factors and Ergonomics. 2015: Las Vegas, Nevada.
6. Verma, G. K., Tiwary, U. S., “Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals,” NeuroImage, Vol. 102, 2014, pp. 162–172.
7. Wilhelm, F., and Grossman, P., “Emotions beyond the laboratory: Theoretical fundaments, study design, and analytic strategies for advanced ambulatory assessment,” Biological Psychology, Vol. 84, 2010, pp. 552–569.
8. Wilson, G. F. and Russell C. A., "Real-time assessment of mental workload using psychophysiological measures and artificial neural networks," Hum. Factors, Vol. 45, No. 4, 2003, pp. 635-643.
9. Novak, D., Mihelj, M., Munih, M., “A survey of methods for data fusion and system adaption using autonomic nervous system responses in physiological computing,” Interacting with Computers, Vol. 24, 2012, pp. 154-172.
10. Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis, G., Zivkovic, V. T., Olmstead, R. E., Tremoulet, P. D., Craven, P. L., “EEG Correlates of Task Engagement and Mental Workload in Vigilance, Learning, and Memory Tasks,” Aviation, Space and Environmental Medicine, Vol. 78, No. 5, 2007, Section II.
11. Hart, S. G. and Staveland, L. E., “Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research,” P. A. Hancock and N. Meshkati (Eds.) Human Mental Workload, North Holland Press, Amsterdam, 1988.
12. Taylor, R. M., “Situation awareness rating technique (SART): the development of a tool for aircrew systems design,” Situational Awareness in Aerospace Operations, Neuilly sur-Seine, France, NATO-AGARD-CP-478, 1990.
13. Li, F., “Improving Engagement Assessment by Model Individualization and Deep Learning,” Dissertation, Old Dominion University, 2015.
14. Pope, A. T., Bogart, E. H., Bartolome, E. S., “Biocybernetic system evaluates indices of operator engagement in automated task,” Biological Psychology, Vol. 40, 1995, pp. 187-195.
15. Harrivel, A., Weissman, D., Noll, D., Huppert, T. and Peltier, S., “Dynamic filtering improves attentional state prediction with fNIRS,” Biomed. Opt. Express. Vol. 7, 2016, pp. 979-1002.
16. Hirshfield, L. M., Solovey, E. T., Girouard, A., Kebinger, J., Jacob, R. J., Sassaroli, A., “Brain Measurement for Usability Testing and Adaptive Interfaces: An Example of Uncovering Syntactic Workload with Functional Near Infrared Spectroscopy,” Computer-Human Interaction, Boston, MA, 2009.
17. Fairclough, S. H., & Gilleade, K., “Construction of the Biocybernetic Loop: A Case Study,” ICMI 2012 Grand Challenge - Brain-Computer Interfaces, Santa Monica, CA, 2012.
18. Gross, J. J. and Levenson, R. W. Emotion elicitation using films. Cognition & Emotion, Vol. 9, 1995, pp. 87-108.
19. Parasuraman, R., & Davies, D. R., “A Taxonomic Analysis of Vigilance,” R. R. Mackie, Vigilance: Theory, Operational Performance, and Physiological Correlates, Plenum, New York, NY, 1977, pp. 559-574.
20. Santiago-Espada, Y., Myer, R. R., Latorella, K. A., & Comstock, J. R. (2011). The Multi-Attribute Task Battery II (MATB-II) Software for Human Performance and Workload Research: A User's Guide. NASA, Langley Research Center. Hampton: NASA/TM-2011-217164, L-20031, NF1676. L-12800.
21. Stephens, C.L., Christie, I.C., and Friedman, B.H. “Autonomic specificity of basic emotions: Evidence from pattern classification and cluster analysis,” Biological Psychology, Vol. 84, 2010, pp. 463-473.
22. Rivera, J., Jentsch, F., Talone, A. B., Boesser, C. T., and Jimenez, C., “Defining Startle, Surprise, and Distraction: A State-of-the-Art Technical Review to Support the Development of FAA Technical and Advisory Guidance, and of Line-Oriented Simulation Scenarios for Training,” University of Central Florida, 2015, FAA CRA 13-G-007.
23. Von Tscharner, V., “Intensity analysis in time-frequency space of surface myoelectric signals by wavelets of specified resolution,” J. Electromyogr Kinesio, Vol. 6, No. 10, 2000, pp. 433–445.
24. R. N. Khushaba, A. Al-Jumaily, and A. Al-Ani, “Novel Feature Extraction Method based on Fuzzy Entropy and Wavelet Packet Transform for Myoelectric Control”, 7th International Symposium on Communications and Information Technologies ISCIT2007, Sydney, Australia, pp. 352–357.
25. R. N. Khushaba, S. Kodagoa, S. Lal, and G. Dissanayake, “Driver Drowsiness Classification Using Fuzzy Wavelet Packet Based Feature Extraction Algorithm”, IEEE Transaction on Biomedical Engineering, Vol. 58, No. 1, 2011, pp. 121-131.
26. MATLAB, Statistics and Signal Processing Toolboxes, R2015a, The MathWorks, Inc., Natick, MA, United States.
27. Breiman, L., Friedman, J., Olshen, R., Stone, C., “Classification and Regression Trees,” Wadsworth, Belmont, CA, 1984.
28. Lin, J. "Divergence measures based on the Shannon entropy," IEEE Transactions on Information theory,” Vol. 37, No. 1, 1991, pp. 145-151.
29. Gilland, J. Driving, eye-tracking and visual entropy: Exploration of age and task effects. ProQuest, 2008.
30. Hamilton, P. S., Tompkins, W. J., “Quantitative investigation of QRS detection rules using the MIT/BIH arrhythmia database,” IEEE Trans. Biomed. Eng., Vol. 33, No. 12, 1986, pp. 1157-1165.
31. Jasper, H., "Report of the committee on methods of clinical examination in electroencephalography: 1957," Electroencephalography and Clinical Neurophysiology, Vol. 10, No. 2, pp. 370–375. doi:10.1016/0013-4694(58)90053-1
32. Breiman, L., “Random Forests,” 2001, URL:https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf [cited 23 October 2015]
33. Friedman, J. H. “Greedy Function Approximation: A Gradient Boosting Machine,” 2001, URL: https://statweb.stanford.edu/~jhf/ftp/trebst.pdf [cited 23 October 2015]
34. Schölkopf, B., Platt, J., Shawe-Taylor, J., Smola, A. J., and Williamson, R. C., “Estimating the support of a high-dimensional distribution,” Neural Computation, Vol. 13, 2001, pp. 1443-1471.
35. Liu, X., Wu, J., Zhou, Z., “Exploratory Undersampling for Class-Imbalance Learning,” IEEE TRANSACTIONS ON SYSTEMS, MAN AND CYBERNETICS – PART B, 2008.
36. Li, F., “Improving Engagement Assessment by Model Individualization and Deep Learning,” Dissertation, Old Dominion University, 2015.
37. Wilson G. F. and Russell, C. A., "Real-time assessment of mental workload using psychophysiological measures and artificial neural networks," Human Factors, Vol. 45, No. 4, 2003, pp. 635-643.
38. Nickel, P., Nachreiner, F., “Sensitivity and diagnosticity of the 0.1-Hz component of heart rate variability as an indicator of mental workload,” Human Factors, Vol. 45, No. 4, 2003, pp. 575-590.
39. Christensen, J. C., Estepp, J. R., Wilson, G. F., Russel, C. A., “The effects of day-to-day variability of physiological data on operator functional state classification,” NeuroImage, Vol. 59, 2012, pp. 57–63.
40. Malkawi, M. and Murad, O., “Artificial neuro fuzzy logic system for detecting human emotions,” Human-centric Computing and Information Sciences, Vol. 3, No. 3, 2013. doi: 10.1186/2192-1962-3-3
41. Lin, C-T., Ko, L-W., Chung, I-F, Huang, T-Y, Chen, Y-C, Jung, T-P, and Liang, S-F., “Adaptive EEG-Based Alertness Estimation System by Using ICA-Based Fuzzy Neural Networks,” IEEE Transactions on Circuits and Systems—I: Regular Papers, Vol. 53, No. 11, 2006, pp. 2469-2476.
42. Moon, B.S., Lee, H.C., Lee, Y.H., Park, J.C., Oh, I.S., and Lee, J.W., “Fuzzy systems to process ECG and EEG signals for quantification of the mental workload,” Information Sciences, Vol. 142, 2002, pp. 23–35.