The multi-aspect measurement approach: Rationale, technologies, tools, and challenges for systems design

Designing Soldier Systems: Current Issues in Human Factors

Volumn , Issue , 2012, Pages 217-247

  Oie, Kelvin S ^a   Gordon, Stephen ^a   McDowell, Kaleb ^b  

^a ARL HRED ^*

^b Translational Neuroscience Branch   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84900679642     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Chapter

References (116)

1. Allanson, J. and Fairclough, S.H. 2004. A research agenda for physiological computing. Interacting with Computers, 16, 857-78.
2. Andreassi, J.L. 2000. Psychophysiology: Human Behavior and Physiological Response. Mahwah, NJ: Lawrence Erlbaum.
3. Arminian, K. and Najafi, B. 2004. Capturing human motion using body-fixed sensors: Outdoor measurement and clinical applications. Journal of Visualization and Computer Animation, 15(2), 79-94.
4. Aslin, R.N. and Mehler, J. 2005. Near-infrared spectroscopy for functional studies of brain activity in human infants: Promise, prospects, and challenges. Journal of Biomedical Optics, 10(1), 011009: 1-3.
5. Azevedo, F.A.C., Carvalho, L.R.B., Grinberg, L.T., Farfel, J.M., Ferretti, R.E.K., Leite, R.E.P., Filho, W.J., Lent, R. and Herculano-Houzel, S. 2009. Equal numbers of neuronal and nonneuronal cells make the human brain an isomtetrically scaled-up primate brain. Journal of Comparative Neurology, 513, 532-41.
6. Backs, R.W. and Selijos, K.A. 1994. Metabolic and cardiorespiratory measures of mental effort: The effects of level of difficulty in a working memory task. International Journal of Psychophysiology, 16, 57-68.
7. Ball, G. and Breese, J. 1999. Modeling the Emotional State of Computer Users. Redmond, WA: Microsoft Corporation.
8. Barandiaran, X. and Moreno, A. 2006. On what makes certain dynamical systems cognitive: A minimally cognitive organization program. Adaptive Behavior, 14, 171-85.
9. Beatty, J. 1982. Task-evoked pupillary response, processing load, and the structure of processing resources. Psychological Bulletin, 91(2), 276-92.
10. Beatty, J. 2000. The pupillary system, in Handbook of Psychophysiology, edited by J.T. Cacioppo, G. Berntson and L.G. Tasskinary. 2nd Edition. Cambridge: Cambridge University Press, 142-62.
11. Berka, C., Levendowski, D.J., Lumicao, M.N., Yau, A., Davis, G., Zikovic, V.T., Olmstead, R.E., Tremoulet, P.D. and Craven, P.L. 2007. EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, Space, and Environmental Medicine, 78(5), B231-B244.
12. Berlyne, D.E. and McDonnell P. 1965. Effects of stimulus complexity and incongruity on duration of EEG desynchronization. Electroencephalography and Clinical Neurophysiology, 18(2), 156-61.
13. Blascovich, J., Berry-Mendes, W., Hunter, S.B. and Salomon, K. 1999. Social "facilitation" as challenge and threat. Journal of Personality and Social Psychology, 77(1), 216-33.
14. Boiten, F. 1998. The effects of emotional behavior on components of respiratory cycle. Biological Psychology, 49, 29-51.
15. Boucsein, W. 1992. Electrodermal Activity. New York: Plenum Press.
16. Boutcher, S.H., Nugent, F.W., McLaren, P.F. and Weltman, A.L. 1998. Heart period variability of trained and untrained men at rest and during mental challenge. Psychophysiology, 35, 16-22.
17. Bunce, S.C., Izzetoglu, M., Izzetoglu, K., Onaral, B. and Pourrezaei, K. 2006. Functional near-infrared spectroscopy: An emerging neuroimaging modality. IEEE Engineering in Medicine and Biology Magazine, 25(4), 54-62.
18. Cacioppo, J.T., Bush, L.K. and Tassinary, L.G. 1990. Microexpressive facial actions as a function of affective stimuli: Replication and extension. Personality and Social Psychology Bulletin, 18, 515-26.
19. Carlson, B., Singelis, T.M. and Chemtob, C.M. 1997. Facial EMG response to combat-related visual stimuli in veterans with and without post traumatic stress disorder. Applied Psychophysiology and Biofeedback, 22, 247-59.
20. Charlton, S.G. 2002. Measurement of cognitive states in test and evaluation, in Handbook of Human Factors Testing and Evaluation, edited by S.G. Charlton and T.G. O'Brien. Mahwah, NJ: Lawrence Erlbaum Associates, 97-126.
21. Cheng, S.Y. and Trivedi, M.M. 2006. Turn-intent analysis using body pose for intelligent driver assistance. Pervasive Computing, 5(4), 28-37.
22. Christie, B., Delafield, G., Lucas, B., Winwood, M. and Gale, A. 1972. Stimulus complexity and the EEG: Differential effects of the number and the variety of display elements. Canadian Journal of Psychology, 26(2), 155-70.
23. Cohen, I., Sebe, N., Garg, A., Lew, M.S. and Huang, T.S. 2002. Facial expression recognition from video sequences. Proceedings of ICME'02, International Conference on Multimedia and Expo, Lausanne, Switzerland, August 2002, 121-24.
24. Davis, R.C. 1940. Set and Muscular Tension. Bloomington, IN: University of Indiana Press.
25. Devor, A., Ulbert, I., Dunn, A.K., Narayanan, S.N., Jones, S.R., Andermann, M.L., Boas, D.A. and Dale, A.M. 2005. Coupling of the cortical hemodynamic response to cortical and thalamic neuronal activity. Proceedings of the National Academy of Sciences, 102(10), 3822-3827.
26. Donchin, E. and Coles, M.G.H. 1988. Is the P300 component a manifestation of context updating? Behavioral and Brain Sciences, 11, 357-74.
27. Duchowski, A.T. 2002. A breadth-first search of eye-tracking applications. Behavior Research Methods, Instruments and Computers, 34(4), 455-70.
28. Ekman, P. 1992. An argument for basic emotions. Cognition and Emotion, 6, 169-200.
29. Ekman, P. 2009. Lie catching and micro expression, in The Philosophy of Deception, edited by C. Martin. New York: Oxford University Press.
30. Fairclough, S.H. and Venables, L. 2006. Prediction of subjective states from psychophysiology: A multivariate approach. Biological Psychology, 71, 100-110.
31. Foerde, K., Knowlton, B.J. and Poldrack, R.A. 2006. Modulation of competing memory systems by distraction. Proceedings of the National Academy of Sciences, 103(31), 11778-11783.
32. Fogarty, J., Ko, A.J., Aung, H.H., Golden, E., Tang, K.P. and Hudson, S.E. 2005. Examining task engagement in sensor-based statistical models of human interruptibility. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Portland, OR, April 2005, 331-40.
33. Fox, P.T., Raichle, M.E., Mintun, M.A. and Dence, C. 1988. Nonoxidative glucose consumption during focal physiologic neural activity. Science, 241, 462-64.
34. Fréard, D., Jamet, E., Le Bohec, O., Poulain, G. and Botherel, V. 2007. Subjective measurement of workload related to a multimodal interaction task: NASATLX vs. workload profile. Human Computer Interaction, 4552, 60-69.
35. Gawron, V.J. 2000. Human Performance Measures Handbook. Mahwah, NJ: Lawrence Erlbaum.
36. Genno, H. 1997. Non-contact method for measuring facial skin temperature. International Journal of Individual Ergonomics, 19, 147-59.
37. Genno, H., Kanbara, O., Matsumoto, K., Suzuki, R., Osumi, M. and Mabuchi, K. 1995. Relationship between thermal sensation and the temperature of skin and deep tissues in various environments. Journal of the Japanese Society of Thermology, 15(4), 2-7.
38. Gevins, A. and Smith, M.E. 2003. Neurophysiological measures of cognitive workload during human-computer interaction. Theoretical Issues in Ergonomic Science, 4, 113-21.
39. Glenstrup, A.J. and Engell-Nielsen, T. 1995. Eye controlled media: Present and future state. Copenhagen: University of Copenhagen.
40. Gratton, G. and Fabiani, M. 2001. Shedding light on brain function: The eventrelated optical signal. Trends in Cognitive Sciences, 5(8), 357-63.
41. Halchenko, Y.O., Hanson, S.J. and Pearlmutter, B.A. 2005. Multimodal integration: fMRI, MRI, EEG, MEG, in Advanced Image Processing in Magnetic Resonance Imaging, edited by L. Landini, V. Positano and M.F. Santarelii. Boca Raton, FL: CRC Press, 223-65.
42. Hancock, P.A. and Vasmatzidis, I. 2003. Effects of heat stress on cognitive performance: The current state of knowledge. International Journal of Hypothermia, 19(3), 355-72.
43. Hart, S.G. and Staveland, L.E. 1988. Development of NASA-TLX (task load index): Results of empirical and theoretical research, in Human Mental Workload, edited by P.A. Hancock and N. Mechkati. Amsterdam: North Holland Press, 239-50.
44. Ikehara, C.S. and Crosby, M.E. 2005. Assessing cognitive load with physiological sensors. Proceedings of the 38th Hawaii Conference on System Sciences, Big Island HI, January 2005, 295.
45. Ishikawa, K., Genno, H., Kanbara, O., Yasuda, M. and Osumi, M. 1998. Evaluation of stress using facial skin temperature. Proceedings of PIE, 158-59.
46. Izzetoglu, M., Bunce, S., Izzetoglu, K., Onaral, B. and Pourrezaei, K. 2007. Functional brain imaging using near-infrared technology: Assessing cognitive activity in real-life situations. IEEE Engineering in Medicine and Biology Magazine, 26(4), 38-46.
47. Jancke, L. 1996. EMG in an anger-provoking situation: Individual differences in directing anger outwards or inwards. International Journal of Psychophysiology, 23, 207-14.
48. Jennings, J.R., Berg, W.K., Hutchinson, J.S., Obrist, P., Porges, S. and Turpin, G. 1981. Publication guidelines for heart rate studies in man. Psychophysiology, 18(3), 226-31.
49. Jung, T-P., Makeig, S., Humphries, C., Lee, T-W., McKeown, M.J., Iragui, V. and Sejnowski, T.J. 2000. Removing electroencephalographic artifacts by blind source separation. Psychophysiology, 37, 162-78.
50. Jung, T-P., Makeig, S., Humphries, C., Lee, T-W., McKeown, M.J., Iragui, V. and Sejnowski, T.J. 2001. On the design of adaptive automation for complex systems. International Journal of Cognitive Ergonomics, 5(1), 37-57.
51. Kahana, M.J., Seelig, D. and Madsen, J.R. 2001. Theta returns. Current Opinion in Neurobiology, 11, 739-44.
52. Kahana, M.J., Seelig, D. and Madsen, J.R. 2005. Gesture-based affective computing on motion capture data, in Affective Computing II 2005, edited by J. Tao, T. Tan and R.W. Picard. Berlin/Heidelberg: Springer-Verlag, 1-7.
53. Kingstone, A., Smilek, D. and Eastwood, J.D. 2008. Cognitive ethology: A new approach for studying human cognition. British Journal of Psychology, 99, 317-40.
54. Kingstone, A., Smilke, D., Ristic, J., Friesen, C.K. and Eastwood, J.D. 2003. Attention, researchers! It is time to take a look at the real world. Current Directions in Psychological Science, 12, 176-80.
55. Klimesch, W., Freunberger, R. and Sauseng, P. 2010. Oscillatory mechanisms of process binding in memory. Neuroscience and Biobehavioral Reviews, 34, 1002-1014.
56. Kutas, M. and Donchin, E. 1974. Studies of squeezing: Handedness, responding hand, response force, and asymmetry of readiness potential. Science, 186, 545-48.
57. Lange, J. and Lappe, M. 2007. The role of spatial and temporal information in biological motion perception. Advances in Cognitive Psychology, 3(4), 419-28.
58. Lau, E., Stroud, C., Plesch, S. and Phillips, C. 2006. The role of structural prediction in rapid syntactic analysis. Brain and Language, 98(1), 74-88.
59. Lauritzen, M. 2001. Relationship of spikes, synaptic activity, and local changes of cerebral blood flow. Journal of Cerebral Blood Flow and Metabolism, 21, 1367-1383.
60. Levine, J.A., Pavlidis, I. and Cooper, M. (2001). The face of fear. The Lancet, 357(9270), 1757.
61. Liu, A.K., Dale, A.M. and Belliveau, J.W. 2002. Monte carlo simulation studies of EEG and MEG localization accuracy. Human Brain Mapping, 16, 47-62.
62. Luu, P., Geyer A., Fidopiastis, C., Campbell, G., Wheeler, T., Cohn, J. and Tucker, D.M. 2010. Reentrant processing in intuitive processing. PLoS One, 5(3), e9523.
63. Makeig, S. 2009. Mind monitoring via mobile brain-body imaging, in Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience, edited by D. Schmorrow, I.V. Estabrooke and M. Gootjen. Berlin/Heidelberg: Springer, 749-58.
64. Makeig, S. 2009. Linking brain, mind, and behavior: The promise of mobile brain/body imaging (MOBI). International Journal of Psychophysiology, 73(2), 95-100.
65. Malmo, R.B. and Malmo, H.P. 2000. On electromyographic (EMG) gradients and movement-related brain activity: Significance for motor control, cognitive functions, and certain psychopathologies. International Journal of Psychophysiology, 38, 143-207.
66. Mandryk, R.L., Inkpen, K.M. and Calvert, T.W. 2006. Using psychophysiological techniques to measure user experience with entertainment technologies. Journal of Behavior and Information Technology, 25(2), 141-58.
67. Martin, J.H. 1985. Cortical neurons, the EEG, and the mechanisms of epilepsy, in Principles of Neural Science, edited by E.R. Kandel and J.H. Schwartz. 2nd Edition. Amsterdam: Elsevier, 636-47.
68. Moeslund, T.B. and Granum, E. 2001. A survey of computer vision-based human motion capture. Computer Vision and Image Understanding, 81, 231-68.
69. Monsell, S. 2003. Task switching. TRENDS in Cognitive Science, 7(3), 134-40.
70. Nikoleav, A.R., Gepshtein, S., Gong, P. and van Leeuwen, C. 2010. Duration of coherence intervals in electrical brain activity in perceptual organization. Cerbral Cortex, 20, 365-82.
71. Noel, J.B., Bauer, K.W. and Lanning, J.W. 2005. Improving pilot mental workload classification through feature exploitation and combination: A feasibility study. Computers and Operations Research, 32, 2713-2730.
72. Nunez, P.L. and Silberstein, R.B. 2000. On the relationship of synaptic activity to macroscopic measurements: Does co-registration of EEG with fMRI make sense? Brain Topography, 13, 79-96.
73. Nunez, P.L. and Srinivasan, R. 2006. Electric Fields of the Brain: The Neurophysics of EEG. Oxford: Oxford University Press.
74. Oakes, T.R., Johnstone, J., Ores Walsh, K.S., Greischar, L.L., Alexander, A.L., Fox, A.S. and Davidson, R.J. 2005. Comparison of fMRI motion correction software tools. NeuroImage, 28, 529-43.
75. Ohsuga, M., Shimono, F. and Genno, H. 2001. Assessment of phasic work stress using autonomic indices. International Journal of Psychophysiology, 40, 211-20.
76. Oostenveld, R. and Praamstra, P. 2001. The five percent electrode system for high-resolution EEG and ERP measurements. Clinical Neurophysiology, 112, 713-19.
77. Or, C.K.L. and Duffy, V.G. 2007. Development of a facial skin temperature-based methodology for non-intrusive mental workload measurement. Occupational Ergonomics, 7, 82-94.
78. Osterhout, L. and Holcomb, P.J. 1992. Event-related potentials elicited by syntactic anomaly. Journal of Memory and Language, 31(6), 785-806.
79. Parasuraman, R., Bahri, T., Deaton, J.E., Morrison, J.G. and Barnes, M. 1992. Theory and Design of Adaptive Automation in Aviation Systems. (Report No. NAWCADWAR-92033-60). Warminster, PA: US Department of the Navy, Naval Air Warfare Center-Aircraft Division.
80. Partala, T. and Surakka, V. 2003. Pupil size variation as an indication of affective processing. International Journal of Human-Computer Studies, 59, 185-98.
81. Pavlidis, I., Eberhardt, N.L. and Levine, J.A. 2002. Seeing through the face of deception. Nature, 415, 35-37.
82. Perala, C.H. and Sterling, B.S. 2007. Galvanic Skin Response as a Measure of Soldier Stress. (Technical Report ARL-TR-4114). Aberdeen Proving Ground, MD: US Army Research Laboratory.
83. Pfurtshceller, G., Muller, G., Pfurtshceller, J., Gerner, H.J. and Rupp, R. 2003. "Thought"-control of functional electrical stimulation to restore handgrip in a patient with tetraplegia. Neuroscience Letters, 351, 33-36.
84. Picton, T.W. 1992. The P300 wave of the human event-related potential. Journal of Clinical Neurophysiology, 9(4), 456-79.
85. Poh, M-Z., McDuff, D.J. and Picard, R.W. 2010. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express, 18(10), 10762-10774.
86. Reeves, L.M., Schmorrow, D.D. and Stanney, K.M. 2007. Augmented cognition and cognitive state assessment technology-near-term, mid-term, and long-term research objectives, in Foundations of Augmented Cognition, edited by D. Schmorrow and L.M. Reeves. Berlin/Heidelberg: Springer, 220-28.
87. RTO 2004. Operator Functional Assessment. (Report No. RTO-TR-HFM-104). Neuilly-sur-Seine Cedex, France: North Atlantic Treaty Organization, Research and Technology Organisation.
88. Rubio, S., Diaz, E., Martin, J. and Puente, J.M. 2004. Evaluation of subjective mental workload: A comparison of SWAT, NASA-TLX, and workload profile methods. Applied Psychology, 53(1), 61-86.
89. Russo, M.B., Vo, A., Labutta, R., Black, I., Campbell, W., Greene, J., McGhee, J. and Redmond, D. 2005. Human biovibrations: Assessment of human life signs, motor activity, and cognitive performance using wrist-mounted actigraphy. Aviation, Space, and Environmental Medicine, 76(7), C64-C74.
90. Rutkove, S.B. 2007. Introduction to volume conduction, in The Clinical Neurophysiology Primer, edited by A.S. Blum and S.B. Rutkove. Totowa, NJ: Humana Press.
91. Sarter, M., Gehring, W.J. and Kozak, R. 2006. More attention must be paid: The neurobiology of attentional effort. Brain Research Reviews, 51, 145-60.
92. Sauseng P., Griesmayr, B., Freunberger, R. and Klimesch, W. 2010. Control mechanisms in working memory: A possible function of EEG theta oscillations. Neuroscience and Biobehavioral Reviews, 34, 1015-1022.
93. Scallen, S.F., Hancock, P.A. and Duley, J.A. 1995. Pilot performance and preference for short cycles of automation in adaptive function allocation. Applied Ergonomics, 26(6), 387-403.
94. Schalk, G., McFarland, D.J., Hinterberger, T., Birbaumer, N. and Wolpaw, J.R. 2004. BCI2000: A general-purpose brain-computer interface (BCI) system. IEEE Transactions on Biomedical Engineering, 51(6), 1034-1043.
95. Sinha, R., Lovallo, W.R. and Parsons O.A. 1992. Cardiovascular differentiation of emotions. Psychosomatic Medicine, 54, 422-35.
96. Smit, A.S., Eling, P.A.T.M. and Coenen, A.M.L. 2004. Mental effort affects vigilance enduringly: After-effects in EEG and behavior. International Journal of Psychophysiology, 53, 239-43.
97. Smith, P., Shah, M. and da Vitorio Lobo, N. 2003. Determining driver visual attention with one camera. IEEE Transactions on Intelligent Transportation Systems, 4(4), 205-18.
98. Speier, C., Valacich, J.S. and Vessey, I. 1999. The influence of task interruption on individual decision making: An information overload perspective. Decision Sciences, 30(2), 337-60.
99. Sperdin, H.F., Cappe, C., Foxe, J.J. and Murray, M.M. 2009. Early, low-level auditory-somatosensory multisensory interactions impact reaction time speed. Frontiers in Integrative Neuroscience, 3(2), 1-10.
100. Stern, J.A., Boyer, D. and Schroeder, D. 1994. Blink rate: A possible measure of fatigue. Human Factors, 36(2), 285-97.
101. Susskind, J.M., Littlewort, G., Bartlett, M.S., Movellan, J. and Anderson, A.K. 2007. Human and computer recognition of facial expressions of emotion. Neuropsychologica, 45, 152-62.
102. Tsang, P.S. and Velazquez, V.L. 1996. Diagnosticity and multidimensional subjective workload ratings. Ergonomics, 39(3), 358-81.
103. Valentino, D.A., Arruda, J.E. and Gold, S.M. 1993. Comparison of QEEG and response accuracy in good vs poorer performers during a vigilance task. International Journal of Psychophysiology, 15(2), 123-33.
104. Valentino, D.A., Arruda, J.E. and Gold, S.M. 2010. Data River-A software platform for real-time management of multiple data streams. Fourth International BCI Meeting, Asilomar, CA, May-June 2010.
105. Veltman, H.J.A. and Gaillard, A.W.K. 1996. Physiological indices of workload in a simulated flight task. Biological Psychology, 42, 323-42.
106. Veltman, H.J.A. and Gaillard, A.W.K. 1998. Physiological workload reactions to increased levels of task difficulty. Ergonomics, 41(5), 656-69.
107. Verwey, W.B. and Veltman, H.J.A. 1996. Detecting short periods of elevated workload: A comparison of nine workload assessment techniques. Journal of Experimental Psychology: Applied, 2(3), 270-85.
108. Vidulich, M. and Hughes, E.B. 1991. Testing a subjective metric of situation awareness. Proceedings of the 35th Human Factors Society Annual Meeting, San Francisco, CA, September 1991, 1307-1311.
109. Wagner, J., Kim, J. and Andre, E. 2005. From physiological signals to emotions: Implementing and comparing selected methods for feature extractions and classification. Proceedings of the IEEE International Conference on Multimedia and Expo, Amsterdam, July 2005, 940-43.
110. Walter, W.G., Cooper, R., Aldridge, V.J., McCallum, W.C. and Winter, A.L. 1964. Contingent negative variation: An electrical sign of sensory-motor association and expectancy in the human brain. Nature, 203, 380-84.
111. Warren, W.H., Jr., Kay, B.A., Zosh, W.D., Duchon, A.P. and Sahuc, S. 2001. Optic flow is used to control human walking. Nature Neuroscience, 4(2), 213-16.
112. Wingard, J.A. and Maltzman, I. 1980. Interest as a predeterminer of the GSR index of the orienting reflex. Acta Psychologica, 40, 153-60.
113. Wood, C., Torkkola, K. and Kundalkar, S. 2004. Using driver's speech to detect cognitive workload. Proceedings of SPECOM 2004, 9th Conference on Speech and Computer, St. Petersburg, Russia, September 2004.
114. Wu, C. and Liu, Y. 2007. Queueing network model of driver workload and performance. IEEE Transactions on Intelligent Transportation Systems, 8(3), 528-37.
115. Yamada, F. 1998. Frontal midline theta rhythm and eyeblinking activity during a VDT task and a video game: Useful tools of psychophysiology in ergonomics. Ergonomics, 41(5), 678-88.
116. Zak, P.J. 2004. Neuroeconomics. Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences, 359(1451), 1737-1748.