Effect of cognitive load on speech prosody in aviation: Evidence from military simulator flights

Applied Ergonomics

Volumn 42, Issue 2, 2011, Pages 348-357

  Huttunen, Kerttu ^a   Keränen, Heikki ^a   Väyrynen, Eero ^a   Pääkkönen, Rauno ^b   Leino, Tuomo ^a,c  

^a UNIVERSITY OF OULU   (Finland)

^b FINNISH INSTITUTE OF OCCUPATIONAL HEALTH   (Finland)

^c Air Force Command Finland   (Finland)

Author keywords

Pilot; Voice; Workload

Indexed keywords

DECISION MAKING; SPEECH; SPEECH PROCESSING;

FLIGHT INSTRUCTORS; FUNDAMENTAL FREQUENCIES; MILITARY SIMULATORS; PILOT; PROSODIC FEATURES; SITUATION AWARENESS; SPEECH RECORDING; WORKLOAD;

SIMULATORS;

ADULT; AIRPLANE PILOT; ARMY; ARTICLE; AVIATION; AWARENESS; COGNITION; CONTROLLED STUDY; DECISION MAKING; FLIGHT; HUMAN; HUMAN EXPERIMENT; LANGUAGE ABILITY; MALE; SOUND INTENSITY; SPEECH INTELLIGIBILITY; TASK PERFORMANCE; TRAINING; VOICE;

EID: 78349305609     PISSN: 00036870     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.apergo.2010.08.005     Document Type: Article

References (63)

1. Armentrout J.J., Holland D.A., O'Toole K.J., Ercoline W.R. Fatigue and related human factors in the near crash of a large military aircraft. Aviat. Space. Environ. Med. 2006, 77:963-971.
2. Australian Transport Safety Bureau, 2001. Investigation Report 20000377. Pilot and passenger incapacitation. Beech Super King Air 200 VH-SKC, Wernadinga Station, Qld, 4 September 2000. Australian Transport Safety Bureau, Department of Transport and Regional Services, Canberra, Australia, pp. 1-22. Available from: http://www.atsb.gov.au/publications/investigation_reports/2000/AAIR/pdf/aair200003771_001.pdf.
3. Batliner A.R., Fischer K., Huber R., Spilker J. How to find trouble in communication. Speech Comm. 2003, 40:117-143.
4. Berthold A., Jameson A. Interpreting symptoms of cognitive load in speech input. Proceedings of the Seventh International Conference, UM99: User Modeling 1999, 1-10. Springer, Wienna, Available from:. http://www.dfki.de/%7Ejameson/abs/BertholdJ99.html, K. Kay (Ed.).
5. Brenner M., Doherty E.T., Shipp T. Speech measures indicating workload demand. Aviat. Space. Environ. Med. 1994, 65:21-26.
6. Brockx J., Nooteboom S. Intonation and the perceptual separation of simultaneous voices. J. Phonet. 1982, 10:23-36.
7. Casner S.M. Perceived vs. measured effects of advanced cockpit systems on pilot workload and error: are pilot's beliefs misaligned with reality?. Appl. Ergon. 2009, 40:448-456.
8. Clavel C., Vasilescu I., Devillers L., Richard G., Ehrette T. Fear-type emotion recognition for future audio-based surveillance systems. Speech Comm. 2008, 50:487-503.
9. Douglas-Cowie E., Cowie R. The vocal correlates of stress: a summary. Proceedings of the ISCA ITRW: Speech and Emotion, 5-7 September 2000 2000, 219-224. Textflow, Belfast, Available from:. http://www.image.ntua.gr/oresteia/private/contributed_docs/reports/Voice-and-Stress.pdf, R. Cowie, E. Douglas-Cowie, M. Schroeder (Eds.).
10. Elmenhorst E.-M., Vejvoda M., Maass H., Wenzel J., Plath G., Schubert E., Basner M. Pilot workload during approaches: comparison of simulated standard and noise-abatement profiles. Aviat. Space. Environ. Med. 2009, 80:346-370.
11. Endsley M.R. A survey of situation awareness requirements in air-to-air combat fighters. Int. J. Aviat. Psychol. 1993, 3:157-168.
12. Endsley M.R. Situation awareness: some reflections and comments. Situational Awareness in Complex Systems 1994, 315-317. Embry-Riddle Aeronautical University Press, Daytona Beach, FL. R.D. Gilson, D.J. Garland, J.M. Koonce (Eds.).
13. Endsley M.R. Toward a theory of situation awareness in dynamic systems. Hum. Factors. 1995, 37:32-64.
14. Endsley M.R. Situation awareness in aviation systems. Handbook of Aviation Human Factors 1999, 257-276. Lawrence Erlbaum Associates, Mahwah, NJ. D.J. Garland, J.A. Wise, V.D. Hopkin (Eds.).
15. Endsley M.R. Theoretical underpinnings of situation awareness. Situation Awareness Analysis and Measurement 2000, 1-21. Erlbaum, Mahwah, NJ. M.R. Endsley, D.J. Garlands (Eds.).
16. Fernandez R., Pickard M. Modeling driver's speech under stress. Speech Comm. 2003, 40:145-159.
17. Flach J.M. Situation awareness: proceed with caution. Hum. Factors. 1995, 37:149-157.
18. Grant K.W., Walden B.E. Spectral distribution of prosodic information. J. Speech. Hear. Res. 1996, 39:228-238.
19. Griffin G., Williams C. The effects of different levels of task complexity on three vocal measures. Aviat. Space. Environ. Med. 1987, 58:1165-1170.
20. Hagmüller, M., Rank, E., Kubin, G., 2006. Evaluation of the human voice for indications of workload induced stress in the aviation environment. EEC Note No. 18/06, INO-1 AC STSS Project. EUROCONTROL Experimental Centre: Brétigny-sur-Orge CEDEX, France. Available from: http://www.eurocontrol.int/eec/public/standard_page/DOC_Report_2006_023.html.
21. Hannula M., Huttunen K., Koskelo J., Laitinen T., Leino T. Comparison between artificial neural network and multilinear regression models in an evaluation of cognitive workload in a flight simulator. Comput. Biol. Med. 2008, 38:1163-1170.
22. Hannula, M., Koskelo, J., Huttunen, K., Sorri, M., Leino, T. 2007. Artificial neural network analysis of heart rate under cognitive load in a flight simulator. In: Devedžic, V. (Ed.), Proceedings of the International Conference on Artificial Intelligence and Applications as part of the 25th IASTED International Multi-Conference on Applied Informatics. February 12-14, 2007, Innsbruck, Austria. pp. 75-77.
23. Hansen, J.H.H., Swail, C., South, A.J., Moore, R.K., Steeneken, H., Cupples, E.J., Anderson, T., Vloeberghs, C.R.A., Trancoso, I., Verlinde, P., 2000. The impact of speech under " stress" on military speech technology. RTO Technical Report RTO-TR-10, AC/323(IST)TP/5 IST/TG-01, NATO Research & Technology Organization, Neuilly-sur-Seine Cedex, France. Available from: http://www.dtic.mil/cgibin/GetTRDoc%3FAD%3DADA377422&Location%3DU2&doc%3DGetTRDoc.pdf.
24. Hart S.G., Hauser J.R. Inflight application of three pilot workload measurement techniques. Aviat. Space. Environ. Med. 1987, 58:402-410.
25. Hirschberg J. Communication and prosody: functional aspects of prosody. Speech Comm. 2002, 36:31-43.
26. Hutchins, S.G., Morrison, G.J., Kelly, R.T., 1996. Principles for aiding complex military decision making. In: Proceedings of the Seventh International Command and Control Research and Technology Symposium., Monterey, pp. 1-18.
27. Jessen M., Köster O., Gfroerer S. Influence of vocal effort on average and variability of fundamental frequency. Speech Lang. Law 2005, 12:174-213.
28. Johannes B., Salnitski V.P., Gugga H.-C., Kirsch K. Voice stress monitoring in space-possibilities and limits. Aviat. Space. Environ. Med. 2000, 71:A58-A65.
29. Johnstone, T., Scherer, K.R., 1999. The effects of emotions on voice quality. In: Proceedings of the 14th International Congress of Phonetic Sciences (ICPhS'99), San Francisco, pp. 2029-2032. Available from: http://brainimaging.waisman.wisc.edu/%7Etjohnstone/0602.pdf.
30. Johnstone T., Scherer K. Vocal communication of emotion. Handbook of Emotions 2004, 220-235. Guilford Press, New York. M. Lewis, J.-M. Haviland-Jones (Eds.).
31. Jones C., Jonsson I.-M. Using paralinguistic cues in speech to recognise emotions in older car drivers. Affect and emotion in human-computer interaction, in the series: Lecture Notes in Computer Science 2008, vol. 4868:229-240. Springer, Berlin.
32. Junqua J.-C. The influence of acoustics of speech production: a noise-induced stress phenomenon known as the Lombard reflex. Speech Comm. 1996, 20:13-22.
33. Keränen, H., 2005. Features of speech produced by Finnish military fighter pilots. Unpublished MA thesis in English Philology. Department of English, University of Oulu, Finland.
34. Keränen, H., Väyrynen, E., Pääkkönen, R., Leino, T., Kuronen, P., Toivanen, J., Seppänen, T., 2004. Prosodic features of speech produced by military pilots during demanding tasks. In: Seppänen, T., Suomi, K., Toivanen, J. (Eds.), Fonetiikan päivät 2004, Phonet Symp 2004. MediaTeam Oulu and suomen kielen, informaatiotutkimuksen ja logopedian laitos, Oulun yliopisto, Oulu, pp. 88-91. Available from: http://www.mediateam.oulu.fi/publications/pdf/555.pdf.
35. Kuronen, P., 2004. Military aviation noise. Noise-induced hearing impairment and hearing protection. Academic dissertation. Acta Universitatis Ouluensis, Medica, D 795. Oulu: University of Oulu (Finland). Available from: http://herkules.oulu.fi/isbn9514274261/isbn9514274261.pdf.
36. Lahtinen T.M., Koskelo J.P., Laitinen T., Leino T.K. Heart rate and performance during combat missions in a flight simulator. Aviat. Space. Environ. Med. 2007, 78:387-391.
37. Lee Y.-H., Liu B.-S. Inflight workload assessment: comparison of subjective and physiological measures. Aviat. Space. Environ. Med. 2003, 74:1078-1084.
38. Lieberman H.R., Niro P., Tharion W.J., Nindl B.C., Castellani J.W., Montain S.J. Cognition during sustained operations: comparison of a laboratory simulation to field studies. Aviat. Space. Environ. Med. 2006, 77:929-935.
39. Lively S., Pisoni D., Van Summers W., Bernacki R. Effect of cognitive workload on speech production: acoustic analyses and perceptual consequences. J. Acoust. Soc. Am. 1993, 93:2962-2973.
40. McCormack H.M., de L.Horne D.J., Sheater S. Clinical applications of visual analogue scales: a critical review. Psychol. Med. 1988, 18:1007-1019.
41. Mendoza E., Carballo G. Acoustic analysis of induced vocal stress by means of cognitive workload tasks. J. Voice. 1998, 12:263-273.
42. Olsen W.O. Average speech levels and spectra in various speaking/listening conditions: a summary of the Pearson, Bennett, & Fidell (1977) report. Am. J. Audiol. 1998, 7:21-25.
43. Prinzo O.V., Britton T.W. ATC/Pilot Voice Communications: a Survey of the Literature. Final Report 1993, Federal Aviation Administration, Oklahoma City, OK.
44. Protopapas A., Lieberman P. Fundamental frequency of phonation and perceived emotional stress. J. Acoust. Soc. Am. 1997, 101:2267-2277.
45. Rothkrantz L.J.M., Wiggers P., van Wees J.-W.A., van Vark R.J. Voice stress analysis. Text, Speech and Dialogue, in the series: Lecture Notes in Computer Science 2004, vol. 3206:449-456. Springer, Berlin.
46. Saito I., Fujiwara O., Utsuki N., Mizumoto C., Arimori T. Hypoxia-induced fatal aircraft accident revealed by voice analysis. Aviat. Space. Environ. Med. 1980, 514:402-406.
47. Scherer, K.R., Grandjean, D., Johstone, T., Klasmeyer, G., Bänziger, T., 2002. Acoustic correlates of task load and stress. In: Proceedings of the 7th International Conference of Spoken Language Processing (ICSLP2002), Denver, CO, pp. 2017-2020. Available from: http://www.affective-sciences.org/system/files/2002_Scherer_Proc_ICSLP.pdf.
48. Seppänen, T., Väyrynen, E., Toivanen, J., 2003. Prosody-based classification of emotions in spoken Finnish. In: Eighth European Conference on Speech and Communication Technology (Eurospeech 2003 - Interspeech 2003), Geneva, Switzerland, pp. 717-720.
49. Silberstein D., Dietrich R. Cockpit communication under high cognitive workload. Communication in High Risk Environments 2003, 9-56. Helmut Buske Verlag, Hamburg. D. Dietrich (Ed.).
50. Skinner M.J., Simpson P.A. Workload issues in military tactical airlift. Int. J. Aviat. Psychol. 2002, 12:79-93.
51. Stokes A., Kite K. Flight stress: stress, fatigue, and performance in aviation 2003, Ashgate, Burlington.
52. Svensson E., Angelborg-Thanderz M., Sjöberg L., Olsson S. Information complexity-mental workload and performance in combat aircraft. Ergonomics 1997, 40:362-380.
53. Svensson A.I., Wilson G.F. Psychological and psychophysiological models of pilot performance for systems development and mission evaluation. Int. J. Aviat. Psychol. 2002, 12:95-110.
54. Tolkmitt F.J., Scherer K.R. Effect of experimentally induced stress on vocal parameters. J. Exp. Psychol. Hum. Percept. Perform. 1986, 12:302-313.
55. Traunmüller H., Eriksson A. Acoustic effects of variation in vocal effort by men, women, and children. J. Acoust. Soc. Am. 2000, 107:3438-3451.
56. Truong, K.P., van Leeuwen, D.A., Neerincx, M.A., 2007. Unobtrusive multimodal emotion detection in adaptive interfaces: speech and facial expressions. In: Schmorrow, D.D., Reeves, L.M. (Eds.), Augmented Cognition, HCII 2007, LNCS 4565, pp. 354-363.
57. Wewers M.E., Lowe N.K. A critical review of visual analogue scales in the measurement of clinical phenomena. Res. Nurs. Health. 1990, 13:227-236.
58. Wickens C.D. Situation awareness and workload in aviation. Curr. Dir. Psychol. Sci. 2002, 11:128-133.
59. Whitmore J., Fisher S. Speech during sustained operations. Speech Comm. 1996, 20:55-70.
60. Wilson G.F. An analysis of mental workload in pilots during flight using multiple psychophysiological measures. Int. J. Aviat. Psychol. 2002, 12:3-18.
61. Wittels P., Johannes B., Enne R., Kirsch K., Gunga H.C. Voice monitoring to measure emotional load during short-term stress. Eur. J. Appl. Physiol. 2002, 87:278-282.
62. Ylönen H., Lyytinen H., Leino T., Leppäluoto J., Kuronen P. Heart rate responses to real and simulated BA Hawk MK 51 flight. Aviat. Space. Environ. Med. 1997, 68:601-605.
63. Zahorik P., Kelly J.W. Accurate vocal compensation for sound intensity loss with increasing distance in natural environments. J. Acoust. Soc. Am. 2007, 122:EL143-EL150.