Take-over again: Investigating multimodal and directional TORs to get the driver back into the loop

Applied Ergonomics

Volumn 62, Issue , 2017, Pages 204-215

  Petermeijer, Sebastiaan ^a   Bazilinskyy, Pavlo ^b   Bengler, Klaus ^a   de Winter, Joost ^b  

^a TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^b DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

Author keywords

Auditory displays; Highly automated driving; Human machine interfaces; Take over requests; Vibrotactile displays

Indexed keywords

AUTOMATION;

AUDITORY DISPLAY; AUTOMATED DRIVING; HUMAN MACHINE INTERFACE; TAKE-OVER REQUESTS; VIBROTACTILE DISPLAYS;

BEHAVIORAL RESEARCH;

ADULT; ARTICLE; AUDITORY STIMULATION; BRAIN COMPUTER INTERFACE; CONTROLLED STUDY; FEASIBILITY STUDY; FEEDBACK SYSTEM; FEMALE; HUMAN; HUMAN EXPERIMENT; INFORMATION PROCESSING; MALE; QUESTIONNAIRE; SELF REPORT; SIMULATOR; TACTILE STIMULATION; TAKE OVER REQUEST; TASK PERFORMANCE; VIBROTACTILE STIMULATION; ASSOCIATION; CAR; CAR DRIVING; COMPUTER SIMULATION; MAN MACHINE INTERACTION; PHYSICAL STIMULATION; PROCEDURES; REACTION TIME; TOUCH; VIBRATION; YOUNG ADULT;

ACOUSTIC STIMULATION; ADULT; AUTOMOBILE DRIVING; AUTOMOBILES; COMPUTER SIMULATION; CUES; FEMALE; HUMANS; MALE; MAN-MACHINE SYSTEMS; PHYSICAL STIMULATION; REACTION TIME; TASK PERFORMANCE AND ANALYSIS; TOUCH; VIBRATION; YOUNG ADULT;

EID: 85015358574     PISSN: 00036870     EISSN: 18729126     Source Type: Journal    
DOI: 10.1016/j.apergo.2017.02.023     Document Type: Article

References (93)

1. Bazilinskyy, P., De Winter, J.C.F., Auditory interfaces in automated driving: an international survey. PeerJ Comput. Sci., 1, 2015, e13.
2. Bazilinskyy, P., Kyriakidis, M., De Winter, J.C.F., An international crowdsourcing study into people's statements on fully automated driving. (Las Vegas, NV) Proceedings of the 6th International Conference on Applied Human Factors and Ergonomics (AHFE), 2015, 2534–2542.
3. Bazilinskyy, P., Petermeijer, S.M., Petrovych, V., Dodou, D., De Winter, J.C.F., Take-over Requests in Highly Automated Driving: a Crowdsourcing Survey on Auditory, Vibrotactile, and Visual Displays. (Manuscript submitted for publication), 2016.
4. Beggiato, M., Pereira, M., Petzoldt, T., Krems, J., Learning and development of trust, acceptance and the mental model of ACC. A longitudinal on-road study. Transp. Res. Part F Traffic Psychol. Behav. 35 (2015), 75–84.
5. Beruscha, F., Wang, L., Augsburg, K., Wandke, H., Do drivers steer toward or away from lateral directional vibrations at the steering wheel. Proc. Eur. Conf. Hum. Centred Des. Intelligent Transp. Syst. 2 (2010), 227–236.
6. BMW, The BMW 5 Series Gran Turismo. Owner's Manual. Retrieved from, 2015 https://carmanuals2.com/get/bmw-5-series-gran-turismo-2015-owner-s-manual-63422.
7. Boer, E.R., Della Penna, M., Utz, H., Pedersen, L., Sierhuis, M., The role of driving simulators in developing and evaluating autonomous vehicles. (Paris, France) Proceedings of the Driving Simulation Conference Europe, 2015, 3–10.
8. Boer, E.R., Yamamura, T., Kuge, N., Girshick, A., Experiencing the same road twice: a driver centered comparison between simulation and reality. (Paris, France) Proceedings of Driving Simulation Conference, 2000, 33–55.
9. Burke, J.L., Prewett, M.S., Gray, A.A., Yang, L., Stilson, F.R.B., Coovert, M.D., et al. Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis. Proceedings of the 8th International Conference on Multimodal Interfaces, 2006, ACM, New York, NY, 108–117.
10. Calhoun, G.L., Draper, M.H., Guilfoos, B.J., Ruff, H.A., Tactile and aural alerts in high auditory load UAV control environments. Proc. Hum. Factors Ergonomics Soc. Annu. Meet. 49 (2005), 145–149.
11. Cao, Y., Van der Sluis, F., Theune, M., Op den Akker, R., Nijholt, A., Evaluating informative auditory and tactile cues for in-vehicle information systems. Proceedings of the 2nd International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 2010, ACM, 102–109.
12. Citroën, Citroën C5. Owner's Manual. Retrieved from, 2007 https://carmanuals2.com/get/citroen-c5-2007.5-owner-s-manual-63737.
13. Clark, H., Feng, J., Semi-autonomous vehicles: examining driver performance during the take-over. Proc. Hum. Factors Ergonomics Soc. 59th Annu. Meet. 59 (2015), 781–785.
14. Damböck, D., Automationseffekte im Fahrzeug–von der Reaktion zur Übernahme. (Doctoral dissertation), 2013, Technical University of Munich.
15. De Groot, S., De Winter, J.C.F., Mulder, M., Wieringa, P.A., Nonvestibular motion cueing in a fixed-base driving simulator: effects on driver braking and cornering performance. Presence Teleoperators Virtual Environ. 20 (2011), 117–142.
16. De Winter, J.C.F., Predicting self-reported violations among novice license drivers using pre-license simulator measures. Accid. Analysis Prev. 52 (2013), 71–79.
17. De Winter, J.C.F., Happee, R., Martens, M.H., Stanton, N.A., Effects of adaptive cruise control and highly automated driving on workload and situation awareness: a review of the empirical evidence. Transp. Res. Part F Traffic Psychol. Behav. 27 (2014), 196–217.
18. De Winter, J.C.F., Van Leeuwen, P.M., Happee, R., Advantages and disadvantages of driving simulators: a discussion. Utrecht, the Netherlands Proceedings of the Measuring Behavior Conference, 2012, 47–50.
19. Della Penna, M., Van Paassen, M.M., Abbink, D.A., Mulder, M., Mulder, M., Reducing steering wheel stiffness is beneficial in supporting evasive maneuvers. IEEE Int. Conf. Syst. Man Cybern., 2010, 1628–1635.
20. Diaconescu, A.O., Alain, C., McIntosh, A.R., The co-occurrence of multisensory facilitation and cross-modal conflict in the human brain. J. Neurophysiology 106 (2011), 2896–2909.
21. Diederich, A., Colonius, H., Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept. Psychophys. 66 (2004), 1388–1404.
22. Dixit, V.V., Chand, S., Nair, D.J., Autonomous vehicles: disengagements, accidents and reaction times. PLOS ONE, 11, 2016, e0168054.
23. Dodonova, Y.A., Dodonov, Y.S., Is there any evidence of historical slowing of reaction time? No, unless we compare apples and oranges. Intelligence 41 (2013), 674–687.
24. General Motors, Guide du propriétaire Chevrolet Tahoe/Suburban 2015 [Owner's manual Chevrolet Tahoe/Suburban 2015]. 2014 Retrieved from https://carmanuals2.com/get/chevrolet-tahoe-2015-manuel-du-proprietaire-56514.
25. Gold, C., Damböck, D., Lorenz, L., Bengler, K., “Take over!” How long does it take to get the driver back into the loop?. Proc. Hum. Factors Ergonomics Soc. Annu. Meet. 57 (2013), 1938–1942.
26. Gold, C., Berisha, I., Bengler, K., Utilization of drivetime – performing non-driving related tasks while driving highly automated. Proc. Hum. Factors Ergonomics Soc. 59th Annu. Meet. 59 (2015), 1666–1670.
27. Gray, R., Ho, C., Spence, C., A comparison of different informative vibrotactile forward collision warnings: does the warning need to be linked to the collision event. PLOS ONE, 9, 2014, e87070.
28. Hart, S.G., Staveland, L.E., Development of NASA-TLX (task load Index): results of empirical and theoretical research. Hancock, P.A., Meshkati, N., (eds.) Human Mental Workload, 1988, North Holland Press, Amsterdam, the Netherlands, 139–183.
29. Hergeth, S., Lorenz, L., & Krems, J. F. (in press). Prior familiarization with takeover requests affects drivers' takeover performance and automation trust. Hum. Factors J. Hum. Factors Ergonomics Soc.
30. Ho, C., Tan, H.Z., Spence, C., The differential effect of vibrotactile and auditory cues on visual spatial attention. Ergonomics 49 (2006), 724–738.
31. Houtenbos, M., De Winter, J.C.F., Hale, A.R., Wieringa, P.A., Hagenzieker, M.P., Concurrent audio-visual feedback for supporting drivers at intersections: a study using two linked driving simulators. Appl. Ergon. 60 (2017), 30–42.
32. ISO/DTS 14198, Road Vehicles - Ergonomic Aspects of Transport Information and Control Systems - Calibration Tasks for Methods Which Assess Driver Demand Due to the Use of Invehicle Systems. Retrieved from, 2012 http://www.din.de/en/getting-involved/standards-committees/naautomobil/din-spec/wdc-beuth:din21:169391345/toc-1933931/download.
33. Jensen, A.R., Clocking the Mind: Mental Chronometry and Individual Differences. 2006, Elsevier, Amsterdam.
34. Joksch, H.C., Velocity change and fatality risk in a crash—a rule of thumb. Accid. Analysis Prev. 25 (1993), 103–104.
35. Klüver, M., Herrigel, C., Heinrich, C., Schöner, H.P., Hecht, H., The behavioral validity of dual-task driving performance in fixed and moving base driving simulators. Transp. Res. Part F Traffic Psychol. Behav. 37 (2016), 78–96.
36. Kyriakidis, M., Happee, R., De Winter, J.C.F., Public opinion on automated driving: results of an international questionnaire among 5000 respondents. Transp. Res. Part F Traffic Psychol. Behav. 32 (2015), 127–140.
37. Lee, J., McGehee, D., Brown, T., Marshall, D., Effects of adaptive cruise control and alert modality on driver performance. Transp. Res. Rec. J. Transp. Res. Board 1980 (2006), 49–56.
38. Lees, M.N., Cosman, J., Lee, J.D., Vecera, S.P., Dawson, J.D., Rizzo, M., Cross-modal warnings for orienting attention in older drivers with and without attention impairments. Appl. Ergon. 43 (2012), 768–776.
39. Lorenz, L., Kerschbaum, P., Schumann, J., Designing take over scenarios for automated driving How does augmented reality support the driver to get back into the loop?. Proc. Hum. Factors Ergonomics Soc. Annu. Meet. 58 (2014), 1681–1685.
40. Louw, T.L., Merat, N., Jamson, A.H., Engaging with highly automated driving: to be or not to be in the loop?. (Salt Lake City, UT) Proceedings of the 8th International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, 2015, 190–196.
41. Lu, Z., Happee, R., Cabrall, C.D.D., Kyriakidis, M., De Winter, J.C.F., Human factors of transitions in automated driving: a general framework and literature survey. Transp. Res. Part F Traffic Psychol. Behav. 43 (2016), 183–198.
42. Melcher, V., Rauh, S., Diederichs, F., Widlroither, H., Bauer, W., Take-over requests for automated driving. (Las Vegas, NV) Proceedings of the 6th International Conference on Applied Human Factors and Ergonomics (AHFE), 2015, 2867–2873.
43. Meng, F., Spence, C., Tactile warning signals for in-vehicle systems. Accid. Analysis Prev. 75 (2015), 333–346.
44. Merat, N., Jamson, A.H., Lai, F.C., Daly, M., Carsten, O.M., Transition to manual: driver behaviour when resuming control from a highly automated vehicle. Transp. Res. Part F Traffic Psychol. Behav. 27 (2014), 274–282.
45. Mercedes-Benz, E-Class. Sedan. Operator's Manual. Retrieved from, 2015 https://carmanuals2.com/get/bmw-5-series-gran-turismo-2015-owner-s-manual-63422.
46. Mohebbi, R., Gray, R., Tan, H.Z., Driver reaction time to tactile and auditory rear-end collision warnings while talking on a cell phone. Hum. Factors J. Hum. Factors Ergonomics Soc. 51 (2009), 102–110.
47. Mok, B., Johns, M., Lee, K.J., Miller, D., Sirkin, D., Ive, P., Ju, W., Emergency, automation off: unstructured transition timing for distracted drivers of automated vehicles. (Canary Islands, Spain) Proceedings of the 2015 IEEE 18th International Conference on Intelligent Transportation Systems, 2015, 2458–2464.
48. Müsseler, J., Aschersleben, G., Arning, K., Proctor, R.W., Reversed effects of spatial compatibility in natural scenes. Am. J. Psychol. 122 (2009), 325–336.
49. Naujoks, F., Mai, C., Neukum, A., The effect of urgency of take-over requests during highly automated driving under distraction conditions. (Kraków, Poland) Ahram, T., Karwowski, W., Marek, T., (eds.) Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics (AHFE), 2014.
50. Naujoks, F., Neukum, A., Specificity and Timing of Advisory Warnings Based on Cooperative Perception. Butz, A., Koch, M., Schlichter, J., (eds.) Mensch & Computer 2014 – Workshopband, 2014, De Gruyter Oldenbourg, Berlin, 229–238.
51. Naujoks, F., Purucker, C., Neukum, A., Wolter, S., Steiger, R., Controllability of partially automated driving functions – does it matter whether drivers are allowed to take their hands off the steering wheel?. Transp. Res. Part F Traffic Psychol. Behav. 35 (2015), 185–198.
52. Navarro, J., Mars, F., Forzy, J.F., El-Jaafari, M., Hoc, J.M., Objective and subjective evaluation of motor priming and warning systems applied to lateral control assistance. Accid. Analysis Prev. 42 (2010), 904–912.
53. Navarro, J., Mars, F., Hoc, J.-M., Lateral control support for car drivers: a human-machine cooperation approach. Proceedings of the 14th European Conference on Cognitive Ergonomics: Invent! Explore!, 2007, ACM, New York, NY, 249–252.
54. Nukarinen, T., Rantala, J., Farooq, A., Raisamo, R., Delivering directional haptic cues through eyeglasses and a seat. (Evanston, IL) IEEE World Haptics Conf., 2015, 345–350.
55. Oviatt, S., Ten Myths of Multimodal Interaction. Communications of the ACM. vol. 42, 1999, 74–81.
56. Payre, W., Cestac, J., Delhomme, P., Fully automated driving impact of trust and practice on manual control recovery. Hum. Factors J. Hum. Factors Ergonomics Soc. 58 (2016), 229–241.
57. Petermann-Stock, I., Hackenberg, L., Muhr, T., Mergl, C., Wie lange braucht der Fahrer? Eine Analyse zu Übernahmezeiten aus verschiedenen Nebentätigkeiten während einer hochautomatisierten Staufahrt. Tagung Fahrerassistenzsysteme. Der Weg zum automatischen Fahren. TÜV SÜD Akademie GmbH. 2013.
58. Petermeijer, S.M., Abbink, D.A., Mulder, M., De Winter, J.C.F., The effect of haptic support systems on driver performance: a literature survey. IEEE Trans. Haptics 8 (2015), 467–479.
59. Petermeijer, S.M., Cieler, S., De Winter, J.C.F., Comparing spatially static and dynamic vibrotactile take-over requests in the driver seat. Accid. Analysis Prev. 99 (2017), 218–227.
60. Petermeijer, S.M., De Winter, J.C.F., Bengler, K., Vibrotactile displays: a survey with a view on highly automated driving. IEEE Trans. Intelligent Transp. Syst. 17:4 (2016), 897–907.
61. Politis, I., Brewster, S.A., Pollick, F., Evaluating multimodal driver displays under varying situational urgency. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2014, ACM, New York, NY, 4067–4076.
62. Prewett, M.S., Elliott, L.R., Walvoord, A.G., Coovert, M.D., A meta-analysis of vibrotactile and visual information displays for improving task performance. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 42 (2012), 123–132.
63. Radlmayr, J., Gold, C., Lorenz, L., Farid, M., Bengler, K., How traffic situations and non-driving related tasks affect the take-over quality in highly automated driving. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 58, 2014, 2063–2067.
64. Scott, J.J., Gray, R., A comparison of tactile, visual, and auditory warnings for rear-end collision prevention in simulated driving. Hum. Factors J. Hum. Factors Ergonomics Soc. 50 (2008), 264–275.
65. SAE International, Taxonomy and Definitions for Terms Related to Onroad Motor Vehicle Automated Driving Systems (Standard No. J3016). Retrieved from, 2014, SAE International http://standards.sae.org/j3016_201401.
66. Schwalk, M., Kalogerakis, N., Maier, T., Driver support by a vibrotactile seat matrix – recognition, adequacy and workload of tactile patterns in take-over scenarios during automated driving. (Las Vegas, NV) Proceedings of the 6th International Conference on Applied Human Factors and Ergonomics (AHFE), 2015, 1427–1434.
67. Sharek, D., A useable, online NASA-TLX tool. Proc. Hum. Factors Ergonomics Soc. Annu. Meet. 55 (2011), 1375–1379.
68. Shladover, S.E., Road Vehicle Automation History, Opportunities and Challenges. Mini-seminar 'developments Selfdriving Vehicles in USA'. (Delft, the Netherlands), 2015, November.
69. Siegler, I., Reymond, G., Kemeny, A., Berthoz, A., Sensorimotor integration in a driving simulator: contributions of motion cueing in elementary driving tasks. (Sophia-Antipolis, France) Proceedings of the Driving Simulation Conference, 2001, 21–32.
70. Simon, J.R., Hinrichs, J.V., Craft, J.L., Auditory SR compatibility: reaction time as a function of ear-hand correspondence and ear-response-location correspondence. J. Exp. Psychol. 86 (1970), 97–102.
71. Spence, C., Santangelo, V., Capturing spatial attention with multisensory cues: a review. Hear. Res. 258 (2009), 134–142.
72. Stanley, L.M., Haptic and auditory cues for lane departure warnings. Proc. Hum. Factors Ergonomics Soc. Annu. Meet. 50 (2006), 2405–2408.
73. Stanton, N.A., Edworthy, J., Auditory warnings and displays: an overview. Stanton, N.A., Edworthy, J., (eds.) Human Factors in Auditory Warnings, 1999, Ashgate Publishing Limited, Aldershot, 3–30.
74. Stokes, A., Wickens, C., Kite, K., Display Technology-human Factors Concepts (Technical Report No. SAE R-102)., 1990, Society of Automotive Engineers, Inc, Warrendale, PA.
75. Straughn, S.M., Gray, R., Tan, H.Z., To go or not to go: stimulus-response compatibility for tactile and auditory pedestrian collision warnings. IEEE Trans. Haptics 2 (2009), 111–117.
76. Summala, H., Drivers' steering reaction to a light stimulus on a dark road. Ergonomics 24 (1981), 125–131.
77. Talsma, D., Senkowski, D., Soto-Faraco, S., Woldorff, M.G., The multifaceted interplay between attention and multisensory integration. Trends Cognitive Sci. 14 (2010), 400–410.
78. Technical University of Munich, Static Driving Simulator. Retrieved from, 2015 http://www.lfe.mw.tum.de/en/research/methods-and-lab-equipment/static-driving-simulator.
79. Teichner, W.H., Recent studies of simple reaction time. Psychol. Bull. 51 (1954), 128–149.
80. Telpaz, A., Rhindress, B., Zelman, I., Tsimhoni, O., Haptic seat for automated driving: preparing the driver to take control effectively. Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 2015, ACM, New York, NY, 23–30.
81. Tijerina, L., Jackson, J.L., Pomerleau, D.A., Romano, R.A., Petersen, A.D., Driving simulator tests of lane departure collision avoidance systems. Intelligent Transportation: Realizing the Benefits. Proceedings of the 1996 Annual Meeting of ITS America, 1996, 636–648.
82. Umiltá, C., Nicoletti, R., Spatial stimulus-response compatibility. Adv. Psychol. 65 (1990), 89–116.
83. Underwood, S.E., Automated vehicles forecast vehicle symposium opinion survey. (San Francisco, CA) Presentation at the 2014 Automated Vehicles Symposium, 2014.
84. Van der Laan, J.D., Heino, A., De Waard, D., A simple procedure for the assessment of acceptance of advanced transport telematics. Transp. Res. Part C Emerg. Technol. 5 (1997), 1–10.
85. Van Erp, J.B.F., Toet, A., Janssen, J.B., Uni-, bi-and tri-modal warning signals: effects of temporal parameters and sensory modality on perceived urgency. Saf. Sci. 72 (2015), 1–8.
86. Walch, M., Lange, K., Baumann, M., Weber, M., Autonomous driving: investigating the feasibility of car-driver handover assistance. Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 2015, ACM, New York, NY, 11–18.
87. Wang, D.Y.D., Pick, D.F., Proctor, R.W., Ye, Y., Effect of a side collision-avoidance signal on simulated driving with a navigation system. (Stevenson, WA) Proceedings of the 4th International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, 2007, 206–211.
88. Weller, G., Heyne, F., Feige, T., Bretschneider, H., Oeser, H., Schlag, B., Die Wirkung gerichteter Warnungen von Fahrerassis-tenzsystemen auf die Blickzuwendungs- und Reakti-onszeiten von Autofahrern. Brandenburg, E., Doria, L., Gross, A., Günzler, T., Smieszek, H., (eds.) Grundlagen und Anwendungen der Mensch-Maschine-Interaktion 10. Berliner Werkstatt Mensch-Maschine-Systeme, 2013, Universitätsverlag der TU Berlin, Berlin, 376–382.
89. Woodworth, R.S., Schlosberg, H., Experimental Psychology. Revised edition, 1954, Henry Holt & Co, New York, NY.
90. Young, M.S., Attention, Automaticity, and Automation: New Perspectives on Mental Underload and Performance. Doctoral dissertation, 2000, University of Southampton, UK.
91. Zarife, R., Integrative Warning Concept for Multiple Driver Assistance Systems. Doctoral dissertation, 2014, University of Würzburg.
92. Zeeb, K., Buchner, A., Schrauf, M., What determines the take-over time? An integrated model approach of driver take-over after automated driving. Accid. Analysis Prev. 78 (2015), 212–221.
93. Zhang, Y., Yan, X., Yang, Z., Discrimination of effects between directional and nondirectional information of auditory warning on driving behavior. Discrete Dyn. Nat. Soc., 2015 Article ID 980281.