Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning

Experimental Brain Research

Volumn 233, Issue 3, 2015, Pages 909-925

  Sigrist, Roland ^a,b   Rauter, Georg ^a,b   Marchal Crespo, Laura ^a,b   Riener, Robert ^a,b   Wolf, Peter ^a,b  

^a ETH ZURICH   (Switzerland)

^b UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

Augmented feedback; Haptic guidance; Movement sonification; Multimodal feedback; Robot assisted learning; Rowing simulator

Indexed keywords

ADULT; ARTICLE; AUDITORY FEEDBACK; ERROR; FACILITATION; FEMALE; HUMAN; HUMAN EXPERIMENT; LEARNING; MALE; NORMAL HUMAN; PERCEPTION; PRIORITY JOURNAL; ROWING; TACTILE FEEDBACK; TASK PERFORMANCE; ULTRASOUND; VIRTUAL REALITY; VISUAL FEEDBACK; HEARING; MOTOR PERFORMANCE; PHYSIOLOGY; PSYCHOMOTOR PERFORMANCE; SENSORY FEEDBACK; TOUCH; VISION; YOUNG ADULT;

ADULT; AUDITORY PERCEPTION; FEEDBACK, SENSORY; FEMALE; HUMANS; LEARNING; MALE; MOTOR SKILLS; PSYCHOMOTOR PERFORMANCE; TOUCH PERCEPTION; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84925536399     PISSN: 00144819     EISSN: 14321106     Source Type: Journal    
DOI: 10.1007/s00221-014-4167-7     Document Type: Article

References (99)

1. Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14(3):257–262
2. Blandin Y, Toussaint L, Shea CH (2008) Specificity of practice: interaction between concurrent sensory information and terminal feedback. J Exp Psychol Learn Mem Cogn 34(4):994–1000
3. Braun DA, Mehring C, Wolpert DM (2010) Structure learning in action. Behav Brain Res 206(2):157–165
4. Brown R, Palmer C (2012) Auditory–motor learning influences auditory memory for music. Mem Cognit 40(4):567–578
5. Buchanan J, Wang C (2012) Overcoming the guidance effect in motor skill learning: feedback all the time can be beneficial. Exp Brain Res 219(2):305–320
6. Burke JL, Prewett MS, Gray AA, Yang L, Stilson FRB, Coovert MD, Elliot LR, Redden E (2006) Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis. In: Proceedings of the 8th international conference on multimodal interfaces, New York, NY, USA, pp 108–117
7. Carson RG, Kelso JAS (2004) Governing coordination: behavioural principles and neural correlates. Exp Brain Res 154(3):267–274
8. Cesqui B, Aliboni S, Mazzoleni S, Carrozza M, Posteraro F, Micera S (2008) On the use of divergent force fields in robot-mediated neurorehabilitation. In: 2nd IEEE RAS EMBS international conference on biomedical robotics and biomechatronics 2008 BioRob 2008, pp 854–861
9. Chen X, Agrawal S (2013) Assisting versus repelling force-feedback for learning of a line following task in a wheelchair. IEEE Trans Neural Syst Rehabil Eng 21(6):959–968
10. Chen JL, Penhune VB, Zatorre RJ (2008) Listening to musical rhythms recruits motor regions of the brain. Cereb Cortex 18(12):2844–2854
11. Chiviacowsky S, Wulf G (2007) Feedback after good trials enhances learning. Res Q Exerc Sport 78:40–47
12. Chollet D, Madani M, Micallef JP (1992) Effects of two types of biomechanical bio-feedback on crawl performance. In: MacLaren D, Reilly T, Lees A (eds) Biomechanics and medicine in swimming. E & FN Spon, London, pp 57–62
13. Effenberg AO (2005) Movement sonification: effects on perception and action. IEEE Multimedia 12(2):53–59
14. Emken J, Reinkensmeyer DJ (2005) Robot-enhanced motor learning: accelerating internal model formation during locomotion by transient dynamic amplification. IEEE Trans Neural Syst Rehabil Eng 13(1):33–39
15. Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433
16. Freides D (1974) Human information processing and sensory modality: cross-modal functions information complexity memory and deficit. Psychol Bull 81(5):284–310
17. Giese MA, Poggio T (2000) Morphable models for the analysis and synthesis of complex motion patterns. Int J Comput Vis 38(1):59–73
18. Guadagnoli M, Kohl R (2001) Knowledge of results for motor learning: relationship between error estimation and knowledge of results frequency. J Mot Behav 33(2):217–224
19. Guadagnoli MA, Lee TD (2004) Challenge point: a framework for conceptualizing the effects of various practice conditions in motor learning. J Mot Behav 36(2):212–224
20. Hale K, Stanney K (2004) Deriving haptic design guidelines from human physiological psychophysical and neurological foundations. IEEE Comput Graph Appl 24(2):33–39
21. Holden MK (2005) Virtual environments for motor rehabilitation: review. Cyberpsychol Behav 8(3):187–211
22. Huang F, Patton J (2013) Augmented dynamics and motor exploration as training for stroke. IEEE Trans Biomed Eng 60(3):838–844
23. Huang H, Ingalls T, Olson L, Ganley K, Rikakis T, He J (2005) Interactive multimodal biofeedback for task-oriented neural rehabilitation. In: 27th annual international conference of the engineering in medicine and biology society 2005 IEEE-EMBS 2005, Shanghai, pp 2547–2550
24. Hubbard T (2013) Auditory imagery contains more than audition. In: Lacey S, Lawson R (eds) Multisensory imagery. Springer, New York, pp 221–247
25. Huegel J, O’Malley MK (2010) Progressive haptic and visual guidance for training in a virtual dynamic task. In: haptics symposium 2010 IEEE, pp 343–350
26. Israel J, Campbell D, Kahn J, Hornby T (2006) Metabolic costs and muscle activity patterns during robotic-and therapist-assisted treadmill walking in individuals with incomplete spinal cord injury. Phys Ther 86(11):1466–1478
27. Kapur A, Tzanetakis G, Virji-Babul N, Wang G, Cook PR (2005) A framework for sonification of vicon motion capture data. In: Proceedings of the 8th conference on digital audio effects, Madrid, Spain
28. Kennedy D, Boyle J, Shea C (2013) The role of auditory and visual models in the production of bimanual tapping patterns. Exp Brain Res 224(4):507–518
29. Keysers C, Gazzola V (2010) Social neuroscience: mirror neurons recorded in humans. Curr Biol 20(8):R353–R354
30. Kim RS, Seitz AR, Shams L (2008) Benefits of stimulus congruency for multisensory facilitation of visual learning. PLoS One 3(1):e1532
31. Kovacs AJ, Shea CH (2011) The learning of 90° continuous relative phase with and without lissajous feedback: external and internally generated bimanual coordination. Acta Psychol 136(3):311–320
32. Krakauer J, Mazzoni P (2011) Human sensorimotor learning: adaptation skill and beyond. Curr Opin Neurobiol 21(4):636–644
33. Kramer G (1994) Auditory display: sonification audification and auditory interfaces. Addison-Wesley, Reading MA
34. Krebs HI, Palazzolo JJ, Dipietro L, Ferraro M, Krol J, Rannekleiv K, Volpe BT, Hogan N (2003) Rehabilitation robotics: performance-based progressive robot-assisted therapy. Auton Robots 15(1):7–20
35. Lahav A, Saltzman E, Schlaug G (2007) Action representation of sound: audiomotor recognition network while listening to newly acquired actions. J Neurosci 27(2):308–314
36. Liebermann DG, Katz L, Hughes MD, Bartlett RM, McClements J, Franks IM (2002) Advances in the application of information technology to sport performance. J Sports Sci 20(10):755–769
37. Liu D, Todorov E (2007) Evidence for the flexible sensorimotor strategies predicted by optimal feedback control. J Neurosci 27(35):9354–9368
38. Liu J, Wrisberg CA (1997) The effect of knowledge of results delay and the subjective estimation of movement form on the acquisition and retention of a motor skill. Res Q Exerc Sport 68(2):145–151
39. Lüttgen J, Heuer H (2012a) The influence of haptic guidance on the production of spatio-temporal patterns. Hum Mov Sci 31(3):519–528
40. Lüttgen J, Heuer H (2012b) Robotic guidance benefits the learning of dynamic but not of spatial movement characteristics. Exp Brain Res 222(1–2):1–9
41. Lüttgen J, Heuer H (2013) The influence of robotic guidance on different types of motor timing. J Mot Behav 45(3):249–258
42. Marchal-Crespo L, Reinkensmeyer DJ (2008a) Effect of robotic guidance on motor learning of a timing task. In: 2nd IEEE RAS EMBS international conference on biomedical robotics and biomechatronics 2008 BioRob 2008, pp 199–204
43. Marchal-Crespo L, Reinkensmeyer DJ (2008b) Haptic guidance can enhance motor learning of a steering task. J Mot Behav 40(6):545–557
44. Marchal-Crespo L, Reinkensmeyer DJ (2009) Review of control strategies for robotic movement training after neurologic injury. J Neuroeng Rehabil 6(1):20
45. Marchal-Crespo L, Furumasu J, Reinkensmeyer DJ (2010) A robotic wheelchair trainer: design overview and a feasibility study. J Neuroeng Rehabil 7(1):40–51
46. Marchal-Crespo L, Raai M, Rauter G, Wolf P, Riener R (2013) The effect of haptic guidance and visual feedback on learning a complex tennis task. Exp Brain Res 231(3):277–291
47. Marschall F, Bund A, Wiemeyer J (2007) Does frequent feedback really degrade learning? A meta analysis. E-Journal Bewegung und Training 1:75–86
48. Milot MH, Marchal-Crespo L, Green CS, Cramer SC, Reinkensmeyer DJ (2010) Comparison of error-amplification and haptic-guidance training techniques for learning of a timing-based motor task by healthy individuals. Exp Brain Res 201(2):119–131
49. Minogue J, Jones MG (2006) Haptics in Education: exploring an untapped sensory modality. Rev Educ Res 76(3):3–17
50. Patton JL, Stoykov M, Kovic M, Mussa-Ivaldi F (2006) Evaluation of robotic training forces that either enhance or reduce error in chronic hemiparetic stroke survivors. Exp Brain Res 168(3):368–383
51. Patton JL, Wei YJ, Bajaj P, Scheidt RA (2013) Visuomotor learning enhanced by augmenting instantaneous trajectory error feedback during reaching. PLoS One 8(1):e46466
52. Proteau L (1992) On the specificity of learning and the role of visual information for movement control. In: Proteau L, Elliott D (eds) Vision and motor control, vol 85. North-Holland, Amsterdam, pp 67–103
53. Rauter G, von Zitzewitz J, Duschau-Wicke A, Vallery H, Riener R (2010) A tendon based parallel robot applied to motor learning in sports. In: 3rd IEEE RAS and EMBS international conference on biomedical robotics and biomechatronics (BioRob) 2010, Tokyo, Japan, pp 82–87
54. Rauter G, Sigrist R, Marchal-Crespo L, Vallery H, Riener R, Wolf P (2011) Assistance or challenge? Filling a gap in user-cooperative control. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 3068–3073
55. Rauter G, Sigrist R, Koch C, Crivelli F, van Raai M, Riener R, Wolf P (2013) Transfer of complex skill learning from virtual to real rowing. PLoS One 8(12):1–18
56. Reinkensmeyer DJ, Akoner O, Ferris D, Gordon K (2009) Slacking by the human motor system: computational models and implications for robotic orthoses. In: Engineering in medicine and biology society 2009 EMBC 2009 annual international conference of the IEEE, pp 2129–2132
57. Ribeiro DC, Sole G, Abbott JH, Milosavljevic S (2011) Extrinsic feedback and management of low back pain: a critical review of the literature. Man Ther 16(3):231–239
58. Robin C, Toussaint L, Blandin Y, Proteau L (2005) Specificity of learning in a video-aiming task: modifying the salience of dynamic visual cues. J Mot Behav 37(5):367–376
59. Ronsse R, Puttemans V, Coxon JP, Goble DJ, Wagemans J, Wenderoth N, Swinnen SP (2011) Motor learning with augmented feedback: modality-dependent behavioral and neural consequences. Cereb Cortex 21(6):1283–1294
60. Salmoni S (1984) Knowledge of results and motor learning A review and critical reappraisal. Psychol Bull 95(3):355–386
61. Schaffert N, Mattes K, Effenberg AO (2011) An investigation of online acoustic information for elite rowers in on-water training conditions. J Hum Sport Exerc 6(2):392–405
62. Schmidt RA (1991) Frequent augmented feedback can degrade learning: evidence and interpretations. Tutor Motor Neurosci 62:59–75
63. Schmidt RA, Wrisberg C (2008) Motor learning and performance: a situation-based learning approach. Human Kinetics, Champaign, IL
64. Schmidt RA, Wulf G (1997) Continuous concurrent feedback degrades skill learning: implications for training and simulation. Hum Factors 39(4):509–525
65. Schmidt RA, Young DE, Swinnen S, Shapiro DC (1989) Summary knowledge of results for skill acquisition: support for the guidance hypothesis. J Exp Psychol Learn Mem Cogn 15(2):352–359
66. Schmitz G, Mohammadi B, Hammer A, Heldmann M, Samii A, Munte T, Effenberg A (2013) Observation of sonified movements engages a basal ganglia frontocortical network. BMC Neurosci 14(1):1–11
67. Secoli R, Milot M, Rosati G, Reinkensmeyer DJ (2011) Effect of visual distraction and auditory feedback on patient effort during robot-assisted movement training after stroke. J Neuroeng Rehabil 8(1):1–10
68. Seitz AR, Dinse HR (2007) A common framework for perceptual learning. Curr Opin Neurobiol 17(2):148–153
69. Seitz AR, Kim R, Shams R (2006) Sound facilitates visual learning. Curr Biol 16(14):1422–1427
70. Shams L, Seitz AR (2008) Benefits of multisensory learning. Trends Cogn Sci 12(11):411–417
71. Sigrist R, Schellenberg J, Rauter G, Broggi S, Riener R, Wolf P (2011) Visual and auditory augmented concurrent feedback in a complex motor task. Presence Teleop Virt 20(1):15–32
72. Sigrist R, Rauter G, Riener R, Wolf P (2013a) Augmented visual auditory haptic and multimodal feedback in motor learning: a review. Psychon Bull Rev 20:21–53
73. Sigrist R, Rauter G, Riener R, Wolf P (2013b) Terminal feedback outperforms concurrent visual auditory and haptic feedback in learning a complex rowing-type task. J Mot Behav 45(6):455–472
74. Snodgrass SJ, Rivett DA, Robertson VJ, Stojanovski E (2010) Real-time feedback improves accuracy of manually applied forces during cervical spine mobilization. Man Ther 15:19–25
75. Swinnen SP, Schmidt RA, Nicholson DE, Shapiro DC (1990) Information feedback for skill acquisition: instantaneous knowledge of results degrades learning. J Exp Psychol Learn Mem Cogn 16(4):706–716
76. Swinnen SP, Lee TD, Verschueren S, Serrien DJ, Bogaerds H (1997) Interlimb coordination: learning and transfer under different feedback conditions. Hum Mov Sci 16(6):749–785
77. Thoroughman K, Shadmehr R (2000) Learning of action through adaptive combination of motor primitives. Nature 407(6805):742
78. Timmermans AAA, Seelen HAM, Willmann RD, Kingma H (2009) Technology-assisted training of arm-hand skills in stroke: concepts on reacquisition of motor control and therapist guidelines for rehabilitation technology design. J Neuroeng Rehabil 6:1
79. Todorov E (2004) Optimality principles in sensorimotor control. Nat Neurosci 7:907–915
80. Todorov E, Jordan MI (2002) Optimal feedback control as a theory of motor coordination. Nat Neurosci 5:1226–1235
81. Todorov E, Shadmehr R, Bizzi E (1997) Augmented feedback presented in a virtual environment accelerates learning of a difficult motor task. J Mot Behav 29(2):147–158
82. van Beers RJ (2009) Motor learning is optimally tuned to the properties of motor noise. Neuron 63(3):406–417
83. van Beers RJ, Sittig AC, Gon JJ (1999) Integration of proprioceptive and visual position-information: an experimentally supported model. J Neurophysiol 81(3):1355
84. van der Linden DW, Cauraugh JH, Greene TA (1993) The effect of frequency of kinetic feedback on learning an isometric force production task in nondisabled subjects. Phys Ther 73(2):79–87
85. Vlachos M, Hadjieleftheriou M, Gunopulos D, Keogh E (2003) Indexing multi-dimensional time-series with support for multiple distance measures. In: proceedings of the ninth acm sigkdd international conference on knowledge discovery and data mining KDD’03, New York, NY, USA, pp 216–225
86. von Zitzewitz J, Wolf P, Novakovic V, Wellner M, Rauter G, Brunschweiler A, Riener R (2008) Real-time rowing simulator with multimodal feedback. Sports Technol 1(6):257–266
87. Wei K, Körding K (2009) Relevance of error: what drives motor adaptation? J Neurophysiol 101(2):655–664
88. Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88(3):638–667
89. Wickens CD (2002) Multiple resources and performance prediction. Theor Issues Ergon Sci 3(2):159–177
90. Winstein CJ (1991) Knowledge of results and motor learning—implications for physical therapy. Phys Ther 71(2):140–149
91. Winstein CJ, Pohl PS, Cardinale C, Green A, Scholtz L, Waters CS (1996) Learning a partial-weight-bearing skill: effectiveness of two forms of feedback. Phys Ther 76(9):985–993
92. Wishart LR, Lee TD, Cunningham SJ, Murdoch JE (2002) Age-related differences and the role of augmented visual feedback in learning a bimanual coordination pattern. Acta Psychol (Amst) 110(2–3):247–263
93. Wolpert D, Flanagan J (2010) Motor learning. Curr Biol 20(11):R467–R472
94. Wolpert DM, Diedrichsen J, Flanagan JR (2011) Principles of sensorimotor learning. Nat Rev Neurosci 12:739–749
95. Wulf G (2007) Self-controlled practice enhances motor learning: implications for physiotherapy. Physiotherapy 93(2):96–101
96. Wulf G, Shea CH (2002) Principles derived from the study of simple skills do not generalize to complex skill learning. Psychon Bull Rev 9(2):185–211
97. Wulf G, Shea CH, Matschiner S (1998) Frequent feedback enhances complex motor skill learning. J Mot Behav 30(2):180–192
98. Wulf G, Hörger M, Shea CH (1999) Benefits of blocked over serial feedback on complex motor skill learning. J Mot Behav 31(1):95–103
99. Zatorre RJ, Chen JL, Penhune VB (2007) When the brain plays music: auditory–motor interactions in music perception and production. Nat Rev Neurosci 8(7):547–558