Combined auditory and vibrotactile feedback for human-machine-interface control

IEEE Transactions on Neural Systems and Rehabilitation Engineering

Volumn 22, Issue 1, 2014, Pages 62-68

  Thorp, Elias B ^a,b   Larson, Eric ^c   Stepp, Cara E ^a  

^a BOSTON UNIVERSITY   (United States)

^b Northwestern University   (United States)

^c UNIVERSITY OF WASHINGTON   (United States)

Author keywords

Auditory; electromyography; human machine interfaces (HMIs); vibrotactile

Indexed keywords

AUDITORY; AUDITORY FEEDBACK; COGNITIVE COSTS; FACIAL SURFACES; HUMAN MACHINE INTERFACE; OVER TRAININGS; VIBRO-TACTILE FEEDBACKS; VIBROTACTILE;

LINGUISTICS; MAN MACHINE SYSTEMS;

ELECTROMYOGRAPHY;

ELECTROMYOGRAPHY; FACE MUSCLE; FEMALE; HEARING; HUMAN; MALE; MAN MACHINE INTERACTION; PHYSIOLOGY; PROCEDURES; PSYCHOPHYSIOLOGY; REFERENCE VALUE; SENSORY FEEDBACK; TASK PERFORMANCE; TOUCH; VIBRATION; YOUNG ADULT;

AUDITORY PERCEPTION; BIOFEEDBACK, PSYCHOLOGY; ELECTROMYOGRAPHY; FACIAL MUSCLES; FEEDBACK, SENSORY; FEMALE; HUMANS; MALE; MAN-MACHINE SYSTEMS; REFERENCE VALUES; TASK PERFORMANCE AND ANALYSIS; TOUCH; VIBRATION; YOUNG ADULT;

EID: 84892384764     PISSN: 15344320     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNSRE.2013.2273177     Document Type: Article

References (46)

1. T. Hinterberger, A. Kubler, J. Kaiser, N. Neumann, and N. Birbaumer, "A brain-computer interface (BCI) for the locked-in: Comparison of different EEG classifications for the thought translation device," Clin. Neurophysiol., vol. 114, pp. 416-425, Mar. 2003. (Pubitemid 36561111)
2. E. W. Sellers and E. Donchin, "A P300-based brain-computer interface: Initial tests by ALS patients," Clin. Neurophysiol., vol. 117, pp. 538-548, Mar. 2006. (Pubitemid 43343986)
3. N. Birbaumer, N. Ghanayim, T. Hinterberger, I. Iversen, B. Kotchoubey, A. Kubler, J. Perelmouter, E. Taub, and H. Flor, "A spelling device for the paralysed," Nature, vol. 398, pp. 297-298, Mar. 25, 1999. (Pubitemid 129522960)
4. F. Nijboer, E. W. Sellers, J. Mellinger, M. A. Jordan, T. Matuz, A. Furdea, S. Halder, U. Mochty, D. J. Krusienski, T. M. Vaughan, J. R. Wolpaw, N. Birbaumer, and A. Kubler, "A P300-based brain-computer interface for people with amyotrophic lateral sclerosis," Clin. Neurophysiol., vol. 119, pp. 1909-1916, Aug. 2008.
5. L. J. Trejo, R. Rosipal, and B. Matthews, "Brain-computer interfaces for 1-D and 2-D cursor control: Designs using volitional control of the EEG spectrum or steady-state visual evoked potentials," IEEE Trans. Neural Syst. Rehabil. Eng., vol. 14, no. 2, pp. 225-229, Jun. 2006. (Pubitemid 44150248)
6. T. M. Vaughan, D. J. McFarland, G. Schalk, W. A. Sarnacki, D. J. Krusienski, E. W. Sellers, and J. R. Wolpaw, "The Wadsworth BCI research and development program: At home with BCI," IEEE Trans. Neural Syst. Rehabil. Eng., vol. 14, no. 2, pp. 229-233, Jun. 2006.
7. N. Birbaumer, A. Kubler, N. Ghanayim, T. Hinterberger, J. Perelmouter, J. Kaiser, I. Iversen, B. Kotchoubey, N. Neumann, and H. Flor, "The thought translation device (TTD) for completely paralyzed patients," IEEE Trans. Rehabil. Eng., vol. 8, no. 2, pp. 190-193, Jun. 2000. (Pubitemid 30437010)
8. B. Blankertz, G. Dornhege,M. Krauledat, K. R.Muller, V. Kunzmann, F. Losch, and G. Curio, "The Berlin brain-computer interface: EEGbased communication without subject training," IEEE Trans. Neural Syst. Rehabil. Eng., vol. 14, no. 2, pp. 147-152, Jun. 2006. (Pubitemid 44150229)
9. F. Cincotti, L.Kauhanen, F. Aloise, T. Palomaki, N. Caporusso, P. Jylanki, D. Mattia, F. Babiloni, G. Vanacker, M. Nuttin, M. G. Marciani, and R. M. J. Del, "Vibrotactile feedback for brain-computer interface operation," Comput. Intell. Neurosci., 2007.
10. A. Furdea, S. Halder, D. J. Krusienski, D. Bross, F. Nijboer, N. Birbaumer, and A. Kubler, "An auditory oddball (P300) spelling system for brain-computer interfaces," Psychophysiology, vol. 46, pp. 617-625, May 2009.
11. H. Higashi, T. M. Rutkowski, Y. Washizawa, A. Cichocki, and T. Tanaka, "EEG auditory steady state responses classification for the novel BCI," in Proc. IEEE Eng. Med. Biol. Soc. Conf., 2011, vol. 2011, pp. 4576-4579.
12. M.A. Lopez-Gordo, E. Fernandez,S.Romero, F. Pelayo, andA. Prieto, "An auditory brain-computer interface evoked by natural speech," J Neural Eng, vol. 9, Jun. 2012.
13. F. Nijboer, A. Furdea, I. Gunst, J. Mellinger, D. J. McFarland, N. Birbaumer, and A. Kubler, "An auditory brain-computer interface (BCI)," J. Neurosci. Methods, vol. 167, pp. 43-50, Jan. 15, 2008. (Pubitemid 350186060)
14. F. Oscari, R. Secoli, F. Avanzini, G. Rosati, and D. J. Reinkensmeyer, "Substituting auditory for visual feedback to adapt to altered dynamic and kinematic environments during reaching," Exp. Brain Res., vol. 221, pp. 33-41, Aug. 2012.
15. M. Pham, T. Hinterberger, N. Neumann, A. Kubler, N. Hofmayer, A. Grether, B. Wilhelm, J. J. Vatine, and N. Birbaumer, "An auditory brain-computer interface based on the self-regulation of slow cortical potentials," Neurorehabil. Neural Repair, vol. 19, pp. 206-218, Sep. 2005. (Pubitemid 41216296)
16. M. Schreuder, B. Blankertz, and M. Tangermann, "A new auditory multi-class brain-computer interface paradigm: Spatial hearing as an informative cue," PLoS One, vol. 5, 2010.
17. F. H. Guenther, J. S. Brumberg, E. J. Wright, A. Nieto-Castanon, J. A. Tourville, M. Panko, R. Law, S. A. Siebert, J. L. Bartels, D. S. Andreasen, P. Ehirim, H.Mao, and P. R. Kennedy, "A wireless brain-machine interface for real-time speech synthesis," PLoS One, vol. 4, 2009.
18. E. Larson, H. P. Terry, M. M. Canevari, and C. E. Stepp, "Categorical vowel perception enhances the effectiveness and generalization of auditory feedback in human-machine-interfaces," PLoS ONE, vol. 8, p. e59860, 2013.
19. P. Iverson and P. K. Kuhl, "Mapping the perceptual magnet effect for speech using signal detection theory and multidimensional scaling," J. Acoust. Soc. Am., vol. 97, pp. 553-562, Jan. 1995.
20. K. A. Kaczmarek, J. G. Webster, P. Bach-y-Rita, and W. J. Tompkins, "Electrotactile and vibrotactile displays for sensory substitution systems," IEEE Trans. Biomed. Eng., vol. 38, no. 1, pp. 1-16, Jan. 1991.
21. A. M. Brouwer and J. B. van Erp, "A tactile P300 brain-computer interface," Front. Neurosci., vol. 4, p. 19, 2010.
22. G. R. Muller-Putz, R. Scherer, C. Neuper, and G. Pfurtscheller, "Steady-state somatosensory evoked potentials: Suitable brain signals for brain-computer interfaces," IEEE Trans. Neural Syst. Rehabil. Eng., vol. 14, no. 1, pp. 30-37, Mar. 2006.
23. M. van der Waal, M. Severens, J. Geuze, and P. Desain, "Introducing the tactile speller: An ERP-based brain-computer interface for communication," J. Neural Eng., vol. 9, p. 045002, Aug. 2012.
24. A. Chatterjee, V. Aggarwal, A. Ramos, S. Acharya, and N. V. Thakor, "A brain-computer interface with vibrotactile biofeedback for haptic information," J. Neuroeng. Rehabil., vol. 4, p. 40, 2007.
25. C. E. Stepp and Y. Matsuoka, "Vibrotactile sensory substitution for object manipulation: Amplitude versus pulse train frequency modulation," IEEE Trans. Neural Syst. Rehabil. Eng, vol. 20, no. 1, pp. 31-37, Jan. 2012.
26. F. C. Huang, R. B. Gillespie, and A. D. Kuo, "Visual and haptic feedback contribute to tuning and online control during object manipulation," J. Mot. Behav., vol. 39, pp. 179-193, May 2007. (Pubitemid 46884562)
27. D. Morris, T. Hong, T. Chang, and K. Salisbury, "Haptic feedback enhances force skill learning," in 2nd Joint EuroHaptics Conf. Symp. Haptic Interfaces Virtual Environ. Teleoperator Syst., 2007, pp. 21-26. (Pubitemid 47297974)
28. C. E. Stepp and Y. Matsuoka, "Object manipulation improvements with single session training outweigh the differences among stimulation sites during vibrotactile feedback," IEEE Trans. Neural Syst. Rehabil. Eng., vol. 19, no. 6, pp. 677-685, Dec. 2011.
29. A. J. Suminski, D. C. Tkach, A. H. Fagg, and N. G. Hatsopoulos, "Incorporating feedback frommultiple sensorymodalities enhances brainmachine interface control," J. Neurosci., vol. 30, pp. 16777-16787, Dec. 15, 2010.
30. H. S. Vitense, J. A. Jacko, and V. K. Emery, "Multimodal feedback: An assessment of performance and mental workload," Ergonomics, vol. 46, pp. 68-87, Jan. 15, 2003. (Pubitemid 36553160)
31. M. E. Thurlings, A. M. Brouwer, J. B. Van Erp, B. Blankertz, and P. J. Werkhoven, "Does bimodal stimulus presentation increase ERP components usable in BCIs?," J. Neural Eng., vol. 9, p. 045005, Aug. 2012.
32. J.-H. Lee and C. Spence, "Assessing the benefits of multimodal feedback on dual-task performance under demanding conditions," in Proc. 22nd Br. HCI Group Annu. Conf. People Comput.: Culture, Creativity, Interaction, 2008, vol. 1, pp. 185-192.
33. J. L. Burke, M. S. Prewett, A. A. Gray, L. Yang, F. R. B. Stilson, M. D. Coovert, E.L.R., "Comparing the effects of visual-auditory and visual-tactile feedback on user performance: A meta-analysis," in Proc. 8th Int. Conf. Multimodal Interfaces, 2006, pp. 108-117. (Pubitemid 47113436)
34. G. Rosati, F. Oscari, S. Spagnol, F.Avanzini, and S. Masiero, "Effect of task-related continuous auditory feedback during learning of tracking motion exercises," J. Neuroeng. Rehabil., vol. 9, p. 79, Oct. 10, 2012.
35. C. J.De Luca, "The use of surface electromyography in biomechanics," J. Appl. Biomechan., vol. 13, pp. 135-163, 1997. (Pubitemid 27246050)
36. C. E. Stepp, "Surface electromyography for speech and swallowing systems: Measurement, analysis, and interpretation," J. Speech Language Hear. Res., vol. 55, pp. 1232-1246, Aug. 2012.
37. G. Scavone and P. Cook, "RtMIDI, RtAudio, and a synthesis (STK) update," presented at the Int. Comput. Music Conf., Barcelona, Spain, 2005.
38. D. H. Klatt, "Software for a cascade/parallel formant synthesizer," J. Acoust. Soc. Am., vol. 67, pp. 971-995, 1980.
39. R. T. Verrillo, "Vibrotactile thresholds measured at the finger," Perception Psychophys., vol. 9, 1971.
40. R. S. Witte and J. S. Witte, Statistics, 9th ed. Hoboken, NJ: Wiley, 2010.
41. C. E. Stepp, Q. An, and Y. Matsuoka, "Repeated training with augmentative vibrotactile feedback increases object manipulation performance," PLoS One, vol. 7, p. e32743, 2012.
42. S. P.Kim, J.D. Simeral,L.R.Hochberg, J. P. Donoghue, G. M. Friehs, and M. J. Black, "Point-and-click cursor control with an intracortical neural interface system by humans with tetraplegia," IEEE Trans. Neural Syst. Rehabil. Eng., vol. 19, no. 2, pp. 193-203, Apr. 2011.
43. G. Schalk, K. J. Miller, N. R. Anderson, J. A. Wilson, M. D. Smyth, J. G. Ojemann, D. W. Moran, J. R. Wolpaw, and E. C. Leuthardt, "Twodimensional movement control using electrocorticographic signals in humans," J. Neural Eng., vol. 5, pp. 75-84, Mar. 2008. (Pubitemid 351366605)
44. J. R. Wolpaw and D. J. McFarland, "Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans," Proc. Nat. Acad. Sci. USA, vol. 101, pp. 17849-17854, Dec. 21, 2004. (Pubitemid 40051975)
45. J. R.Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, "Brain-computer interfaces for communication and control," Clin. Neurophysiol., vol. 113, pp. 767-791, Jun. 2002. (Pubitemid 34621187)
46. C. Cipriani, M. D'Alonzo, and M. C. Carrozza, "A miniature vibrotactile sensory substitution device for multifingered hand prosthetics," IEEE Trans. Biomed. Eng., vol. 59, no. 2, pp. 400-408, Feb. 2012.