Sensor fusion and computer vision for context-aware control of a multi degree-of-freedom prosthesis

Journal of Neural Engineering

Volumn 12, Issue 6, 2015, Pages

  Markovic, Marko ^a   Dosen, Strahinja ^b   Popovic, Dejan ^c   Graimann, Bernhard ^a   Farina, Dario ^b  

^a Otto Bock HealthCare GmbH   (Germany)

^b BERNSTEIN CENTER FOR COMPUTATIONAL NEUROSCIENCE   (Germany)

^c University of Belgrade   (Serbia)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL LIMBS; COMPUTER VISION; DEGREES OF FREEDOM (MECHANICS); MUSCLE; PROSTHETICS; SENSORY PERCEPTION; ULTRASONIC DEVICES; VISION;

ACTIVITIES OF DAILY LIVING; MULTI DEGREE-OF-FREEDOM; MULTIPLE DEGREES OF FREEDOM; MYOELECTRIC CONTROL; PROPORTIONAL CONTROL; PROSTHESIS CONTROL; SYNERGISTIC ACTION; VISUAL PERCEPTION;

MYOELECTRICALLY CONTROLLED PROSTHETICS;

ARTICLE; BIOLOGY; COMPUTER; CONTROLLED STUDY; DAILY LIFE ACTIVITY; EXERCISE; GRIP STRENGTH; MOTION; MOVEMENT (PHYSIOLOGY); MYOELECTRIC CONTROL; MYOELECTRICITY; PRIORITY JOURNAL; PROPRIOCEPTION; PROSTHESIS; SENSOR; VISION; WRIST; ADULT; AMPUTEE; AWARENESS; COMPUTER INTERFACE; DEVICES; ELECTROMYOGRAPHY; FEMALE; FOREARM; HAND STRENGTH; HUMAN; LIMB PROSTHESIS; MALE; MIDDLE AGED; PHYSIOLOGY; PROCEDURES; PROSTHESIS DESIGN;

ACTIVITIES OF DAILY LIVING; ADULT; AMPUTEES; ARTIFICIAL LIMBS; AWARENESS; ELECTROMYOGRAPHY; FEMALE; FOREARM; HAND STRENGTH; HUMANS; MALE; MIDDLE AGED; PROSTHESIS DESIGN; USER-COMPUTER INTERFACE;

EID: 84969135111     PISSN: 17412560     EISSN: 17412552     Source Type: Journal    
DOI: 10.1088/1741-2560/12/6/066022     Document Type: Article

References (75)

1. MacKenzie C L and Iberall T 1994 Advances in Psychology: The Grasping Hand vol 104 (Amsterdam: North Holland)
2. Cutkosky M R 1989 On grasp choice, grasp models, and the design of hands for manufacturing tasks IEEE Trans. Robot. Autom. 5 269-79
3. Churchill A, Hopkins B, Rönnqvist L and Vogt S 2000 Vision of the hand and environmental context in human prehension Exp. Brain Res. 134 81-9
4. Connolly J D and Goodale M A 1999 The role of visual feedback of hand position in the control of manual prehension Exp. Brain Res. 125 281-6
5. Ingram H A, Van Donkelaar P, Cole J, Vercher J L, Gauthier G M and Miall R C 2000 The role of proprioception and attention in a visuomotor adaptation task Exp. Brain Res. 132 114-26
6. Ziegler-Graham K, MacKenzie E J, Ephraim P L, Travison T G and Brookmeyer R 2008 Estimating the prevalence of limb loss in the United States: 2005 to 2050 Arch. Phys. Med. Rehabil. 89 422-9
7. Fougner A, Stavdahl O, Kyberd P J, Losier Y G and Parker P A 2012 Control of upper limb prostheses: terminology and proportional myoelectric control-a review. IEEE Trans. Neural Syst. Rehabil. Eng. 20 663-77
8. Belter J T, Segil J L, Dollar A M and Weir R F 2013 Mechanical design and performance specifications of anthropomorphic prosthetic hands: A review J. Rehabil. Res. Dev. 50 599-618
9. Otto Bock HealthCare GmbH, Sensor Hand Speed® (http://ottobock.de/cps/rde/xchg/ob-com-en/hs.xsl/3652.html)
10. Otto Bock HealthCare GmbH, Michelangelo® (http://ottobock.com/cps/rde/xchg/ob-com-en/hs.xsl/49464.html)
11. Touch Bionics iLimb (http://touchbionics.com/products/active-prostheses/)
12. DARPA (http://darpa.mil/Our-Work/BTO/Programs/Revolutionizing-Prosthetics.aspx)
13. Ning J, Dosen S, Muller K-R and Farina D 2012 Myoelectric control of artificial limbs-is there a need to change focus? IEEE Signal Process. Mag. 29 152-150
14. Amsuess S, Goebel P, Graimann B and Farina D 2014 Extending mode switching to multiple degrees of freedom in hand prosthesis control is not efficient Proc. Ann. Int. Conf. IEEE Engineering in Medical and Biology Society pp 658-61
15. Fifer M S et al 2014 Simultaneous neural control of simple reaching and grasping with the modular prosthetic limb using intracranial EEG IEEE Trans. Neural Syst. Rehabil. Eng. 22 695-705
16. Xiao R and Ding L 2013 Evaluation of EEG features in decoding individual finger movements from one hand Comput. Math. Methods Med. 2013 243257
17. Suzuki R, Ogino K, Nobuaki K, Kogure K and Tanaka K 2013 Development of meal support system with voice input interface for upper limb disabilities Proc. of the IEEE 8th Conf. on Industrial Electronics and Applications (ICIEA) pp 714-8
18. Carrozza M C, Persichetti A, Laschi C, Vecchi F, Lazzarini R, Vacalebri P and Dario P 2007 A wearable biomechatronic interface for controlling robots with voluntary foot movements IEEE/ASME Trans. Mechatronics 12 1-11
19. Coapt (http://coaptengineering.com/)
20. Baker J J, Scheme E, Englehart K, Hutchinson D T and Greger B 2010 Continuous detection and decoding of dexterous finger flexions with implantable myoelectric sensors IEEE Trans. Neural Syst. Rehabil. Eng. 18 424-32
21. Kuiken T A, Dumanian G A, Lipschutz R D, Miller L A and Stubblefield K A 2004 The use of targeted muscle reinnervation for improved myoelectric prosthesis control in a bilateral shoulder disarticulation amputee Prosthet. Orthot. Int. 28 245-53
22. Amsuess S, Gobel P, Graimann B and Farina D 2014 A multiclass proportional myocontrol algorithm for upper limb prosthesis control: validation in real-life scenarios on amputees IEEE Trans. Neural Syst. Rehabil. Eng. 23 827-36
23. Jiang N, Englehart K B and Parker P A 2009 Extracting simultaneous and proportional neural control information for multiple-DOF prostheses from the surface electromyographic signal IEEE Trans. Biomed. Eng. 56 1070-80
24. Castellini C, Passig G and Zarka E 2012 Using ultrasound images of the forearm to predict finger positions IEEE Trans. Neural Syst. Rehabil. Eng. 20 788-97
25. Bodenhagen L, Fugl A R, Jordt A, Willatzen M, Andersen K A, Olsen M M, Koch R, Petersen H G and Kruger N 2014 An adaptable robot vision system performing manipulation actions with flexible objects IEEE Trans. Autom. Sci. Eng. 11 749-65
26. Chitta S, Jones E, Ciocarlie M and Hsiao K 2012 Mobile manipulation in unstructured environments: perception, planning, and execution IEEE Robot. Autom. Mag. 19 58-71
27. Swain I D and Nightingale J M 1980 An adaptive control system for a complete hand/arm prosthesis J. Biomed. Eng. 2 163-6
28. Tomovic R and Boni G 1962 An adaptive artificial hand IRE Trans. Autom. Control 7 3-10
29. Dillon S S 1995 Automatic control of wrist rotation in artificial arm Proc. of the MyoElectric Controls Prosthetics Symp. pp 110-1
30. Banziger E 1996 Wrist rotation activation in myoelectric prosthetics-an innovative approach: a case study ACPOC News 30 12-4
31. Nightingale J M 1985 Microprocessor control of an artificial arm J. Microcomput. Appl. 8 167-73
32. Chappell P H, Nightingale J M, Kyberd P J and Barkhordar M 1987 Control of a single degree of freedom artificial hand J. Biomed. Eng. 9 273-7
33. Tomovic R, Bekey G and Karplus W 1987 A strategy for grasp synthesis with multifingered robot hands Proc. of the IEEE Int. Conf. on Robotics and Automation vol 4 pp 83-9
34. Wettels N, Parnandi A R, Loeb G E and Sukhatme G S 2009 Grip control using biomimetic tactile sensing systems IEEE/ASME Trans. Mechatronics 14 718-23
35. Light C M, Chappell P H, Hudgins B and Engelhart K 2002 Intelligent multifunction myoelectric control of hand prostheses J. Med. Eng. Technol. 26 139-46
36. Goto S, Nakamura M and Sugi T 2008 Development of meal assistance orthosis for disabled persons using EOG signal and dish image Int. J. Adv. Mechatron. Syst. 1 107
37. Novak D and Riener R 2013 Enhancing patient freedom in rehabilitation robotics using gaze-based intention detection Proc. of the IEEE Int. Conf. on Rehabilitation Robotics
38. Muelling K, Venkatraman A, Valois J-S, Downey J, Weiss J, Javdani S, Hebert M, Schwartz A B, Collinger J L and Bagnell J A 2015 Autonomy infused teleoperation with application to BCI manipulation (arXiv:1503.05451)
39. Controzzi M, Cipriani C, Popovic D B and Carrozza M C 2013 Controlling hand-assistive devices: utilizing electrooculography as a substitute for vision IEEE Robot. Autom. Mag. 20 40-52
40. Novak D and Riener R 2014 A survey of sensor fusion methods in wearable robotics Rob. Auton. Syst. 73 155-70
41. McMullen D et al 2014 Demonstration of a semi-autonomous hybrid brain-machine interface using human intracranial EEG, eye tracking, and computer vision to control a robotic upper limb prosthetic IEEE Trans. Neural Syst. Rehabil. Eng. 22 784-96
42. Johnson M, Bradshaw J M, Feltovich P J, Jonker C M, Van Riemsdijk M B and Sierhuis M 2014 Coactive design: designing support for interdependence in joint activity J. Human-Robot Interact. 3 43
43. EUrobotics 2015 Robotics 2020 Multi-Annual Roadmap pp 178-228
44. Muzumdar A (ed) 2004 Powered Upper Limb Prostheses (Berlin: Springer) pp 35-54
45. Creative Technology Ltd, Senz3D® (http://de.creative.com/p/web-cameras/creative-senz3d)
46. Dosen S, Markovic M, Popović D B, Farina D and Graimann B 2013 Sensor fusion for control of upper limb protheses: integration of myoelectric control with stereovision, augmented reality and inertial sensing EP 13171671.4
47. Microsoft Corporation, Microsoft HoloLens® (https://microsoft.com/microsoft-hololens/en-us)
48. Meta Inc., Meta 1 developer kit (https://getameta.com/)
49. Markovic M, Dosen S, Cipriani C, Popovic D and Farina D 2014 Stereovision and augmented reality for closed-loop control of grasping in hand prostheses J. Neural Eng. 11 046001
50. Dosen S, Cipriani C, Kostić M, Controzzi M, Carrozza M C and Popović D B 2010 Cognitive vision system for control of dexterous prosthetic hands: experimental evaluation J. Neuroeng. Rehabil. 7 42
51. Matija Š, Kočović S, Marković M and Popović D B 2014 Microsoft kinect-based artificial perception system for control of functional electrical stimulation assisted grasping Biomed. Res. Int. 2014 12
52. Marković M, Graimann B, Došen S and Farina D 2014 Computer vision for automatic control of orientation and preshape in a hand prosthesis with an active wrist MEC2014 pp 143-5
53. Luo R C, Chang C C and Lai C C 2011 Multisensor fusion and integration: theories, applications, and its perspectives IEEE Sens. J. 11 3122-38
54. Pugh A 1991 The robot story Comput. Contr. Eng. J. 2 13-21
55. Smith L H, Kuiken T A and Hargrove L J 2014 Real-time simultaneous and proportional myoelectric control using intramuscular EMG J. Neural Eng. 11 066013
56. Kermorgant O and Chaumette F 2011 Multi-sensor data fusion in sensor-based control: application to multi-camera visual servoing Proc. of the IEEE Int. Conf. on Robotics and Automation pp 4518-23
57. Jones L E and Davidson J H 1999 Save that arm: a study of problems in the remaining arm of unilateral upper limb amputees Prosthet. Orthot. Int. 23 55-8
58. Hart S G and Staveland L E 1988 Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research (Amsterdam: Elsevier) vol 52
59. Gonzalez J, Soma H, Sekine M and Yu W 2012 Psychophysiological assessment of a prosthetic hand sensory feedback system based on an auditory display: a preliminary study J. Neuroeng. Rehabil. 9 33
60. Touch Bionics, Inc., Grip chips (http://touchbionics.com/sites/default/files/files/Grip Chip datasheet May 2014.pdf)
61. IBT, morph (http://i-biomed.com/morph.html)
62. Vilarino M 2014 Technical Note: A Novel Wireless Controller for Switching among Modes for an Upper-Limb Prosthesis (www.oandp.org/AcademyTODAY/2014Jan/4.asp)
63. Vilarino M, Moon J, Rogner K, Varghese J, Ryan T, Thakor N V and Kaliki R 2015 Outcomes and perception of a conventional and alternative myoelectric control strategy: a study of experienced and new multiarticulating hand users Am. Acad. Orthot. Prosthetics 7 53-62
64. Krigslund R, Dosen S, Popovski P, Dideriksen J L, Pedersen G F and Farina D 2013 A novel technology for motion capture using passive UHF RFID tags IEEE Trans. Biomed. Eng. 60 1453-7
65. Aulinas J, Petillot Y, Salvi J and Lladó X 2008 The SLAM problem: a survey in Frontiers Artif. Intell. Appl. 184 363-71
66. Stein S C, Worgotter F, Schoeler M, Papon J and Kulvicius T 2014 Convexity based object partitioning for robot applications Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA) pp 3213-20
67. Shumway-Cook A and Woollacott M H 2007 Motor Control: Translating Research Into Clinical Practice (Philadelphia, PA: Lippincott Williams and Williams)
68. SensoMotoric Instruments Inc., SMI Eye Tracking Glasses® (http://eyetracking-glasses.com/)
69. ™ SDK (https://software.intel.com/en-us/intel-realsense-sdk/details)
70. Sun L, Liu G and Liu Y 2014 3D hand tracking with head mounted gaze-directed camera IEEE Sens. J. 14 1380-90
71. El-Gohary M and McNames J 2015 Human joint angle estimation with inertial sensors and validation with a robot arm IEEE Trans. Biomed. Eng. 62 1759-67
72. Cipriani C, Zaccone F, Micera S and Carrozza M C 2008 On the shared control of an EMG-controlled prosthetic hand: analysis of user-prosthesis interaction IEEE Trans. Robot. 24 170-84
73. Hill W, Kyberd P, Hermanssom L, Hubbarad S, Stavdahl O, Swanson S and Cooper M 2009 Upper limb prosthetic outcome measures (UPLOM): a working group and their findings J. Prosthetics Orthot. 21 69-82
74. Hill W, Stavdahl Ø, Liselotte H, Kyberd P, Swanson S and Hubbarad S 2009 'Functional outcomes in the WHO-ICF model: establishment of the upper limb prosthetic outcome measures group J. Prosthetics Orthot. 21 115-9
75. Light C M, Chappell P H and Kyberd P J 2002 Establishing a standardized clinical assessment tool of pathologic and prosthetic hand function: normative data, reliability, and validity Arch. Phys. Med. Rehabil. 83 776-83