Towards autonomous neuroprosthetic control using Hebbian reinforcement learning

Journal of Neural Engineering

Volumn 10, Issue 6, 2013, Pages

  Mahmoudi, Babak ^a   Pohlmeyer, Eric A ^b   Prins, Noeline W ^b   Geng, Shijia ^b   Sanchez, Justin C ^b,c  

^a EMORY UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b University of Miami College of Engineering   (United States)

^c UNIVERSITY OF MIAMI   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTIVE CONTROLLERS; CLOSED-LOOP CONTROL; CONNECTIONIST NETWORKS; CONVERGENCE PROPERTIES; OPEN-LOOP SIMULATIONS; ROBUST PERFORMANCE; SATISFACTORY CONTROL; SEQUENTIAL LEARNING;

ALGORITHMS; REINFORCEMENT LEARNING;

CONTROLLERS;

ALGORITHM; ARTICLE; BINOCULAR CONVERGENCE; FEEDBACK SYSTEM; HEBBIAN REINFORCEMENT LEARNING; LEARNING; NEUROPROSTHESIS; NONHUMAN; PRIORITY JOURNAL; SIMULATION;

ALGORITHMS; ANIMALS; ARTIFICIAL INTELLIGENCE; CALLITHRIX; LEARNING; NEURAL PROSTHESES; RANDOM ALLOCATION; REINFORCEMENT (PSYCHOLOGY);

EID: 84889056477     PISSN: 17412560     EISSN: 17412552     Source Type: Journal    
DOI: 10.1088/1741-2560/10/6/066005     Document Type: Article

References (61)

1. Collinger J L et al 2012 High-performance neuroprosthetic control by an individual with tetraplegia Lancet 381 557-64
2. Hochberg L R et al 2012 Reach and grasp by people with tetraplegia using a neurally controlled robotic arm Nature 485 372-5
3. Mulliken G H, Musallam S and Andersen R A 2008 Decoding trajectories from posterior parietal cortex ensembles J. Neurosci. 28 12913-26
4. Velliste M et al 2008 Cortical control of a prosthetic arm for self-feeding Nature 453 1098-101 (Pubitemid 351871735)
5. Ethier C et al 2012 Restoration of grasp following paralysis through brain-controlled stimulation of muscles Nature 485 368-71
6. Francis J T and Song W 2011 Neuroplasticity of the sensorimotor cortex during learning Neural Plast. 2011 310737
7. Heliot R et al 2010 Learning in closed-loop brain-machine interfaces: modeling and experimental validation IEEE Trans. Syst. Man Cybern. B 40 1387-97
8. Prasad A et al 2012 Comprehensive characterization and failure modes of tungsten microwire arrays in chronic neural implants J. Neural Eng. 9 056015
9. Polikov V S, Tresco P A and Reichert W M 2005 Response of brain tissue to chronically implanted neural electrodes J. Neurosci. Methods 148 1-18 (Pubitemid 41423453)
10. Wu W and Hatsopoulos N G 2008 Real-time decoding of nonstationary neural activity in motor cortex IEEE Trans. Neural Syst. Rehabil. Eng. 16 213 (Pubitemid 351832433)
11. Hatsopoulos N G and Donoghue J P 2009 The science of neural interface systems Annu. Rev. Neurosci. 32 249-66
12. Li Z et al 2011 Adaptive decoding for brain-machine interfaces through Bayesian parameter updates Neural Comput. 23 3162-204
13. Schwartz A B 2004 Cortical neural prosthetics Annu. Rev. Neurosci. 27 487-507 (Pubitemid 39050411)
14. Rotermund D, Ernst U A and Pawelzik K R 2006 Towards on-line adaptation of neuro-prostheses with neuronal evaluation signals Biol. Cybern. 95 243-57 (Pubitemid 44267893)
15. Gage G J et al 2005 Naive coadaptive cortical control J. Neural Eng. 2 52-63 (Pubitemid 40771006)
16. Gilja V et al 2012 A high-performance neural prosthesis enabled by control algorithm design Nat. Neurosci. 15 1752-7
17. Orsborn A L et al 2012 Closed-loop decoder adaptation on intermediate time-scales facilitates rapid BMI performance improvements independent of decoder initialization conditions IEEE Trans. Neural Syst. Rehabil. Eng. 20 468-77
18. Gurel T and Mehring C 2012 Unsupervised adaptation of brain-machine interface decoders Front. Neurosci. 6 16
19. Sutton R S and Barto A G 1998 Reinforcement Learning: An Introduction (Cambridge, MA: MIT Press)
20. DiGiovanna J et al 2009 Coadaptive brain-machine interface via reinforcement learning IEEE Trans. Biomed. Eng. 56 54-64
21. Mahmoudi B and Sanchez J C 2011 A symbiotic brain-machine interface through value-based decision making PLoS One 6 e14760
22. Joel D, Niv Y and Ruppin E 2002 Actor-critic models of the basal ganglia: new anatomical and computational perspectives Neural Netw. 15 535-47 (Pubitemid 34947463)
23. Khamassi M et al 2005 Actor-critic models of reinforcement learning in the basal ganglia: from natural to artificial rats Adapt. Behav. 13 131-48 (Pubitemid 40907085)
24. Houk J C, Davis J L and Beiser D G 1995 Models of Information Processing in the Basal Ganglia (Cambridge, MA: MIT Press)
25. Sanchez J C et al 2012 Dynamically repairing and replacing neural networks: using hybrid computational and biological tools Pulse IEEE 3 57-9
26. Samejima K and Doya K 2007 Multiple representations of belief states and action values in corticobasal ganglia loops Reward and Decision Making in Corticobasal Ganglia Networks (Oxford: Blackwell) pp 213-28 (Pubitemid 47092747)
27. Schultz W, Tremblay L and Hollerman J R 1998 Reward prediction in primate basal ganglia and frontal cortex Neuropharmacology 37 421-9 (Pubitemid 28306931)
28. Izawa E I, Aoki N and Matsushima T 2005 Neural correlates of the proximity and quantity of anticipated food rewards in the ventral striatum of domestic chicks Eur. J. Neurosci. 22 1502-12 (Pubitemid 41442421)
29. Kaelbling L P 1994 Associative reinforcement learning: a generate and test algorithm Mach. Learn. 15 299-319
30. Gullapalli V 1991 Associative reinforcement learning of real-valued functions in systems, man, and cybernetics IEEE Int. Conf. on Decision Aiding for Complex Systems, Conf. Proc. (1991,) pp 1453-8
31. Pennartz C M A 1997 Reinforcement learning by Hebbian synapses with adaptive thresholds Neuroscience 81 303-19 (Pubitemid 27381989)
32. Vasilaki E et al 2009 Spike-based reinforcement learning in continuous state and action space: when policy gradient methods fail PLoS Comput. Biol. 5 e1000586
33. Williams R J 1988 Toward a theory of reinforcement-learning connectionist systems Technical Report NU-CCS-88-3, Northeastern University, College of Computer Science
34. Sehnke F et al 2010 Parameter-exploring policy gradients Neural Netw. 23 551-9
35. Sutton R.S et al 2000 Policy gradient methods for reinforcement learning with function approximation Adv. Neural Inf. Process. Syst. 12 1057-63
36. Peters J and Schaal S 2008 Natural actor-critic Neurocomputing 71 1180-90
37. Williams R J 1992 Simple statistical gradient-following algorithms for connectionist reinforcement learning Mach. Learn. 8 229-56
38. Narendra K S and Thathachar M A L 1989 Learning Automata: An Introduction (Englewood Cliffs, NJ: Prentice-Hall) p 476
39. Barto A G and Anandan P 1985 Pattern-recognizing stochastic learning automata IEEE Trans. Syst. Man Cybern. SMC-15 360-75
40. Hassoun M H 1995 Fundamentals of Artificial Neural Networks (Cambridge, MA: MIT Press)
41. Lotte F et al 2007 A review of classification algorithms for EEG-based brain-computer interfaces J. Neural Eng. 4 R1-13 (Pubitemid 46603810)
42. Kaelbling L P, Littman M L and Moore A W 1996 Reinforcement learning: a survey J. Artif. Intell. Res. 4 237-85 (Pubitemid 126646155)
43. Adam S, Busoniu L and Babuska R 2012 Experience replay for real-time reinforcement learning control IEEE Trans. Syst. Man Cybern. C 42 201-12
44. Wawrzyński P 2009 Real-time reinforcement learning by sequential actor-critics and experience replay Neural Netw. 22 1484-97
45. Izhikevich E M 2003 Simple model of spiking neurons IEEE Trans. Neural Netw. 14 1569-72
46. Sanchez J C et al 2004 Ascertaining the importance of neurons to develop better brain-machine interfaces IEEE Trans. Biomed. Eng. 51 943-53 (Pubitemid 38661446)
47. Erdogmus D et al 2005 Linear-least-squares initialization of multilayer perceptrons through backpropagation of the desired response IEEE Trans. Neural Netw. 16 325-37 (Pubitemid 40390816)
48. Guenther F H et al 2009 A wireless brain-machine interface for real-time speech synthesis PLoS One 4 e8218
49. Xueliang H and Ghovanloo M 2009 Using unconstrained tongue motion as an alternative control mechanism for wheeled mobility IEEE Trans. Biomed. Eng. 56 1719-26
50. Schultz W 2000 Multiple reward signals in the brain Nature Rev. Neurosci. 1 199-207
51. MacDonald A W et al 2000 Dissociating the role of the dorsolateral prefrontal and anterior cingulate cortex in cognitive control Science 288 1835-8 (Pubitemid 30399023)
52. Ferrez P W and del R Millan J 2008 Error-related EEG potentials generated during simulated brain-computer interaction IEEE Trans. Biomed. Eng. 55 923-9 (Pubitemid 351301222)
53. Mogenson G J, Jones D L and Yim C Y 1980 From motivation to action: functional interface between the limbic system and the motor system Prog. Neurobiol. 14 69-97 (Pubitemid 10053405)
54. Prins N et al 2013 Feature extraction and unsupervised classification of neural population reward signals for reinforcement based BMI Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. (EMBC) (Osaka, Japan, 2013) at press
55. Mazzoni P, Andersen R and Jordan M 1991 A more biologically plausible learning rule for neural networks Proc. Natl Acad. Sci. USA 88 4433-7 (Pubitemid 21914554)
56. Pohlmeyer E A et al 2012 Brain-machine interface control of a robot arm using actor-critic reinforcement learning Annu. Int. Conf. Proc. IEEE Eng. Med. Biol. Soc. pp 4108-11
57. Thoroughman K A and Shadmehr R 2000 Learning of action through adaptive combination of motor primitives Nature 407 742-7
58. Hart C B and Giszter S F 2010 A neural basis for motor primitives in the spinal cord J. Neurosci. 30 1322-36
59. Ganguly K and Carmena J M 2009 Emergence of a stable cortical map for neuroprosthetic control PLoS Biol. 7 e1000153
60. Ribas-Fernandes J J F et al 2011 A neural signature of hierarchical reinforcement learning Neuron 71 370-9
61. Barto A G and Mahadevan S 2003 Recent advances in hierarchical reinforcement learning Discrete Event Dyn. Syst. 13 41-77