Reward-based training of recurrent neural networks for cognitive and value-based tasks

eLife

Volumn 6, Issue , 2017, Pages

  Song, H Francis ^a   Yang, Guangyu R ^a   Wang, Xiao Jing ^a,b  

^a NEW YORK UNIVERSITY   (United States)

^b NEW YORK UNIVERSITY SHANGHAI   (China)

Author keywords

[No Author keywords available]

Indexed keywords

LEARNING; MODEL; NERVOUS SYSTEM; REWARD; ANIMAL; ANIMAL BEHAVIOR; BIOLOGICAL MODEL; COGNITION; CONDITIONING; DECISION MAKING;

ANIMALS; BEHAVIOR, ANIMAL; CHOICE BEHAVIOR; COGNITION; CONDITIONING (PSYCHOLOGY); DECISION MAKING; LEARNING; MODELS, NEUROLOGICAL; REWARD;

EID: 85012005486     PISSN: None     EISSN: 2050084X     Source Type: Journal    
DOI: 10.7554/eLife.21492     Document Type: Article

References (103)

1. Baird L, Moore A. 1999. Gradient descent for general reinforcement learning. Advances in Neural Information Processing Systems 11:968–974 https://papers.nips.cc/paper/1576-gradient-descent-for-general-reinforcement-learning.pdf.
2. Barak O, Sussillo D, Romo R, Tsodyks M, Abbott LF. 2013. From fixed points to chaos: three models of delayed discrimination. Progress in Neurobiology 103:214–222. doi: 10.1016/j.pneurobio.2013.02.002, PMID: 23438479.
3. Barto AG, Sutton RS, Anderson CW. 1983. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics SMC-13:834–846. doi: 10.1109/TSMC.1983.6313077.
4. Baxter J, Bartlett PL. 2001. Infinite-horizon policy-gradient estimation. The Journal of Artificial Intelligence Research 15:319–350. doi: 10.1613/jair.806.
5. Bayer HM, Glimcher PW. 2005. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:129–141. doi: 10.1016/j.neuron.2005.05.020, PMID: 15996553.
6. Bengio Y, Simard P, Frasconi P. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5:157–166. doi: 10.1109/72.279181, PMID: 18267787.
7. Bernacchia A, Seo H, Lee D, Wang XJ. 2011. A reservoir of time constants for memory traces in cortical neurons. Nature Neuroscience 14:366–372. doi: 10.1038/nn.2752, PMID: 21317906.
8. Brea J, Gerstner W. 2016. Does computational neuroscience need new synaptic learning paradigms? Current Opinion in Behavioral Sciences 11:61–66. doi: 10.1016/j.cobeha.2016.05.012.
9. Brosch T, Neumann H, Roelfsema PR. 2015. Reinforcement learning of linking and tracing contours in recurrent neural networks.. PLoS Computational Biology 11:e1004489. doi: 10.1371/journal.pcbi.1004489.
10. Cadieu CF, Hong H, Yamins DLK, Pinto N, Ardila D, Solomon EA, Majaj NJ, DiCarlo JJ. 2014. Deep neural networks rival the representation of primate IT cortex for core visual object recognition. PLoS Computational Biology 10:e1003963. doi: 10.1371/journal.pcbi.1003963.
11. Carnevale F, de Lafuente V, Romo R, Barak O, Parga N. 2015. Dynamic control of response criterion in premotor cortex during perceptual detection under temporal uncertainty. Neuron 86:1067–1077. doi: 10.1016/j.neuron.2015.04.014, PMID: 25959731.
12. Cho K, van Merrienboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv. http://arxiv.org/abs/1406.1078.
13. Chung J, Gulcehre C, Cho K, Bengio Y. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv. http://arxiv.org/abs/1412.3555.
14. Dayan P, Balleine BW. 2002. Reward, motivation, and reinforcement learning. Neuron 36:285–298. doi: 10.1016/S0896-6273(02)00963-7, PMID: 12383782.
15. Dayan P, Daw ND. 2008. Decision theory, reinforcement learning, and the brain. Cognitive, Affective, & Behavioral Neuroscience 8:429–453. doi: 10.3758/CABN.8.4.429, PMID: 19033240.
16. Doya K. 2000. Reinforcement learning in continuous time and space. Neural Computation 12:219–245. doi: 10.1162/089976600300015961, PMID: 10636940.
17. Eccles JC, Fatt P, Koketsu K. 1954. Cholinergic and inhibitory synapses in a pathway from motor-axon collaterals to motoneurones. The Journal of Physiology 126:524–562. doi: 10.1113/jphysiol.1954.sp005226, PMID: 13222354.
18. Engel TA, Chaisangmongkon W, Freedman DJ, Wang XJ. 2015. Choice-correlated activity fluctuations underlie learning of neuronal category representation. Nature Communications 6:6454. doi: 10.1038/ncomms7454, PMID: 25759251.
19. Fiete IR, Seung HS. 2006. Gradient learning in spiking neural networks by dynamic perturbation of conductances. Physical Review Letters 97:048104. doi: 10.1103/PhysRevLett.97.048104, PMID: 16907616.
20. Fiete IR, Fee MS, Seung HS. 2007. Model of birdsong learning based on gradient estimation by dynamic perturbation of neural conductances. Journal of Neurophysiology 98:2038–2057. doi: 10.1152/jn.01311.2006, PMID: 17652414.
21. Frank MJ, Claus ED. 2006. Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychological Review 113:300–326. doi: 10.1037/0033-295X.113.2.300, PMID: 16637763.
22. Friedrich J, Lengyel M. 2016. Goal-Directed decision making with spiking neurons. Journal of Neuroscience 36: 1529–1546. doi: 10.1523/JNEUROSCI.2854-15.2016, PMID: 26843636.
23. Frémaux N, Sprekeler H, Gerstner W. 2010. Functional requirements for reward-modulated spike-timing-dependent plasticity. Journal of Neuroscience 30:13326–13337. doi: 10.1523/JNEUROSCI.6249-09.2010, PMID: 20926659.
24. Gao P, Ganguli S. 2015. On simplicity and complexity in the brave new world of large-scale neuroscience. Current Opinion in Neurobiology 32:148–155. doi: 10.1016/j.conb.2015.04.003, PMID: 25932978.
25. Gold JI, Shadlen MN. 2007. The neural basis of decision making. Annual Review of Neuroscience 30:535–574. doi: 10.1146/annurev.neuro.29.051605.113038, PMID: 17600525.
26. Graves A. 2013. Generating sequences with recurrent neural networks. arXiv. http://arxiv.org/abs/1308.0850.
27. Grondman I, Busoniu L, Lopes GAD, Babuska R. 2012. A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C 42:1291–1307. doi: 10.1109/TSMCC.2012.2218595.
28. Hennequin G, Vogels TP, Gerstner W. 2014. Optimal control of transient dynamics in balanced networks supports generation of complex movements. Neuron 82:1394–1406. doi: 10.1016/j.neuron.2014.04.045, PMID: 24945778.
29. Hikosaka O, Kim HF, Yasuda M, Yamamoto S. 2014. Basal ganglia circuits for reward value-guided behavior. Annual Review of Neuroscience 37:289–306. doi: 10.1146/annurev-neuro-071013-013924, PMID: 25032497.
30. Hoerzer GM, Legenstein R, Maass W. 2014. Emergence of complex computational structures from chaotic neural networks through reward-modulated hebbian learning. Cerebral Cortex 24:677–690. doi: 10.1093/cercor/bhs348, PMID: 23146969.
31. Hong H, Yamins DL, Majaj NJ, DiCarlo JJ. 2016. Explicit information for category-orthogonal object properties increases along the ventral stream. Nature Neuroscience 19:613–622. doi: 10.1038/nn.4247, PMID: 26900926.
32. Houk JC, Adams JL, Barto AG. 1995. A model of how the basal ganglia generates and uses neural signals that predict reinforcement. In: Houk J. C, Davis J. L, Beisberb D. G (Eds). Models of Information Processing in the Basal Ganglia. Cambridge, MA: MIT Press. p 249–274.
33. Izhikevich EM. 2007. Solving the distal reward problem through linkage of STDP and dopamine signaling. Cerebral Cortex 17:2443–2452. doi: 10.1093/cercor/bhl152, PMID: 17220510.
34. Jaderberg M, Czarnecki WM, Osindero S, Vinyals O, Graves A, Kavukcuoglu K. 2016. Decoupled neural interfaces using synthetic gradients. arXiv. http://arxiv.org/abs/1608.05343.
35. Joel D, Niv Y, Ruppin E. 2002. Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Networks 15:535–547. doi: 10.1016/S0893-6080(02)00047-3, PMID: 12371510.
36. Kaelbling LP, Littman ML, Cassandra AR. 1998. Planning and acting in partially observable stochastic domains. Artificial Intelligence 101:99–134. doi: 10.1016/S0004-3702(98)00023-X
37. Kepecs A, Uchida N, Zariwala HA, Mainen ZF. 2008. Neural correlates, computation and behavioural impact of decision confidence. Nature 455:227–231. doi: 10.1038/nature07200, PMID: 18690210.
38. Kiani R, Hanks TD, Shadlen MN. 2008. Bounded integration in parietal cortex underlies decisions even when viewing duration is dictated by the environment. Journal of Neuroscience 28:3017–3029. doi: 10.1523/JNEUROSCI.4761-07.2008, PMID: 18354005.
39. Kiani R, Shadlen MN. 2009. Representation of confidence associated with a decision by neurons in the parietal cortex. Science 324:759–764. doi: 10.1126/science.1169405, PMID: 19423820.
40. Kingma DP, Ba JL. 2015. Adam: A method for stochastic optimization. Int. Conf. Learn. Represent. arXiv. https://arxiv.org/abs/1412.6980.
41. Laje R, Buonomano DV. 2013. Robust timing and motor patterns by taming chaos in recurrent neural networks. Nature Neuroscience 16:925–933. doi: 10.1038/nn.3405, PMID: 23708144.
42. Lak A, Costa GM, Romberg E, Koulakov AA, Mainen ZF, Kepecs A. 2014. Orbitofrontal cortex is required for optimal waiting based on decision confidence. Neuron 84:190–201. doi: 10.1016/j.neuron.2014.08.039, PMID: 25242219.
43. Li J, Daw ND. 2011. Signals in human striatum are appropriate for policy update rather than value prediction. Journal of Neuroscience 31:5504–5511. doi: 10.1523/JNEUROSCI.6316-10.2011, PMID: 21471387.
44. Lillicrap TP, Cownden D, Tweed DB, Akerman CJ. 2016. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications 7:13276. doi: 10.1038/ncomms13276, PMID: 27 824044.
45. Machens CK, Romo R, Brody CD. 2010. Functional, but not anatomical, separation of “what” and “when” in prefrontal cortex. Journal of Neuroscience 30:350–360. doi: 10.1523/JNEUROSCI.3276-09.2010, PMID: 20053 916.
46. Maia TV. 2010. Two-factor theory, the actor-critic model, and conditioned avoidance. Learning & Behavior 38: 50–67. doi: 10.3758/LB.38.1.50, PMID: 20065349.
47. Mante V, Sussillo D, Shenoy KV, Newsome WT. 2013. Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature 503:78–84. doi: 10.1038/nature12742, PMID: 24201281.
48. Marblestone AH, Wayne G, Kording KP. 2016. Toward an integration of deep learning and neuroscience. Frontiers in Computational Neuroscience 10:94. doi: 10.3389/fncom.2016.00094, PMID: 27683554.
49. Martens J, Sutskever I. 2011. Learning recurrent neural networks with Hessian-free optimization. Proceedings of the 28th International Conference on Machine Learning, Washington, USA. http://www.icml-2011.org/papers/532_icmlpaper.pdf.
50. Mastrogiuseppe F, Ostojic S. 2016. Intrinsically-generated fluctuating activity in excitatory-inhibitory networks. arXiv. http://arxiv.org/abs/1605.04221.
51. Mazurek ME, Roitman JD, Ditterich J, Shadlen MN. 2003. A role for neural integrators in perceptual decision making. Cerebral Cortex 13:1257–1269. doi: 10.1093/cercor/bhg097, PMID: 14576217.
52. Miconi T. 2016. Biologically plausible learning in recurrent neural networks for flexible decision tasks. bioRxiv. https://doi.org/10.1101/057729.
53. Mnih V, Hess N, Graves A, Kavukcuoglu K. 2014. Recurrent models of visual attention. Advances in neural information processing systems. https://papers.nips.cc/paper/5542-recurrent-models-of-visual-attention.pdf.
54. Mnih V, Mirza M, Graves A, Harley T, Lillicrap TP, Silver D. 2016. Asynchronous methods for deep reinforcement learning. arXiv. http://arxiv.org/abs/1602.01783.
55. Niv Y, Langdon A. 2016. Reinforcement learning with Marr. Current Opinion in Behavioral Sciences 11:67–73. doi: 10.1016/j.cobeha.2016.04.005.
56. O’Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ. 2004. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452–454. doi: 10.1126/science.1094285, PMID: 15087550.
57. Padoa-Schioppa C, Assad JA. 2006. Neurons in the orbitofrontal cortex encode economic value. Nature 441: 223–226. doi: 10.1038/nature04676, PMID: 16633341.
58. Pascanu R, Gulcehre C, Cho K, Bengio Y. 2013a. How to construct deep recurrent neural networks. arXiv. http://arxiv.org/abs/1312.6026.
59. Pascanu R, Mikolov T, Bengio Y. 2013b. On the difficulty of training recurrent neural networks. Proceedings of the 30th International Conference on Machine Learning. http://jmlr.org/proceedings/papers/v28/pascanu13.pdf.
60. Peters J, Schaal S. 2008. Reinforcement learning of motor skills with policy gradients. Neural Networks 21:682–697. doi: 10.1016/j.neunet.2008.02.003, PMID: 18482830.
61. Rajan K, Harvey CD, Tank DW. 2015. Recurrent network models of sequence generation and memory. Neuron 90:128–142. doi: 10.1016/j.neuron.2016.02.009.
62. Ranzato M, Chopra S, Auli M, Zaremba W. 2016. Sequence level training with recurrent neural networks. arXiv. http://arxiv.org/abs/1511.06732.
63. Rao RP. 2010. Decision making under uncertainty: a neural model based on partially observable markov decision processes. Frontiers in Computational Neuroscience 4:146. doi: 10.3389/fncom.2010.00146, PMID: 21152255.
64. Raposo D, Sheppard JP, Schrater PR, Churchland AK. 2012. Multisensory decision-making in rats and humans. Journal of Neuroscience 32:3726–3735. doi: 10.1523/JNEUROSCI.4998-11.2012, PMID: 22423093.
65. Raposo D, Kaufman MT, Churchland AK. 2014. A category-free neural population supports evolving demands during decision-making. Nature Neuroscience 17:1784–1792. doi: 10.1038/nn.3865, PMID: 25383902.
66. Rigotti M, Ben Dayan Rubin D, Wang XJ, Fusi S. 2010. Internal representation of task rules by recurrent dynamics: the importance of the diversity of neural responses. Frontiers in Computational Neuroscience 4:24. doi: 10.3389/fncom.2010.00024, PMID: 21048899.
67. Rigotti M, Barak O, Warden MR, Wang XJ, Daw ND, Miller EK, Fusi S. 2013. The importance of mixed selectivity in complex cognitive tasks. Nature 497:585–590. doi: 10.1038/nature12160, PMID: 23685452.
68. Roitman JD, Shadlen MN. 2002. Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. Journal of Neuroscience 22:9475–9489. PMID: 12417672.
69. Romo R, Brody CD, Hernández A, Lemus L. 1999. Neuronal correlates of parametric working memory in the prefrontal cortex. Nature 399:470–473. doi: 10.1038/20939, PMID: 10365959.
70. Rumelhart DE, Hinton GE, Williams RJ. 1986. Learning internal representations by error propagation. In: Rumelhart DE, McClelland JL (Eds). Parallel Distributed Processing. Cambridge, MA: MIT Press. 1 p 318–362.
71. Scellier B, Bengio Y. 2016. Towards a biologically plausible backprop. arXiv. http://arxiv.org/abs/1602.05179.
72. Schoenbaum G, Takahashi Y, Liu TL, McDannald MA. 2011. Does the orbitofrontal cortex signal value? Annals of the New York Academy of Sciences 1239:87–99. doi: 10.1111/j.1749-6632.2011.06210.x, PMID: 22145878.
73. Schultz W, Dayan P, Montague PR. 1997. A neural substrate of prediction and reward. Science 275:1593–1599. doi: 10.1126/science.275.5306.1593, PMID: 9054347.
74. Schultz W, Tremblay L, Hollerman JR. 2000. Reward processing in primate orbitofrontal cortex and basal ganglia. Cerebral Cortex 10:272–283. doi: 10.1093/cercor/10.3.272, PMID: 10731222.
75. Seung HS. 2003. Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron 40:1063–1073. doi: 10.1016/S0896-6273(03)00761-X, PMID: 14687542.
76. Soltani A, Lee D, Wang XJ. 2006. Neural mechanism for stochastic behaviour during a competitive game. Neural Networks 19:1075–1090. doi: 10.1016/j.neunet.2006.05.044, PMID: 17015181.
77. Soltani A, Wang XJ. 2010. Synaptic computation underlying probabilistic inference. Nature Neuroscience 13: 112–119. doi: 10.1038/nn.2450, PMID: 20010823.
78. Song HF, Yang GR, Wang XJ. 2016. Training excitatory-inhibitory recurrent neural networks for cognitive tasks: A simple and flexible framework. PLoS Computational Biology 12:e1004792. doi: 10.1371/journal.pcbi.1004792, PMID: 26928718.
79. Stalnaker TA, Cooch NK, Schoenbaum G. 2015. What the orbitofrontal cortex does not do. Nature Neuroscience 18:620–627. doi: 10.1038/nn.3982, PMID: 25919962.
80. Sugrue LP, Corrado GS, Newsome WT. 2005. Choosing the greater of two goods: neural currencies for valuation and decision making. Nature Reviews Neuroscience 6:363–375. doi: 10.1038/nrn1666, PMID: 15832198.
81. Sussillo D, Abbott LF. 2009. Generating coherent patterns of activity from chaotic neural networks. Neuron 63: 544–557. doi: 10.1016/j.neuron.2009.07.018, PMID: 19709635.
82. Sussillo D, Barak O. 2013. Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Computation 25:626–649. doi: 10.1162/NECO_a_00409, PMID: 23272922.
83. Sussillo D. 2014. Neural circuits as computational dynamical systems. Current Opinion in Neurobiology 25:156–163. doi: 10.1016/j.conb.2014.01.008, PMID: 24509098.
84. Sussillo D, Churchland MM, Kaufman MT, Shenoy KV. 2015. A neural network that finds a naturalistic solution for the production of muscle activity. Nature Neuroscience 18:1025–1033. doi: 10.1038/nn.4042, PMID: 26075643.
85. Sutton RS, Barto AG. 1998. Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
86. Sutton RS, Mcallester D, Singh S, Mansour Y. 2000. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems 12:1057–1063 http://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf.
87. Takahashi Y, Schoenbaum G, Niv Y. 2008. Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model. Frontiers in Neuroscience 2:86–99. doi: 10.3389/neuro.01.014.2008, PMID: 18982111.
88. Takahashi YK, Roesch MR, Wilson RC, Toreson K, O’Donnell P, Niv Y, Schoenbaum G. 2011. Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nature Neuroscience 14:1590–1597. doi: 10.1038/nn.2957, PMID: 22037501.
89. The Theano Development Team. 2016. Theano: A python framework for fast computation of mathematical expressions. arXiv. http://arxiv.org/abs/1605.02688.
90. Todd MT, Niv Y, Cohen JD. 2008. Learning to use working memory in partially observable environments through dopaminergic reinforcement. Advances in Neural Information Processing Systems. http://papers.nips.cc/paper/3508-learning-to-use-working-memory-in-partially-observable-environments-through-dopaminergic-reinforcement.pdf.
91. Turner RS, Desmurget M. 2010. Basal ganglia contributions to motor control: a vigorous tutor. Current Opinion in Neurobiology 20:704–716. doi: 10.1016/j.conb.2010.08.022, PMID: 20850966.
92. Urbanczik R, Senn W. 2009. Reinforcement learning in populations of spiking neurons. Nature Neuroscience 12: 250–252. doi: 10.1038/nn.2264, PMID: 19219040.
93. Wallis JD. 2007. Orbitofrontal cortex and its contribution to decision-making. Annual Review of Neuroscience 30:31–56. doi: 10.1146/annurev.neuro.30.051606.094334, PMID: 17417936.
94. Wang XJ. 2002. Probabilistic decision making by slow reverberation in cortical circuits. Neuron 36:955–968. doi: 10.1016/S0896-6273(02)01092-9, PMID: 12467598.
95. Wang XJ. 2008. Decision making in recurrent neuronal circuits. Neuron 60:215–234. doi: 10.1016/j.neuron.2008.09.034, PMID: 18957215.
96. Wei Z, Wang XJ. 2015. Confidence estimation as a stochastic process in a neurodynamical system of decision making. Journal of Neurophysiology 114:99–113. doi: 10.1152/jn.00793.2014, PMID: 25948870.
97. Wierstra D, Forster A, Peters J, Schmidhuber J. 2009. Recurrent policy gradients. Logic Journal of IGPL 18:620–634. doi: 10.1093/jigpal/jzp049.
98. Williams RJ. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8:229–256. doi: 10.1007/BF00992696.
99. Wong KF, Wang XJ. 2006. A recurrent network mechanism of time integration in perceptual decisions. Journal of Neuroscience 26:1314–1328. doi: 10.1523/JNEUROSCI.3733-05.2006, PMID: 16436619.
100. Xu K, Ba JL, Kiros R, Cho K, Courville A, Salakhutdinov R, Zemel RS, Bengio Y. 2015. Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the 32 nd International Conference on Machine Learning. http://jmlr.org/proceedings/papers/v37/xuc15.pdf.
101. Yamins DL, Hong H, Cadieu CF, Solomon EA, Seibert D, DiCarlo JJ. 2014. Performance-optimized hierarchical models predict neural responses in higher visual cortex. PNAS 111:8619–8624. doi: 10.1073/pnas.1403112111, PMID: 24812127.
102. Zaremba W, Sutskever I. 2016. Reinforcement learning neural turing machines. arXiv. http://arxiv.org/abs/1505.00521.
103. Zipser D, Andersen RA. 1988. A back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons. Nature 331:679–684. doi: 10.1038/331679a0, PMID: 3344044