Neuromodulatory adaptive combination of correlation-based learning in cerebellum and reward-based learning in basal ganglia for goal-directed behavior control

Frontiers in Neural Circuits

Volumn 8, Issue October, 2014, Pages

  Dasgupta, Sakyasingha ^a,b   Wörgötter, Florentin ^a,b   Manoonpong, Poramate ^b,c  

^a UNIVERSITY OF GÖTTINGEN   (Germany)

^b BERNSTEIN CENTER FOR COMPUTATIONAL NEUROSCIENCE   (Germany)

^c UNIVERSITY OF SOUTHERN DENMARK   (Denmark)

Author keywords

Basal ganglia; Cerebellum; Classical conditioning; Correlation learning; Decision making; Neuromodulation; Operant conditioning; Recurrent neural networks

Indexed keywords

ACTOR CRITIC RL NETWORK; ARTICLE; BASAL GANGLION; BEHAVIOR CONTROL; CEREBELLUM; CONTROLLED STUDY; CORRELATION BASED LEARNING; DECISION MAKING; FORAGING BEHAVIOR; INPUT CORRELATION LEARNING; LEARNING; NERVE CELL NETWORK; NERVE CELL PLASTICITY; NEUROMODULATION; POSITIVE FEEDBACK; REWARD BASED LEARNING; REWARD MODULATED HETEROSYNAPTIC PLASTICITY; SENSORY FEEDBACK; TASK PERFORMANCE; ALGORITHM; ANIMAL; BIOLOGICAL MODEL; COMPUTER SIMULATION; MOTIVATION; PHYSIOLOGY; REWARD; STATISTICS;

ALGORITHMS; ANIMALS; BASAL GANGLIA; CEREBELLUM; COMPUTER SIMULATION; GOALS; LEARNING; MODELS, NEUROLOGICAL; REWARD; STATISTICS AS TOPIC;

EID: 84908366806     PISSN: 16625110     EISSN: None     Source Type: Journal    
DOI: 10.3389/fncir.2014.00126     Document Type: Article

References (113)

1. Akkal, D., Dum, R. P., and Strick, P. L. (2007). Supplementary motor area and presupplementary motor area: targets of basal ganglia and cerebellar output. J. Neurosci. 27, 10659-10673. doi: 10.1523/JNEUR0SCI.3134-07.2007
2. Allen, G., and Tsukahara, N. (1974). Cerebrocerebellar communication systems. Physiol. Rev. 54, 957-1006
3. Anderson, M. E., and Turner, R. S. (1991). Activity of neurons in cerebellar-receiving and pallidal-receiving areas of the thalamus of the behaving monkey. J. Neurophysiol. 66, 879-893
4. Bailey, C. H., Giustetto, M., Huang, Y.-Y., Hawkins, R. D., and Kandel, E. R. (2000). Is heterosynaptic modulation essential for stabilizing hebbian plasiticity and memory. Nat. Rev. Neurosci. 1, 11-20. doi: 10.1038/35036191
5. Barnard, C. J. (2004). Animal Behaviour: Mechanism, Development, Function and Evolution. Essex: Pearson Education.
6. Baxter, D. A., and Byrne, J. H. (2006). Feeding behavior of aplysia: a model system for comparing cellular mechanisms of classical and operant conditioning. Learn. Mem. 13, 669-680. doi: 10.1101/lm.339206
7. Bernacchia, A., Seo, H., Lee, D., and Wang, X.-J. (2011). A reservoir of time con­stants for memory traces in cortical neurons. Nat. Neurosci. 14, 366-372. doi: 10.1038/nn.2752
8. Boedecker, J., Lampe, T., and Riedmiller, M. (2013). Modeling effects of intrin­sic and extrinsic rewards on the competition between striatal learning systems. Front. Psychol. 4:739. doi: 10.3389/fpsyg.2013.00739
9. Bosch-Bouju, C., Hyland, B. I., and Parr-Brownlie, L. C. (2013). Motor thalamus integration of cortical, cerebellar and basal ganglia information: implications for normal and parkinsonian conditions. Front. Comput. Neurosci. 7:163. doi: 10.3389/fncom.2013.00163
10. Bostan, A. C., Dum, R. P., and Strick, P. L. (2010). The basal ganglia commu­nicate with the cerebellum. Proc. Natl. Acad. Sci. U.S.A. 107, 8452-8456. doi: 10.1073/pnas.1000496107
11. Brembs, B., Baxter, D. A., and Byrne, J. H. (2004). Extending in vitro conditioning in aplysia to analyze operant and classical processes in the same preparation. Learn. Mem. 11, 412-420. doi: 10.1101/lm.74404
12. Brembs, B., and Heisenberg, M. (2000). The operant and the classical in condi­tioned orientation of drosophila melanogaster at the flight simulator. Learn. Mem. 7, 104-115. doi: 10.1101/lm.7.2.104
13. Brembs, B., Lorenzetti, F. D., Reyes, F. D., Baxter, D. A., and Byrne, J.H. (2002). 0perant reward learning in aplysia: neuronal correlates and mechanisms. Science 296, 1706-1709. doi: 10.1126/science.1069434
14. Burguiere, E., Arabo, A., Jarlier, F., Zeeuw, C. I. D., and Rondi-Reig, L. (2010). Role of the cerebellar cortex in conditioned goal-directed behavior. J. Neurosci. 30, 13265-13271. doi: 10.1523/JNEUR0SCI.2190-10.2010
15. Chistiakova, M., and Volgushev, M. (2009). Heterosynaptic plasticity in the neocortex. Exp. Brain Res. 199, 377-390. doi: 10.1007/s00221-009-1859-5
16. Christian, K. M., and Thompson, R. F. (2003). Neural substrates of eye-blink conditioning: acquisition and retention. Learn. Mem. 10, 427-455. doi: 10.1101/lm.59603
17. Clark, R. E., and Squire, L. R. (1998). Classical conditioning and brain systems: the role of awareness. Science 280, 77-81. doi: 10.1126/science.280.5360.77
18. Cleland, G. G., and Davey, G. C. (1983). Autoshaping in the rat: The effects of localizable visual and auditory signals for food. J. Exp. Anal. Behav. 40, 47-56. doi: 10.1901/jeab.1983.40-47
19. Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B., and Uchida, N. (2012). Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 482, 85-88. doi: 10.1038/nature10754
20. Dasgupta, S., Manoonpong, P., and Wôrgôtter, F. (2014). Reservoir of neurons with adaptive time constants: a hybrid model for robust motor-sensory tem­poral processing. BMC Neurosci. 15(Suppl. 1):P9. doi: 10.1186/1471-2202-15-S1-P9
21. Dasgupta, S., Wôrgôtter, F., and Manoonpong, P. (2013a). Information dynamics based self-adaptive reservoir for delay temporal memory tasks. Evol. Syst. 4, 235-249. doi: 10.1007/s12530-013-9080-y
22. Dasgupta, S., Wôrgôtter, F., Morimoto, J., and Manoonpong, P. (2013b). “Neural combinatorial learning of goal-directed behavior with reservoir critic and reward modulated hebbian plasticity,” in Systems, Man, and Cybernetics (SMC), 2013, IEEE International Conference on (Manchester, UK), 993-1000
23. Dayan, P., and Balleine, B. W. (2002). Reward, motivation, and reinforcement learning. Neuron 36, 285-298. doi: 10.1016/S0896-6273(02)00963-7
24. de Wit, S., Barker, R. A., Dickinson, A. D., and Cools, R. (2011). Habitual ver­sus goal-directed action control in parkinson disease. J. Cogn. Neurosci. 23, 1218-1229. doi: 10.1162/jocn.2010.21514
25. Desiraju, T., and Purpura, D. (1969). Synaptic convergence of cerebellar and lenticular projections to thalamus. Brain Res. 15, 544-547. doi: 10.1016/0006-8993(69)90180-2
26. Doya, K. (1999). What are the computations of the cerebellum, the basal gan­glia and the cerebral cortex? Neural Netw. 12, 961-974. doi: 10.1016/S0893-6080(99)00046-5
27. Doya, K. (2000a). Complementaryroles ofbasal ganglia andcerebellum in learning and motor control. Curr. Opin. Neurobiol. 10, 732-739. doi: 10.1016/S0959-4388(00)00153-7
28. Doya, K. (2000b). Reinforcement learning in continuous time and space. Neural Comput. 12, 219-245. doi: 10.1162/089976600300015961
29. Dreher, J.-C., and Grafman, J. (2002). The roles of the cerebellum and basal ganglia in timing and error prediction. Eur. J. Neurosci. 16, 1609-1619. doi: 10.1046/j.1460-9568.2002.02212.x
30. Freeman, J. H., and Steinmetz, A. B. (2011). Neural circuitry and plasticity mech­anisms underlying delay eyeblink conditioning. Learn. Mem. 18, 666-677. doi: 10.1101/lm.2023011
31. Fremaux, N., Sprekeler, H., and Gerstner, W. (2013). Reinforcement learning using a continuous time actor-critic framework with spiking neurons. PLoS Comput. Biol. 9:e1003024. doi: 10.1371/journal.pcbi.1003024
32. García-Cabezas, M. Á., Rico, B., Sánchez-González, M. Á., and Cavada, C. (2007). Distribution of the dopamine innervation in the macaque and human thalamus. Neuroimage 34, 965-984. doi: 10.1016/j.neuroimage.2006.07.032
33. Gurney, K., Prescott, T. J., and Redgrave, P. (2001). A computational model of action selection in the basal ganglia. i. a new functional anatomy. Biol. Cybern. 84, 401-410. doi: 10.1007/PL00007984
34. Gurney, K., Prescott, T. J., Wickens, J. R., and Redgrave, P. (2004). Computational models of the basal ganglia: from robots to membranes. Trends Neurosci. 27, 453-459. doi: 10.1016/j.tins.2004.06.003
35. Haber, S. N., and Calzavara, R. (2009). The cortico-basal ganglia integra­tive network: the role of the thalamus. Brain Res. Bull. 78, 69-74. doi: 10.1016/j.brainresbull.2008.09.013
36. Haykin, S. S. (2002). Adaptive filter theory. Upper Saddle River, NJ: Prentice Hall.
37. Herreros, I., and Verschure, P. F. (2013). Nucleo-olivary inhibition balances the interaction between the reactive and adaptive layers in motor control. Neural Netw. 47, 64-71. doi: 10.1016/j.neunet.2013.01.026
38. Hinaut, X., and Dominey, P. F. (2013). Real-time parallel processing of gram­matical structure in the fronto-striatal system: a recurrent network simulation study using reservoir computing. PLoS ONE 8:e52946. doi: 10.1371/jour-nal.pone.0052946
39. Hoerzer, G. M., Legenstein, R., and Maass, W. (2012). Emergence of complex com­putational structures from chaotic neural networks through reward-modulated hebbian learning. Cereb. Cortex 24, 677-690. doi: 10.1093/cercor/bhs348
40. Hofstoetter, C., Mintz, M., and Verschure, P. F. (2002). The cerebellum in action: a simulation and robotics study. Eur. J. Neurosci. 16, 1361-1376. doi: 10.1046/j.1460-9568.2002.02182.x
41. Hoshi, E., Tremblay, L., Féger, J., Carras, P. L., and Strick, P. L. (2005). The cere­bellum communicates with the basal ganglia. Nat. Neurosci. 8, 1491-1493. doi: 10.1038/nn1544
42. Hosp, J. A., Pekanovic, A., Rioult-Pedotti, M. S., and Luft, A. R. (2011). Dopaminergic projections from midbrain to primary motor cor­tex mediate motor skill learning. J. Neurosci. 31, 2481-2487. doi: 10.1523/JNEUR0SCI.5411-10.2011
43. Houk, J., Bastianen, C., Fansler, D., Fishbach, A., Fraser, D., Reber, P., et al. (2007). Action selection and refinement in subcortical loops through basal gan­glia and cerebellum. Philos. Trans. R. Soc. B Biol. Sci. 362, 1573-1583. doi: 10.1098/rstb.2007.2063
44. Houk, J. C., Adams, J. L., and Barto, A. G. (1995). “A model of how the basal gan­glia generate and use neural signals that predict reinforcement," in Models of Information Processing in the Basal Ganglia, eds J. C. Houk, J. L. Davis, and D. G. Beiser (Cambridge, MA: The MIT Press), 249-270
45. Humphries, M. D., Stewart, R. D., and Gurney, K. N. (2006). A physiologically plau­sible model of action selection and oscillatory activity in the basal ganglia. J. Neurosci. 26, 12921-12942. doi: 10.1523/JNEUR0SCI.3486-06.2006
46. Ishikawa, M., Otaka, M., Huang, Y. H., Neumann, P. A., Winters, B. D., Grace, A. A., et al. (2013). Dopamine triggers heterosynaptic plasticity. J. Neurosci. 33, 6759-6765. doi: 10.1523/JNEUR0SCI.4694-12.2013
47. Jaeger, H., and Haas, H. (2004). Harnessing nonlinearity: Predicting chaotic sys­tems and saving energy in wireless communication. Science 304, 78-80. doi: 10.1126/science.1091277
48. Joel, D., Niv, Y., and Ruppin, E. (2002). Actor-critic models of the basal ganglia: new anatomical and computational perspectives. NeuralNetw. 15, 535-547. doi: 10.1016/S0893-6080(02)00047-3
49. Joel, D., and Weiner, I. (2000). The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum. Neuroscience 96,451-474. doi: 10.1016/S0306-4522(99)00575-8
50. Jones, E. G., Steriade, M., and McCormick, D. (1985). The thalamus. New York, NY: Plenum Press. doi: 10.1007/978-1-4615-1749-8
51. Kawato, M. (1999). Internal models for motor control and trajectory planning. Curr. Opin. Neurobiol. 9, 718-727. doi: 10.1016/S0959-4388(99)00028-8
52. Kawato, M., Kuroda, S., and Schweighofer, N. (2011). Cerebellar supervised learn­ing revisited: biophysical modeling and degrees-of-freedom control. Curr. Opin. Neurobiol. 21, 791-800. doi: 10.1016/j.conb.2011.05.014
53. Kim, J. J., and Thompson, R. E. (1997). Cerebellar circuits and synaptic mecha­nisms involved in classical eyeblink conditioning. Trends Neurosci. 20, 177-181. doi: 10.1016/S0166-2236(96)10081-3
54. Kitazawa, S., Kimura, T., and Yin, P.-B. (1998). Cerebellar complex spikes encode both destinations and errors in arm movements. Nature 392, 494-497. doi: 10.1038/33141
55. Klopf, A. H. (1988). A neuronal model of classical conditioning. Psychobiology 16, 85-125
56. Knudsen, E. (1994). Supervised learning in the brain. J. Neurosci. 14, 3985-3997
57. Kolodziejski, C., Porr, B., and Worgotter, F. (2008). Mathematical properties of neu­ronal td-rules and differential hebbian learning: a comparison. Biol. Cybern. 98, 259-272. doi: 10.1007/s00422-007-0209-6
58. Koprinkova-Hristova, P., Oubbati, M., and Palm, G. (2010). “Adaptive critic design with echo state network," in Systems Man and Cybernetics (SMC), 2010, IEEE International Conference on (Istanbul), 1010-1015
59. Kreitzer, A. C., and Malenka, R. C. (2008). Striatal plasticity and basal ganglia circuit function. Neuron 60, 543-554. doi: 10.1016/j.neuron.2008.11.005
60. Krupa, D. J., Thompson, J. K., and Thompson, R. F. (1993). Localization of a memory trace in the mammalian brain. Science 260, 989-991. doi: 10.1126/sci-ence.8493536
61. Kuramoto, E., Furuta, T., Nakamura, K. C., Unzai, T., Hioki, H., and Kaneko, T. (2009). Two types of thalamocortical projections from the motor thalamic nuclei of the rat: a single neuron-tracing study using viral vectors. Cereb. Cortex 19, 2065-2077. doi: 10.1093/cercor/bhn231
62. Lazar, A., Pipa, G., and Triesch, J. (2007). Fading memory and time series predic­tion in recurrent networks with different forms of plasticity. Neural Netw. 20, 312-322. doi: 10.1016/j.neunet.2007.04.020
63. Legenstein, R., Pecevski, D., and Maass, W. (2008). A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback. PLoS Comput. Biol. 4:e1000180. doi: 10.1371/journal.pcbi.1000180
64. Lisberger, S., andThach, T. (2013). “The cerebellum," in Principles of Neural Science, eds E. R. Kandel, J. H. Schwartz, T. M. Jessel, S. A. Siegelbaum, and A. J. Hudspeth (New York, NY: McGraw-Hill), 960-981
65. Lovibond, P. F. (1983). Facilitation of instrumental behavior by apavlovian appeti­tive conditioned stimulus. J. Exp. Psychol. Anim. Behav. Process. 9, 225-247. doi: 10.1037/0097-7403.9.3.225
66. Maass, W., Natschlaeger, T., and Markram, H. (2002). Real-time computing with­out stable states: a new framework for neural computation based on perturba­tions. Neural Comput. 14, 2531-2560. doi: 10.1162/089976602760407955
67. Manoonpong, P., Geng, T., Kulvicius, T., Porr, B., and Worgotter, F. (2007). Adaptive, fast walking in a biped robot under neuronal control and learning. PLoS Comput. Biol. 3:e134. doi: 10.1371/journal.pcbi.0030134
68. Manoonpong, P., Kolodziejski, C., Worgotter, F., and Morimoto, J. (2013). Combining correlation-based and reward-based learning in neural con­trol for policy improvement. Adv. Comp. Syst. 16, 1350015-1350052. doi: 10.1142/S021952591350015X
69. McFarland, N. R., and Haber, S. N. (2002). Thalamic relay nuclei of the basal ganglia form both reciprocal and nonreciprocal cortical connections, linking multiple frontal cortical areas. J. Neurosci. 22, 8117-8132
70. Mehler, W. R. (1971). Idea of a new anatomy of the thalamus. J. Psychiatr. Res. 8, 203-217. doi: 10.1016/0022-3956(71)90019-7
71. Meyer, P. J., Cogan, E. S., and Robinson, T. E. (2014). The form of a conditioned stimulus can influence the degree to which it acquires incentive motivational properties. PLoS ONE 9:e98163. doi: 10.1371/journal.pone.0098163
72. Middleton, F. A., and Strick, P. L. (1994). Anatomical evidence for cerebellar and basal ganglia involvement in higher cognitive function. Science 266, 458-461. doi: 10.1126/science.7939688
73. Morimoto, J., and Doya, K. (1998). “Reinforcement learning of dynamic motor sequence: Learning to stand up," in Intelligent Robots and Systems,1998, Proceedings., 1998 IEEE/RSJ International Conference on (Victoria, BC), 1721-1726
74. Morimoto, J., andDoya, K. (2001). Acquisition of stand-up behavior by areal robot using hierarchical reinforcement learning. Robot. Auton. Syst. 36, 37-51. doi: 10.1016/S0921-8890(01)00113-0
75. Neychev, V. K., Fan, X., Mitev, V., Hess, E. J., and Jinnah, H. (2008). The basal ganglia and cerebellum interact in the expression of dystonic movement. Brain 131, 2499-2509. doi: 10.1093/brain/awn168
76. Ni, Z., Gunraj, C., Kailey, P., Cash, R. F., and Chen, R. (2014). Heterosynaptic mod­ulation of motor cortical plasticity in human. J. Neurosci. 34, 7314-7321. doi: 10.1523/JNEUR0SCI.4714-13.2014
77. Pavlov, I. P. (1927). Conditioned Reflexes: An Investigation of the Physiological Activity of the Cerebral Cortex. London: Oxford University Press, Humphrey Milford.
78. Percheron, G., Francois, C., Talbi, B., Yelnik, J., and Fenelon, G. (1996). The primate motor thalamus. Brain Res. Rev. 22, 93-181. doi: 10.1016/0165-0173(96)00003-3
79. Pierce, W. D., and Cheney, C. D. (2013). Behavior Analysis and Learning. New York, NY: Psychology Press.
80. Porr, B., and Wôrgôtter, F. (2006). Strongly improved stability and faster conver­gence of temporal sequence learning by utilising input correlations only. Neural Comput. 18, 1380-1412. doi: 10.1162/neco.2006.18.6.1380
81. Prescott, T. J., Gonzaelez, F. M., Gurney, K., Humphries, M. D., and Redgrave, P. (2006). A robot model of the basal ganglia: behavior and intrinsic processing. Neural Netw. 19, 31-61. doi: 10.1016/j.neunet.2005.06.049
82. Proville, R. D., Spolidoro, M., Guyon, N., Dugué, G. P., Selimi, F., Isope, P., et al. (2014). Cerebellum involvement in cortical sensorimotor circuits for the control of voluntary movements. Nat. Neurosci. 17, 1233-1239. doi: 10.1038/nn.3773
83. Puig, M. V., and Mille, E. K. (2012). The role of prefrontal dopamine d1 recep­tors in the neural mechanisms of associative learning. Neuron 74, 874-886. doi: 10.1016/j.neuron.2012.04.018
84. Rajan, K., Abbott, L., and Sompolinsky, H. (2010). Stimulus-dependent sup­pression of chaos in recurrent neural networks. Phys. Rev. E 82:011903. doi: 10.1103/PhysRevE.82.011903
85. Redgrave, P., Rodriguez, M., Smith, Y., Rodriguez-Oroz, M. C., Lehericy, S., Bergman, H., et al. (2010). Goal-directed and habitual control in the basal gan­glia: implications for parkinson’s disease. Nat. Rev. Neurosci. 11, 760-772. doi: 10.1038/nrn2915
86. Rescorla, R. A., and Solomon, R. L. (1967). Two-process learning theory: relation­ships between pavlovian conditioning and instrumental learning. Psychol. Rev. 74, 151-182. doi: 10.1037/h0024475
87. Sakai, S. T., Stepniewska, I., Qi, H. X., and Kaas, J. H. (2000). Pallidal and cere­bellar afferents to pre-supplementary motor area thalamocortical neurons in the owl monkey: a multiple labeling study. J. Comp. Neurol. 417, 164-180. doi: 10.1002/(SICI)1096-9861(20000207)417:2<164::AID-CNE3>3.0.C0;2-6
88. Salmon, D. P., and Butters, N. (1995). Neurobiology of skill and habit learning. Curr. Opin. Neurobiol. 5, 184-190. doi: 10.1016/0959-4388(95)80025-5
89. Schultz, W., and Dickinson, A. (2000). Neuronal coding of prediction errors. Ann.Rev. Neurosci. 23, 473-500. doi: 10.1146/annurev.neuro.23.1.473
90. Shettleworth, S. J. (2009). Cognition, Evolution, and Behavior. New York, NY: 0xford University Press.
91. Skinner, B. F. (1938). The Behavior of Organisms: An Experimental Analysis. New York, NY: Appleton-Century.
92. Soltoggio, A., Lemme, A., Reinhart, F., and Steil, J. J. (2013). Rare neural correla­tions implement robotic conditioning with delayed rewards and disturbances. Front Neurorobot. 7:6. doi: 10.3389/fnbot.2013.00006
93. Sompolinsky, H., Crisanti, A., and Sommers, H. (1988). Chaos in random neural networks. Phys. Rev. Lett. 61, 259-262. doi: 10.1103/PhysRevLett.61.259
94. Staddon, J. E. (1983). Adaptive Behaviour and Learning. Cambridge, UK: CUP Archive.
95. Stepniewska, I., Preuss, T. M., and Kaas, J. H. (1994). Thalamic connections of the primary motor cortex (m1) of owl monkeys. J. Comp. Neurol. 349, 558-582. doi: 10.1002/cne.903490405
96. Sugrue, L. P., Corrado, G. S., and Newsome, W. T. (2004). Matching behavior and the representation of value in the parietal cortex. Science 304, 1782-1787. doi: 10.1126/science.1094765
97. Sul, J. H., Jo, S., Lee, D., and Jung, M. W. (2011). Role of rodent secondary motor cortex in value-based action selection. Nat. Neurosci. 14, 1202-1208. doi: 10.1038/nn.2881
98. Suri, R. E., and Schultz, W. (2001). Temporal difference model repro­duces anticipatory neural activity. Neural Comput. 13, 841-862. doi: 10.1162/089976601300014376
99. Sussillo, D., and Abbott, L. F. (2009). Generating coherent patterns of activity from chaotic neural networks. Neuron 63, 544-557. doi: 10.1016/j.neuron.2009.07.018
100. Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Mach. Learn. 3, 9-44. doi: 10.1007/BF00115009
101. Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
102. Takikawa, Y., Kawagoe, R., and Hikosaka, O. (2004). A possible role of midbrain dopamine neurons in short-and long-term adaptation of saccades to position-reward mapping. J. Neurophysiol. 92, 2520-2529. doi: 10.1152/jn.00238.2004
103. Thompson, R., and Steinmetz, J. (2009). The role of the cerebellum in classical conditioning of discrete behavioral responses. Neuroscience 162, 732-755. doi: 10.1016/j.neuroscience.2009.01.041
104. Triesch, J. (2005). “A gradient rule for the plasticity of a neurons intrinsic excitabil­ity,” in Artificial Neural Networks: Biological Inspirations-ICANN 2005, eds W. Duch, J. Kacprzyk, E. Oja, and S. Zadrozny (Warsaw: Springer), 65-70
105. Varela, C. (2014). Thalamic neuromodulation and its implications for executive networks. Front. Neural Circuits 8:69. doi: 10.3389/fncir.2014.00069
106. Verschure, P. F., and Mintz, M. (2001). A real-time model of the cerebel­lar circuitry underlying classical conditioning: a combined simulation and robotics study. Neurocomputing 38, 1019-1024. doi: 10.1016/S0925-2312(01) 00377-0
107. Vitureira, N., Letellier, M., and Goda, Y. (2012). Homeostatic synaptic plasticity: from single synapses to neural circuits. Curr. Opin. Neurobiol. 22, 516-521. doi: 10.1016/j.conb.2011.09.006
108. Williams, D. R., and Williams, H. (1969). Auto-maintenance in the pigeon: Sustained pecking despite contingent non-reinforcement. J. Exp. Anal. Behav. 12, 511-520. doi: 10.1901/jeab.1969.12-511
109. Winstanley, C. A., Baunez, C., Theobald, D. E., and Robbins, T. W. (2005). Lesions to the subthalamic nucleus decrease impulsive choice but impair autoshap­ing in rats: the importance of the basal ganglia in pavlovian conditioning and impulse control. Eur. J. Neurosci. 21, 3107-3116. doi: 10.1111/j.1460-9568.2005.04143.x
110. Woodruff-Pak, D. S., and Disterhoft, J. F. (2008). Where is the trace in trace conditioning? Trends Neurosci. 31, 105-112. doi: 10.1016/j.tins.2007.11.006
111. Wôrgôtter, F., and Porr, B. (2005). Temporal sequence learning, prediction, and control: a review of different models and their relation to biological mecha­nisms. Neural Comput. 17,245-319. doi: 10.1162/0899766053011555
112. Yeo, C. H., and Hesslow, G. (1998). Cerebellum and conditioned reflexes. Trends Cogn. Sci. 2, 322-330. doi: 10.1016/S1364-6613(98)01219-4
113. Yin, H. H., and Knowlton, B. J. (2006). The role of the basal ganglia in habit formation. Nat. Rev. Neurosci. 7, 464-476. doi: 10.1038/nrn1919