Habits, action sequences and reinforcement learning

European Journal of Neuroscience

Volumn 35, Issue 7, 2012, Pages 1036-1051

  Dezfouli, Amir ^a   Balleine, Bernard W ^a  

^a UNIVERSITY OF SYDNEY   (Australia)

Author keywords

Action sequence; Goal directed action; Habitual action; Reinforcement learning

Indexed keywords

ADAPTIVE BEHAVIOR; ARTICLE; BEHAVIOR; CORPUS STRIATUM; DECISION MAKING; DORSOLATERAL STRIATUM; GOAL DIRECTED ACTION; HABIT; HUMAN; INSTRUMENTAL CONDITIONING; LEARNING; NONHUMAN; PRIORITY JOURNAL; REACTION TIME; REINFORCEMENT; REINFORCEMENT LEARNING; REWARD; SENSORIMOTOR CORTEX; SENSORIMOTOR STRIATUM; STIMULUS RESPONSE; TASK PERFORMANCE;

ANIMALS; GOALS; HABITS; HUMANS; LEARNING; RANDOM ALLOCATION; REINFORCEMENT (PSYCHOLOGY);

EID: 84859341150     PISSN: 0953816X     EISSN: 14609568     Source Type: Journal    
DOI: 10.1111/j.1460-9568.2012.08050.x     Document Type: Article

References (116)

1. Adams, C.D. (1982) Variations in the sensitivity of instrumental responding to reinforcer devaluation. Q. J. Exp. Psychol. B, 34B, 77-98.
2. Alexander, G.E. & Crutcher, M.D. (1990) Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends Neurosci., 13, 266-271.
3. Astrom, K.J. & Murray, R.M. (2008) Feedback Systems: An Introduction for Scientists and Engineers. Princeton University Press, Princeton, NJ.
4. Badre, D. & Frank, M.J. (2012) Mechanisms of hierarchical reinforcement learning in cortico-striatal circuits 2: evidence from fMRI. Cereb. Cortex, 22, 527-536.
5. Bailey, K.R. & Mair, R.G. (2006) The role of striatum in initiation and execution of learned action sequences in rats. J. Neurosci., 26, 1016-1025.
6. Bailey, K.R. & Mair, R.G. (2007) Effects of frontal cortex lesions on action sequence learning in the rat. Eur. J. Neurosci., 25, 2905-2915.
7. Baird, L.C. (1993) Advantage Updating (No. WL--TR-93-1146). Wright Laboratory, Wright-Patterson Air Force Base Ohio.
8. Balleine, B.W. & O'Doherty, J.P. (2010) Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology, 35, 48-69.
9. Balleine, B.W., Delgado, M.R. & Hikosaka, O. (2007) The role of the dorsal striatum in reward and decision-making. J. Neurosci., 27, 8161-8165.
10. Balleine, B.W., Liljeholm, M. & Ostlund, S.B. (2009) The integrative function of the basal ganglia in instrumental conditioning. Behav. Brain Res., 199, 43-52.
11. Bapi, R.S. & Doya, K. (2001) Multiple forward model architecture for sequence processing. In Sun, R. & Giles, C.L. (Eds), Sequence learning: paradigms, algorithms, and applications. Springer Verlag, NY, pp. 309-320.
12. Barnes, T.D., Kubota, Y., Hu, D., Jin, D.Z. & Graybiel, A.M. (2005) Activity of striatal neurons reflects dynamic encoding and recoding of procedural memories. Nature, 437, 1158-1161.
13. Barto, A.G. & Mahadevan, S. (2003) Recent advances in hierarchical reinforcement learning. Discrete Event Dyn. S., 13, 41-77.
14. Berns, G.S. & Sejnowski, T.J. (1998) A computational model of how the basal ganglia produce sequences. J. Cogn. Neurosci., 10, 108-121.
15. Bissmarck, F., Nakahara, H., Doya, K. & Hikosaka, O. (2008) Combining modalities with different latencies for optimal motor control. J. Cogn. Neurosci., 20, 1966-1979.
16. Botvinick, M.M. (2008) Hierarchical models of behavior and prefrontal function. Trends Cogn. Sci., 12, 201-208.
17. Botvinick, M.M., Niv, Y. & Barto, A.G. (2009) Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition, 113, 262-280.
18. Buitrago, M.M., Ringer, T., Schulz, J.B., Dichgans, J. & Luft, A.R. (2004a) Characterization of motor skill and instrumental learning time scales in a skilled reaching task in rat. Behav. Brain Res., 155, 249-256.
19. Buitrago, M.M., Schulz, J.B., Dichgans, J. & Luft, A.R. (2004b) Short and long-term motor skill learning in an accelerated rotarod training paradigm. Neurobiol. Learn. Mem., 81, 211-216.
20. Chang, Q. & Gold, P.E. (2004) Inactivation of dorsolateral striatum impairs acquisition of response learning in cue-deficient, but not cue-available, conditions. Behav. Neurosci., 118, 383-388.
21. Costa, R.M., Cohen, D. & Nicolelis, M.A.L. (2004) Differential corticostriatal plasticity during fast and slow motor skill learning in mice. Curr. Biol., 14, 1124-1134.
22. Costa, R.M., Lin, S.-C., Sotnikova, T.D., Cyr, M., Gainetdinov, R.R., Caron, M.G. & Nicolelis, M.A.L. (2006) Rapid alterations in corticostriatal ensemble coordination during acute dopamine-dependent motor dysfunction. Neuron, 52, 359-369.
23. Daw, N.D. & Touretzky, D.S. (2000) Behavioral considerations suggest an average reward TD model of the dopamine system. Neurocomputing, 32-33, 679-684.
24. Daw, N.D. & Touretzky, D.S. (2002) Long-term reward prediction in TD models of the dopamine system. Neural Comput., 14, 2567-2583.
25. Daw, N.D., Niv, Y. & Dayan, P. (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci., 8, 1704-1711.
26. Dayan, P. & Balleine, B.W. (2002) Reward, motivation, and reinforcement learning. Neuron, 36, 285-298.
27. Dayan, P. & Kakade, S. (2001) Explaining away in weight space. In Leen, T.K., Dietterich, T.G. & Tresp, V. (Eds), Advances in Neural Information Processing Systems 13. MIT Press, Denver, CO, USA, pp. 451-457.
28. Dayan, P., Kakade, S. & Montague, P.R. (2000) Learning and selective attention. Nat. Neurosci., 3, 1218-1223.
29. Denny, M.R., Wells, R.H. & Maatsch, J.L. (1957) Resistance to extinction as a function of the discrimination habit established during fixed-ratio reinforcement. J. Exp. Psychol., 54, 451-456.
30. Desmurget, M. & Turner, R.S. (2010) Motor sequences and the basal ganglia: kinematics, not habits. J. Neurosci., 30, 7685-7690.
31. Devan, B.D. & White, N.M. (1999) Parallel information processing in the dorsal striatum: relation to hippocampal function. J. Neurosci., 19, 2789-2798.
32. Devan, B.D., McDonald, R.J. & White, N.M. (1999) Effects of medial and lateral caudate-putamen lesions on place- and cue-guided behaviors in the water maze: relation to thigmotaxis. Behav. Brain Res., 100, 5-14.
33. Dickinson, A. (1994) Instrumental conditioning. In Mackintosh, N.J. (Ed.) Animal Cognition and Learning. Academic Press, London, pp. 4-79.
34. Dickinson, A., Squire, S., Varga, Z. & Smith, J.W. (1998) Omission learning after instrumental pretraining. Q. J. Exp. Psychol. B, 51, 271-286.
35. Endress, A.D. & Wood, J. (2011) From movements to actions: two mechanisms for learning action sequences. Cogn. Psychol., 63, 141-171.
36. Faure, A., Haberland, U., Condé, F. & El Massioui, N. (2005) Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation. J. Neurosci., 25, 2771-2780.
37. Featherstone, R.E. & McDonald, R.J. (2004) Dorsal striatum and stimulus-response learning: lesions of the dorsolateral, but not dorsomedial, striatum impair acquisition of a stimulus-response-based instrumental discrimination task, while sparing conditioned place preference learning. Neuroscience, 124, 23-31.
38. Featherstone, R.E. & McDonald, R.J. (2005) Lesions of the dorsolateral striatum impair the acquisition of a simplified stimulus-response dependent conditional discrimination task. Neuroscience, 136, 387-395.
39. Frank, M.J. & O'Reilly, R.C. (2006) A mechanistic account of striatal dopamine function in human cognition: psychopharmacological studies with cabergoline and haloperidol. Behav. Neurosci., 120, 497-517.
40. Graybiel, A.M. (1998) The basal ganglia and chunking of action repertoires. Neurobiol. Learn. Mem., 70, 119-136.
41. Hansen, E.A., Barto, A.G. & Zilberstein, S. (1996) Reinforcement learning for mixed open-loop and closed loop control. In Mozer, M., Jordan, M.I. & Petsche, T. (Eds). Advances in Neural Information Processing Systems NIPS, Vol. 9. The MIT Press, Denver, CO, USA, pp. 1026-1032.
42. Haruno, M. & Kawato, M. (2006) Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning. Neural. Netw., 19, 1242-1254.
43. Hernandez, P.J., Schiltz, C.A. & Kelley, A.E. (2006) Dynamic shifts in corticostriatal expression patterns of the immediate early genes Homer 1a and Zif268 during early and late phases of instrumental training. Learn. Mem., 13, 599-608.
44. Hikosaka, O., Rand, M.K., Miyachi, S. & Miyashita, K. (1995) Learning of sequential movements in the monkey: process of learning and retention of memory. J. Neurophysiol., 74, 1652-1661.
45. Hull, C.L. (1943) Principles of Behavior. Appleton, New York, NY.
46. Iba, G.A. (1989) A heuristic approach to the discovery of macro-operators. Mach. Learn., 3, 285-317.
47. Jin, X. & Costa, R.M. (2010) Start/stop signals emerge in nigrostriatal circuits during sequence learning. Nature, 466, 457-462.
48. Jog, M.S., Kubota, Y., Connolly, C.I., Hillegaart, V. & Graybiel, A.M. (1999) Building neural representations of habits. Science, 286, 1745-1749.
49. Johnson, A., van der Meer, M.A.A. & Redish, A.D. (2007) Integrating hippocampus and striatum in decision-making. Curr. Opin. Neurobiol., 17, 692-697.
50. Keele, S.W., Ivry, R., Mayr, U., Hazeltine, E. & Heuer, H. (2003) The cognitive and neural architecture of sequence representation. Psychol. Rev., 110, 316-339.
51. Kelly, R.M. & Strick, P.L. (2004) Macro-architecture of basal ganglia loops with the cerebral cortex: use of rabies virus to reveal multisynaptic circuits. Prog. Brain Res., 143, 449-459.
52. Keramati, M., Dezfouli, A. & Piray, P. (2011) Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput. Biol., 7, e1002055.
53. Kolter, Z., Plagemann, C., Jackson, D.T., Ng, A. & Thrun, S. (2010) A probabilistic approach to mixed open-loop and closed-loop control, with application to extreme autonomous driving. Proc. of the IEEE Int.Conf.on Robotics & Automation (ICRA ). Anchorage, Alaska, USA.
54. Korf, R.E. (1985) Macro-operators: a weak method for learning. Artif. Intell., 26, 35-77.
55. Kubota, Y., Liu, J., Hu, D., DeCoteau, W.E., Eden, U.T., Smith, A.C. & Graybiel, A.M. (2009) Stable encoding of task structure coexists with flexible coding of task events in sensorimotor striatum. J. Neurophysiol., 102, 2142-2160.
56. Lashley, K.S. (1951) The problem of serial order in behavior. In Jeffress, L.A. (Ed.) Cerebral Mechanisms in Behavior. Wiley, NY, pp. 112-136.
57. Lehéricy, S., Benali, H., Van de Moortele, P.-F., Pélégrini-Issac, M., Waechter, T., Ugurbil, K. & Doyon, J. (2005) Distinct basal ganglia territories are engaged in early and advanced motor sequence learning. Proc. Nat. Acad. Sci. USA, 102, 12566-12571.
58. Matsumoto, N., Hanakawa, T., Maki, S., Graybiel, A.M. & Kimura, M. (1999) Nigrostriatal dopamine system in learning to perform sequential motor tasks in a predictive manner. J. Neurophysiol., 82, 978-998.
59. Mahadevan, S. (1996) Average reward reinforcement learning: foundations, algorithms, and empirical results. Mach. Learn., 22, 159-195.
60. Matsuzaka, Y., Picard, N. & Strick, P.L. (2007) Skill representation in the primary motor cortex after long-term practice. J. Neurophysiol., 97, 1819-1832.
61. Mcgovern, E.A. (2002) Autonomous Discovery of Temporal Abstractions From Interaction With An Environment. University of Massachusetts, Amherst.
62. Miyachi, S., Hikosaka, O., Miyashita, K., Kárádi, Z. & Rand, M.K. (1997) Differential roles of monkey striatum in learning of sequential hand movement. Exp. Brain Res., 115, 1-5.
63. Miyachi, S., Hikosaka, O. & Lu, X. (2002) Differential activation of monkey striatal neurons in the early and late stages of procedural learning. Exp. Brain Res., 146, 122-126.
64. Miyashita, K., Rand, M.K., Miyachi, S. & Hikosaka, O. (1996) Anticipatory saccades in sequential procedural learning in monkeys. J. Neurophysiol., 76, 1361-1366.
65. Montague, P.R., Dayan, P. & Sejnowski, T.J. (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci., 16, 1936-1947.
66. Moussa, R., Poucet, B., Amalric, M. & Sargolini, F. (2011) Behavior contributions of dorsal striatal subregions to spatial alternation behavior. Learn. Mem., 18, 444-451.
67. Mowrer, O.H. & Jones, H. (1945) Habit strength as a function of the pattern of reinforcement. J. Exp. Psychol., 35, 293-311.
68. Nakahara, H., Doya, K. & Hikosaka, O. (2001) Parallel cortico-basal ganglia mechanisms for acquisition and execution of visuomotor sequences - a computational approach. J. Cogn. Neurosci., 13, 626-647.
69. Neuringer, A. (2004) Reinforced variability in animals and people: implications for adaptive action. Am. Psychol., 59, 891-906.
70. Neuringer, A. & Jensen, G. (2010) Operant variability and voluntary action. Psychol. Rev., 117, 972-993.
71. Newell, K.M., Liu, Y.T. & Mayer-Kress, G. (2001) Time scales in motor learning and development. Psychol. Rev., 108, 57-82.
72. Nissen, M.J. & Bullemer, P. (1987) Attentional requirements of learning: performance measures evidence from. Cogn. Psychol., 19, 1-32.
73. Niv, Y., Daw, N.D., Joel, D. & Dayan, P. (2007) Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology, 191, 507-520.
74. O'Doherty, J.P., Dayan, P., Schultz, J., Deichmann, R., Friston, K. & Dolan, R.J. (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304, 452-454.
75. O'Reilly, R.C., Herd, S.A. & Pauli, W.M. (2010) Computational models of cognitive control. Curr. Opin. Neurobiol., 20, 257-261.
76. Ostlund, S.B., Winterbauer, N.E. & Balleine, B.W. (2009) Evidence of action sequence chunking in goal-directed instrumental conditioning and its dependence on the dorsomedial prefrontal cortex. J. Neurosci., 29, 8280-8287.
77. Overmann, S.R. & Denny, M.R. (1974) The free-operant partial reinforcement effect: a discrimination analysis. Learn. Motiv., 5, 248-257.
78. Packard, M.G. & McGaugh, J.L. (1996) Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neurobiol. Learn. Mem., 65, 65-72.
79. Platt, J.R. & Day, R.B. (1979) A hierarchical response-unit analysis of resistance to extinction following fixed-number and fixed-consecutive-number reinforcement. J. Exp. Psychol. Anim. Behav. Process., 5, 307-320.
80. Poldrack, R.A., Sabb, F.W., Foerde, K., Tom, S.M., Asarnow, R.F., Bookheimer, S.Y. & Knowlton, B.J. (2005) The neural correlates of motor skill automaticity. J. Neurosci., 25, 5356-5364.
81. Puterman, M.L. (1994) Markov Decision Processes. Wiley Interscience, New York.
82. Randløv, J. (1998) Learning macro-actions in reinforcement learning. In Kearns, M.J., Solla, S.A. & Cohn, D.A. (Eds). Advances in Neural Information Processing Systems 11. MIT Press, Denver, CO, USA, pp. 1045-1051.
83. Rangel, A., Camerer, C. & Montague, P.R. (2008) A framework for studying the neurobiology of value-based decision making. Nat. Rev. Neurosci., 9, 545-556.
84. Redish, A.D., Jensen, S. & Johnson, A. (2008) A unified framework for addiction: vulnerabilities in the decision process. Behav. Brain Sci., 31, 415-437; discussion 437-487.
85. Restle, F. (1957) Discrimination of cues in mazes: a resolution of the place-vs.-response question. Psychol. Rev., 64, 217-228.
86. Reynolds, J.N., Hyland, B.I. & Wickens, J.R. (2001) A cellular mechanism of reward-related learning. Nature, 413, 67-70.
87. Ribas-Fernandes, J.J.F., Solway, A., Diuk, C., McGuire, J.T., Barto, A.G., Niv, Y. & Botvinick, M.M. (2011) A neural signature of hierarchical reinforcement learning. Neuron, 71, 370-379.
88. Ritchie, B.F., Aeschliman, B. & Pierce, P. (1950) Studies in spatial learning: VIII. Place performance and the acquisition of place dispositions. J. Comp. Physiol. Psychol., 43, 73-85.
89. Rondi-Reig, L., Petit, G.H., Tobin, C., Tonegawa, S., Mariani, J. & Berthoz, A. (2006) Impaired sequential egocentric and allocentric memories in forebrain-specific-NMDA receptor knock-out mice during a new task dissociating strategies of navigation. J. Neurosci., 26, 4071-4081.
90. Sakai, K., Kitaguchi, K. & Hikosaka, O. (2003) Chunking during human visuomotor sequence learning. Exp. Brain Res., 152, 229-242.
91. Schmitzer-Torbert, N. & Redish, A.D. (2004) Neuronal activity in the rodent dorsal striatum in sequential navigation: separation of spatial and reward responses on the multiple T task. J. Neurophysiol., 91, 2259-2272.
92. Schultz, W. & Dickinson, A. (2000) Neuronal coding of prediction errors. Annu. Rev. Neurosci., 23, 473-500.
93. Schultz, W., Dayan, P. & Montague, P.R. (1997) A neural substrate of prediction and reward. Science, 275, 1593-1599.
94. Schwarting, R.K.W. (2009) Rodent models of serial reaction time tasks and their implementation in neurobiological research. Behav. Brain Res., 199, 76-88.
95. Shah, A. (2008) Biologically-Based Functional Mechanisms of Motor Skill Acquisition. University of Massachusetts, Amherst.
96. Shah, A. & Barto, A.G. (2009). Effect on movement selection of an evolving sensory representation: a multiple controller model of skill acquisition. Brain Res., 1299, 55-73. Elsevier B.V.
97. Shull, R.L. & Grimes, J.A. (2003) Bouts of responding from variable-interval reinforcement of lever pressing by rats. J. Exp. Anal. Behav., 80, 159-171.
98. Shull, R.L., Gaynor, S.T. & Grimes, J.A. (2002) Response rate viewed as engagement bouts: resistance to extinction. J. Exp. Anal. Behav., 77, 211-231.
99. Skinner, B.F. (1938) The Behavior of Organisms. Appleton-Century-Crofts, New York.
100. Spence, K.W. (1956) Behavior Theory and Conditioning. Yale University Press, New Haven.
101. Sutton, R.S. & Barto, A.G. (1998) Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA.
102. Tang, C., Pawlak, A.P., Prokopenko, V. & West, M.O. (2007) Changes in activity of the striatum during formation of a motor habit. Eur. J. Neurosci., 25, 1212-1227.
103. Terrace, H.S. (1991) Chunking during serial learning by a pigeon: I. Basic evidence. J. Exp. Psychol. Anim. Behav. Process., 17, 81-93.
104. Tolman, E.C., Ritchie, B.F. & Kalish, D. (1946) Studies in spatial learning: II. Place learning versus response learning. J. Exp. Psychol., 36, 221-229.
105. Tsitsiklis, J.N. & Roy, B.V. (1999) Average cost temporal-difference learning. Automatica, 35, 1799-1808.
106. Turner, R.S. & Desmurget, M. (2010) Basal ganglia contributions to motor control: a vigorous tutor. Curr. Opin. Neurobiol., 20, 704-716.
107. Watkins, C.J.C.H. (1989) Learning from Delayed Rewards. Cambridge University, Cambridge.
108. Willingham, D.B. (1998) A neuropsychological theory of motor skill learning. Psychol. Rev., 105, 558-584.
109. Willingham, D.B., Nissen, M.J. & Bullemer, P. (1989) On the development of procedural knowledge. J. Exp. Psychol. Learn. Mem. Cogn., 15, 1047-1060.
110. Wolpert, D.M. & Flanagan, J.R. (2010) Motor learning. Curr. Biol., 20, R467-R472.
111. Wolpert, D.M., Ghahramani, Z. & Flanagan, J.R. (2001) Perspectives and problems in motor learning. Trends Cogn. Sci., 5, 487-494.
112. Yin, H.H. & Knowlton, B.J. (2004) Contributions of striatal subregions to place and response learning. Learn. Mem., 11, 459-463.
113. Yin, H.H., Knowlton, B.J. & Balleine, B.W. (2004) Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur. J. Neurosci., 19, 181-189.
114. Yin, H.H., Knowlton, B.J. & Balleine, B.W. (2005) Blockade of NMDA receptors in the dorsomedial striatum prevents action-outcome learning in instrumental conditioning. Eur. J. Neurosci., 22, 505-512.
115. Yin, H.H., Knowlton, B.J. & Balleine, B.W. (2006) Inactivation of dorsolateral striatum enhances sensitivity to changes in the action-outcome contingency in instrumental conditioning. Behav. Brain Res., 166, 189-196.
116. Yin, H.H., Ostlund, S.B. & Balleine, B.W. (2008) Reward-guided learning beyond dopamine in the nucleus accumbens: the integrative functions of cortico-basal ganglia networks. Eur. J. Neurosci., 28, 1437-1448.