Investigating habits: Strategies, technologies and models

Frontiers in Behavioral Neuroscience

Volumn 8, Issue FEB, 2014, Pages

  Smith, Kyle S ^a   Graybiel, Ann M ^b  

^a Dartmouth College   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Action; Chunking; Cortex; Flexibility; Learning; Reinforcement; Reward; Striatum

Indexed keywords

EID: 84894064742     PISSN: 16625153     EISSN: None     Source Type: Journal    
DOI: 10.3389/fnbeh.2014.00039     Document Type: Review

References (176)

1. Adams, C. D. (1982). Variations in the sensitivity of instrumental responding to reinforcer devaluation. Q. J. Exp. Psychol. B 34, 77-98.
2. Ahmari, S. E., Spellman, T., Douglass, N. L., Kheirbek, M. A., Simpson, H. B., Deisseroth, K., et al. (2013). Repeated cortico-striatal stimulation generates persistent OCD-like behavior. Science 340, 1234-1239. doi: 10.1126/science. 1234733
3. Albin, R. L., Young, A. B., and Penney, J. B. (1989). The functional anatomy of basal ganglia disorders. Trends Neurosci. 12, 366-375. doi: 10.1016/0166-2236(89)90074-x
4. Aldridge, J. W., Berridge, K. C., and Rosen, A. R. (2004). Basal ganglia neural mechanisms of natural movement sequences. Can. J. Physiol. Pharmacol. 82, 732-739. doi: 10.1139/y04-061
5. Alexander, G. E., DeLong, M. R., and Strick, P. L. (1986). Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu. Rev. Neurosci. 9, 357-381. doi: 10.1146/annurev.neuro.9.1.357
6. Aosaki, T., Graybiel, A. M., and Kimura, M. (1994). Effect of the nigrostriatal dopamine system on acquired neural responses in the striatum of behaving monkeys. Science 265, 412-415. doi: 10.1126/science.8023166
7. Apicella, P. (2007). Leading tonically active neurons of the striatum from reward detection to context recognition. Trends Neurosci. 30, 299-306. doi: 10.1016/j. tins.2007.03.011
8. Balleine, B., and Dickinson, A. (1992). Signalling and incentive processes in instrumental reinforcer devaluation. Q. J. Exp. Psychol. B 45, 285-301.
9. Balleine, B. W., and Dickinson, A. (1998). Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37, 407-419. doi: 10.1016/s0028-3908(98)00033-1
10. Balleine, B. W., Garner, C., Gonzalez, F., and Dickinson, A. (1995). Motivational control of heterogeneous instrumental chains. J. Exp. Psychol. Anim. Behav. Process. 21, 203-217. doi: 10.1037//0097-7403.21.3.203
11. Balleine, B. W., and Killcross, S. (2006). Parallel incentive processing: an integrated view of amygdala function. Trends Neurosci. 29, 272-279. doi: 10.1016/j.tins.2006.03.002
12. Balleine, B. W., Liljeholm, M., and Ostlund, S. B. (2009). The integrative function of the basal ganglia in instrumental conditioning. Behav. Brain Res. 199, 43-52. doi: 10.1016/j.bbr.2008.10.034
13. Balleine, B. W., and O'Doherty, J. P. (2010). Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 35, 48-69. doi: 10.1038/npp.2009.131
14. Barker, J. M., Torregrossa, M. M., and Taylor, J. R. (2013). Bidirectional modulation of infralimbic dopamine D1 and D2 receptor activity regulates flexible reward seeking. Front. Neurosci. 7:126. doi: 10.3389/fnins.2013.00126
15. Barnes, T. D., Kubota, Y., Hu, D., Jin, D. Z., and Graybiel, A. M. (2005). Activity of striatal neurons reflects dynamic encoding and recoding of procedural memories. Nature 437, 1158-1161. doi: 10.1038/nature04053
16. Belin, D., Jonkman, S., Dickinson, A., Robbins, T. W., and Everitt, B. J. (2009). Parallel and interactive learning processes within the basal ganglia: relevance for the understanding of addiction. Behav. Brain Res. 199, 89-102. doi: 10.1016/j. bbr.2008.09.027
17. Berke, J. D., Breck, J. T., and Eichenbaum, H. (2009). Striatal versus hippocampal representations during win-stay maze performance. J. Neurophysiol. 101, 1575-1587. doi: 10.1152/jn.91106.2008
18. Berke, J. D. (2011). Functional properties of striatal fast-spiking interneurons. Front. Syst. Neurosci. 5:45. doi: 10.3389/fnsys.2011.00045
19. Berns, G. S., and Sejnowski, T. J. (1998). A computational model of how the basal ganglia produce sequences. J. Cogn. Neurosci. 10, 108-121. doi: 10. 1162/089892998563815
20. Bernstein, J. G., and Boyden, E. S. (2011). Optogenetic tools for analyzing the neural circuits of behavior. Trends Cogn. Sci. 15, 592-600. doi: 10.1016/j.tics.2011. 10.003
21. Berridge, K. C., Aldridge, J. W., Houchard, K. R., and Zhuang, X. (2005). Sequential super-stereotypy of an instinctive fixed action pattern in hyper-dopaminergic mutant mice: a model of obsessive compulsive disorder and Tourette's. BMC Biol. 3:4. doi: 10.1186/1741-7007-3-4
22. Berridge, K. C., and O'Doherty, J. P. (2013). "From experienced utility to decision utility," in Neuroeconomics: Decisions and the Brain. 2nd Edn., eds P. Glimcher and E. Fehr (London, England: Academic Press), 335-354.
23. Berridge, K. C. (2004). Motivation concepts in behavioral neuroscience. Physiol. Behav. 81, 179-209. doi: 10.1016/j.physbeh.2004.02.004
24. Berridge, K. C. (2009). Wanting and liking: observations from the neuroscience and psychology laboratory. Inquiry (Oslo) 52, 378-398. doi: 10.1080/00201740903087359
25. Berridge, K. C. (2012). From prediction error to incentive salience: mesolimbic computation of reward motivation. Eur. J. Neurosci. 35, 1124-1143. doi: 10. 1111/j.1460-9568.2012.07990.x
26. Bindra, D. (1978). How adaptive behavior is produced: a perceptual-motivation alternative to response reinforcement. Behav. Brain Sci. 1, 41-52. doi: 10. 1017/s0140525x00059380
27. Bolles, R. C. (1972). Reinforcement, expectancy, and learning. Psychol. Rev. 79, 394-409. doi: 10.1037/h0033120
28. Bornstein, A. M., and Daw, N. D. (2011). Multiplicity of control in the basal ganglia: computational roles of striatal subregions. Curr. Opin. Neurobiol. 21, 374-380. doi: 10.1016/j.conb.2011.02.009
29. Bossert, J. M., Stern, A. L., Theberge, F. R., Cifani, C., Koya, E., Shaham, Y., et al. (2011). Ventral medial prefrontal cortex neuronal ensembles mediate context-induced relapse to heroin. Nat. Neurosci. 14, 420-422. doi: 10.1038/nn. 2758
30. Boyden, E. S., Zhang, F., Bamberg, E., Nagel, G., and Deisseroth, K. (2005). Millisecond-timescale, genetically targeted optical control of neural activity. Nat. Neurosci. 8, 1263-1268. doi: 10.1038/nn1525
31. Bradfield, L. A., Bertran-Gonzalez, J., Chieng, B., and Balleine, B. W. (2013). The thalamostriatal pathway and cholinergic control of goal-directed action: interlacing new with existing learning in the striatum. Neuron 79, 153-166. doi: 10.1016/j.neuron.2013.04.039
32. Brainard, M. S., and Doupe, A. J. (2002). What songbirds teach us about learning. Nature 417, 351-358. doi: 10.1038/417351a
33. Brembs, B. (2009). Mushroom bodies regulate habit formation in Drosophila. Curr. Biol. 19, 1351-1355. doi: 10.1016/j.cub.2009.06.014
34. Burguiere, E., Monteiro, P., Feng, G., and Graybiel, A. M. (2013). Optogenetic stimulation of lateral orbitofronto-striatal pathway suppresses compulsive behaviors. Science 340, 1243-1246. doi: 10.1126/science.1232380
35. Canales, J. J., and Graybiel, A. M. (2000). A measure of striatal function predicts motor stereotypy. Nat. Neurosci. 3, 377-383. doi: 10.1038/73949
36. Carelli, R. M., Wolske, M., and West, M. O. (1997). Loss of lever press-related firing of rat striatal forelimb neurons after repeated sessions in a lever pressing task. J. Neurosci. 17, 1804-1814.
37. Carr, H., and Watson, J. B. (1908). Orientation in the white rat. J. Comp. Neurol. Psychol. 18, 27-44. doi: 10.1002/cne.920180103
38. Corbit, L. H., and Janak, P. H. (2007). Inactivation of the lateral but not medial dorsal striatum eliminates the excitatory impact of Pavlovian stimuli on instrumental responding. J. Neurosci. 27, 13977-13981. doi: 10.1523/jneurosci.4097-07.2007
39. Coutureau, E., and Killcross, S. (2003). Inactivation of the infralimbic prefrontal cortex reinstates goal-directed responding in overtrained rats. Behav. Brain Res. 146, 167-174. doi: 10.1016/j.bbr.2003.09.025
40. Covington, H. E. 3rd, Lobo, M. K., Maze, I., Vialou, V., Hyman, J. M., Zaman, S., et al. (2010). Antidepressant effect of optogenetic stimulation of the medial prefrontal cortex. J. Neurosci. 30, 16082-16090. doi: 10.1523/JNEUROSCI.1731-10.2010
41. Crittenden, J. R., and Graybiel, A. M. (2011). Basal Ganglia disorders associated with imbalances in the striatal striosome and matrix compartments. Front. Neuroanat. 5:59. doi: 10.3389/fnana.2011.00059
42. Daw, N. D., Niv, Y., Dayan, P. (2005a). "Actions, policies, values and the basal ganglia," in Recent Breakthroughs in Basal Ganglia Research, ed E. Bezard (Hauppauge, NY: Nova Science Publishers), 91-106.
43. Daw, N. D., Niv, Y., and Dayan, P. (2005b). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704-1711. doi: 10.1038/nn1560
44. DeCoteau, W. E., Thorn, C., Gibson, D. J., Courtemanche, R., Mitra, P., Kubota, Y., et al. (2007a). Learning-related coordination of striatal and hippocampal theta rhythms during acquisition of a procedural maze task. Proc. Natl. Acad. Sci. U S A 104, 5644-5649. doi: 10.1073/pnas.0700818104
45. DeCoteau, W. E., Thorn, C., Gibson, D. J., Courtemanche, R., Mitra, P., Kubota, Y., et al. (2007b). Oscillations of local field potentials in the rat dorsal striatum during spontaneous and instructed behaviors. J. Neurophysiol. 97, 3800-3805. doi: 10.1152/jn.00108.2007
46. Desmurget, M., and Turner, R. S. (2010). Motor sequences and the basal ganglia: kinematics, not habits. J. Neurosci. 30, 7685-7690. doi: 10.1523/JNEUROSCI.0163-10.2010
47. Desrochers, T. M., Jin, D. Z., Goodman, N. D., and Graybiel, A. M. (2010). Optimal habits can develop spontaneously through sensitivity to local cost. Proc. Natl. Acad. Sci. U S A 107, 20512-20517. doi: 10.1073/pnas.1013470107
48. de Wit, S., Barker, R. A., Dickinson, A. D., and Cools, R. (2011). Habitual versus goal-directed action control in Parkinson disease. J. Cogn. Neurosci. 23, 1218-1229. doi: 10.1162/jocn.2010.21514
49. Dezfouli, A., and Balleine, B. W. (2012). Habits, action sequences and reinforcement learning. Eur. J. Neurosci. 35, 1036-1051. doi: 10.1111/j.1460-9568.2012.08050.x
50. Dickinson, A. (1985). Actions and habits: the development of behavioral autonomy. Philos. Trans. R. Soc. Lond. B Biol. Sci. 308, 67-78. doi: 10.1098/rstb.1985.0010
51. Dickinson, A., and Balleine, B. W. (2009). "Hedonics: the cognitive-motivational interface," in Pleasures of the Brain, eds M. L. Kringelbach and K. C. Berridge (Oxford, UK: Oxford University Press), 74-84.
52. Dickinson, A., Nichols, D. J., and Adams, C. D. (1983). The effect of the instrumental training contingency on susceptibility to reinforcer devaluation. Q. J. Exp. Psychol. B 35, 35-51.
53. Dolan, R. J., and Dayan, P. (2013). Goals and habits in the brain. Neuron 80, 312-325. doi: 10.1016/j.neuron.2013.09.007
54. Doya, K., Samejima, K., Katagiri, K., and Kawato, M. (2002). Multiple modelbased reinforcement learning. Neural Comput. 14, 1347-1369. doi: 10. 1162/089976602753712972
55. Fanelli, R. R., Klein, J. T., Reese, R. M., and Robinson, D. L. (2013). Dorsomedial and dorsolateral striatum exhibit distinct phasic neuronal activity during alcohol self-administration in rats. Eur. J. Neurosci. 38, 2637-2648. doi: 10. 1111/ejn.12271
56. Faure, A., Haberland, U., Conde, F., and El Massioui, N. (2005). Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation. J. Neurosci. 25, 2771-2780. doi: 10.1523/jneurosci.3894-04.2005
57. Fee, M. S., and Goldberg, J. H. (2011). A hypothesis for basal ganglia-dependent reinforcement learning in the songbird. Neuroscience 198, 152-170. doi: 10. 1016/j.neuroscience.2011.09.069
58. Fenno, L., Yizhar, O., and Deisseroth, K. (2011). The development and application of optogenetics. Annu. Rev. Neurosci. 34, 389-412. doi: 10.1146/annurev-neuro061010-113817
59. Ferguson, S. M., and Neumaier, J. F. (2012). Grateful DREADDs: engineered receptors reveal how neural circuits regulate behavior. Neuropsychopharmacology 37, 296-297. doi: 10.1038/npp.2011.179
60. Fujii, N., and Graybiel, A. M. (2003). Representation of action sequence boundaries by macaque prefrontal cortical neurons. Science 301, 1246-1249. doi: 10. 1126/science.1086872
61. Fujii, N., and Graybiel, A. M. (2005). Time-varying covariance of neural activities recorded in striatum and frontal cortex as monkeys perform sequentialsaccade tasks. Proc. Natl. Acad. Sci. U S A 102, 9032-9037. doi: 10.1073/pnas. 0503541102
62. Fujimoto, H., Hasegawa, T., and Watanabe, D. (2011). Neural coding of syntactic structure in learned vocalizations in the songbird. J. Neurosci. 31, 10023-10033. doi: 10.1523/JNEUROSCI.1606-11.2011
63. Galarce, E. M., Crombag, H. S., and Holland, P. C. (2007). Reinforcer-specificity of appetitive and consummatory behavior of rats after Pavlovian conditioning with food reinforcers. Physiol. Behav. 91, 95-105. doi: 10.1016/j.physbeh.2007.01.021
64. Glickman, S. E., and Schiff, B. B. (1967). A biological theory of reinforcement. Psychol. Rev. 74, 81-109. doi: 10.1037/h0024290
65. Goldberg, J. A., and Reynolds, J. N. (2011). Spontaneous firing and evoked pauses in the tonically active cholinergic interneurons of the striatum. Neuroscience 198, 27-43. doi: 10.1016/j.neuroscience.2011.08.067
66. Gourley, S. L., Olevska, A., Gordon, J., and Taylor, J. R. (2013). Cytoskeletal determinants of stimulus-response habits. J. Neurosci. 33, 11811-11816. doi: 10.1523/JNEUROSCI.1034-13.2013
67. Graybiel, A. M. (1995). The basal ganglia. Trends Neurosci. 18, 60-62. doi: 10. 1016/0166-2236(95)80019-X
68. Graybiel, A. M. (1997). The basal ganglia and cognitive pattern generators. Schizophr. Bull. 23, 459-469. doi: 10.1093/schbul/23.3.459
69. Graybiel, A. M. (1998). The basal ganglia and chunking of action repertoires. Neurobiol. Learn. Mem. 70, 119-136. doi: 10.1006/nlme.1998.3843
70. Graybiel, A. M. (2008). Habits, rituals and the evaluative brain. Annu. Rev. Neurosci. 31, 359-387. doi: 10.1146/annurev.neuro.29.051605.112851
71. Gremel, C. M., and Costa, R. M. (2013). Orbitofrontal and striatal circuits dynamically encode the shift between goal-directed and habitual actions. Nat. Commun. 4:2264. doi: 10.1038/ncomms3264
72. Hershberger, W. A. (1986). An approach through the looking glass. Anim. Learn. Behav. 14, 443-451. doi: 10.3758/bf03200092
73. Hikosaka, O., and Isoda, M. (2010). Switching from automatic to controlled behavior: cortico-basal ganglia mechanisms. Trends Cogn. Sci. 14, 154-161. doi: 10.1016/j.tics.2010.01.006
74. Hitchcott, P. K., Quinn, J. J., and Taylor, J. R. (2007). Bidirectional modulation of goal-directed actions by prefrontal cortical dopamine. Cereb. Cortex 17, 2820-2827. doi: 10.1093/cercor/bhm010
75. Holland, P. C. (2004). Relations between Pavlovian-instrumental transfer and reinforcer devaluation. J. Exp. Psychol. Anim. Behav. Process. 30, 104-117. doi: 10. 1037/0097-7403.30.2.104
76. Holland, P. C. (2008). Cognitive versus stimulus-response theories of learning. Learn. Behav. 36, 227-241. doi: 10.3758/lb.36.3.227
77. Holland, P. C., Lasseter, H., and Agarwal, I. (2008). Amount of training and cueevoked taste-reactivity responding in reinforcer devaluation. J. Exp. Psychol. Anim. Behav. Process. 34, 119-132. doi: 10.1037/0097-7403.34.1.119
78. Holland, P. C., and Rescorla, R. A. (1975). The effect of two ways of devaluing the unconditioned stimulus after firstand second-order appetitive conditioning. J. Exp. Psychol. Anim. Behav. Process. 1, 355-363. doi: 10.1037//0097-7403.1.4.355
79. Holland, P. C., and Straub, J. J. (1979). Differential effects of two ways of devaluing the unconditioned stimulus after Pavlovian appetitive conditioning. J. Exp. Psychol. Anim. Behav. Process. 5, 65-78. doi: 10.1037//0097-7403.5.1.65
80. Holland, P. C., and Wheeler, D. S. (2009). "Representation-mediated food aversions," in Conditioned Taste Aversion: Behavioral and Neural Processes, eds S. Reilly and T. Schachtman (Oxford: Oxford University Press), 196-225.
81. Holtzheimer, P. E., and Mayberg, H. S. (2011). Deep brain stimulation for psychiatric disorders. Annu. Rev. Neurosci. 34, 289-307. doi: 10.1146/annurev-neuro061010-113638
82. Hull, C. L. (1943). Principles of Behavior. New York: Appleton-Century Crofts.
83. Hurley, K. M., Herbert, H., Moga, M. M., and Saper, C. B. (1991). Efferent projections of the infralimbic cortex of the rat. J. Comp. Neurol. 308, 249-276. doi: 10.1016/0006-8993(91)91677-s
84. Hyman, S. E., Malenka, R. C., and Nestler, E. J. (2006). Neural mechanisms of addiction: the role of reward-related learning and memory. Annu. Rev. Neurosci. 29, 565-598. doi: 10.1146/annurev.neuro.29.051605.113009
85. James, W. (1890). The Principles of Psychology. New York, NY: Cosimo. 696 pp.
86. Jin, D. Z., Fujii, N., and Graybiel, A. M. (2009). Neural representation of time in cortico-basal ganglia circuits. Proc. Natl. Acad. Sci. U S A 106, 19156-19161. doi: 10.1073/pnas.0909881106
87. Jin, X., and Costa, R. M. (2010). Start/stop signals emerge in nigrostriatal circuits during sequence learning. Nature 466, 457-462. doi: 10.1038/nature09263
88. Jog, M. S., Kubota, Y., Connolly, C. I., Hillegaart, V., and Graybiel, A. M. (1999). Building neural representations of habits. Science 286, 1745-1749. doi: 10. 1126/science.286.5445.1745
89. Johnson, A., and Redish, A. D. (2007). Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176-12189. doi: 10.1523/jneurosci.3761-07.2007
90. Karni, A., Meyer, G., Jezzard, P., Adams, M. M., Turner, R., and Ungerleider, L. G. (1995). Functional MRI evidence for adult motor cortex plasticity during motor skill learning. Nature 377, 155-158. doi: 10.1038/377155a0
91. Killcross, S., and Coutureau, E. (2003). Coordination of actions and habits in the medial prefrontal cortex of rats. Cereb. Cortex 13, 400-408. doi: 10. 1093/cercor/13.4.400
92. Kimchi, E. Y., Torregrossa, M. M., Taylor, J. R., and Laubach, M. (2009). Neuronal correlates of instrumental learning in the dorsal striatum. J. Neurophysiol. 102, 475-489. doi: 10.1152/jn.00262.2009
93. Kimura, M., Yamada, H., and Matsumoto, N. (2003). Tonically active neurons in the striatum encode motivational contexts of action. Brain Dev. 25(Suppl. 1), S20-S23. doi: 10.1016/s0387-7604(03)90003-9
94. Konorski, J. (1967). Integrative Activity of the Brain. Chicago: University of Chicago Press.
95. Kravitz, A. V., Freeze, B. S., Parker, P. R., Kay, K., Thwin, M. T., Deisseroth, K., et al. (2010). Regulation of parkinsonian motor behaviours by optogenetic control of basal ganglia circuitry. Nature 466, 622-626. doi: 10.1038/nature09159
96. Kravitz, A. V., Tye, L. D., and Kreitzer, A. C. (2012). Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat. Neurosci. 15, 816-818. doi: 10.1038/nn.3100
97. Kubota, Y., Liu, J., Hu, D., DeCoteau, W. E., Eden, U. T., Smith, A. C., et al. (2009). Stable encoding of task structure coexists with flexible coding of task events in sensorimotor striatum. J. Neurophysiol. 102, 2142-2160. doi: 10.1152/jn.00522. 2009
98. Lee, A. S., Duman, R. S., and Pittenger, C. (2008). A double dissociation revealing bidirectional competition between striatum and hippocampus during learning. Proc. Natl. Acad. Sci. U S A 105, 17163-17168. doi: 10.1073/pnas.0807749105
99. Lenz, J. D., and Lobo, M. K. (2013). Optogenetic insights into striatal function and behavior. Behav. Brain Res. 255, 44-54. doi: 10.1016/j.bbr.2013.04.018
100. Lingawi, N. W., and Balleine, B. W. (2012). Amygdala central nucleus interacts with dorsolateral striatum to regulate the acquisition of habits. J. Neurosci. 32, 1073-1081. doi: 10.1523/jneurosci.4806-11.2012
101. Locurto, C., Terrace, H. C., and Gibbon, J. (1976). Autoshaping, random control and omission training in the rat. J. Exp. Anal. Behav. 26, 451-462. doi: 10. 1901/jeab.1976.26-451
102. Lorenz, K., and Leyhausen, P. (1973). Motivation of Human and Animal Behavior; an Ethological View. New York: Van Nostrand-Reinhold. 423 pp.
103. Marder, E. (2011). Variability, compensation and modulation in neurons and circuits. Proc. Natl. Acad. Sci. U S A 108(Suppl. 3), 15542-15548. doi: 10. 1073/pnas.1010674108
104. Matsumoto, N., Hanakawa, T., Maki, S., Graybiel, A. M., and Kimura, M. (1999). Role of [corrected] nigrostriatal dopamine system in learning to perform sequential motor tasks in a predictive manner. J. Neurophysiol. 82, 978-998.
105. McDannald, M. A., Takahashi, Y. K., Lopatina, N., Pietras, B. W., Jones, J. L., and Schoenbaum, G. (2012). Model-based learning and the contribution of the orbitofrontal cortex to the model-free world. Eur. J. Neurosci. 35, 991-996. doi: 10.1111/j.1460-9568.2011.07982.x
106. McDonald, R. J., and White, N. M. (1993). A triple dissociation of memory systems: hippocampus, amygdala and dorsal striatum. Behav. Neurosci. 107, 3-22. doi: 10. 1037//0735-7044.107.1.3
107. Mei, Y., and Zhang, F. (2012). Molecular tools and approaches for optogenetics. Biol. Psychiatry 71, 1033-1038. doi: 10.1016/j.biopsych.2012.02.019
108. Milad, M. R., Rauch, S. L., Pitman, R. K., and Quirk, G. J. (2006). Fear extinction in rats: implications for human brain imaging and anxiety disorders. Biol. Psychol. 73, 61-71. doi: 10.1016/j.biopsycho.2006.01.008
109. Milad, M. R., and Rauch, S. L. (2012). Obsessive-compulsive disorder: beyond segregated cortico-striatal pathways. Trends Cogn. Sci. 16, 43-51. doi: 10.1016/j. tics.2011.11.003
110. Miller, G. A. (1956). The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychol. Rev. 63, 81-97. doi: 10. 1037/h0043158
111. Morgan, M. A., Romanski, L. M., and LeDoux, J. E. (1993). Extinction of emotional learning: contribution of medial prefrontal cortex. Neurosci. Lett. 163, 109-113. doi: 10.1016/0304-3940(93)90241-c
112. Morgan, M. J. (1974). Resistance to satiation. Anim. Behav. 22, 449-466. doi: 10. 1016/s0003-3472(74)80044-8
113. Muenzinger, K. F. (1938). Vicarious trial and error at a point of choice: I. A general survey of its relation to learning efficiency. J. Genet. Psychol. 53, 75-86. doi: 10. 1080/08856559.1938.10533799
114. Nudo, R. J., Wise, B. M., SiFuentes, F., and Milliken, G. W. (1996). Neural substrates for the effects of rehabilitative training on motor recovery after ischemic infarct. Science 272, 1791-1794. doi: 10.1126/science.272.5269.1791
115. Ostlund, S. B., Winterbauer, N. E., and Balleine, B. W. (2009). Evidence of action sequence chunking in goal-directed instrumental conditioning and its dependence on the dorsomedial prefrontal cortex. J. Neurosci. 29, 8280-8287. doi: 10.1523/jneurosci.1176-09.2009
116. Packard, M. G. (2009). Exhumed from thought: basal ganglia and response learning in the plus-maze. Behav. Brain Res. 199, 24-31. doi: 10.1016/j.bbr.2008.12.013
117. Packard, M. G., and Goodman, J. (2013). Factors that influence the relative use of multiple memory systems. Hippocampus 23, 1044-1052. doi: 10.1002/hipo. 22178
118. Packard, M. G., and McGaugh, J. L. (1996). Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neurobiol. Learn. Mem. 65, 65-72. doi: 10.1006/nlme.1996.0007
119. Pascoli, V., Turiault, M., and Luscher, C. (2011). Reversal of cocaine-evoked synaptic potentiation resets drug-induced adaptive behaviour. Nature 481, 71-75. doi: 10.1038/nature10709
120. Pavlov, I. (1927). Conditioned Reflexes. Mineola, NY: Dover Publications. 448 pp.
121. Peters, J., Kalivas, P. W., and Quirk, G. J. (2009). Extinction circuits for fear and addiction overlap in prefrontal cortex. Learn. Mem. 16, 279-288. doi: 10. 1101/lm.1041309
122. Peters, J., LaLumiere, R. T., and Kalivas, P. W. (2008). Infralimbic prefrontal cortex is responsible for inhibiting cocaine seeking in extinguished rats. J. Neurosci. 28, 6046-6053. doi: 10.1523/jneurosci.1045-08.2008
123. Plautz, E. J., Milliken, G. W., and Nudo, R. J. (2000). Effects of repetitive motor training on movement representations in adult squirrel monkeys: role of use versus learning. Neurobiol. Learn. Mem. 74, 27-55. doi: 10.1006/nlme.1999.3934
124. Poldrack, R. A., Sabb, F. W., Foerde, K., Tom, S. M., Asarnow, R. F., Bookheimer, S.Y., et al. (2005). The neural correlates of motor skill automaticity. J. Neurosci. 25, 5356-5364. doi: 10.1523/jneurosci.3880-04.2005
125. Ragozzino, M. E. (2007). The contribution of the medial prefrontal cortex, orbitofrontal cortex and dorsomedial striatum to behavioral flexibility. Ann. N Y Acad. Sci. 1121, 355-375. doi: 10.1196/annals.1401.013
126. Rangel, A., Camerer, C., and Montague, P. R. (2008). A framework for studying the neurobiology of value-based decision making. Nat. Rev. Neurosci. 9, 545-556. doi: 10.1038/nrn2357
127. Redish, A. D., Jensen, S., and Johnson, A. (2008). A unified framework for addiction: vulnerabilities in the decision process. Behav. Brain Sci. 31, 415-437; discussion 437-487. doi: 10.1017/s0140525x0800472x
128. Rescorla, R. A. (1996). Response-outcome associations remain functional through interference treatments. Anim. Learn. Behav. 24, 450-458. doi: 10. 3758/bf03199016
129. Rhodes, S. E., and Killcross, S. (2004). Lesions of rat infralimbic cortex enhance recovery and reinstatement of an appetitive Pavlovian response. Learn. Mem. 11, 611-616. doi: 10.1101/lm.79704
130. Rich, E. L., and Shapiro, M. (2009). Rat prefrontal cortical neurons selectively code strategy switches. J. Neurosci. 29, 7208-7219. doi: 10.1523/jneurosci.6068-08. 2009
131. Robinson, M. J., and Berridge, K. C. (2013). Instant transformation of learned repulsion into motivational "wanting". Curr. Biol. 23, 282-289. doi: 10.1016/j. cub.2013.01.016
132. Root, D. H., Tang, C. C., Ma, S., Pawlak, A. P., and West, M. O. (2010). Absence of cue-evoked firing in rat dorsolateral striatum neurons. Behav. Brain Res. 211, 23-32. doi: 10.1016/j.bbr.2010.03.001
133. Rudebeck, P. H., and Murray, E. A. (2011). Balkanizing the primate orbitofrontal cortex: distinct subregions for comparing and contrasting values. Ann. N Y Acad. Sci. 1239, 1-13. doi: 10.1111/j.1749-6632.2011.06267.x
134. Salamone, J. D., and Correa, M. (2012). The mysterious motivational functions of mesolimbic dopamine. Neuron 76, 470-485. doi: 10.1016/j.neuron.2012.10.021
135. Schneck, N., and Vezina, P. (2012). Enhanced dorsolateral striatal activity in drug use: the role of outcome in stimulus-response associations. Behav. Brain Res. 235, 136-142. doi: 10.1016/j.bbr.2012.07.042
136. Sherrington, C. S. (1906). The Integrative Action of the Nervous System. New York: C. Scribner's sons. xvi, 411 pp.
137. Smith, K. S., Berridge, K. C., and Aldridge, J. W. (2011). Disentangling pleasure from incentive salience and learning signals in brain reward circuitry. Proc. Natl. Acad. Sci. U S A 108, E255-E264. doi: 10.1073/pnas.1101920108
138. Smith, K. S., and Graybiel, A. M. (2013a). A dual operator view of habitual behavior reflecting cortical and striatal dynamics. Neuron 79, 361-374. doi: 10.1016/j. neuron.2013.05.038
139. Smith, K. S., and Graybiel, A. M. (2013b). Using optogenetics to study habits. Brain Res. 1511, 102-114. doi: 10.1016/j.brainres.2013.01.008
140. Smith, K. S., Virkud, A., Deisseroth, K., and Graybiel, A. M. (2012). Reversible online control of habitual behavior by optogenetic perturbation of medial prefrontal cortex. Proc. Natl. Acad. Sci. U S A 109, 18932-18937. doi: 10. 1073/pnas.1216264109
141. Stalnaker, T. A., Calhoon, G. G., Ogawa, M., Roesch, M. R., and Schoenbaum, G. (2010). Neural correlates of stimulus-response and response-outcome associations in dorsolateral versus dorsomedial striatum. Front. Integr. Neurosci. 4:12. doi: 10.3389/fnint.2010.00012
142. Stiers, M., and Silberberg, A. (1974). Lever-contact responses in rats: automaintenance with and without a negative response-reinforcer dependency. J. Exp. Anal. Behav. 22, 497-506. doi: 10.1901/jeab.1974.22-497
143. Stuber, G. D., Britt, J. P., and Bonci, A. (2012). Optogenetic modulation of neural circuits that underlie reward seeking. Biol. Psychiatry 71, 1061-1067. doi: 10. 1016/j.biopsych.2011.11.010
144. Stuber, G. D., Hnasko, T. S., Britt, J. P., Edwards, R. H., and Bonci, A. (2010). Dopaminergic terminals in the nucleus accumbens but not the dorsal striatum corelease glutamate. J. Neurosci. 30, 8229-8233. doi: 10.1523/jneurosci.1754-10.2010
145. Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
146. Tang, C., Pawlak, A. P., Prokopenko, V., and West, M. O. (2007). Changes in activity of the striatum during formation of a motor habit. Eur. J. Neurosci. 25, 1212-1227. doi: 10.1111/j.1460-9568.2007.05353.x
147. Tang, C. C., Root, D. H., Duke, D. C., Zhu, Y., Teixeria, K., Ma, S., et al. (2009). Decreased firing of striatal neurons related to licking during acquisition and overtraining of a licking task. J. Neurosci. 29, 13952-13961. doi: 10. 1523/jneurosci.2824-09.2009
148. Tecuapetla, F., Patel, J. C., Xenias, H., English, D., Tadros, I., Shah, F., et al. (2010). Glutamatergic signaling by mesolimbic dopamine neurons in the nucleus accumbens. J. Neurosci. 30, 7105-7110. doi: 10.1523/jneurosci.0265-10.2010
149. Thorn, C. A., Atallah, H. E., Howe, M. W., and Graybiel, A. M. (2010). Differential dynamics of activity changes in dorsolateral and dorsomedial striatal loops during learning. Neuron 66, 781-795. doi: 10.1016/j.neuron.2010.04.036
150. Thorn, C. A., and Graybiel, A. M. (in press). Differential learning-related dynamics of spiking activity and local field eEntrainment among neuronal subpopulations in the sensorimotor and associative striatum. J. Neurosci.
151. Thorndike, E. L. (1898). Animal Intelligence: An Experimental Study of the Associative Processes in Animals. New York: Macmillan.
152. Tinbergen, N. (1950). The hierarchical organization of nervous mechanisms underlying instinctive behaviour. Symp. Soc. Exp. Biol. 4, 305-312.
153. Tindell, A. J., Berridge, K. C., Zhang, J., Peciña, S., and Aldridge, J. W. (2005). Ventral pallidal neurons code incentive motivation: amplification by mesolimbic sensitization and amphetamine. Eur. J. Neurosci. 22, 2617-2634. doi: 10.1111/j. 1460-9568.2005.04411.x
154. Toates, F. M. (1986). Motivational Systems. Cambridge: Cambridge University Press.
155. Tolman, E. C. (1932). Purposive Behavior in Animals and Men. New York: Century.
156. Tolman, E. C. (1948). Cognitive maps in rats and men. Psychol. Rev. 55, 189-208. doi: 10.1037/h0061626
157. Tolman, E. C., and Gleitman, H. (1949). Studies in learning and motivation; equal reinforcements in both end-boxes; followed by shock in one end-box. J. Exp. Psychol. 39, 810-819. doi: 10.1037/h0062845
158. Tolman, E. C., Ritchie, B. F., and Kalish, D. (1947). Studies in spatial learning: V. response learning versus place learning by the non-correction method. J. Exp. Psychol. 37, 285-292. doi: 10.1037/h0057434
159. Tort, A. B., Kramer, M. A., Thorn, C., Gibson, D. J., Kubota, Y., Graybiel, A. M., et al. (2008). Dynamic cross-frequency couplings of local field potential oscillations in rat striatum and hippocampus during performance of a T-maze task. Proc. Natl. Acad. Sci. U S A 105, 20517-20522. doi: 10.1073/pnas.0810524105
160. Tricomi, E., Balleine, B. W., and O'Doherty, J. P. (2009). A specific role for posterior dorsolateral striatum in human habit learning. Eur. J. Neurosci. 29, 2225-2232. doi: 10.1111/j.1460-9568.2009.06796.x
161. Vertes, R. P. (2004). Differential projections of the infralimbic and prelimbic cortex in the rat. Synapse 51, 32-58. doi: 10.1002/syn.10279
162. Wang, L. P., Li, F., Wang, D., Xie, K., Shen, X., and Tsien, J. Z. (2011). NMDA receptors in dopaminergic neurons are crucial for habit learning. Neuron 72, 1055-1066. doi: 10.1016/j.neuron.2011.10.019
163. Welch, J. M., Lu, J., Rodriguiz, R. M., Trotta, N. C., Peca, J., Ding, J. D., et al. (2007). Cortico-striatal synaptic defects and OCD-like behaviours in Sapap3-mutant mice. Nature 448, 894-900. doi: 10.1038/nature06104
164. Wickens, J. R., Arbuthnott, G. W., and Shindou, T. (2007). Simulation of GABA function in the basal ganglia: computational models of GABAergic mechanisms in basal ganglia function. Prog. Brain Res. 160, 313-329. doi: 10.1016/s0079-6123(06)60018-6
165. Williams, D. R., and Williams, H. (1969). Auto-maintenance in the pigeon: sustained pecking despite contingent non-reinforcement. J. Exp. Anal. Behav. 12, 511-520. doi: 10.1901/jeab.1969.12-511
166. Witten, I. B., Lin, S. C., Brodsky, M., Prakash, R., Diester, I., Anikeeva, P., et al. (2010). Cholinergic interneurons control local circuit activity and cocaine conditioning. Science 330, 1677-1681. doi: 10.1126/science.1193771
167. Wood, W., and Neal, D. T. (2007). A new look at habits and the habit-goal interface. Psychol. Rev. 114, 843-863. doi: 10.1037/0033-295x.114.4.843
168. Wunderlich, K., Dayan, P., and Dolan, R. J. (2012). Mapping value based planning and extensively trained choice in the human brain. Nat. Neurosci. 15, 786-791. doi: 10.1038/nn.3068
169. Yin, H. H., Knowlton, B. J., and Balleine, B. W. (2004). Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur. J. Neurosci. 19, 181-189. doi: 10.1111/j.1460-9568.2004.03095.x
170. Yin, H. H., Knowlton, B. J., and Balleine, B. W. (2005a). Blockade of NMDA receptors in the dorsomedial striatum prevents action-outcome learning in instrumental conditioning. Eur. J. Neurosci. 22, 505-512. doi: 10.1111/j.1460-9568.2005.04219.x
171. Yin, H. H., Knowlton, B. J., and Balleine, B. W. (2006). Inactivation of dorsolateral striatum enhances sensitivity to changes in the action-outcome contingency in instrumental conditioning. Behav. Brain Res. 166, 189-196. doi: 10.1016/j.bbr. 2005.07.012
172. Yin, H. H., and Knowlton, B. J. (2006). The role of the basal ganglia in habit formation. Nat. Rev. Neurosci. 7, 464-476. doi: 10.1038/nrn1919
173. Yin, H. H., Mulcare, S. P., Hilario, M. R., Clouse, E., Holloway, T., Davis, M. I., et al. (2009). Dynamic reorganization of striatal circuits during the acquisition and consolidation of a skill. Nat. Neurosci. 12, 333-341. doi: 10.1038/nn.2261
174. Yin, H. H., Ostlund, S. B., Knowlton, B. J., and Balleine, B. W. (2005b). The role of the dorsomedial striatum in instrumental conditioning. Eur. J. Neurosci. 22, 513-523. doi: 10.1111/j.1460-9568.2005.04218.x
175. Yu, C., Gupta, J., Chen, J. F., and Yin, H. H. (2009). Genetic deletion of A2A adenosine receptors in the striatum selectively impairs habit formation. J. Neurosci. 29, 15100-15103. doi: 10.1523/jneurosci.4215-09.2009
176. Zhang, J., Berridge, K. C., Tindell, A. J., Smith, K. S., and Aldridge, J. W. (2009). A neural computational model of incentive salience. PLoS Comput. Biol. 5:e1000437. doi: 10.1371/journal.pcbi.1000437