Still at the choice-point: Action selection and initiation in instrumental conditioning

Annals of the New York Academy of Sciences

Volumn 1104, Issue , 2007, Pages 147-171

  Balleine, Bernard W ^a   Ostlund, Sean B ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Associative learning; Goal directed action; Habit; Reinforcement; Reward

Indexed keywords

BASAL GANGLION; BEHAVIOR; BRAIN CORTEX; CONFERENCE PAPER; FEEDBACK SYSTEM; HABIT; HUMAN; INSTRUMENTAL CONDITIONING; LEARNING; NONHUMAN; OUTCOME ASSESSMENT; REINFORCEMENT; REWARD; STATE DEPENDENT LEARNING; STIMULUS RESPONSE; TASK PERFORMANCE; ANIMAL; ANIMAL BEHAVIOR; BRAIN; CONDITIONED REFLEX; DECISION MAKING; DISCRIMINATION LEARNING; HISTOLOGY; MEMORY; PHYSIOLOGY; RAT; REVIEW;

RATTUS;

ANIMALS; ASSOCIATION LEARNING; BEHAVIOR, ANIMAL; BRAIN; CHOICE BEHAVIOR; CONDITIONING, CLASSICAL; CONDITIONING, OPERANT; DISCRIMINATION LEARNING; EXTINCTION, PSYCHOLOGICAL; HUMANS; LEARNING; MEMORY; RATS; REINFORCEMENT (PSYCHOLOGY); REWARD;

EID: 34447632824     PISSN: 00778923     EISSN: 17496632     Source Type: Book Series    
DOI: 10.1196/annals.1390.006     Document Type: Conference Paper

References (78)

1. BECHARA, A. & M. VAN DER LINDEN. 2005. Decision-making and impulse control after frontal lobe injuries. Curr. Opin. Neurol. 18: 734-739.
2. DALLEY, J.W., R.N. CARDINAL & T.W. ROBBINS. 2004. Prefrontal executive and cognitive functions in rodents: neural and neurochemical substrates. Neurosci. Biobehav. Rev. 28: 771-784.
3. DAW, N.D., Y. NIV & P. DAYAN. 2005. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8: 1704-1711.
4. GLIMCHER, P.W. 2005. Indeterminacy in brain and behavior. Annu. Rev. Psychol. 56: 25-56.
5. MA, W.J., J.M. BECK, P.E. LATHAM & A. POUGET. 2006. Bayesian inference with probabilistic population codes. Nat. Neurosci. 9: 1432-1438.
6. MONTAGUE, P.R., B. KING-CASAS & J.D. COHEN. 2006. Imaging valuation models in human choice. Annu. Rev. Neurosci. 29: 417-448.
7. GUTHRIE, E.R. 1935. The Psychology of Learning. Harpers. New York.
8. TOLMAN, E.C. 1932. Purposive Behavior in Animals. Century Books. New York.
9. HULL, C.L. 1943. Principles of Behavior. Appleton. New York.
10. ADAMS, C.D. & A. DICKINSON. 1981. Instrumental responding following reinforcer devaluation. Q. J. Exp. Psychol. 33B: 109-121.
11. HOLMAN, E.W. 1975. Some conditions for the dissociation of consummatory and instrumental behavior in rats. Learn. Motiv. 6: 358-366.
12. COLWILL, R.M. & R.A. RESCORLA. 1986. Associative structures in instrumental learning. In: The Psychology of Learning and Motivation. Vol. 20. G.H. Bower, Ed.: 55-104. Academic Press. Orlando, FL.
13. BALLEINE, B.W. & A. DICKINSON. 1998. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37: 407-419.
14. CORBIT, L.H. & B.W. BALLEINE. 2000. The role of the hippocampus in instrumental conditioning. J. Neurosci. 20: 4233-4239.
15. BALLEINE, B.W. 2001. Incentive processes in instrumental conditioning. In: Handbook of Contemporary Learning Theories. R.M.S. Klein, Ed.: 307-366. LEA. Hillsdale, NJ.
16. DICKINSON, A. & B.W. BALLEINE. 1994. Motivational control of goal-directed action. Anim. Learn. Behav. 22: 1-18.
17. RESCORLA, R.A. & R.L. SOLOMON. 1967. Two-process learning theory: relationships between Pavlovian conditioning and instrumental learning. Psychol. Rev. 74: 151-182.
18. TRAPOLD, M.A. & J.B. OVERMIER. 1972. The second learning process in instrumental conditioning. In: Classical Conditioning: II. Current Research and Theory. A.A. Black & W.F. PROKASY, Eds.: 427-452. Appleton-Century-Crofts. New York.
19. SUTTON, R.S. & A.G. BARTO. 1998. Reinforcement Learning. MIT Press. Cambridge, MA.
20. O'DOHERTY, J., et al. 2004. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304: 452-454.
21. DICKINSON, A., J. CAMPOS, Z.I. VARGA & B. BALLEINE. 1996. Bidirectional instrumental conditioning. Q. J. Exp. Psychol. B. 49: 289-306.
22. COLWILL, R.M. & R.A. RESCORLA. 1988. Associations between the discriminative stimulus and the reinforcer in instrumental learning. J. Exp. Psychol. Anim. Behav. Process. 14: 155-164.
23. CORBIT, L.H., J.L. MUIR & B.W. BALLEINE. 2001. The role of the nucleus accumbens in instrumental conditioning: evidence of a functional dissociation between accumbens core and shell. J. Neurosci. 21: 3251-3260.
24. CORBIT, L.H. & B.W. BALLEINE. 2003. Instrumental and Pavlovian incentive processes have dissociable effects on components of a heterogeneous instrumental chain. J. Exp. Psychol. Anim. Behav. Process. 29: 99-106.
25. RESCORLA, R.A. 1992. Response-outcome versus outcome-response associations in instrumental learning. Anim. Learn. Behav. 20: 223-232.
26. RESCORLA, R.A. 1991. Associative relations in instrumental learning: the Eighteenth Bartlett Memorial Lecture. Q. J. Exp. Psychol. 43: 1-23.
27. DICKINSON, A. & S. DEWIT. 2003. The interaction between discriminative stimuli and outcomes during instrumental learning. Q. J. Exp. Psychol. B. 56: 127-139.
28. BOTVINICK, M.M., J.D. COHEN & C.S. CARTER. 2004. Conflict monitoring and anterior cingulate cortex: an update. Trends Cogn. Sci. 8: 539-546.
29. DE WIT, S., Y. KOSAKI, B.W. BALLEINE & A. DICKINSON. 2006. Dorsomedial prefrontal cortex resolves response conflict in rats. J. Neurosci. 26: 5224-5229.
30. DONEGAN, N.H., J.W. WHITLOW JR., & A.R. WAGNER. 1977. Posttrial reinstatement of the CS in Pavlovian conditioning: facilitation or impairment of acquisition as a function of individual differences in responsiveness to the CS. J. Exp. Psychol. Anim. Behav. Process. 3: 357-376.
31. OSTLUND, S.B. & B.W. BALLEINE. 2005. Lesions of medial prefrontal cortex disrupt the acquisition but not the expression of goal-directed learning. J. Neurosci. 25: 7763-7770.
32. OSTLUND, S.B. & B.W. BALLEINE. 2007. Instrumental reinstatement depends on sensory- and motivationally-specific features of the instrumental outcome. Learn. Behav. 35(1).
33. DICKINSON, A., B.W. BALLEINE, A. WATT, F. GONZALES & R.A. BOAKES. 1995. Overtraining and the motivational control of instrumental action. Anim. Learn. Behav. 22: 197-206.
34. ADAMS, C.D. 1981. Variations in the sensitivity of instrumental responding to reinforcer devaluation. Q. J. Exp. Psychol. 34B: 77-98.
35. COLWILL, R.M. & R.A. RESCORLA. 1990. Effect of reinforcer devaluation on discriminative control of instrumental behavior. J. Exp. Psychol. Anim. Behav. Process. 16: 40-47.
36. RESCORLA, R.A. 1994. Transfer of instrumental control mediated by a devalued outcome. Anim. Learn. Behav. 22: 27-33.
37. HOLLAND, P.C. 2004. Relations between Pavlovian-instrumental transfer and reinforcer devaluation. J. Exp. Psychol. Anim. Behav. Process. 30: 104-117.
38. DICKINSON, A. 1985. Actions and habits: the development of behavioural autonomy. Philos. Trans. R. Soc. Lond. B. 308: 67-78.
39. DICKINSON, A. 1994. Instrumental conditioning. In: Animal Cognition and Learning. N.J. Mackintosh, Ed.: 4-79. Academic Press. London.
40. DICKINSON, A. & B.W. BALLEINE. 1993. Actions and responses: the dual psychology of behaviour. In: Spatial Representation. N. Eilan, R. McCarthy & M.W. Brewer, Eds.: 277-293. Basil Blackwell Ltd. Oxford.
41. BALLEINE, B.W. 2005. Neural bases of food seeking: affect, arousal and reward in corticostriatolimbic circuits. Physiol. Behav. 86: 717-730.
42. DAYAN, P. & B.W. BALLEINE. 2002. Reward, motivation, and reinforcement learning. Neuron. 36: 285-298.
43. BALLEINE, B.W. 2004. Incentive Behavior. In: The Behavior of the Laboratory Rat: A Handbook with Tests. I.Q. Whishaw & B. Kolb, Eds.: 436-446. Oxford University Press. Oxford.
44. KELLEY, A.E. 2004. Ventral striatal control of appetitive motivation: role in ingestive behavior and reward-related learning. Neurosci. Biobehav. Rev. 27: 765-776.
45. HARUNO, M. & M. KAWATO. 2006. Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning. Neural. Netw. 19: 1242-1254.
46. GOTO, Y. & A.A. GRACE. 2005. Dopaminergic modulation of limbic and cortical drive of nucleus accumbens in goal-directed behavior. Nat. Neurosci. 8: 805-812.
47. ATALLAH, H.E., D. LOPEZ-PANIAGUA, J.W. RUDY & R. O'REILLY. 2007. C. Separate neural substrates for skill learning and performance in the ventral and dorsal striatum. Nat. Neurosci. 10: 126-131.
48. KOECHLIN, E., A. DANEK, Y. BURNOD & J. GRAFMAN. 2002. Medial prefrontal and subcortical mechanisms underlying the acquisition of motor and cognitive action sequences in humans. Neuron 35: 371-381.
49. HAMILTON, A.F. & S.T. GRAFTON. 2006. Goal representation in human anterior intraparietal sulcus. J. Neurosci. 26: 1133-1137.
50. BUCCINO, G., F. BINKOFSKI & L. RIGGIO. 2004. The mirror neuron system and action recognition. Brain Lang. 89: 370-376.
51. CORBIT, L.H. & B.W. BALLEINE. 2003. The role of prelimbic cortex in instrumental conditioning. Behav. Brain Res. 146: 145-157.
52. CORBIT, L.H., J.L. MUIR & B.W. BALLEINE. 2003. Lesions of mediodorsal thalamus and anterior thalamic nuclei produce dissociable effects on instrumental conditioning in rats. Eur. J. Neurosci. 18: 1286-1294.
53. DE BORCHGRAVE, R., J.N. RAWLINS, A. DICKINSON & B.W. BALLEINE. 2002. Effects of cytotoxic nucleus accumbens lesions on instrumental conditioning in rats. Exp. Brain. Res. 144: 50-68.
54. BALLEINE, B. & S. KILLCROSS. 1994. Effects of ibotenic acid lesions of the nucleus accumbens on instrumental action. Behav. Brain Res. 65: 181-193.
55. BALLEINE, B.W., A.S. KILLCROSS & A. DICKINSON. 2003. The effect of lesions of the basolateral amygdala on instrumental conditioning. J. Neurosci. 23: 666-675.
56. WANG, S.H., S.B. OSTLUND, K. NADER & B.W. BALLEINE. 2005. Consolidation and reconsolidation of incentive learning in the amygdala. J. Neurosci. 25: 830-835.
57. YIN, H.H., S.B. OSTLUND, B.J. KNOWLTON & B.W. BALLEINE. 2005. The role of the dorsomedial striatum in instrumental conditioning. Eur. J. Neurosci. 22: 513-523.
58. YIN, H.H., B.J. KNOWLTON & B.W. BALLEINE. 2005. Blockade of NMDA receptors in the dorsomedial striatum prevents action-outcome learning in instrumental conditioning. Eur. J. Neurosci. 22: 505-512.
59. POLDRACK, R.A. & M.G. PACKARD. 2003. Competition among multiple memory systems: converging evidence from animal and human brain studies. Neuropsychologia 41: 245-251.
60. GRAYBIEL, A.M. 1995. Building action repertoires: memory and learning functions of the basal ganglia. Curr. Opin. Neurobiol. 5: 733-741.
61. MCDONALD, R.J. & N.M. WHITE. 1993. A triple dissociation of memory systems: hippocampus, amygdala, and dorsal striatum. Behav. Neurosci. 107: 3-22.
62. YIN, H.H., B.J. KNOWLTON & B.W. BALLEINE. 2004. Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur. J. Neurosci. 19: 181-189.
63. YIN, H.H., B.J. KNOWLTON & B.W. BALLEINE. 2006. Inactivation of dorsolateral striatum enhances sensitivity to changes in the action-outcome contingency in instrumental conditioning. Behav. Brain Res. 166: 189-196.
64. KELLEY, A.E., V.B. DOMESICK & W.J. NAUTA. 1982. The amygdalostriatal projection in the rat - an anatomical study by anterograde and retrograde tracing methods. Neuroscience 7: 615-630.
65. VERTES, R.P. 2006. Interactions among the medial prefrontal cortex, hippocampus and midline thalamus in emotional and cognitive processing in the rat. Neuroscience 142: 1-20.
66. ALHEID, G.F. 2003. Extended amygdala and basal forebrain. Ann. N. Y. Acad. Sci. 985: 185-205.
67. SESACK, S.R., D.B. CARR, N. OMELCHENKO & A. PINTO. 2003. Anatomical substrates for glutamate-dopamine interactions: evidence for specificity of connections and extrasynaptic actions. Ann. N. Y. Acad. Sci. 1003: 36-52.
68. JOEL, D. & I. WEINER. 2000. The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum. Neuroscience 96: 451-474.
69. FUDGE, J.L. & A.B. EMILIANO. 2003. The extended amygdala and the dopamine system: another piece of the dopamine puzzle. J. Neuropsychiatry Clin. Neurosci. 15: 306-316.
70. BALLEINE, B.W. & S. KILLCROSS. 2006. Parallel incentive processing: an integrated view of amygdala function. Trends Neurosci. 29: 272-279.
71. ALEXANDER, G.E., M.R. DELONG & P.L. STRICK. 1986. Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu. Rev. Neurosci. 9: 357-381.
72. ALEXANDER, G.E. & M.D. CRUTCHER. 1990. Trends Neurosci. 13: 266-271.
73. NAKAHARA, H., K. DOYA & O. HIKOSAKA. 2001. Parallel cortico-basal ganglia mechanisms for acquisition and execution of visuomotor sequences - a computational approach. J. Cogn. Neurosci. 13: 626-647.
74. HABER, S.N. 2003. The primate basal ganglia: parallel and integrative networks. J. Chem. Neuroanat. 26: 317-330.
75. PERCHERON, G. & M. FILION. 1991. Parallel processing in the basal ganglia: up to a point. Trends Neurosci. 14: 55-59.
76. BAR-GAD, I., G. MORRIS & H. BERGMAN. 2003. Information processing, dimensionality reduction and reinforcement learning in the basal ganglia. Prog. Neurobiol. 71: 439-473.
77. YELNIK, J. 2002. Functional anatomy of the basal ganglia. Mov. Disord. 17 (Suppl. 3): S15-S21.
78. NADJAR, A., et al. 2006. Phenotype of striatofugal medium spiny neurons in parkinsonian and dyskinetic nonhuman primates: a call for a reappraisal of the functional organization of the basal ganglia. J. Neurosci. 26: 8653-8661.