Correlates of reward-predictive value in learning-related hippocampal neural activity

Hippocampus

Volumn 19, Issue 5, 2009, Pages 487-506

  Okatan, Murat ^a  

^a BOSTON UNIVERSITY   (United States)

Author keywords

Action selection; Joint probability model; Machine learning; Prefrontal cortex; Reward pathway

Indexed keywords

ARTICLE; BEHAVIORAL SCIENCE; CONTROLLED STUDY; HIPPOCAMPUS; LEARNING; LEARNING ALGORITHM; NERVE CELL; NONHUMAN; PRIORITY JOURNAL; RHESUS MONKEY; TASK PERFORMANCE;

ACTION POTENTIALS; ALGORITHMS; ANIMALS; ASSOCIATION LEARNING; BEHAVIOR, ANIMAL; COMPUTER SIMULATION; DECISION MAKING; HIPPOCAMPUS; LIKELIHOOD FUNCTIONS; MACACA MULATTA; MODELS, NEUROLOGICAL; NEURONS; POISSON DISTRIBUTION; PROBABILITY; REGRESSION ANALYSIS; REWARD; TIME FACTORS;

EID: 65949092315     PISSN: 10509631     EISSN: 10981063     Source Type: Journal    
DOI: 10.1002/hipo.20535     Document Type: Article

References (88)

1. Ainge JA, Tamosiunaite M, Woergoetter F, Dudchenko PA. 2007. Hippocampal CA1 place cells encode intended destination on a maze with multiple choice points. J Neurosci 27:9769-9779. (Pubitemid 47492189)
2. Apicella P, Scamati E, Ljungberg T, Schultz W. 1992. Neuronal activity in monkey striatum related to the expectation of predictable environmental events. J Neurophysiol 68:945-960.
3. Attwell D, Iadecola C. 2002. The neural basis of functional brain imaging signals. Trends Neurosci 25:621-625. (Pubitemid 35356509)
4. Barraclough DJ, Conroy ML, Lee D. 2004. Prefrontal cortex and decision making in a mixed-strategy game. Nat Neurosci 7:404-410. (Pubitemid 38429181)
5. Barto AG, Sutton RS. 1982. Simulation of anticipatory responses in classical conditioning by a neuron-like adaptive element. Behav Brain Res 4:221-235. (Pubitemid 12180133)
6. Barto AG, Sutton RS, Anderson CW. 1983. Neuronlike adaptive elements that can solve difficult learning problems. IEEE Trans Syst Man Cybern 13:834-846. (Pubitemid 14138646)
7. Bellebaum C, Koch B, Schwarz M, Daum I. 2008. Focal basal ganglia lesions are associated with impairments in reward-based reversal learning. Brain 131:829-841. (Pubitemid 350216193)
8. Brown EN, Frank LM, Tang D, Quirk M, Wilson MA. 1998. A statistical paradigm for neural spike train decoding applied to position prediction from ensemble firing patterns of rat hippocampal place cells. J Neurosci 18:7411-7425. (Pubitemid 28425825)
9. Brown EN, Nguyen DP, Frank LM, Wilson MA, Solo V. 2001. An analysis of neural receptive field plasticity by point process adaptive filtering. PNAS 98:12261-12266. (Pubitemid 32959861)
10. Burnham KP, Anderson DR. 2002. Model Selection and Multimodel Inference. New York: Springer.
11. Buzsaki G. 2002. Theta oscillations in the hippocampus. Neuron 33:325-340. (Pubitemid 34158839)
12. Cacucci F, Yi M, Wills TJ, Chapman P, O'Keefe J. 2008. Place cell firing correlates with memory deficits and amyloid plaque burden in Tg2576 Alzheimer mouse model. PNAS 105:7863-7868.
13. Cahusac PM, Rolls ET, Miyashita Y, Niki H. 1993. Modification of the responses of hippocampal neurons in the monkey during the learning of a conditional spatial response task. Hippocampus 3:29- 42. (Pubitemid 23129125)
14. Chapman PF, White GL, Jones MW, Cooper-Blacketer D, Marshall VJ, Irizarry M, Younkin L, Good MA, Bliss TVP, Hyman BT, Younkin SG, Hsiao KK. 1999. Impaired synaptic plasticity and learning in aged amyloid precursor protein transgenic mice. Nat Neurosci 2:271-276. (Pubitemid 30491299)
15. Chen LL, Wise SP. 1995. Neuronal activity in the supplementary eye field during acquisition of conditional oculomotor associations. J Neurophysiol 73:1101-1121.
16. Chornoboy ES, Schramm LP, Karr AF. 1988. Maximum likelihood identification of neuronal point process systems. Biol Cybern 59:265-275. (Pubitemid 18226217)
17. Daley D, Vere-Jones D. 2003. An Introduction to the Theory of Point Processes, 2nd ed. New York: Springer-Verlag.
18. Dempster AP, Laird NM, Rubin DB. 1977. Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc B 39:1- 38.
19. Efron B, Tibshirani RJ. 1998. An Introduction to the Bootstrap. New York: Chapman & Hall/CRC.
20. Eichenbaum H. 1999. The hippocampus and mechanisms of declarative memory. Behav Brain Res 103:123-133. (Pubitemid 29417996)
21. Eichenbaum H, Fortin NJ. 2005. Bridging the gap between brain and behavior: Cognitive and neural mechanisms of episodic memory. J Exp Anal Behav 84:619-629.
22. Eichenbaum H, Fagan A, Cohen NJ. 1986. Normal olfactory discrimination learning set and facilitation of reversal learning after medial-temporal damage in rats: Implications for an account of preserved learning abilities in amnesia. J Neurosci 6:1876-1884. (Pubitemid 16064951)
23. Eichenbaum H, Dudchenko P, Wood E, Shapiro M, Tanila H. 1999. The hippocampus, memory, and place cells: is it spatial memory or a memory space? Neuron 23:209-226.
24. Foster DJ, Wilson MA. 2006. Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 440: 680-683.
25. Foster TC, Castro CA, McNaughton BL. 1989. Spatial selectivity of rat hippocampal neurons: Dependence on preparedness for movement. Science 244:1580-1582. (Pubitemid 19189736)
26. Friedman DP, Aggleton JP, Saunders RC. 2002. Comparison of hippocampal, amygdala, and perirhinal projections to the nucleus accumbens: Combined anterograde and retrograde tracing study in the macaque brain. J Comp Neurol 450:345-365.
27. Funahashi S, Bruce CJ, Goldman-Rakic P. 1989. Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex. J Neurophysiol 61:331-349. (Pubitemid 19063448)
28. Gheusi G, Goodall G, Dantzer R. 1997. Individually distinctive odours represent individual conspecifics in rats. Anim Behav 53:935-944. (Pubitemid 27218685)
29. Goldman-Rakic PS. 1987. Circuitry of primate prefrontal cortex and regulation of behavior by representational memory. In: Mountcastle VB, Plum F, Geiger SR, editors. Handbook of Physiology. The Nervous System, Vol. 5. Bethesda, MD: American Physiological Society. pp 373-417.
30. Goldman-Rakic PS, Selemon LD, Schwartz ML. 1984. Dual pathways connecting the dorsolateral prefrontal cortex with the hippocampal formation and parahippocampal cortex in the rhesus monkey. Neuroscience 12:719-743. (Pubitemid 14092488)
31. Haykin S. 1996. Adaptive Filter Theory. Englewood Cliffs, NJ: Prentice- Hall.
32. Houk JC, Adams JL, Barto AG. 1995. A model of how the basal ganglia generate and use reward signals that predict reinforcement. In: Houk JC, Davis JL, Beiser DG, editors. Models of Information Processing in the Basal Ganglia. Cambridge: MIT Press.
33. Hölscher C, Jacob W, Mallot HA. 2003. Reward modulates neuronal activity in the hippocampus of the rat. Behav Brain Res 142:181-191. (Pubitemid 36677605)
34. Jung Y, Hong S. 2003. Organization of projections from the medial temporal cortical areas to the ventral striatum in macaque monkeys. Korean J Biol Sci 7:237-248. (Pubitemid 37193695)
35. Kaelbling LP, Littman ML, Moore AW. 1996. Reinforcement learning: A survey. J Artif Intell Res 4:237-285. (Pubitemid 126646155)
36. Kakade S, Dayan P. 2000. Dopamine bonuses. NIPS 13:131-137.
37. Knepper BR, Kurylo DD. 1998. Effects of nitric oxide synthase inhibitor NG-Nitro-l-Arginine Methyl Ester on spatial and cued leaning. Neuroscience 83:837-841. (Pubitemid 28068904)
38. Kunishio K, Ohmoto T, Haber SN. 1996. Topographic organization of the ventral striatum afferent projection from amygdaloid complex and hippocampal formation. No To Shinkei 48:534-542. (Pubitemid 26198804)
39. Lee I, Griffin AL, Zilli EA, Eichenbaum E, Hasselmo ME. 2006. Gradual translocation of spatial correlates of neuronal firing in the hippocampus toward prospective reward locations. Neuron 51:639-650. (Pubitemid 44301823)
40. Leutgeb S, Leutgeb JK, Barnes CA, Moser EI, McNaughton BL, Moser MB. 2005. Independent codes for spatial and episodic memory in hippocampal neuronal ensembles. Science 309:619- 623. (Pubitemid 41033647)
41. Liu Z, Richmond BJ. 2000. Response differences in monkey TE and perirhinal cortex: Stimulus association related to reward schedules. J Neurophysiol 83:1677-1692. (Pubitemid 30159589)
42. Ljungberg T, Apicella P, Schultz W. 1992. Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophysiol 67:145-163.
43. McClure SM, Berns GS, Montague PR. 2003. Temporal prediction errors in a passive learning task activate human striatum. Neuron 38:339-346. (Pubitemid 36555658)
44. McClure SM, York MK, Montague PR. 2004. The neural substrates of reward processing in humans: The modern role of fMRI. Neuroscientist 10:260-268. (Pubitemid 38657092)
45. McCullagh P, Nelder JA. 1989. Generalized Linear Models,2nd ed. New York: Chapman & Hall.
46. Mehta MR, Barnes CA, McNaughton BL. 1997. Experience-dependent, asymmetric expansion of hippocampal place fields. PNAS 94:8918-8921. (Pubitemid 27335140)
47. Montague PR, Dayan P, Person C, Sejnowski TJ. 1995. Bee foraging in uncertain environments using predictive hebbian learning. Nature 377:725-728. (Pubitemid 26450238)
48. Montague PR, Dayan P, Sejnowski TJ. 1996. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16:1936-1947. (Pubitemid 26145969)
49. Montague PR, Hyman SE, Cohen JD. 2004. Computational roles for dopamine in behavioral control. Nature 431:760-767. (Pubitemid 39434064)
50. Montague PR, King-Casas B, Cohen JD. 2006. Imaging valuation models in human choice. Annu Rev Neurosci 29:417-448. (Pubitemid 44476852)
51. Niv Y, Duff MO, Dayan P. 2005. Dopamine, uncertainty and TD learning. Behav Brain Funct 1:6.
52. O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ. 2003. Temporal difference models and reward-related learning in the human brain. Neuron 28:329-337. (Pubitemid 36555657)
53. O'Doherty JP, Buchanan TW, Seymour B, Dolan RJ. 2006. Predictive neural coding of reward preference involves dissociable responses in human ventral midbrain and ventral striatum. Neuron 49:157- 166. (Pubitemid 43017193)
54. Okatan M, Wilson MA, Brown EN. 2005. Analyzing functional connectivity using a network likelihood model of ensemble neural spiking activity. Neural Comput 17:1927-1961. (Pubitemid 41057057)
55. O'Keefe J, Dostrovsky J. 1971. The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Res 34:171-175.
56. O'Keefe J, Nadel L. 1978. The Hippocampus as a Cognitive Map. Oxford: Clarendon Press.
57. Pastalkova E, Itskov V, Amarasingham A, Buzsa'ki G. 2008. Internally generated cell assembly sequences in the rat hippocampus. Science 321:1322-1327.
58. Rolls ET. 1999. Spatial view cells and the representation of place in the primate hippocampus. Hippocampus 9:467-480. (Pubitemid 29419633)
59. Rolls ET, Xiang JZ. 2005. Reward-spatial view representations and learning in the primate hippocampus. J Neurosci 25:6167- 6174. (Pubitemid 40923483)
60. Romo R, Schultz W. 1990. Dopamine neurons of the monkey midbrain: Contingencies of responses to active touch during self-initiated arm movements. J Neurophysiol 63:592-606. (Pubitemid 20115486)
61. Samejima K, Ueda Y, Doya K, Kimura M. 2005. Representation of action-specific reward values in the striatum. Science 310:1337- 1340. (Pubitemid 41698867)
62. Sappington BF, Goldman L. 1994. Discrimination learning and concept formation in the Arabian horse. J Anim Sci 72:3080-3087.
63. Schacter DL, Tulving E. 1994. What are the memory systems of 1994? In: Schacter DL, Tulving E, Editors. Memory Systems. Cambridge: MIT Press. p 1-38.
64. Schoenbaum G, Chiba AA, Gallagher M. 1999. Neural encoding in orbitofrontal cortex and basolateral amygdala during olfactory discrimination learning. J Neurosci 19:1876-1884. (Pubitemid 29094203)
65. Schultz W. 1998. Predictive reward signal of dopamine neurons. J Neurophysiol 80:1-27. (Pubitemid 28326237)
66. Schultz W. 2000. Multiple reward signals in the brain. Nat Rev Neurosci 1:199-207.
67. Schultz W. 2004. Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology. Curr Opin Neurobiol 14:139-147. (Pubitemid 38479929)
68. Schultz W, Apicella P, Ljungberg T. 1993. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J Neurosci 13:900-913. (Pubitemid 23067580)
69. Schultz W, Dayan P, Montague RR. 1997. A neural substrate of prediction and reward. Science 275:1593-1599. (Pubitemid 27120526)
70. Seymour B, O'Doherty JP, Dayan P, Koltzenburg M, Jones AK, Dolan RJ, Friston KJ, Frackowiak RS. 2004. Temporal difference models describe higher-order learning in humans. Nature 429:664-667. (Pubitemid 38812318)
71. Siegel S, Castellan NJ. 1988. Nonparametric Statistics for the Behavioral Sciences. New York: McGraw-Hill.
72. Singh S, Bertsekas D. 1997. Reinforcement learning for dynamic channel allocation in cellular telephone systems. In Advances in Neural Information Processing Systems: Proceedings of the 1996 Conference. Cambridge, MA: MIT Press. p 974-980.
73. Smith AC, Brown EN. 2003. State-space estimation from point process observations. Neural Comput 15:965-991. (Pubitemid 37049798)
74. Smith AC, Frank LM, Wirth S, Yanike M, Hu D, Kubota Y, Graybiel AM, Suzuki WA, Brown EN. 2004. Dynamic analysis of learning in behavioral experiments. J Neurosci 24:447-461. (Pubitemid 38112376)
75. Snyder D, Miller M. 1991. Random Point Processes in Time and Space 2nd ed. New York: Springer-Verlag.
76. Squire LR, Knowlton B, Musen G. 1993. The structure and organization of memory. Annu Rev Psychol 44:453-495.
77. Suri RE, Schultz W. 1999. A neural network model with dopaminelike reinforcement signal that learns a spatial delayed response task. Neuroscience 91:871-890. (Pubitemid 29225621)
78. Suri RE, Schultz W. 2001. Temporal difference model reproduces anticipatory neural activity. Neural Comput 13:841-862. (Pubitemid 33594307)
79. Sutton RS. 1988. Learning to predict by the method of temporal difference. Machine Learn 3:9-44.
80. Sutton RS, Barto AG. 1981. Toward a modern theory of adaptive networks: Expectation and prediction. Psychol Rev 88:135-170.
81. Sutton RS, Barto AG. 1987. A temporal-difference model of classical conditioning. Proceedings of the Ninth Annual Conference of the Cognitive Science Society, Seattle, WA. pp 355-378.
82. Sutton RS, Barto AG. 1998. Reinforcement Learning: An Introduction. Cambridge: MIT Press. p 261-290.
83. Tanaka SC, Doya K, Okada G, Ueda K, Okamoto Y, Yamawaki S. 2004. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat Neurosci 7:887-893. (Pubitemid 38989513)
84. Tesauro G. 1994. TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Comput 6:215-219.
85. Waelti P, Dickinson A, Schultz W. 2001. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412:43- 48. (Pubitemid 32851502)
86. Wilson MA, McNaughton BL. 1993. Dynamics of the hippocampal ensemble code for space. Science 261:1055-1058. (Pubitemid 23300867)
87. Wirth S, Yanike M, Frank LM, Smith AC, Brown EN, Suzuki WA. 2003. Single neurons in the monkey hippocampus and learning of new associations. Science 300:1578-1581. (Pubitemid 36682890)
88. Yin HH, Knowlton BJ. 2006. The role of the basal ganglia in habit formation. Nat Rev Neurosci 7:464-476. (Pubitemid 44012286)