The successor representation and temporal context

Neural Computation

Volumn 24, Issue 6, 2012, Pages 1553-1568

  Gershman, Samuel J ^a   Moore, Christopher D ^a   Todd, Michael T ^a   Norman, Kenneth A ^a   Sederberg, Per B ^b  

^a PRINCETON UNIVERSITY   (United States)

^b OHIO STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTICLE; LEARNING; PHYSIOLOGY; PSYCHOLOGICAL MODEL; REPRODUCIBILITY; TIME;

ALGORITHMS; LEARNING; MODELS, PSYCHOLOGICAL; REPRODUCIBILITY OF RESULTS; TIME FACTORS;

EID: 84865306110     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/NECO_a_00282     Document Type: Letter

References (58)

1. Addis, D., Wong, A., & Schacter, D. (2007). Remembering the past and imagining the future: Common and distinct neural substrates during event construction and elaboration. Neuropsychologia, 45, 1363-1377.
2. Altmann, G., & Kamide, Y. (1999). Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73, 247-264.
3. Anderson, J., & Bower, G. (1972). Recognition and retrieval processes in free recall. Psychological Review, 79, 97-123.
4. Atance, C.,&O'Neill, D. (2001). Episodic future thinking. Trends in Cognitive Sciences, 5, 533-539.
5. Bellman, R. (1957). Dynamic programming. Princeton, NJ: Princeton University Press.
6. Botvinick, M., Niv, Y., & Barto, A. (2009). Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition, 113, 262- 280.
7. Brown, G., Neath, I., & Chater, N. (2007). A temporal ratio model of memory. Psychological Review, 114, 539-576.
8. Brown, S.D.,&Heathcote, A. (2008). The simplest complete model of choice response time: Linear ballistic accumulation. Cognitive Psychology, 57, 153-178.
9. Brown, T., Kairiss, E., & Keenan, C. (1990). Hebbian synapses: Biophysical mechanisms and algorithms. Annual Review of Neuroscience, 13, 475-511.
10. Byrne, P., Becker, S., & Burgess, N. (2007). Remembering the past and imagining the future: A neural model of spatialmemory and imagery. Psychological Review, 114, 340-375.
11. Dayan, P. (1993). Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5, 613-624.
12. Deese, J., & Kaufman, R. (1957). Serial effects in recall of unorganized and sequentially organized verbal material. Journal of Experimental Psychology, 54, 180-187.
13. Doya, K. (2002). Metalearning and neuromodulation. Neural Networks, 15, 495-506.
14. Estes, W. (1955). Statistical theory of spontaneous recovery and regression. Psychological Review, 62, 145-154.
15. Glascher, J., Daw, N., Dayan, P., & O'Doherty, J. (2010). States versus rewards: Dissociable neural prediction error signals underlying model-based and modelfree reinforcement learning. Neuron, 66, 585-595.
16. Glenberg, A., & Swanson, N. (1986). A temporal distinctiveness theory of recency and modality effects. Journal of Experimental Psychology: Learning, Memory, and Cognition, 12, 3-15.
17. Hassabis, D., Kumaran,D.,Vann, S.,&Maguire, E. (2007). Patientswith hippocampal amnesia cannot imagine new experiences. Proceedings of the National Academy of Sciences, 104, 1726-1731.
18. Hassabis, D.,&Maguire, E. (2009). The construction system of the brain. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1521), 1263-1271.
19. Howard, M., Fotedar, M., Datey, A., & Hasselmo, M. (2005). The temporal context model in spatial navigation and relational learning: Explaining medial temporal lobe function across domains. Psychological Review, 112, 75-116.
20. Howard, M., Jing, B., Rao, V., Provyn, J., & Datey, A. (2009). Bridging the gap: Transitive associations between items presented in similar temporal contexts. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 391-407.
21. Howard, M., & Kahana, M. (1999). Contextual variability and serial position effects in free recall. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 923-941.
22. Howard,M., & Kahana, M. (2002). A distributed representation of temporal context. Journal of Mathematical Psychology, 46, 269-299.
23. Howard, M., Shankar, K., & Jagadisan, U. (2011). Constructing semantic representations from a gradually-changing representation of temporal context. Topics in Cognitive Science, 3, 48-73.
24. Jones, R. (1993). Entorhinal-hippocampal connections: A speculative view of their function. Trends in Neurosciences, 16, 58-64.
25. Kahana,M. (1996). Associate retrieval processes in free recall. Memory and Cognition, 24, 103-109.
26. Kahana, M., & Howard, M. (2005). Spacing and lag effects in free recall of pure lists. Psychonomic Bulletin and Review, 12, 159-164.
27. Kahana, M., Howard, M., & Polyn, S. (2008). Associative retrieval processes in episodic memory. In H. Roediger (Ed.), Learning and memory: A comprehensive reference. Orlando, FL: Academic Press.
28. Kamide, Y., Altmann, G., & Haywood, S. (2003). The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye movements. Journal of Memory and Language, 49, 133-156.
29. Kemeny, J., & Snell, J. (1976). Finite Markov chains. New York: Springer.
30. Kumaran, D., & Maguire, E. (2007). Which computational mechanisms operate in the hippocampus during novelty detection? Hippocampus, 17,735-748.
31. Levy,W., Hocking, A., &Wu, X. (2005). Interpreting hippocampal function as recoding and forecasting. Neural Networks, 18, 1242-1264.
32. Lisman, J., & Otmakhova, N. (2001). Storage, recall, and novelty detection of sequences by the hippocampus: elaborating on the SOCRATIC model to account for normal and aberrant effects of dopamine. Hippocampus, 11, 551-568.
33. Lisman, J., & Redish, A. (2009). Prediction, sequences & the hippocampus. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1521), 1193-1201.
34. Mahadevan, S., & Maggioni, M. (2007). Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes. Journal of Machine Learning Research, 8, 2169-2231.
35. Mensink, G., & Raaijmakers, J. (1988). A model for interference and forgetting. Psychological Review, 95, 434-455.
36. Mozer, M., Pashler, H., Cepeda, N., Lindsey, R., & Vul, E. (2009). Predicting the optimal spacing of study: A multiscale context model of memory. Advances in neural information processing systems, 22 (pp. 1321-1329). Cambridge, MA: MIT Press.
37. Polyn, S., Norman, K., & Kahana, M. (2009). A context maintenance and retrieval model of organizational processes in free recall. PsychologicalReview, 116, 129-156.
38. Rainer, G., Rao, S., & Miller, E. (1999). Prospective coding for objects in primate prefrontal cortex. Journal of Neuroscience, 19, 5493-5505.
39. Rescorla, R. A., & Wagner, A. R. (1972). A theory of of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In A. Black &W. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64-99). New York: Appleton-Century-Crofts.
40. Ribas-Fernandes, J., Solway,A.,Diuk,C.,McGuire, J., Barto,A.,Niv,Y., Botvinick,M., et al. (2011). A neural signature of hierarchical reinforcement learning. Neuron, 71, 370-379.
41. Sakai, K., & Miyashita, Y. (1991). Neural organization for the long-term memory of paired associates. Nature, 354(6349), 152-155.
42. Schacter, D., Addis, D., & Buckner, R. (2007). Remembering the past to imagine the future: the prospective brain. Nature Reviews Neuroscience, 8, 657-661.
43. Schank, R. (1982). Dynamic memory. Cambridge: Cambridge University Press.
44. Sederberg, P., Howard, M., & Kahana, M. (2008). A context-based theory of recency and contiguity in free recall. Psychological Review, 115, 893-912.
45. Sederberg, P., Miller, J., Howard, M., & Kahana, M. (2010). Temporal contiguity between recalls predicts episodic memory performance. Psychonomic Bulletin and Review, 38, 689-699.
46. Shankar, K., Jagadisan, U.,&Howard,M. (2009). Sequential learning using temporal context. Journal of Mathematical Psychology, 53, 474-485.
47. Socher, R., Gershman, S., Perotte, A., Sederberg, P., Blei, D., & Norman, K. (2009). A bayesian analysis of dynamics in free recall. In Y. Bengio, D. Schuurmans, J. Lafferty, C.K.I. Williams, & A. Culotta (Eds.), Advances in neural information processing systems, 22 (pp. 1714-1722). Cambridge, MA: MIT Press.
48. Sutton,R. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9-44.
49. Sutton, R. (1995). TD models: Modeling the world at a mixture of time scales. In Proceedings of the International Conference of Machine Learning (pp. 531-539). San Francisco: Morgan Kaufmann.
50. Sutton, R., & Barto, A. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
51. Sutton, R., Precup, D., & Singh, S. (1999). Between MDPS and semi-MDPS: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112, 181-211.
52. Szpunar, K., Watson, J., & McDermott, K. (2007). Neural substrates of envisioning the future. Proceedings of the National Academy of Sciences, 104, 642-647.
53. Unsworth, N. (2007). Individual differences in working memory capacity and episodic retrieval: Examining the dynamics of delayed and continuous distractor free recall. Journal of Experimental Psychology: Learning, Memory, and Cognition, 33, 1020-1034.
54. Usher, M., & McClelland, J. L. (2001). The time course of perceptual choice: The leaky, competing accumulator model. Psychological Review, 108, 550-592.
55. Vinogradova, O. (2001). Hippocampus as comparator: Role of the two input and two output systems of the hippocampus in selection and registration of information. Hippocampus, 11, 578-598.
56. White, L. (1995). Temporal difference learning: Eligibility traces and the successor representation for actions. Unpublished master's thesis, University of Toronto.
57. Wingfield, A., Lindfield, K., & Kahana, M. (1998). Adult age differences in the temporal characteristics of category free recall. Psychology and Aging, 13, 256-266.
58. Zilli, E., & Hasselmo,M. (2008).Modeling the role of working memory and episodic memory in behavioral tasks. Hippocampus, 18, 193-209.