Analyses of Markov decision process structure regarding the possible strategic use of interacting memory systems

Frontiers in Computational Neuroscience

Volumn 2, Issue DEC, 2008, Pages

  Zilli, Eric A ^a   Hasselmo, Michael E ^a  

^a BOSTON UNIVERSITY   (United States)

Author keywords

Content addressable sequential retrieval; Gated active maintenance; Multiple memory systems; Partially observable Markov decision process; Reinforcement learning

Indexed keywords

CONTENT-ADDRESSABLE SEQUENTIAL RETRIEVAL; EPISODIC-LIKE MEMORY; INTERMEDIATE LEVEL; MARKOV DECISION PROCESSES; MEMORY STRATEGIES; MEMORY SYSTEMS; PARTIALLY OBSERVABLE MARKOV DECISION PROCESS; SEMANTIC ASSOCIATIONS;

MARKOV PROCESSES; REINFORCEMENT LEARNING;

SEMANTICS;

EID: 69949100349     PISSN: 16625188     EISSN: None     Source Type: Journal    
DOI: 10.3389/neuro.10.006.2008     Document Type: Article

References (44)

1. Baddeley, A. D. (1986). Working Memory. Oxford, Clarendon Press.
2. Baddeley, A. D., and Hitch, G. (1974). Working memory. In The Psychology of Learning and Motivation: Advances in Research and Theory, G. H. Bower, (New York, Academic Press), pp. 47-89.
3. Cabeza, R. (2008). Role of parietal regions in episodic memory retrieval: the dual attentional processes hypothesis. Neuropsychologia 46, 1813-1827.
4. Ciaramelli, E., Grady, C. L., and Moscovitch, M. (2008). Top-down and bottom-up attention to memory: a hypothesis (AtoM) on the role of the posterior parietal cortex in memory retrieval. Neuropsychologia 46, 1828-1851.
5. Cowan, N. (2001). The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behav. Brain Sci. 24, 87-185.
6. Dayan, P. (1992). The convergence of TD(T) for general). Mach. Learn. 8, 341-362.
7. Dayan, P. (2007). Bilinearity, rules, and prefrontal cortex. Front. Comput. Neurosci. 1, 1.
8. Deco G., and Rolls, E. T. (2005). Synaptic and spiking dynamics underlying reward reversal in the orbitofrontal cortex. Cereb. Cortex 15, 15-30.
9. Drosopoulos, S., Windau, E., Wagner, U., and Born, J. (2007). Sleep enforces the temporal order in memory. PLoS ONE 2, e376. doi: 10.1371/journal. pone.0000376.
10. Eichenbaum, H., and Cohen, N. J. (2001). From Conditioning to Conscious Recollection. New York, Oxford University Press.
11. Fass, C., Turtle, B., and Ginelli, M. (1996). Six degrees of Kevin Bacon. New York, Plume.
12. Frank, M. J., Loughry, B., and O'Reilly, R. C. (2001). Interactions between frontal cortex and basal ganglia in working memory: a computational model. Cogn. Affect. Behav. Neurosci. 1, 137-160.
13. Fransén, E., Alonso, A. A., and Hasselmo, M. E. (2002). Simulations of the role of the muscarinic-activated calcium-sensitive nonspecific cation current INCM in entorhinal neuronal activity during delayed matching tests. J. Neurosci. 22, 1081-1097.
14. Fuster, J. M. (1995). Memory in the Cerebral Cortex. Cambridge, MA, MIT Press.
15. Hasselmo, M. E. (2007). Arc length coding by interference of theta frequency oscillations may underlie contextdependent hippocampal unit data and episodic memory function. Learn. Mem. 14, 782-794.
16. Hasselmo, M. E., and Eichenbaum, H. (2005). Hippocampal mechanisms for the context-dependent retrieval of episodes. Neural Netw. 18, 1172-1190.
17. Jensen, O., and Lisman, J. E. (2005). Hippocampal sequence-encoding driven by a cortical multi-item working memory buffer. Trends Neurosci. 28, 67-72.
18. Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artif. Intell. 101, 99-134.
19. Li, S.-C., and Lewandowsky, S. (1995). Forward and backward recall: different retrieval processes. J. Exp. Psychol. Learn. Mem. Cogn. 21, 837-847.
20. Lisman, J. E., Fellous, J. M., and Wang, X. J. (1998). A role for NMDA-receptor channels in working memory. Nat. Neurosci. 1, 273-275.
21. Marr, D. (1982). Vision. San Francisco, W. H. Freeman.
22. Miller, E. K., Erickson, C. A., and Desimone, R. (1996). Neural mechanisms of visual working memory in prefrontal cortex of the macaque. J. Neurosci. 16, 5154-5167.
23. Miller, G. A. (1956). The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychol. Rev. 63, 81-97.
24. Miyake, A., and Shah, P. (1999). Models of working memory: mechanisms of active maintenance and executive control. New York, Cambridge University Press.
25. Monahan, G. E. (1982). A survey of partially observable Markov decision processes. Manage. Sci. 28, 16.
26. Moustafa, A. A., and Maida, A. S. (2007). Using TD learning to simulate working memory performance in a model of the prefrontal cortex and basal ganglia. Cogn. Syst. Res. 8, 262-281.
27. O'Reilly, R. C., and Frank, M. J. (2006). Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput. 18, 283-328.
28. Phillips, J. L., and Noelle, D. C. (2005). A biologically inspired working memory framework for robots. In Proceedings of the 27th Annual Meeting of the Cognitive Science Society, Stresa, Italy.
29. Polyn, S. M., Natu, V. S., Cohen, J.D., and Norman, K. A. (2005). Categoryspecific cortical activity precedes retrieval during memory search. Science 310, 1963-1966.
30. Prescott, T. J., Gurney, K., and Redgrave, P. (2003). Basal ganglia. In The Handbook of Brain Theory and Neural Networks, 2nd, M. A. Arbib (Cambridge, MA, MIT Press), pp. 147-151.
31. Schacter, D. L., and Tulving, E. (1994). What are the memory systems of 1994? In Memory Systems, D. L. Schacter and E. Tulving, eds. (Cambridge, MA, MIT Press), pp. 1-38.
32. Schwartz, B. L., Hoffman, M. L., and Evans, S. (2005). Episodic-like memory in a gorilla: a review and new findings. Learn. Motiv. 36, 226-244.
33. Squire, L. R. (2004). Memory systems of the brain: a brief history and current perspective. Neurobiol. Learn. Mem. 82, 171-177.
34. Squire, L. R., and Alvarez, P. (1995). Retrograde amnesia and memory consolidation: a neurobiological perspective. Curr. Opin. Neurobiol. 5, 169-77.
35. Squire, L. R., and Zola-Morgan, S. (1991). The medial temporal lobe memory system. Science 253, 1380-1386.
36. Sutton, R. S., and Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MIT Press.
37. Tsitsiklis J. N. (1994). Asynchronous stochastic approximation and Q-learning. Mach. Learn. 16, 185-202.
38. Tulving, E. (1972). Episodic and semantic memory. In Organization of Memory, E. Tulving and W. Donaldson, eds (New York, Academic Press), pp. 381-403.
39. Tulving, E. (1985). How many memory systems are there? Am. Psychol. 40, 385-398.
40. Tulving, E. (2002). Episodic memory: from mind to brain. Annu. Rev. Psychol. 53, 1-25.
41. Watkins, C. J. C. H., and Dayan, P. (1992). Q-learning. Mach. Learn. 8, 279-292.
42. Zilli, E. A., and Hasselmo, M. E. (2008a). The influence of Markov decision process structure on the possible strategic use of working memory and episodic memory. PLoS ONE 3, e2756.
43. Zilli, E. A., and Hasselmo, M. E. (2008b). Modeling the role of working memory and episodic memory in behavioral tasks. Hippocampus 18, 193-209.
44. Zipser, D., Kehoe, B., Littlewort, G., and Fuster, J. (1993). A spiking network model of short-term active memory. J. Neurosci. 13, 3406-3420.