Active inference, epistemic value, and vicarious trial and error

Learning and Memory

Volumn 23, Issue 7, 2016, Pages 322-338

  Pezzulo, Giovanni ^a   Cartoni, Emilio ^a,d   Rigoli, Francesco ^b   Pio Lopez, Léo ^c   Friston, Karl ^b  

^a INSTITUTE OF COGNITIVE SCIENCES AND TECHNOLOGIES   (Italy)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c UNIVERSITÉ BLAISE PASCAL   (France)

^d SAPIENZA UNIVERSITY OF ROME   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

ACTIVE INFERENCE; ANIMAL MODEL; ARTICLE; ASSOCIATION; BEHAVIOR; EPISODIC MEMORY; EPISTEMIC VALUE; EXPLORATORY BEHAVIOR; INFORMATION; MAZE TEST; NEUROPHYSIOLOGY; NONHUMAN; PRIORITY JOURNAL; RAT; REWARD; VICARIOUS TRIAL AND ERROR BEHAVIOR; ANIMAL; BAYES THEOREM; DECISION MAKING; HUMAN; PSYCHOLOGICAL MODEL;

ANIMALS; BAYES THEOREM; DECISION MAKING; HUMANS; MAZE LEARNING; MODELS, PSYCHOLOGICAL; RATS; REWARD;

EID: 84978410979     PISSN: 10720502     EISSN: 15495485     Source Type: Journal    
DOI: 10.1101/lm.041780.116     Document Type: Article

References (87)

1. Attias H. 2003. Planning by Probabilistic Inference. In: Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics. Key West, FL.
2. Balleine BW, Dickinson A. 1998. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37:407-419.
3. Beal MJ. 2003. Variational algorithms for approximate Bayesian inference. University of London, London, UK.
4. Blumenthal A, Steiner A, Seeland K, Redish AD. 2011. Effects of pharmacological manipulations of NMDA-receptors on deliberation in the multiple-T task. Neurobiol Learn Mem 95:376-384.
5. Botvinick M, Toussaint M. 2012. Planning as inference. Trends Cogn Sci 16:485-488.
6. Bower GH. 1959. Choice-point behavior. In Studies of mathematical learning (ed. Bush RR, Estes WK), pp. 109-124. Stanford University Press, Stanford, CA.
7. Buckner RL. 2010. The role of the hippocampus in prediction and imagination. Annu Rev Psychol 61:27-48, C1-8.
8. Charnov E. 1976. Optimal foraging, the marginal value theorem. Theor Popul Biol 9:129-136.
9. Cohen JD, McClure SM, Yu AJ. 2007. Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philos Trans R Soc Lond B Biol Sci 362:933-942.
10. Collins AG, Frank MJ. 2013. Cognitive control over learning: creating, clustering, and generalizing task-set structure. Psychol Rev 120:190.
11. Collins A, Koechlin E. 2012. Reasoning, learning, and creativity: frontal lobe function and human decision-making. PLoS Biol 10:e1001293.
12. Daw ND, Niv Y, Dayan P. 2005. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8:1704-1711.
13. Daw ND, O'Doherty JP, Dayan P, Seymour B, Dolan RJ. 2006. Cortical substrates for exploratory decisions in humans. Nature 441:876-879.
14. Dayan P, Hinton GE. 1997. Using expectation-maximization for reinforcement learning. Neural Comput 9:271-278.
15. Dayan P, Sejnowski TJ. 1996. Exploration bonus and dual control. Mach Learn 25:5-22.
16. Dayan P, Hinton GE, Neal RM, Zemel RS. 1995. The Helmholtz machine. Neural Comput 7:889-904.
17. Diba K, Buzsáki G. 2007. Forward and reverse hippocampal place-cell sequences during ripples. Nat Neurosci 10:1241-1242.
18. Dolan RJ, Dayan P. 2013. Goals and habits in the brain. Neuron 80:312-325.
19. Donnarumma F, Maisto D, Pezzulo G. 2016. Problem solving as probabilistic inference with subgoaling: explaining human successes and pitfalls in the Tower of Hanoi. PLoS Comput Biol 12:e1004864.
20. Dragoi G, Tonegawa S. 2011. Preplay of future place cell sequences by hippocampal cellular assemblies. Nature 469:397-401.
21. Eichenbaum H. 2000. A cortical-hippocampal system for declarative memory. Nat Rev Neurosci 1:41-50.
22. Feng T, Silva D, Foster DJ. 2015. Dissociation between the experience-dependent development of hippocampal θ sequences and single-trial phase precession. J Neurosci 35:4890-4902.
23. Foster DJ, Wilson MA. 2007. Hippocampal θ sequences. Hippocampus 17:1093-1099.
24. Friston K. 2010. The free-energy principle: a unified brain theory? Nat Rev Neurosci 11:127-138.
25. Friston K, Adams RA, Perrinet L, Breakspear M. 2012a. Perceptions as hypotheses: saccades as experiments. Front Psychol 3:151.
26. Friston K, Samothrakis S, Montague R. 2012b. Active inference and agency: optimal control without cost functions. Biol Cybern 106:523-541.
27. Friston K, Schwartenbeck P, Fitz Gerald T, Moutoussis M, Behrens T, Dolan RJ. 2013. The anatomy of choice: active inference and agency. Front Hum Neurosci 7:598.
28. Friston K, Schwartenbeck P, FitzGerald T, Moutoussis M, Behrens T, Dolan RJ. 2014. The anatomy of choice: dopamine and decision-making. Philos Trans R Soc Lond B Biol Sci 369:20130481.
29. Friston K, Rigoli F, Ognibene D, Mathys C, Fitzgerald T, Pezzulo G. 2015. Active inference and epistemic value. Cogn Neurosci 6:187-214.
30. Gallistel CR. 2012. Extinction from arationalist perspective. Behav Processes 90:66-80.
31. Gallistel CR, Matzel LD. 2013. The neuroscience of learning: beyond the Hebbian synapse. Annu Rev Psychol 64:169-200.
32. Gershman SJ, Niv Y. 2012. Exploring a latent cause theory of classical conditioning. Learn Behav 40:255-268.
33. Graybiel AM. 2008. Habits, rituals, and the evaluative brain. Annu Rev Neurosci 31:359-387.
34. Gupta AS, van der Meer MAA, Touretzky DS, Redish AD. 2010. Hippocampal replay is not a simple function of experience. Neuron 65:695-705.
35. Gupta AS, van der Meer MA, Touretzky DS, Redish AD. 2012. Segmentation of spatial experience by hippocampal theta sequences. Nat Neurosci 15:1032-1039.
36. Hassabis D, Maguire EA. 2009. The construction system of the brain. Philos Trans R Soc Lond B Biol Sci 364:1263-1271.
37. Helmholtz H. von. 1866. Concerning the perceptionsin general. In Treatise on physiological optics (ed. Southall JPC). Dover, New York.
38. Hills TT, Todd PM, Lazer D, Redish AD, Couzin ID. 2015. Exploration versus exploitation in space, mind, and society. Trends Cogn Sci 19:46-54.
39. Hobson JA, Friston KJ. 2012. Waking and dreaming consciousness: neurobiological and functional considerations. Prog Neurobiol 98:82-98.
40. Howard R. 1966. Information value theory. IEEE Trans Syst Sci Cybern 2:22-26.
41. Hu DAN, Amsel A. 1995. A simple test of the vicarious trial-and-error hypothesis ofhippocampal function. Proc Natl Acad Sci 92:5506-5509.
42. Johnson A, Redish AD. 2007. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J Neurosci 27:12176-12189.
43. Johnson A, van der Meer MA, Redish AD. 2007. Integrating hippocampus and striatum in decision-making. Curr Opin Neurobiol 17:692-697.
44. Johnson A, Varberg Z, Benhardus J, Maahs A, Schrater P. 2012. The hippocampus and exploration: dynamically evolving behavior and neural representations. Front Hum Neurosci 6:216.
45. Kappen HJ, Gómez V, Opper M. 2012. Optimal controlas a graphical model inference problem. Mach Learn 87:159-182.
46. Keramati M, Dezfouli A, Piray P. 2011. Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput Biol 7:e1002055.
47. Lansink CS, Goltstein PM, Lankelma JV, McNaughton BL, Pennartz CM. 2009. Hippocampus leads ventral striatum in replay of place-reward information. PLoS Biol 7:e1000173.
48. Lee SW, Shimojo S, O'Doherty JP. 2014. Neural computations underlying arbitration between model-based and model-free learning. Neuron 81:687-699.
49. Lengyel M, Dayan P. 2008. Hippocampal contributions to control: the third way. In Advances in neural information processing systems 20 (ed. Platt JC, Koller D, Singer Y, Roweis S), pp. 889-896. MIT Press, Cambridge, MA.
50. Lisman JE, Otmakhova NA. 2001. Storage, recall, and novelty detection of sequences by the hippocampus: elaborating on the SOCRATIC model to account for normal and aberrant effects of dopamine. Hippocampus 11:551-568.
51. Maisto D, Donnarumma F, Pezzulo G. 2016. Nonparametric problem-space clustering: learning efficient codes for cognitive control tasks. Entropy 18:61.
52. McClelland JL, McNaughton BL, O'Reilly RC. 1995. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol Rev 102:419-457.
53. Muenzinger KF. 1938. Vicarious trial and error at a point of choice: I. A general survey of its relation to learning efficiency. Pedagog. Semin J Genet Psychol 53:75-86.
54. O'Doherty JP, Lee SW, McNamee D. 2015. The structure of reinforcement-learning mechanisms in the human brain. Curr Opin Behav Sci 1:94-100.
55. O'keefe J, Nadel L. 1978. The hippocampus as a cognitive map. Clarendon Press Oxford.
56. Papale AE, Stott JJ, Powell NJ, Regier PS, Redish AD. 2012. Interactions between deliberation and delay-discounting in rats. Cogn Affect Behav Neurosci 12:513-526.
57. Pennartz CMA, Ito R, Verschure PFMJ, Battaglia FP, Robbins TW. 2011. The hippocampal-striatal axis in learning, prediction and goal-directed behavior. Trends Neurosci 34:548-559.
58. Penny WD, Zeidman P, Burgess N. 2013. Forward and backward inference in spatial cognition. PLoS Comput Biol 9:e1003383.
59. Pezzulo A, Cisek P. 2016. Navigating the affordance landscape: feedback control as a process model of behavior and cognition. Trends Cogn Sci, doi:10.1016/j.tics.2016.03.013.
60. Pezzulo G, Rigoli F, Chersi F. 2013. The mixed instrumental controller: using value of information to combine habitual choice and mental simulation. Front Cogn 4:92.
61. Pezzulo G, van der Meer MAA, Lansink CS, Pennartz CMA. 2014. Internally generated sequences in learning and executing goal-directed behavior. Trends Cogn Sci 18:647-657.
62. Pezzulo G, Rigoli F, Friston KJ. 2015. Active Inference, homeostatic regulation and adaptive behavioural control. Prog Neurobiol 134:17-35.
63. Pfeiffer BE, Foster DJ. 2013. Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497:74-79.
64. Redish AD. 2013. The mind within the brain: how we make decisions and how those decisions go wrong, 1st ed. Oxford University Press, Oxford.
65. Regier PS, Amemyia S, Redish AD. 2015. Hippocampus and subregions of the dorsal striatum respond differently to a behavioral strategy change on a spatial navigation task. J Neurophysiol 114:1399-1416.
66. Rushworth MFS, Behrens TEJ. 2008. Choice, uncertainty and value in prefrontal and cingulate cortex. Nat Neurosci 11:389-397.
67. Schacter DL, Addis DR. 2007. The cognitive neuroscience of constructive memory: remembering the past and imagining the future. Philos Trans R Soc Lond B Biol Sci 362:773-786.
68. Schacter DL, Addis DR, Buckner RL. 2007. Remembering the past to imagine the future: the prospective brain. Nat Rev Neurosci 8:657-661.
69. Schacter DL, Addis DR, Hassabis D, Martin VC, Spreng RN, Szpunar KK. 2012. The future of memory: remembering, imagining, and the brain. Neuron 76:677-694.
70. Schmidt B, Papale A, Redish AD, Markus EJ. 2013. Conflict between place and response navigation strategies: effects on vicarious trial and error (VTE) behaviors. Learn Mem 20:130-138.
71. Schrier AM, Povar ML. 1979. Eye movements of stumptailed monkeys during discrimination learning: VTE revisited. Anim Learn Behav 7:239-245.
72. Solway A, Botvinick MM. 2012. Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates. Psychol Rev 119:120-154.
73. Spence KW. 1960. Conceptional models of spatial and non-spatial selective learning. Prentice-Hall, Englewood Cliffs, NJ.
74. Steiner AP, Redish AD. 2014. Behavioral and neurophysiological correlates of regret in rat decision-making on a neuroeconomic task. Nat Neurosci 17:995-1002.
75. Stoianov I, Genovesio A, Pezzulo G. 2015. Prefrontal goal codes emerge as latent states in probabilistic value learning. J Cogn Neurosci 28:140-157.
76. Sutton RS. 1990. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In: Proceedings of the Seventh International Conference on Machine Learning. Morgan Kaufmann, San Mateo, CA, pp. 216-224.
77. Tolman EC. 1938. The determiners of behavior at a choice point. Psychol Rev 46:318-336.
78. Tolman EC. 1939. Prediction of vicarious trial and error by means of the schematic sowbug. Psychol Rev 46:318.
79. Tolman EC. 1948. Cognitive maps in rats and men. Psychol Rev 55:189-208.
80. Tulving E. 1972. Episodic and semantic memory, In Organization of memory (ed. Tulving E, Donaldson W). Academic Press, New York.
81. van der Meer MAA, Redish AD. 2010. Expectancies in decision making, reinforcement learning, and ventral striatum. Front Neurosci 4:6.
82. van der Meer MAA, Johnson A, Schmitzer-Torbert NC, Redish AD. 2010. Triple dissociationofinformation processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task. Neuron 67:25-32.
83. van der Meer M, Kurth-Nelson Z, Redish AD. 2012. Information processing in decision-making systems. The Neuroscientist 18:342-359.
84. Vinogradova OS. 2001. Hippocampus as comparator: role of the two input and two output systems of the hippocampus in selection and registration of information. Hippocampus 11:578-598.
85. Voss JL, Warren DE, Gonsalves BD, Federmeier KD, Tranel D, Cohen NJ. 2011. Spontaneous revisitation during visual exploration as a link among strategic behavior, learning, and the hippocampus. Proc Natl Acad Sci 108:E402-E409.
86. Wikenheiser AM, Redish AD. 2013. The balance of forward and backward hippocampal sequences shifts across behavioral states. Hippocampus 23:22-29.
87. Yin HH, Knowlton BJ. 2006. The role of the basal ganglia in habit formation. Nat Rev Neurosci 7:464-476.