Model-based hierarchical reinforcement learning and human action control

Philosophical Transactions of the Royal Society B: Biological Sciences

Volumn 369, Issue 1655, 2014, Pages

  Botvinick, Matthew ^a   Weinstein, Ari ^a  

^a PRINCETON UNIVERSITY   (United States)

Author keywords

Goal directed behaviour; Hierarchy; Reinforcement learning

Indexed keywords

DECISION MAKING; HIERARCHICAL SYSTEM; HUMAN BEHAVIOR; LEARNING; NUMERICAL MODEL;

BIOLOGICAL MODEL; DECISION MAKING; HUMAN; LEARNING; MOTIVATION; PHYSIOLOGY; REINFORCEMENT;

DECISION MAKING; GOALS; HUMANS; LEARNING; MODELS, NEUROLOGICAL; REINFORCEMENT (PSYCHOLOGY);

EID: 84907487070     PISSN: 09628436     EISSN: 14712970     Source Type: Journal    
DOI: 10.1098/rstb.2013.0480     Document Type: Article

References (69)

1. Bromberg-Martin ES, Matsumoto M, Hikosaka O. 2010 Dopamine in motivational control: rewarding, aversive and alerting. Neuron 68, 815–834. (doi:10.1016/j.neuron.2010.11.022)
2. Dayan P, Balleine BW. 2002 Reward, motivation, and reinforcement learning. Neuron 36, 285–298. (doi:10.1016/S0896-6273(02)00963-7)
3. Daw ND, Niv Y, Dayan P. 2005 Uncertainty-based competition between prefrontal and striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711. (doi:10.1038/nn1560)
4. Dolan RJ, Dayan P. 2013 Goals and habits in the brain. Neuron 80, 312–325. (doi:10.1016/j.neuron.2013.09.007)
5. Doll BB, Simon DA, Daw ND. 2012 The ubiquity of model-based reinforcement learning. Curr. Opin. Neurobiol. 22, 1075–1081. (doi:10.1016/j.conb.2012.08.003)
6. Morris R, Ward G. 2004 The cognitive psychology of planning. Hove, UK: Psychology Press.
7. Solway A, Botvinick MM. 2012 Goal directed decision making as probabilistic inference: a computational framework and potential neural correlates. Psychol. Rev. 119, 120–154. (doi:10.1037/a0026435)
8. Daw ND, Dayan P. 2014 The algorithmic anatomy of model-based evaluation. Phil. Trans. R. Soc. B 369, 20130478. (doi:10.1098/rstb.2013.0478)
9. Barto A, Mahadevan S. 2003 Recent advances in hierarchical reinforcement learning. Discrete Event Dyn. Syst. 13, 341–379. (doi:10.1023/A:1025696116075)
10. Botvinick M., Niv Y, Barto AC. 2009 Hierarchically organized behavior and its neural foundations: a reinforcement-learning perspective. Cognition 113, 262–280. (doi:10.1016/j.cognition.2008.08.011)
11. Barto AG, Konidaris GD, Vigorito CM. 2013 Behavioral hierarchy: exploration and representation. In Computational and robotic models of hierarchical organization of behavior (eds G Baldassarre, M Mirolli), pp. 13–46. Heidelberg, Germany: Springer.
12. Dezfouli A, Balleine BW. 2012 Habits, action sequences and reinforcement learning. Eur. J. Neurosci. 35, 1036–1051. (doi:10.1111/j.1460-9568.2012.08050.x)
13. Shenhav A, Botvinick M., Cohen JD. 2013 The expected value of control: an integrative theory of anterior cingulate cortex function. Neuron 79, 217–240. (doi:10.1016/j.neuron.2013.07.007)
14. Holroyd CB, Yeung N. 2012 Motivation of extended behaviors by anterior cingulate cortex. Trends Cogn. Sci. 16, 122–128. (doi:10.1016/j.tics.2011.12.008)
15. Frank MJ, Badre D. 2012 Mechanisms of hierarchical reinforcement learning in corticostraital circuits 1: computational analysis. Cereb. Cortex 22, 509–526. (doi:10.1093/cercor/bhr114)
16. Sutton RS, Precup D, Singh S. 1999 Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif. Intell. 112, 181–211. (doi:10.1016/S0004-3702(99) 00052-1)
17. Silver D, Ciosek K. 2012 Compositional planning using optimal option models. (http://arxiv.org/abs/1206.6473).
18. Moore AW, Baird L, Kaelbling L. 1999 Multi-value-functions: efficient automatic action hierarchies for multiple goal MDPs. In Proc. Int. Joint Conf. on Artificial Intelligence, Stockholm, Sweden, 31 July 1999, pp. 1316–1323. San Francisco, CA: Morgan Kaufmann.
19. Mann TA, Mannor S. 2013 The advantage of planning with options. In First Multidisciplinary Conf. on Reinforcement Learning and Decision Making, Princeton, NJ, 25–27 October 2013.
20. Botvinick MM. 2012 Hierarchical reinforcement learning and decision making. Curr. Opin. Neurobiol. 22, 956–962. (doi:10.1016/j.conb.2012.05.008)
21. Otto AR, Gershman SJ, Markman AB, Daw ND. 2013 The curse of planning: dissecting multiple reinforcement-learning systems by taxing the central executive. Psychol. Sci. 24, 751–761. (doi:10.1177/0956797612463080)
22. Dietterich TG. 2000 State abstraction in MAXQ hierarchical reinforcement learning. In Advances in Neural Information Processing, Denver, Colorado, 28 November 2000, pp. 994–1000. Cambridge, MA: MIT Press.
23. Konidaris G, Kaelbling L, Lozano-Perez T. 2013 Symbol acquisition for task-level planning. In The AAAI 2013 Workshop on Learning Rich Representations from Low-Level Sensors.
24. Kaelbling L., Lozano-Pérez T. 2011 Hierarchical task and motion planning in the now. In IEEE Int. Conf. on Robotics and Automation, Shanghai, China, 9 May 2011, pp. 1470–1477. Piscataway, NJ: IEEE Press.
25. Solway A, Diuk C, Cordova N, Yee D, Barto AG, Niv Y, Botvinick M. 2014 Optimal behavioral hierarchy. PLoS Comput. Biol. 10, 1–10.
26. Ribas-Fernandes JJF, Solway A, Diuk C, Barto AG, NIv Y, Botvinick M. 2011 A neural signature of hierarchical reinforcement learning. Neuron 71, 370–379. (doi:10.1016/j.neuron.2011.05.042)
27. Diuk C, Tsai K, Wallis J, Botvinick M, Niv Y. 2013 Two simultaneous, but separable, prediction errors in human ventral striatum. J. Neurosci. 33, 5797–5805. (doi:10.1523/JNEUROSCI.5445-12.2013)
28. Anderson JR, Bothell D, Byrne MD, Scott D, Lebiere C, Qin Y. 2004 An integrated theory of mind. Psychol. Rev. 111, 1036–1060. (doi:10.1037/0033-295X.111.4.1036)
29. Laird JE, Newell A, Rosenbloom PS. 1987 SOAR: an architecture for general intelligence. Artif. Intell. 33, 1–64. (doi:10.1016/0004-3702(87)90050-6)
30. Logan GD, Crump MJC. 2011 Hierarchical control of cognitive processes: the case for skilled typewriting. In The psychology of learning and motivation: advances in research and theory (ed. BH Ross), pp. 2–19. New York, NY: Academic Press.
31. Miller GA, Galanter E, Pribram KH. 1960 Plans and the structure of behavior. New York, NY: Holt, Rinehart and Winston.
32. Botvinick M, Plaut DC. 2004 Doing without schema hierarchies: a recurrent connectionist approach to normal and impaired routine sequential action. Psychol. Rev. 111, 395–429. (doi:10.1037/0033-295X.111.2.395)
33. Cooper R, Shallice T. 2000 Contention scheduling and the control of routine activities. Cogn. Neuropsychol. 17, 297–338. (doi:10.1080/026432900380427)
34. Botvinick M, Plaut DC. 2006 Such stuff as habits are made on: a reply to Cooper and Shallice (2006). Psychol. Rev. 113, 917–928. (doi:10.1037/0033-295X.113.4.917)
35. James W. 1890 The principles of psychology. New York, NY: Holt.
36. Reason JT. 1992 Human error. Cambridge, UK: Cambridge University Press.
37. Tolman EC. 1932 Purposive behavior in animals and men. New York, NY: Century.
38. Sacerdoti ED. 1974 Planning in a hierarchy of abstraction spaces. Artif. Intell. 5, 115–135. (doi:10.1016/0004-3702(74)90026-5)
39. Hayes-Roth B, Hayes-Roth F. 1979 A cognitive model of planning. Cogn. Sci. 3, 275–310. (doi:10.1207/s15516709cog0304_1)
40. Nau D, Au T-C, Ilghami O, Kuter U, Murdock JW, Wu D, Yaman F. 2003 SHOP2: an HTN Planning System. J. Artif. Intell. Res. 20, 379–404.
41. Huys Q, Lally N, Falkner P, Gershman S, Dayan P, Roiser J. 2013 Hierarchical deconstruction and memoization of goal-directed plans. Poster presented at First Multidisciplinary Conf. on Reinforcement Learning and Decision Making, Princeton, NJ, 25–27 October 2013.
42. Dezfouli A, Balleine BW. 2013 Actions, action sequences and decision-making: evidence that goal-directed and habitual action control are hierarchically organized. PLoS Comput. Biol. 9, e1003364. (doi:10.1371/journal.pcbi.1003364)
43. Dezfouli A, Lingawi NW, Balleine BW. 2014 Habits as action sequences: hierarchical action control and changes in outcome value. Phil. Trans. R. Soc. B 369, 20130482. (doi:10.1098/rstb.2013.0482)
44. Ostlund SB, Winterbauer NE, Balleine BW. 2009 Evidence of action sequence chunking in goal-directed instrumental conditioning and its dependence on the dorsomedial prefrontal cortex. J. Neurosci. 29, 8280–8287. (doi:10.1523/JNEUROSCI.1176-09.2009)
45. Fredrickson BL, Kahneman D. 1993 Duration neglect in retrospective evaluations of affective episodes. J. Pers. Soc. Psychol. 65, 45–55. (doi:10.1037/0022-3514.65.1.45)
46. Ariely D, Carmon Z. 2000 Gestalt characteristics of experiences: the defining features of summarized events. J. Behav. Decis. Making 13, 191–201. (doi:10.1002/(SICI)1099-0771(200004/06)13: 2,191::AID-BDM330.3.0.CO;2-A)
47. Geng X, Chen Z, Lam W, Zheng Q. 2013 Hedonic evaluation over short and long retention intervals: the mechanism of the peak–end rule. J. Behav. Decis. Making 26, 225–236. (doi:10.1002/bdm.1755)
48. Griffiths TL, Tenenbaum JB. 2006 Optimal predictions in everyday cognition. Psychol. Sci. 17, 767–773. (doi:10.1111/j.1467-9280.2006.01780.x)
49. Griffiths TL, Tenenbaum JB. 2011 Predicting the future as Bayesian Inference: people combine prior knowledge with observations when estimating duration and extent. J. Exp. Psychol. Gen. 140, 725–743. (doi:10.1037/a0024899)
50. Roy MM, Christenfeld NJ, McKenzie CRM. 2005 Underestimating the duration of future events: memory incorrectly used or memory bias? Psychol. Bull. 131, 738–756. (doi:10.1037/0033-2909.131.5.738)
51. Kruglanski AW, Bélanger JJ, Chen X, Köpetz C, Pierro A, Mannetti L. 2012 The energetics of motivated cognition: a force-field analysis. Psychol. Rev. 119, 1–20. (doi:10.1037/a0025488)
52. Wiener JM, Mallot HA. 2003 ‘Fine-to-coarse’ route planning and navigation in regionalized environments. Spat. Cogn. Comput. 3, 331–358. (doi:10.1207/s15427633scc0304_5)
53. Jonsson A, Barto AG. 2006 Casual graph based decomposition of factored MDPs. J. Mach. Learn. Res. 7, 2259–2301.
54. Vigorito CM, Barto AG. 2010 Intrinsically motivated hieararchical skill learning in structured environments. IEEE Trans. Auton. Ment. Dev. (T-AMD) 2, 83–90. (doi:10.1109/TAMD.2010.2051436)
55. Hengst B. 2002 Discovering hierarchy in reinforcement learning with HEXQ. Proc. Int. Conf. Mach. Learn. 19, 243–250.
56. Mugan J, Kuipers B. 2012 Autonomous learning of high-level states and actions in continuous environments. IEEE Trans. Auton. Ment. Dev. 4, 70–86. (doi:10.1109/TAMD.2011.2160943)
57. Botvinick M, Cohen JD. In press. Computational models of executive control: charted territory and new frontiers. Cogn. Sci.
58. Schapiro A, Cordova N, Turk-Browne N, Rogers TT, Botvinick MM. 2013 Neural representations of events arise from temporal community structure. Nat. Neurosci. 16, 486–492. (doi:10.1038/nn.3331)
59. Dayan P. 1993 Improving generalization for temporal difference learning: the successor representation. Neural Comput. 5, 613–624. (doi:10.1162/neco.1993.5.4.613)
60. Glascher J, Daw N, Dayan P, O’Doherty JP. 2010 States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595. (doi:10.1016/j.neuron.2010.04.016)
61. Badre D. 2008 Cognitive control, hierarchy, and the rostro–caudal organization of the frontal lobes. Trends Cogn. Sci. 12, 193–200. (doi:10.1016/j.tics.2008.02.004)
62. Koechlin E, Ody C, Kouneiher F. 2003 The architecture of cognitive control in the human prefrontal cortex. Science 302, 1181–1185. (doi:10.1126/science.1088545)
63. Crittenden BM, Duncan J. 2012 Task difficulty manipulation reveals multiple demand activity but no frontal lobe hierarchy. Cereb. Cortex 24, 532–540. (doi:10.1093/cercor/bhs333)
64. Nee DE, Jahn A, Brown JW. 2013 Prefrontal cortex organization: dissociating effects of temporal abstraction, relational abstraction, and integration with fMRI. Cereb. Cortex 24, 2377–2387. (doi:10.1093/cercor/bht091)
65. Reynolds JR, O’Reilly RC, Cohen JD, Braver TS. 2012 The function and organization of lateral prefrontal cortex: a test of competing hypotheses. PLoS ONE 7, e30284. (doi:10.1371/journal.pone.0030284)
66. Buckner RL, Carroll DC. 2006 Self-projection and the brain. Trends Cogn. Sci. 11, 49–57. (doi:10.1016/j.tics.2006.11.004)
67. O’Keefe J, Nadel L. 1978 The hippocampus as a cognitive map. Oxford, UK: Oxford University Press.
68. Sutton RS. 1990 Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proc. Seventh International Conference on Machine Learning, Austin, Texas, 21 June 1990, pp. 216–224. San Francisco, CA: Morgan Kaufmann.
69. Baldassarre G. 2002 A biologically plausible model of human planning based on neural networks and Dyna-PI models. In Workshop on Adaptive Behaviour in Anticipatory Learning Systems, pp. 40–60.