Divide et impera: Subgoaling reduces the complexity of probabilistic inference and problem solving

Journal of the Royal Society Interface

Volumn 12, Issue 104, 2015, Pages

  Maisto, Domenico ^a   Donnarumma, Francesco ^b   Pezzulo, Giovanni ^b  

^a CNR   (Italy)

^b INSTITUTE OF COGNITIVE SCIENCES AND TECHNOLOGIES   (Italy)

Author keywords

Active inference; Hierarchies; Model based reinforcement learning; Planning as inference; Problem solving; Subgoals

Indexed keywords

REINFORCEMENT LEARNING; SOCIAL NETWORKING (ONLINE);

ACTIVE INFERENCE; COMPUTATIONAL FRAMEWORK; DESCRIPTIVE COMPLEXITY; HIERARCHIES; INFORMATION RESOURCE; MODEL-BASED REINFORCEMENT LEARNING; PROBABILISTIC INFERENCE; SUBGOALS;

PROBLEM SOLVING;

ALGORITHM; ANIMAL EXPERIMENT; ANIMAL MODEL; ARTICLE; BEHAVIOR; CONTROLLED STUDY; DYNAMICS; EXECUTIVE FUNCTION; LEARNING; MATHEMATICAL ANALYSIS; MOVEMENT (PHYSIOLOGY); NONHUMAN; PREFRONTAL CORTEX; PROBABILITY; PROBLEM SOLVING; REINFORCEMENT; SAMPLING; STRATEGIC PLANNING; VALIDATION STUDY; WORKING MEMORY; ANIMAL; BAYES THEOREM; BRAIN; COMPUTER SIMULATION; HAPLORHINI; HUMAN; INTERNET; METABOLISM; MONTE CARLO METHOD; NERVE CELL; NEUROSCIENCE; PHYSIOLOGY; PROCEDURES;

ANIMALIA;

ALGORITHMS; ANIMALS; BAYES THEOREM; BRAIN; COMPUTER SIMULATION; HAPLORHINI; HUMANS; INTERNET; LEARNING; MARKOV CHAINS; MONTE CARLO METHOD; NEURONS; NEUROSCIENCES; PREFRONTAL CORTEX; PROBABILITY; PROBLEM SOLVING;

EID: 84923247065     PISSN: 17425689     EISSN: 17425662     Source Type: Journal    
DOI: 10.1098/rsif.2014.1335     Document Type: Article

References (55)

1. Newell A, Simon HA. 1972 Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.
2. Fuster JM. 1997 The prefrontal cortex: anatomy, physiology, and neuropsychology of the frontal lobe. Philadelphia, PA: Lippincott-Raven.
3. Miller EK, Cohen JD. 2001 An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 24, 167-202. (doi:10. 1146/annurev. neuro. 24. 1. 167)
4. Passingham RE, Wise SP. 2012 The neurobiology of the prefrontal cortex: anatomy, evolution, and the origin of insight, vol. 50. Oxford, UK: Oxford University Press.
5. Pezzulo G, Castelfranchi C. 2009 Thinking as the control of imagination: a conceptual framework for goal-directed systems. Psychol. Res. 73, 559-577. (doi:10. 1007/s00426-009-0237-z)
6. Mushiake H, Saito N, Sakamoto K, Itoyama Y, Tanji J. 2006 Activity in the lateral prefrontal cortex reflects multiple steps of future events in action plans. Neuron 50, 631-641. (doi:10. 1016/j. neuron. 2006. 03. 045)
7. Saito N, Mushiake H, Sakamoto K, Itoyama Y, Tanji J. 2005 Representation of immediate and final behavioral goals in the monkey prefrontal cortex during an instructed delay period. Cereb. Cortex 15, 1535-1546. (doi:10. 1093/cercor/bhi032)
8. Shallice T. 1982 Specific impairments of planning. Phil. Trans. R. Soc. Lond. B 298, 199-209. (doi:10. 1098/rstb. 1982. 0082)
9. Botvinick MM. 2008 Hierarchical models of behavior and prefrontal function. Trends Cogn. Sci. 12, 201-208. (doi:10. 1016/j. tics. 2008. 02. 009)
10. Solway A, Diuk C, Cordova N, Yee D, Barto AG, Niv Y, Botvinick MM. 2014 Optimal behavioral hierarchy. PLoS Comput. Biol. 10, e1003779. (doi:10. 1371/ journal. pcbi. 1003779)
11. Sutton RS, Precup D, Singh S. 1999 Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif. Intell. 112, 181-211. (doi:10. 1016/S0004-3702(99) 00052-1)
12. Barto AG, Mahadevan S. 2003 Recent advances in hierarchical reinforcement learning. Discr. Event Dyn. Syst. 13, 341-379. (doi:10. 1023/A:102569 6116075)
13. Botvinick M, Niv Y, Barto A. 2009 Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 119, 262-280. (doi:10. 1016/j. cognition. 2008. 08. 011)
14. van Dijk SG, Polani D, Nehaniv CL. 2011 Hierarchical behaviours: getting the most bang for your bit. In Advances in artificial life. Darwin Meets von Neumann, pp. 342-349. Berlin, Germany: Springer.
15. Rosenbloom PS, Laird JE, Newell A. 1992 The soar papers: research on integrated intelligence, vols. 1 and 2. Cambridge, MA: MIT Press.
16. Hauskrecht M, Meuleau N, Kaelbling LP, Dean T, Boutilier C. 1998 Hierarchical solution of Markov decision processes using macro-actions. In Proc. 14th Conf. on Uncertainty in artificial intelligence, Madison, WI, 24-26 July, pp. 220-229. San Francisco, CA: Morgan Kaufmann Publishers Inc.
17. Lipovetzky N, Geffner H. 2012 Width and serialization of classical planning problems. In ECAI 2012: 20th European Conf. on Artificial Intelligence, Montpellier, France, 27-31 August, pp. 540-545. Clifton, VA: IOS Press. (doi:10. 3233/978-1-61499-098-7-540)
18. Donnarumma F, Prevete R, Trautteur G. 2012 Programming in the brain: a neural network theoretical framework. Connect. Sci. 24, 71-90. (doi:10. 1080/09540091. 2012. 684670)
19. Schmidhuber J. 1997 Discovering neural nets with low Kolmogorov complexity and high generalization capability. Neural Netw. 10, 10-15. (doi:10. 1016/ S0893-6080(96)00127-X)
20. Mumford MD, Schultz RA, Van Doorn JR. 2001 Performance in planning: processes, requirements, and errors. Rev. Gen. Psychol. 5, 213-240. (doi:10. 1037/1089-2680. 5. 3. 213)
21. Spitz HH, Minsky SK, Bessellieu CL. 1984 Subgoal length versus full solution length in predicting Tower of Hanoi problem-solving performance. Bull. Psychon. Soc. 22, 301-304. (doi:10. 3758/ BF03333826)
22. Simon DA, Daw ND. 2011 Neural correlates of forward planning in a spatial decision task in humans. J. Neurosci. 31, 5526-5539. (doi:10. 1523/ JNEUROSCI. 4647-10. 2011)
23. Li M, Vitaanyi PMD. 2008 An introduction to Kolmogorov complexity and its applications. Berlin, Germany: Springer.
24. Attias H. 2003 Planning by probabilistic inference. In Proc. 9th Int. Workshop on Artificial Intelligence and Statistics, Key West, Florida, 3-6 January. New Jersey: Society for Artificial Intelligence and Statistics.
25. Botvinick M, Toussaint M. 2012 Planning as inference. Trends Cogn. Sci. 16, 485-488. (doi:10. 1016/j. tics. 2012. 08. 006)
26. Pezzulo G, Rigoli F, Chersi F. 2013 The mixed instrumental controller: using value of information to combine habitual choice and mental simulation. Front. Psychol. 4, 92. (doi:10. 3389/fpsyg. 2013. 00092)
27. Toussaint M, Storkey A. 2006 Probabilistic inference for solving discrete and continuous state Markov decision processes. In Proc. 23rd Int. Conf. on Machine learning, Pittsburgh, Pennsylvania, 25-29 June, pp. 945-952. New York, NY: ACM.
28. Botvinick MM, An J. 2008 Goal-directed decision making in prefrontal cortex: a computational framework. In Advances in Neural Information Processing Systems (NIPS), Vancouver, Canada, 8-11 December. Cambridge, MA: MIT Press.
29. Solway A, Botvinick MM. 2012 Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates. Psychol. Rev. 119, 120-154. (doi:10. 1037/a0026435)
30. Friston KJ, Daunizeau J, Kiebel SJ. 2009 Reinforcement learning or active inference? PLoS ONE 4, e6421. (doi:10. 1371/journal. pone. 0006421)
31. Murphy KP. 2002 Dynamic Bayesian networks: representation, inference and learning. PhD thesis, Computer Science Division, UC Berkeley, USA.
32. Sutton RS, Barto AG. 1998 Reinforcement learning: an introduction. Cambridge, MA: MIT Press.
33. Verma D, Rao RPN. 2006 Planning and acting in uncertain environments using probabilistic inference. In IROS, Beijing, China, 9-15 October, pp. 2382-2387. Piscataway, NJ: IEEE.
34. Solomonoff RJ. 1997 The discovery of algorithmic probability. J. Comp. Syst. Sci. 55, 73-88. (doi:10. 1006/jcss. 1997. 1500)
35. van Dijk SG, Polani DP. 2011 Grounding subgoals in information transitions. In Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), 2011 IEEE Symposium, Paris, France, 11-15 April, pp. 105-111. New York, NY: IEEE.
36. Doucet A, Godsill S, Andrieu C. 2000 On sequential Monte Carlo sampling methods for Bayesian filtering. Stat. Comput. 10, 197-208. (doi:10. 1023/ A:1008935410038)
37. Geffner H. 2013 Computational models of planning. Wiley Interdiscip. Rev. Cogn. Sci. 4, 341-356. (doi:10. 1002/wcs. 1233)
38. Doya K, Ishii S, Pouget A, Rao RPN (eds). 2007 Bayesian brain: probabilistic approaches to neural coding, 1 edn. Cambridge, MA: MIT Press.
39. Friston K. 2010 The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127-138. (doi:10. 1038/nrn2787)
40. Genovesio A, Tsujimoto S, Wise SP. 2012 Encoding goals but not abstract magnitude in the primate prefrontal cortex. Neuron 74, 656-662. (doi:10. 1016/j. neuron. 2012. 02. 023)
41. Friston K, Samothrakis S, Montague R. 2012 Active inference and agency: optimal control without cost functions. Biol. Cybern. 106, 523-541. (doi:10. 1007/s00422-012-0512-8)
42. Lee D, Rushworth MFS, Walton ME, Watanabe M, Sakagami M. 2007 Functional specialization of the primate frontal cortex during decision making. J. Neurosci. 27, 8170-8173. (doi:10. 1523/ JNEUROSCI. 1561-07. 2007)
43. Ortega PA, Braun DA. 2013 Thermodynamics as a theory of decision-making with informationprocessing costs. Proc. R. Soc. A 469, 20120683. (doi:10. 1098/rspa. 2012. 0683)
44. Anderson C, Boomsma JJ, Bartholdi JJ. 2002 Task partitioning in insect societies: bucket brigades. Insectes Soc. 49, 171-180. (doi:10. 1007/s00040-002-8298-7)
45. Hamann H, Karsai I, Schmickl T. 2013 Time delay implies cost on task switching: a model to investigate the efficiency of task partitioning. Bull. Math. Biol. 75, 1181-1206. (doi:10. 1007/s11538-013-9851-4)
46. Karsai I, Schmickl T. 2011 Regulation of task partitioning by a common stomach: a model of nest construction in social wasps. Behav. Ecol. 22, 819-830. (doi:10. 1093/beheco/arr060)
47. Seeley TD, Visscher PK, Schlegel T, Hogan PM, Franks NR, Marshall JAR. 2012 Stop signals provide cross inhibition in collective decision-making by honeybee swarms. Science 335, 108-111. (doi:10. 1126/science. 1210361)
48. Trianni V, Tuci E, Passino KM, Marshall JAR. 2011 Swarm cognition: an interdisciplinary approach to the study of self-organising biological collectives. Swarm Intell. 5, 3-18. (doi:10. 1007/s11721-010-0050-8)
49. Daw ND, Niv Y, Dayan P. 2005 Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704-1711. (doi:10. 1038/nn1560)
50. Pezzulo G, Rigoli F. 2011 The value of foresight: how prospection affects decision-making. Front. Neurosci. 5, 79. (doi:10. 3389/fnins. 2011. 00079)
51. Pezzulo G, Verschure P, Balkenius C, Pennartz C. 2014 The principles of goaldirected decision-making: from neural mechanisms to computation and robotics. Phil. Trans. R. Soc. B 369, 20130470. (doi:10. 1098/ rstb. 2013. 0470)
52. Pezzulo G, van der Meer MA, Lansink CS, Pennartz CMA. 2014 Internally generated sequences in learning and executing goal-directed behavior. Trends Cogn. Sci. 18, 647-657. (doi:10. 1016/j. tics. 2014. 06. 011)
53. Verschure P, Pennartz C, Pezzulo G. 2014 The why, what, where, when and how of goal directed choice: neuronal and computational principles. Phil. Trans. R. Soc. B 369, 20130483. (doi:10. 1098/rstb. 2013. 0483)
54. Barlow HB. 1961 Possible principles underlying the transformation of sensory messages. In Sensory communication (ed. WA Rosenblith). Cambridge, MA: MIT Press.
55. Kiebel SJ, Daunizeau J, Friston KJ. 2008 A hierarchy of time-scales and the brain. PLoS Comput. Biol. 4, e1000209. (doi:10. 1371/journal. pcbi. 1000209)