A biologically based model for the integration of sensory-motor contingencies in rules and plans: A prefrontal cortex based extension of the Distributed Adaptive Control architecture

Brain Research Bulletin

Volumn 85, Issue 5, 2011, Pages 289-304

  Duff, Armin ^a,b   Sanchez Fibla, Marti ^a   Verschure, Paul F M J ^a,c  

^a UNIVERSITAT POMPEU FABRA   (Spain)

^b Institute of Neuroinformatics Zurich   (Switzerland)

^c INSTITUCIÓ CATALANA DE RECERCA I ESTUDIS AVANÇATS ICREA   (Spain)

Author keywords

Autonomous synthetic agents; Distributed Adaptive Control; Model; Planning; Prefrontal cortex; Rule learning and switching

Indexed keywords

ARTICLE; BIOMIMETICS; COGNITION; LEARNING; LONG TERM MEMORY; NERVE CELL PLASTICITY; PREFRONTAL CORTEX; PRIORITY JOURNAL; SENSORIMOTOR FUNCTION; TASK PERFORMANCE;

ANIMALS; HUMANS; INTELLIGENCE; LEARNING; MODELS, NEUROLOGICAL; NEUROPSYCHOLOGICAL TESTS; PREFRONTAL CORTEX; PSYCHOMOTOR PERFORMANCE; USER-COMPUTER INTERFACE;

EID: 79958804703     PISSN: 03619230     EISSN: 18732747     Source Type: Journal    
DOI: 10.1016/j.brainresbull.2010.11.008     Document Type: Article

References (112)

1. Amit D. The Hebbian paradigm reintegrated: local reverberations as internal representations. Behav. Brain Sci. 1995, 18(4):617-657.
2. Amit D.J., Brunel N. Learning internal representations in an attractor neural-network with analog neurons. Network 1995, 6(3):359-388.
3. Anderson J.R., Bothell D., Byrne M.D., Douglass S., Lebiere C., Qin Y. An integrated theory of the mind. Psychol. Rev. 2004, 111(4):1036-1060.
4. Asaad W.F., Rainer G., Miller E.K. Neural activity in the primate prefrontal cortex during associative learning. Neuron 1998, 21(6):1399-1407.
5. Asaad W.F., Rainer G., Miller E.K. Task-specific neural activity in the primate prefrontal cortex. J. Neurophysiol. 2000, 84(1):451-459.
6. Aston-Jones G., Rajkowski J., Kubiak P., Alexinsky T. Locus coeruleus neurons in monkey are selectively activated by attended cues in a vigilance task. J. Neurosci. 1994, 14(7):4467-4480.
7. Averbeck B., Sohn J., Lee D. Activity in prefrontal cortex during dynamic selection of action sequences. Nat. Neurosci. 2006, 9:276-282.
8. Baddeley A.D. Working Memory 1988, Oxford University Press, New York and Oxford.
9. Badoni D., Bertazzoni S., Buglioni S., Salina G., Amit D., Fusi S. Electronic implementation of an analogue attractor neural network with stochastic learning. Network 1995, 6(2):125-157.
10. Barbas H., Pandya D.N. Architecture and intrinsic connections of the prefrontal cortex in the rhesus monkey. J. Comp. Neurol. 1989, 286(3):353-375.
11. Barraclough D.J., Conroy M.L., Lee D. Prefrontal cortex and decision making in a mixed-strategy game. Nat. Neurosci. 2004, 7(4):404-410.
12. Bates J. Virtual reality, art, and entertainment. Presence Teleoper Virtual Environ. 1992, 1(1):133-138.
13. Bernardet U., Bermúdez i Badia S., Verschure P. A model for the neuronal substrate of dead reckoning and memory in arthropods: a comparative computational and behavioral study. Theory Biosci. 2008, 127(2).
14. Bouret S., Sara S.J. Reward expectation, orientation of attention and locus coeruleus-medial frontal cortex interplay during learning. Eur. J. Neurosci. 2004, 20(3):791-802.
15. Braitenberg V. Vehicles, Experiments in Synthetic Psychology 1984, MIT Press, Cambridge, USA.
16. Brooks R. Intelligence without representation. Artif. Intell. 1991, 47(991):139-159.
17. Bunge S.A. How we use rules to select actions: a review of evidence from cognitive neuroscience. Cogn. Affect. Behav. Neurosci. 2004, 4(4):564-579.
18. Cassandra A., Littman M.L., Zhang N.L. Incremental pruning: a simple, fast, exact method for partially observable Markov decision processes. Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence 1997, 54-61. Morgan Kaufmann Publishers.
19. Chicca E., Badoni D., Dante V., D'Andreagiovanni M., Salina G., Carota L., Fusi S., Del Giudice P. A VLSI recurrent network of integrate-and-fire neurons connected by plastic synapses with long-term memory. IEEE Trans. Neural Netw. 2003, 14(5):1297-1307.
20. Cisek P., Kalaska J.F. Neural correlates of mental rehearsal in dorsal premotor cortex. Nature 2004, 431(7011):993-996.
21. Compte A. Computational and in vitro studies of persistent activity: edging towards cellular and synaptic mechanisms of working memory. Neuroscience 2006, 139(1):135-151.
22. Compte A., Brunel N., Goldman-Rakic P.S., Wang X.J. Synaptic mechanisms and network dynamics underlying spatial working memory in a cortical network model. Cereb. Cortex 2000, 10(9):910-923.
23. Constantinidis C., Wang X.-J. A neural circuit basis for spatial working memory. Neuroscientist 2004, 10(6):553-565.
24. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8(12):1704-1711.
25. Decety J., Grezes J. The power of simulation: imagining one's own and other's behavior. Brain Res. 2006, 1079(1):4-14.
26. Deco G., Rolls E.T. Attention and working memory: a dynamical model of neuronal activity in the prefrontal cortex. Eur. J. Neurosci. 2003, 18(8):2374-2390.
27. Deco G., Rolls E.T. Decision-making and Weber's law: a neurophysiological model. Eur. J. Neurosci. 2006, 24(3):901-916.
28. Del Giudice P., Fusi S., Mattia M. Modelling the formation of working memory with networks of integrate-and-fire neurons connected by plastic synapses. J. Physiol. Paris 2003, 97(4-6):659-681.
29. D'Esposito M. From cognitive to neural models of working memory. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2007, 362(1481):761-772.
30. Douglas R., Martin K. Neocortex. The Synaptic Organization of the Brain 1998, 459-509. Oxford University Press, Oxford and New York. S.G. Shepherd (Ed.).
31. Doya K., Metalearning and neuromodulation Neural Netw. 2002, 15(4-6):495-506.
32. Duff A., Rennó-Costa C., Marcos E., Luvizotto A.L., Giovannucci A., Sanchez-Fibla M., Bernardet U., Verschure P.F.M.J. Distributed adaptive control: a proposal on the neuronal organization of adaptive goal oriented behavior. From Motor to Interaction Learning in Robots 2009, Springer, New York, pp. 15-41.
33. Duff A., Verschure P.F.M.J. Unifying perceptual and behavioral learning with a correlative subspace learning rule. Neurocomputing 2010, 73(10-12):1818-1830.
34. Duff A., Wyss R., Verschure P.F.M.J. Learning temporally stable representations from natural sounds: temporal stability as a general objective underlying sensory processing. Lecture Notes in Computer Science: Artificial Neural Networks ICANN 2007 2007, 129-138. J. Marques de Sá, L.A. Alexandre, W. Duch, D. Mandic (Eds.).
35. Duncan J., Emslie H., Williams P., Johnson R., Freer C. Intelligence and the frontal lobe: the organization of goal-directed behavior. Cogn. Psychol. 1996, 30(3):257-303.
36. Durstewitz D., Seamans J.K., Sejnowski T.J. Neurocomputational models of working memory. Nat. Neurosci. 2000, 3(Suppl.):1184-1191.
37. Florin-Lechner S.M., Druhan J.P., Aston-Jones G., Valentino R.J. Enhanced norepinephrine release in prefrontal cortex with burst stimulation of the locus coeruleus. Brain Res. 1996, 742(1-2):89-97.
38. Fusi S., Asaad W.F., Miller E.K., Wang X.-J. A neural circuit model of flexible sensorimotor mapping: learning and forgetting on multiple timescales. Neuron 2007, 54(2):319-333.
39. Fuster J., Alexander G. Neuron activity related to short-term memory. Science 1971, 173(3997):652-654.
40. Fuster J.M. The Prefrontal Cortex: Anatomy, Physiology, and Neurophysiology of the Frontal Lobe 1997, Lippincott-William & Wilkins, Philadelphia.
41. Gallistel C.R. The Organization of Learning 1990, MIT Press, Cambridge, USA.
42. Gibson J.J. The Ecological Approach to Visual Perception 1979, Lawrence Erlbaum, New Jersey, USA.
43. Gilbert C.D., Wiesel T.N. Columnar specificity of intrinsic horizontal and corticocortical connections in cat visual cortex. J. Neurosci. 1989, 9(7):2432-2442.
44. Harnad S. The symbol grounding problem. Physica D 1990, 42(1-3):335-346.
45. Hasselmo M.E. A model of prefrontal cortical mechanisms for goal-directed behavior. J. Cogn. Neurosci. 2005, 17(7):1115-1129.
46. Hawkins J., George D., Niemasik J. Sequence memory for prediction, inference and behaviour. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2009, 364(1521):1203-1209.
47. Hazy T.E, Frank M.J., O'reilly R.C. Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2007, 362(1485):1601-1613.
48. Herremans A.H.J., Hijzen T.H. The delayed-conditional-discrimination task improves measurement of working memory in rats. Neurosci. Biobehav. Rev. 1997, 21(3):371-379.
49. Hofstötter C., Mintz M., Verschure P.F.M.J. The cerebellum in action: a simulation and robotics study. Eur. J. Neurosci. 2002, 16(7):1361-1376.
50. Hopfield J.J. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 1982, 79(8):2554-2558.
51. Koene R.A., Hasselmo M.E. An integrate-and-fire model of prefrontal cortex neuronal activity during performance of goal-directed decision making. Cereb. Cortex 2005, 15(12):1964-1981.
52. Laird J. Using a computer game to develop advanced AI. Computer 2001, 34(7):70-75.
53. Lewis D.A., Melchitzky D.S., Burgos G.-G. Specificity in the functional architecture of primate prefrontal cortex. J. Neurocytol. 2002, 31(3-5):265-276.
54. Li W., Gilbert C.D. Global contour saliency and local collinear interactions. J. Neurophysiol. 2002, 88(5):2846-2856.
55. Lisman J.E. Role of the dual entorhinal inputs to hippocampus: a hypothesis based on cue/action (non-self/self) couplets. Prog. Brain Res. 2007, 163:615-625.
56. Lisman J.E., Grace A.A. The hippocampal-VTA loop: controlling the entry of information into long-term memory. Neuron 2005, 46(5):703-713.
57. Machens C.K., Wehr M.S., Zador A.M. Linearity of cortical receptive fields measured with natural sounds. J. Neurosci. 2004, 24(5):1089-1100.
58. Mackintosh N.J. Conditioning and Associative Learning 1983, Oxford University Press, USA.
59. Mantz J., Milla C., Glowinski J., Thierry A.M. Differential effects of ascending neurons containing dopamine and noradrenaline in the control of spontaneous activity and of evoked responses in the rat prefrontal cortex. Neuroscience 1988, 27(2):517-526.
60. Matsumoto K., Tanaka K. The role of the medial prefrontal cortex in achieving goals. Curr. Opin. Neurobiol. 2004, 14(2):178-185.
61. Miller E.K. The prefrontal cortex and cognitive control. Nat. Rev. Neurosci. 2000, 1(1):59-65.
62. Miller E.K., Cohen J.D. An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 2001, 24:167-202.
63. Miller E.K., Freedman D.J., Wallis J.D. The prefrontal cortex: categories, concepts and cognition. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2002, 357(1424):1123-1136.
64. Miller E.K, Li L., Desimone R. Activity of neurons in anterior inferior temporal cortex during a short-term memory task. J. Neurosci. 1993, 13(4):1460-1478.
65. Montague P.R., Hyman S.E., Cohen J.D. Computational roles for dopamine in behavioural control. Nature 2004, 431(7010):760-767.
66. Moore T.L., Schettler S.P., Killiany R.J., Rosene D.L., Moss M.B. Effects on executive function following damage to the prefrontal cortex in the rhesus monkey (Macaca mulatta). Behav. Neurosci. 2009, 123(2):231-241.
67. Nason S., Laird J.E. Soar-RL: integrating reinforcement learning with Soar. Cogn. Syst. Res. 2005, 6(1):51-59.
68. Newell A. Unified Theories of Cognition 1990, Harvard University Press, Cambridge, USA.
69. Oades R.D., Halliday G.M. Ventral tegmental (A10) system: neurobiology. 1. Anatomy and connectivity. Brain Res. 1987, 434(2):117-165.
70. O'Regan J.K., Noe A. A sensorimotor account of vision and visual consciousness. Behav. Brain Sci. 2001, 24(5):939-973.
71. O'Reilly R.C. Biologically based computational models of high-level cognition. Science 2006, 314(5796):91-94.
72. O'Reilly R.C., Frank M.J. Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput. 2006, 18(2):283-328.
73. Padoa-Schioppa C., Assad J.A. Neurons in the orbitofrontal cortex encode economic value. Nature 2006, 441(7090):223-226.
74. Pan X., Sawa K., Tsuda I., Tsukada M., Sakagami M. Reward prediction based on stimulus categorization in primate lateral prefrontal cortex. Nat. Neurosci. 2008, 11(6):703-712.
75. Prada R., Paiva A. Teaming up humans with autonomous synthetic characters. Artif. Intell. 2009, 173(1):80-103.
76. Pudovkina O.L., Kawahara Y., de Vries J., Westerink B.H. The release of noradrenaline in the locus coeruleus and prefrontal cortex studied with dual-probe microdialysis. Brain Res. 2001, 906(1-2):38-45.
77. Redgrave P., Gurney K. The short-latency dopamine signal: a role in discovering novel actions?. Nat. Rev. Neurosci. 2006, 7(12):967-975.
78. Rodriguez A., Whitson J., Granger R. Derivation and analysis of basic computational operations of thalamocortical circuits. J. Cogn. Neurosci. 2004, 16(5):856-877.
79. Rossetti Z.L., Carboni S. Noradrenaline and dopamine elevations in the rat prefrontal cortex in spatial working memory. J. Neurosci. 2005, 25(9):2322-2329.
80. Rougier N.P., Noelle D.C., Braver T.S., Cohen J.D., O'Reilly R.C. Prefrontal cortex and flexible cognitive control: rules without symbols. Proc. Natl. Acad. Sci. U.S.A. 2005, 102(20):7338-7343.
81. Samejima K., Ueda Y., Doya K., Kimura M. Representation of action-specific reward values in the striatum. Science 2005, 310(5752):1337-1340.
82. Sanchez-Montanes M.A., König P., Verschure P.F.M.J. Learning sensory maps with real-world stimuli in real time using a biophysically realistic learning rule. IEEE Trans. Neural Netw. 2002, 13(3):619-632.
83. Sara S.J. The locus coeruleus and noradrenergic modulation of cognition. Nat. Rev. Neurosci. 2009, 10(3):211-223.
84. Schultz W. Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 2006, 57:87-115.
85. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275(5306):1593-1599.
86. Shallice T. Specific impairments of planning. Philos. Trans. R. Soc. Lond. B Biol. Sci. 1982, 298(1089):199-209.
87. Sherburne L., Zentall T. Coding of feature and no-feature events by pigeons performing a delayed conditional discrimination. Anim. Learn. Behav. 1993, 21(2):92-100.
88. Stettler D.D., Das A., Bennett J., Gilbert C.D. Lateral connectivity and contextual interactions in macaque primary visual cortex. Neuron 2002, 36(4):739-750.
89. Stone C.P., Nyswander D. The reliability of rat learning scores from the multiple T-maze as determined by four different methods. J. Genet. Psychol. 1927, 34:397-524.
90. Sutton R., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press, Cambridge, USA.
91. Thorndike E. Animal Intelligence 1911, Macmillan, New York.
92. Thrun S. Monte Carlo POMDPs. Advances in Neural Information Processing Systems 12 2000, 1064-1070. MIT Press, Cambridge, USA. S. Solla, T. Leen, K.-R. Müller (Eds.).
93. Tremblay L., Schultz W. Modifications of reward expectation-related neuronal activity during learning in primate orbitofrontal cortex. J. Neurophysiol. 2000, 83(4):1877-1885.
94. Unterrainer J.M., Rahm B., Halsband U., Kaller C.P. What is in a name: comparing the Tower of London with one of its variants. Brain Res. Cogn. Brain Res. 2005, 23(2-3):418-428.
95. Usher M., Cohen J.D., Servan-Schreiber D., Rajkowski J., Aston-Jones G. The role of locus coeruleus in the regulation of cognitive performance. Science 1999, 283(5401):549-554.
96. van Duuren E., Escamez F.A.N., Joosten R.N.J.M.A., Visser R., Mulder A.B., Pennartz C.M.A. Neural coding of reward magnitude in the orbitofrontal cortex of the rat during a five-odor olfactory discrimination task. Learn. Mem. 2007, 14(6):446-456.
97. van Duuren E., Lankelma J., Pennartz C.M.A. Population coding of reward magnitude in the orbitofrontal cortex of the rat. J. Neurosci. 2008, 28(34):8590-8603.
98. Vasilaki E., Fusi S., Wang X.-J., Senn W. Learning flexible sensori-motor mappings in a complex network. Biol. Cybern. 2009, 100(2):147-158.
99. Vernon D., Metta G., Sandini G. A survey of artificial cognitive systems: implications for the autonomous development of mental capabilities in computational agents. IEEE Trans. Evol. Comput. 2007, 11(2):151-180.
100. Verschure P.F.M.J. Synthetic epistemology: the acquisition, retention, and expression of knowledge in natural and synthetic systems. Proceedings of IEEE World Conference on Computational Intelligence 1998, 147-152.
101. Verschure P.F.M.J., Althaus P. A real-world rational agent: unifying old and new AI. Cogn. Sci. 2003, 27:561-590.
102. Verschure P.F.M.J., Coolen A.C.C. Adaptive fields: distributed representations of classically conditioned associations. Network 1991, 2(2):189-206.
103. Verschure P.F.M.J., Krose B., Pfeifer R. Distributed adaptive control: the self-organization of structured behavior. Rob. Auton. Syst. 1993, 9:181-196.
104. Verschure P.F.M.J., Voegtlin T., Douglas R.J. Environmentally mediated synergy between perception and behaviour in mobile robots. Nature 2003, 425(6958):620-624.
105. Voegtlin T., Verschure P.F.M.J. What can robots tell us about brains? A synthetic approach towards the study of learning and problem solving. Rev. Neurosci. 1999, 10(3-4):291-310.
106. Wang X.-J. Synaptic reverberation underlying mnemonic persistent activity. Trends Neurosci. 2001, 24(8):455-463.
107. Wang X.-J. Decision making in recurrent neuronal circuits. Neuron 2008, 60(2):215-234.
108. Ward G., Allport A. Planning and problem solving using the five disc Tower of London task. Q. J. Exp. Psychol. A 1997, 50(1):49-78.
109. White I.M., Wise S.P. Rule-dependent neuronal activity in the prefrontal cortex. Exp. Brain Res. 1999, 126(3):315-335.
110. R. Wyss, Sensory and motor coding in the organization of behavior, Ph.D. thesis, ETHZ, 2003.
111. Wyss R., König P., Verschure P.F.M.J. A model of the ventral visual system based on temporal stability and local memory. PLoS Biol. 2006, 4(5):e120.
112. Yen S.-C., Baker J., Gray C.M. Heterogeneity in the responses of adjacent neurons to natural stimuli in cat striate cortex. J. Neurophysiol. 2007, 97(2):1326-1341.