Flexible shaping: How learning in small steps helps

Cognition

Volumn 110, Issue 3, 2009, Pages 380-394

  Krueger, Kai A ^a   Dayan, Peter ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Computational modeling; Gating; PFC; Sequence learning; Shaping

Indexed keywords

ARTICLE; COGNITION; COGNITIVE DEVELOPMENT; COMPREHENSION; LEARNING; MENTAL FUNCTION; MODEL; NERVE CELL NETWORK; PRIORITY JOURNAL; TASK PERFORMANCE; TEACHING; WORKING MEMORY;

ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; HUMANS; LEARNING; MEMORY; NEURAL NETWORKS (COMPUTER); REVERSAL LEARNING;

EID: 59649113160     PISSN: 00100277     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.cognition.2008.11.014     Document Type: Article

References (51)

1. Badre D., Poldrack R.A., Pare-Blagoev E., Juliana amd Insler R.Z., and Wagner A.D. Dissociable controlled retrieval and generalized selection mechanisms in ventrolateral prefrontal cortex. Neuron 47 6 (2005) 907-918
2. Bakker, B., & Schmidhuber, J. (2004). Hierarchical reinforcement learning based on subgoal discovery and subpolicy specialization. In F. Groen, N. Amato, A. Bonarini, E. Yoshida, & B. Kroese, (Eds.), Proceedings of the 8th conference on intelligent autonomous systems, Vol. IAS-8, pp. 438-445.
3. Barto A.G., and Mahadevan S. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems: Theory and Applications 13 4 (2003) 341-379
4. Breland K., and Breland M. The misbehavior of organisms. American Psychologist 16 11 (1961) 681-684
5. Brown S.L., Joseph J., and Stopfer M. Encoding a temporally structured stimulus with a temporally structured neural representation. Nature Neuroscience 8 11 (2005) 1568-1576
6. Butter C.M. Perseveration in extinction and in discrimination reversal tasks following selective frontal ablations in Macaca mulatta. Physiology & Behaviour 4 2 (1969) 163-171
7. Dahlin E., Neely A.S., Larsson A., Backman L., and Nyberg L. Transfer of learning after updating training mediated by the striatum. Science 320 5882 (2008) 1510-1512
8. Dayan P. Images, frames, and connectionist hierarchies. Neural Computation 18 10 (2006) 2293-2319
9. Dayan P., and Yu A.J. Phasic norepinephrine: A neural interrupt signal for unexpected events. Network: Computation in Neural Systems 17 4 (2006) 335-350
10. Dorigo M., and Colombetti M. Robot shaping: An experiment in behavior engineering (1998), MIT Press/Bradford Books
11. Duncan J. Attention, intelligence, and the frontal lobes. In: Gazzaniga M.S. (Ed). The new cognitive neurosciences (1995), The MIT Press 721-733
12. Elman J.L. Learning and development in neural networks: The importance of starting small. Cognition 48 1 (1993) 71-99
13. Frank, M. J., Loughry, B., & O'Reilly, R. C. (2001). Interactions between frontal cortex and basal ganglia in working memory: A computational model. Technical Report, Department of Psychology, University of Colorado.
14. Gers F.A., Schmidhuber J., and Cummins F. Learning to forget: Continual prediction with LSTM. Neural Computation 12 10 (2000) 2451-2471
15. Gilbert S.J., Frith C.D., and Burgess P.W. Involvement of rostral prefrontal cortex in selection between stimulus-oriented and stimulus-independent thought. European Journal of Neuroscience 21 5 (2005) 1423-1431
16. Grossberg S. How does a brain build a cognitive code?. Psychological Review 87 1 (1980) 1-51
17. Haruno M., Wolpert D.M., and Kawato M. MOSAIC model for sensorimotor learning and control. Neural Computation 13 10 (2001) 2201-2220
18. Hazy T.E., Frank M.J., and O'Reilly R.C. Towards an executive without a homunculus: Computational models of the prefrontal cortex/basal ganglia system. Philosophical Transactions of the Royal Society B. Biological Sciences 362 1485 (2007) 1601-1613
19. Hinton G.E., and Ghahramani Z. Generative models for discovering sparse distributed representations. Philosophical Transactions of the Royal Society B 352 1358 (1997) 1177-1190
20. Hinton G.E., Osindero S., and Teh Y.-W. A fast learning algorithm for deep belief nets. Neural Computation 18 7 (2006) 1527-1554
21. Hochreiter S., and Schmidhuber J. Long short-term memory. Neural Computation 9 8 (1997) 1735-1780
22. Iversen S.D., and Mishkin M. Perseverative interference in monkeys following selective lesions of the inferior prefrontal convexity. Experimental Brain Research 11 4 (1970) 376-386
23. Jacobs R., Jordan M.I., Nowlan S.J., and Hinton G.E. Adaptive mixtures of local experts. Neural Computation 3 1 (1991) 79-87
24. Jones B., and Mishkin M. Limbic lesions and the problem of stimulus-reinforcement associations. Experimental Neurology 36 2 (1972) 362-377
25. Koechlin E., Ody C., and Kouneiher F. The architecture of cognitive control in the human prefrontal cortex. Science 302 5648 (2003) 1181-1185
26. Koechlin E., and Summerfield C. An information theoretical approach to prefrontal executive function. Trends in Cognitive Sciences 11 6 (2007) 229-235
27. McGovern A., and Barto A.G. Automatic discovery of subgoals in reinforcement learning using diverse density. Proceedings of the 18th international conference on machine learning (2001), Morgan Kaufmann, San Francisco, CA 361-368
28. McNab F., and Klingberg T. Prefrontal cortex and basal ganglia control access to working memory. Nature Neuroscience 11 1 (2008) 103-107
29. Newport E.L. Constraints on learning and their role in language acquisition: Studies of the acquisition of american sign language. Language Sciences 10 1 (1988) 147-172
30. Newport E.L. Maturational constraints on language learning. Cognitive Science 14 1 (1990) 11-28
31. O'Reilly R.C., and Frank M.J. Making working memory work: A computational model of learning in the prefrontal cortex and basal ganglia. Neural Computation 18 2 (2005) 283-328
32. Parr R., and Russell S. Reinforcement learning with hierarchies of machines. In: Jordan M.I., Kearns M.J., and Solla S.A. (Eds). Advances in neural information processing systems (1997), The MIT Press
33. Peterson G.B. A day of great illumination: B.F. Skinner's discovery of shaping. Journal of the Experimental Analysis of Behavior 82 3 (2004) 317-328
34. Premack D. The codes of man and beasts. Behavioral and Brain Sciences 6 (1983) 125-167
35. Redish A.D., Jensen S., Johnson A., and Kurth-Nelson Z. Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling. Psychological Review 114 3 (2007) 784-805
36. Reynolds, J. (2007). Computational, behavioral, neuro-imaging methods investigating the hierarchical organization of pfc and goal-oriented behavior. unpublished/NIPS workshop on hierarchical organization of behavior.
37. Reynolds J.R., Zacks J.M., and Braver T.S. A computational model of event segmentation from perceptual prediction. Cognitive Science 31 4 (2007) 613-643
38. Rohde D.L.T., and Plaut D.C. Language acquisition in the absence of explicit negative evidence: How important is starting small?. Cognition 72 1 (1999) 67-109
39. Saksida L.M., Raymond S.M., and Touretzky D.S. Shaping robot behavior using principles from instrumental conditioning. Robotics and Autonomous Systems 22 3 (1997) 231-249
40. Savage T. Shaping: The link between rats and robots. Connection Science 10 3 (1998) 321-340
41. Savage T. Shaping: A multiple contingencies analysis and its relevance to behaviour-based robotics. Connection Science 13 3 (2001) 199-234
42. Shallice T., and Burgess P.W. Deficits in strategy application following frontal lobe damage in man. Brain 114 2 (1991) 727-741
43. Sheffield, F. (1965). Relation between classical conditioning and instrumental learning. In W. F. Prokasy, (Ed.), Classical conditioning: A symposium. Appleton-Century-Crofts, 302-322.
44. Singh S.P. Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning 8 (1992) 323-339
45. Skinner B.F. The behavior of organisms: An experimental analysis. Appleton-Century-Crofts (1938)
46. Sowell E.R., Thompson P.M., Holmes C., Jernigan T.L., and Toga A.W. In vivo evidence for post-adolescent brain maturation in frontal and striatal regions. Nature Neuroscience 2 (1999) 859-861
47. Thompson R., Oden D., and Boysen S. Language-naive chimpanzees (Pan troglodytes) judge relations between relations in a conceptual matching-to-sample task. Journal of Experimental Psychology: Animal Behavior Processes 23 1 (1997) 31-43
48. Watkins, C. J. C. H. (1989). Learning from delayed rewards. Ph.D. Thesis, Cambridge, UK.
49. Williams D.R., and Williams H. Auto-maintenance in the pigeon: Sustained pecking despite contingent non-reinforcement. Journal of the Experimental Analysis of Behavior 12 4 (1969) 511-520
50. Yu A.J., and Dayan P. Uncertainty, neuromodulation, and attention. Neuron 46 (2005) 681-692
51. Zacks J.M., Speer N.K., Swallow K.M., Braver T.S., and Reynolds J.R. Event perception: A mind-brain perspective. Psychological Bulletin 133 2 (2007) 273-293