A Unifying Probabilistic View of Associative Learning

PLoS Computational Biology

Volumn 11, Issue 11, 2015, Pages

  Gershman, Samuel J ^a  

^a HARVARD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

REINFORCEMENT LEARNING;

ASSOCIATIVE LEARNING; BAYESIAN; BAYESIAN LEARNING; DESIGN PRINCIPLES; LEARNER TRACKING; PROBABILISTICS; RATIONAL DESIGN; REINFORCEMENT LEARNING THEORIES; REINFORCEMENT LEARNINGS; UNCERTAINTY;

ANIMALS;

ANIMAL EXPERIMENT; LEARNING THEORY; NONHUMAN; PREDICTION; REINFORCEMENT; REWARD; STATE DEPENDENT LEARNING; SYNTHESIS; UNCERTAINTY; ALGORITHM; ANIMAL; BAYES THEOREM; BIOLOGICAL MODEL; BIOLOGY; HUMAN; LEARNING; PHYSIOLOGY; PROCEDURES;

ALGORITHMS; ANIMALS; ASSOCIATION LEARNING; BAYES THEOREM; COMPUTATIONAL BIOLOGY; HUMANS; MODELS, NEUROLOGICAL;

EID: 84949266762     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1004567     Document Type: Article

References (78)

1. Shanks DR, The Psychology of Associative Learning. Cambridge University Press; 1995.
2. Pearce JM, Bouton ME, Theories of associative learning in animals. Annual Review of Psychology. 2001;52:111–139. doi: 10.1146/annurev.psych.52.1.111 11148301
3. Dayan P, Kakade S, Leen TK, Dietterich TG, Tresp V, Explaining Away in Weight Space. In: Advances in Neural Information Processing Systems 13. MIT Press; 2001. p. 451–457.
4. Kakade S, Dayan P, Acquisition and extinction in autoshaping. Psychological Review. 2002;109:533–544. doi: 10.1037/0033-295X.109.3.533 12088244
5. Courville AC, Daw ND, Touretzky DS, Bayesian theories of conditioning in a changing world. Trends in Cognitive Sciences. 2006;10:294–300. doi: 10.1016/j.tics.2006.05.004 16793323
6. Kruschke JK, Bayesian approaches to associative learning: From passive to active learning. Learning & Behavior. 2008;36:210–226. doi: 10.3758/LB.36.3.210
7. Gershman SJ, Blei DM, Niv Y, Context, learning, and extinction. Psychological Review. 2010;117:197–209. doi: 10.1037/a0017808 20063968
8. Gershman SJ, Niv Y, Exploring a latent cause theory of classical conditioning. Learning & Behavior. 2012;40:255–268. doi: 10.3758/s13420-012-0080-8
9. Sutton RS, Barto AG, Gabriel M, Moore J, Time-derivative models of Pavlovian reinforcement. In: Learning and Computational Neuroscience: Foundations of Adaptive Networks. MIT Press; 1990. p. 497–537.
10. Niv Y, Reinforcement learning in the brain. Journal of Mathematical Psychology. 2009;53:139–154. doi: 10.1016/j.jmp.2008.12.005
11. Ludvig EA, Sutton RS, Kehoe EJ, Evaluating the TD model of classical conditioning. Learning & Behavior. 2012;40:305–319. doi: 10.3758/s13420-012-0082-6
12. Rescorla RA, Wagner AR, Black AH, Prokasy WF, A theory of of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: Classical Conditioning II: Current Research and theory. New York, NY: Appleton-Century-Crofts; 1972. p. 64–99.
13. Geist M, Pietquin O, Kalman temporal differences. Journal of Artificial Intelligence Research. 2010;39:483–532.
14. Keramati M, Dezfouli A, Piray P, Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Computational Biology. 2011;7:e1002055. doi: 10.1371/journal.pcbi.1002055 21637741
15. Holland PC, Ross RT, Within-compound associations in serial compound conditioning. Journal of Experimental Psychology: Animal Behavior Processes. 1981;7:228–241.
16. Miller RR, Barnet RC, Grahame NJ, Assessment of the Rescorla-Wagner model. Psychological Bulletin. 1995;117:363–386. doi: 10.1037/0033-2909.117.3.363 7777644
17. Pearce JM, Hall G, A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychological Review. 1980;87:532–552. doi: 10.1037/0033-295X.87.6.532 7443916
18. Van Hamme LJ, Wasserman EA, Cue competition in causality judgments: The role of nonpresentation of compound stimulus elements. Learning and Motivation. 1994;25:127–151. doi: 10.1006/lmot.1994.1008
19. Schmajuk NA, Larrauri JA, Experimental challenges to theories of classical conditioning: application of an attentional model of storage and retrieval. Journal of Experimental Psychology: Animal Behavior Processes. 2006;32:1–20. 16435961
20. Robbins H, Monro S, A stochastic approximation method. The Annals of Mathematical Statistics. 1951;22:400–407. doi: 10.1214/aoms/1177729586
21. Yuille AL, Saul LK, Weiss Y, Bottou L, The Rescorla-Wagner Algorithm and Maximum Likelihood Estimation of Causal Parameters. In: Advances in Neural Information Processing Systems 17. MIT Press; 2005. p. 1585–1592.
22. Widrow B, Hoff ME, Adaptive switching circuits. Proc Of WESCON Conv Rec, part 4. 1960;p. 96–140.
23. Bach DR, Dolan RJ, Knowing how much you don’t know: a neural organization of uncertainty estimates. Nature Reviews Neuroscience. 2012;13:572–586. 22781958
24. Pouget A, Beck JM, Ma WJ, Latham PE, Probabilistic brains: knowns and unknowns. Nature Neuroscience. 2013;16:1170–1178. doi: 10.1038/nn.3495 23955561
25. Rescorla RA, Reduction in the effectiveness of reinforcement after prior excitatory conditioning. Learning and Motivation. 1970;1:372–381. doi: 10.1016/0023-9690(70)90101-3
26. Pavlov IP, Conditioned Reflexes. Oxford University Press; 1927.
27. Rescorla RA, Pavlovian conditioned inhibition. Psychological Bulletin. 1969;72:77–94. doi: 10.1037/h0027760
28. Kamin LJ, Attention-like associative processes in classical conditioning. In: Miami symposium on the prediction of behavior: Aversive stimulation. University of Miami Press, Miami, FL; 1968. p. 9–31.
29. Lubow RE, Latent inhibition. Psychological Bulletin. 1973;79:398–407. doi: 10.1037/h0034425 4575029
30. Rizley RC, Rescorla RA, Associations in second-order conditioning and sensory preconditioning. Journal of Comparative and Physiological Psychology. 1972;81:1–11. doi: 10.1037/h0033333 4672573
31. Shanks DR, Forward and backward blocking in human contingency judgement. The Quarterly Journal of Experimental Psychology. 1985;37:1–21. doi: 10.1080/14640748508402082
32. Chapman GB, Trial order affects cue interaction in contingency judgment. Journal of Experimental Psychology: Learning, Memory, and Cognition. 1991;17:837–854. 1834767
33. Miller RR, Matute H, Biological significance in forward and backward blocking: Resolution of a discrepancy between animal conditioning and human causal judgment. Journal of Experimental Psychology: General. 1996;125:370–386. doi: 10.1037/0096-3445.125.4.370
34. Matzel LD, Schachtman TR, Miller RR, Recovery of an overshadowed association achieved by extinction of the overshadowing stimulus. Learning and Motivation. 1985;16:398–412. doi: 10.1016/0023-9690(85)90023-2
35. Blaisdell AP, Gunther LM, Miller RR, Recovery from blocking achieved by extinguishing the blocking CS. Animal Learning & Behavior. 1999;27:63–76. doi: 10.3758/BF03199432
36. Blaisdell AP, Denniston JC, Miller RR, Recovery from the overexpectation effect: Contrasting performance-focused and acquisition-focused models of retrospective revaluation. Animal Learning & Behavior. 2001;29:367–380. doi: 10.3758/BF03192902
37. Kepecs A, Mainen ZF, A computational framework for the study of confidence in humans and animals. Philosophical Transactions of the Royal Society B: Biological Sciences. 2012;367:1322–1337. doi: 10.1098/rstb.2012.0037
38. Behrens TE, Woolrich MW, Walton ME, Rushworth MF, Learning the value of information in an uncertain world. Nature Neuroscience. 2007;10:1214–1221. doi: 10.1038/nn1954 17676057
39. Gershman SJ, The penumbra of learning: A statistical theory of synaptic tagging and capture. Network: Computation in Neural Systems. 2014;25:97–115.
40. Aguado L, Symonds M, Hall G, Interval between preexposure and test determines the magnitude of latent inhibition: Implications for an interference account. Animal Learning & Behavior. 1994;22:188–194. doi: 10.3758/BF03199919
41. Hallam SC, Matzel LD, Sloat JS, Miller RR, Excitation and inhibition as a function of posttraining extinction of the excitatory cue used in Pavlovian inhibition training. Learning and Motivation. 1990;21:59–84. doi: 10.1016/0023-9690(90)90004-8
42. Dickinson A, Burke J, Within compound associations mediate the retrospective revaluation of causality judgements. The Quarterly Journal of Experimental Psychology: Section B. 1996;49:60–80. doi: 10.1080/713932614
43. Stout SC, Miller RR, Sometimes-competing retrieval (SOCR): A formalization of the comparator hypothesis. Psychological Review. 2007;114:759–783. doi: 10.1037/0033-295X.114.3.759 17638505
44. Hull CL, The problem of stimulus equivalence in behavior theory. Psychological Review. 1939;46:9–30. doi: 10.1037/h0054032
45. Desmond J, Moore J, Adaptive timing in neural networks: The conditioned response. Biological Cybernetics. 1988;58:405–415. doi: 10.1007/BF00361347 3395634
46. Grossberg S, Schmajuk NA, Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural Networks. 1989;2:79–102. doi: 10.1016/0893-6080(89)90026-9
47. Williams BA, Conditioned reinforcement: Experimental and theoretical issues. The Behavior Analyst. 1994;17:261–285. 22478192
48. Schultz W, Dayan P, Montague PR, A neural substrate of prediction and reward. Science. 1997;275:1593–1599. doi: 10.1126/science.275.5306.1593 9054347
49. Sutton RS, Barto AG, Reinforcement Learning: An Introduction. MIT Press; 1998.
50. Daw ND, Courville AC, Touretzky DS, Representation and timing in theories of the dopamine system. Neural Computation. 2006;18:1637–1677. doi: 10.1162/neco.2006.18.7.1637 16764517
51. Ludvig EA, Sutton RS, Kehoe EJ, Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural Computation. 2008;20:3034–3054. doi: 10.1162/neco.2008.11-07-654 18624657
52. Gershman SJ, Moustafa AA, Ludvig EA, Time representation in reinforcement learning models of the basal ganglia. Frontiers in Computational Neuroscience. 2013;7.
53. Dearden R, Friedman N, Russell S. Bayesian Q-learning. In: Proceedings of the AAAI; 1998. p. 761–768.
54. Engel Y, Mannor S, Meir R. Bayes meets Bellman: The Gaussian process approach to temporal difference learning. In: International Conference on Machine Learning. vol. 20; 2003.
55. Denniston JC, Savastano HI, Blaisdell AP, Miller RR, Cue competition as a retrieval deficit. Learning and Motivation. 2003;34:1–31. doi: 10.1016/S0023-9690(02)00505-2
56. Rashotte ME, Griffin RW, Sisk CL, Second-order conditioning of the pigeon’s keypeck. Animal Learning & Behavior. 1977;5:25–38. doi: 10.3758/BF03209127
57. Leyland CM, Higher order autoshaping. The Quarterly Journal of Experimental Psychology. 1977;29:607–619. doi: 10.1080/14640747708400636
58. Rescorla RA, Aspects of the reinforcer learned in second-order Pavlovian conditioning. Journal of Experimental Psychology: Animal Behavior processes. 1979;5(1):79–95. 528880
59. Nairne JS, Rescorla RA, Second-order conditioning with diffuse auditory reinforcers in the pigeon. Learning and Motivation. 1981;12:65–91. doi: 10.1016/0023-9690(81)90025-4
60. Holland PC, Rescorla RA, Second-order conditioning with food unconditioned stimulus. Journal of Comparative and Physiological Psychology. 1975;88:459–467. doi: 10.1037/h0076219 1120816
61. Gibbs CM, Kehoe EJ, Gormezano I, Conditioning of the rabbit’s nictitating membrane response to a CSA-CSB-US serial compound: Manipulations of CSB’s associative character. Journal of Experimental Psychology: Animal Behavior Processes. 1991;17:423–432. 1744596
62. Shevill I, Hall G, Retrospective revaluation effects in the conditioned suppression procedure. Quarterly Journal of Experimental Psychology Section B. 2004;57:331–347.
63. Egger MD, Miller NE, Secondary reinforcement in rats as a function of information value and reliability of the stimulus. Journal of Experimental Psychology. 1962;64(2):97–104. doi: 10.1037/h0040364 13889429
64. Schmajuk NA, Mechanisms in Classical Conditioning: A Computational Approach. Cambridge University Press; 2010.
65. Pearce JM, A model for stimulus generalization in Pavlovian conditioning. Psychological Review. 1987;94:61–73. doi: 10.1037/0033-295X.94.1.61 3823305
66. O’Reilly JX, Jbabdi S, Behrens TE, How can a Bayesian approach inform neuroscience? European Journal of Neuroscience. 2012;35:1169–1179. doi: 10.1111/j.1460-9568.2012.08010.x 22487045
67. Shadmehr R, Mussa-Ivaldi S, Biological Learning and Control: How the Brain Builds Representations, Predicts Events, and Makes Decisions. MIT Press; 2012.
68. Swan JA, Pearce JM, The orienting response as an index of stimulus associability in rats. Journal of Experimental Psychology: Animal Behavior Processes. 1988;14:292–301. 3404083
69. Nassar MR, Wilson RC, Heasly B, Gold JI, An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. The Journal of Neuroscience. 2010;30:12366–12378. doi: 10.1523/JNEUROSCI.0822-10.2010 20844132
70. Mathys C, Daunizeau J, Friston KJ, Stephan KE, A Bayesian foundation for individual learning under uncertainty. Frontiers in Human Neuroscience. 2011;5. doi: 10.3389/fnhum.2011.00039 21629826
71. Gershman SJ, Niv Y, Learning latent structure: carving nature at its joints. Current Opinion in Neurobiology. 2010;20:251–256. doi: 10.1016/j.conb.2010.02.008 20227271
72. Pearce JM, Similarity and discrimination: a selective review and a connectionist model. Psychological Review. 1994;101:587–607. doi: 10.1037/0033-295X.101.4.587 7984708
73. O’Reilly RC, Rudy JW, Conjunctive representations in learning and memory: principles of cortical and hippocampal function. Psychological Review. 2001;108:311–345. doi: 10.1037/0033-295X.108.2.311 11381832
74. Redish AD, Jensen S, Johnson A, Kurth-Nelson Z, Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling. Psychological Review. 2007;114:784–805. doi: 10.1037/0033-295X.114.3.784 17638506
75. Soto FA, Gershman SJ, Niv Y, Explaining Compound Generalization in Associative and Causal Learning Through Rational Principles of Dimensional Generalization. Psychological Review. 2014;121:526–558. doi: 10.1037/a0037018 25090430
76. Park J, Sandberg IW, Universal approximation using radial-basis-function networks. Neural Computation. 1991;3:246–257. doi: 10.1162/neco.1991.3.2.246
77. Joel D, Niv Y, Ruppin E, Actor-critic models of the basal ganglia: New anatomical and computational perspectives. Neural Networks. 2002;15:535–547. doi: 10.1016/S0893-6080(02)00047-3 12371510
78. Ghavamzadeh M, Engel Y. Bayesian actor-critic algorithms. In: Proceedings of the 24th international conference on Machine learning. ACM; 2007. p. 297–304.