Discovering latent causes in reinforcement learning

Current Opinion in Behavioral Sciences

Volumn 5, Issue , 2015, Pages 43-50

  Gershman, Samuel J ^a   Norman, Kenneth A ^b   Niv, Yael ^b  

^a HARVARD UNIVERSITY   (United States)

^b PRINCETON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ASSOCIATION; COGNITION; CONDITIONING; LATENT PERIOD; LEARNING; REINFORCEMENT; REVIEW; STIMULUS RESPONSE; THEORETICAL STUDY;

EID: 84938863250     PISSN: None     EISSN: 23521546     Source Type: Journal    
DOI: 10.1016/j.cobeha.2015.07.007     Document Type: Review

References (62)

1. Polo M. The Travels of Marco Polo 1918, JM Dent & Sons.
2. Braun D.A., Mehring C., Wolpert D.M. Structure learning in action. Behav Brain Res 2010, 206:157-165.
3. Gershman S.J., Niv Y. Learning latent structure: carving nature at its joints. Curr Opin Neurobiol 2010, 20:251-256.
4. McCallum R.A. Hidden state and reinforcement learning with instance-based state identification. IEEE Trans Syst Man Cybernet, Part B 1996, 26:464-473.
5. Givan R., Dean T., Greig M. Equivalence notions and model minimization in Markov decision processes. Artif Intell 2003, 147:163-223.
6. Roy N., Gordon G.J., Thrun S. Finding approximate POMDP solutions through belief compression. J Artif Intell Res 2005, 23:1-40.
7. Courville A.C., Daw N.D., Touretzky D.S. Bayesian theories of conditioning in a changing world. Trends Cogn Sci 2006, 10:294-300.
8. Gershman D.J., Blei D.M., Niv Y. Context, learning, and extinction. Psychol Rev 2010, 117:197-209.
9. Gershman S.J., Niv Y. Exploring a latent cause theory of classical conditioning. Learn Behav 2012, 40:255-268.
10. Gershman D.J., Radulescu A., Norman K.A., Niv Y. Statistical computations underlying the dynamics of memory updating. PLOS Comput Biol 2014, 10:e1003939.
11. Gershman S.J., Hartley C. Individual differences in learning predict the return of fear. Learn Behav 2015, 43:243-250.
12. Lloyd K., Leslie D.S. Context-dependent decision-making: a simple Bayesian model. J R Soc Interface 2013, 10:20130069.
13. Soto F.A., Gershman S.J., Niv Y. Explaining compound generalization in associative and causal learning through rational principles of dimensional generalization. Psychol Rev 2014, 121:526-558.
14. Austerweil J.L., Gershman S.J., Tenenbaum J.B., Griffiths T.L. Structure and flexibility in Bayesian models of cognition. Oxford Handbook of Computational and Mathematical Psychology 2015, Oxford University Press, Oxford. J.R. Busemeyer, J.T. Townsend, Z. Wang, A. Eidels (Eds.).
15. Rescorla R.A., Wagner A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. Classical Conditioning II: Current Research and Theory 1972, 64-99. Appleton-Century-Crofts, New York, NY. A.H. Black, W.F. Prokasy (Eds.).
16. Bouton M.E. Context and behavioral processes in extinction. Learn Memory 2004, 11:485-494.
17. Chater N., Vitányi P. Simplicity: a unifying principle in cognitive science?. Trends Cogn Sci 2003, 7:19-22.
18. Gershman S.J., Niv Y. Perceptual estimation obeys Occam's razor. Front Psychol 2013, 4.
19. Gershman S.J., Blei D.M. A tutorial on Bayesian nonparametric models. J Math Psychol 2012, 56:1-12.
20. Gershman S.J., Jones C.E., Norman K.A., Monfils M.-H., Niv Y. Gradual extinction prevents the return of fear: implications for the discovery of state. Front Behav Neurosci 2013, 7.
21. Rescorla R.A. Spontaneous recovery. Learn Memory 2004, 11:501-509.
22. Rescorla R.A., Heth C.D. Reinstatement of fear to an extinguished conditioned stimulus. J Exp Psychol: Anim Behav Process 1975, 1:88-96.
23. Bouton M.E., Bolles R.C. Role of conditioned contextual stimuli in reinstatement of extinguished fear. J Exp Psychol: Anim Behav Process 1979, 5:368-378.
24. Miller R.R., Matzel L.D. Retrieval failure versus memory loss in experimental amnesia: definitions and processes. Learn Memory 2006, 13:491-497.
25. Kagerer F.A., Contreras-Vidal J.L., Stelmach G.E. Adaptation to gradual as compared with sudden visuo-motor distortions. Exp Brain Res 1997, 115:557-561.
26. Pekny S.E., Criscimagna-Hemminger S.E., Shadmehr R. Protection and expression of human motor memories. J Neurosci 2011, 31:13829-13839.
27. Taylor J.A., Wojaczynski G.J., Ivry R.B. Trial-by-trial analysis of intermanual transfer during visuomotor adaptation. J Neurophysiol 2011, 106:3157-3172.
28. Wallis G., Bülthoff H.H. Effects of temporal association on recognition memory. Proc Natl Acad Sci U S A 2001, 98:4800-4804.
29. Preminger S., Sagi D., Tsodyks M. The effects of perceptual history on memory of visual objects. Vis Res 2007, 47:965-973.
30. Preminger S., Blumenfeld B., Sagi D., Tsodyks M. Mapping dynamic memories of gradually changing objects. Proc Natl Acad Sci U S A 2009, 106:5371-5376.
31. Pearce J.M. A model for stimulus generalization in Pavlovian conditioning. Psychol Rev 1987, 94:61-73.
32. Wagner A.R. Context-sensitive elemental theory. Q J Exp Psychol: Sect B 2003, 56:7-29.
33. Harris J.A. Elemental representations of stimuli in associative learning. Psychol. Rev. 2006, 113:584-605.
34. Melchers K.G., Shanks D.R., Lachnit H. Stimulus coding in human associative learning: flexible representations of parts and wholes. Behav Process 2008, 77:413-427.
35. Pearce J.M. Similarity and discrimination: a selective review and a connectionist model. Psychol Rev 1994, 101:587-607.
36. Kehoe E.J., Horne A.J., Horne P.S., Macrae M. Summation and configuration between and within sensory modalities in classical conditioning of the rabbit. Anim Learn Behav 1994, 22:19-26.
37. Rescorla R.A., Coldwell S.E. Summation in autoshaping. Anim Learn Behav 1995, 23:314-326.
38. Glautier S. Spatial separation of target and competitor cues enhances blocking of human causality judgements. Q J Exp Psychol: Sect B 2002, 55:121-135.
39. Livesey E.J., Boakes R.A. Outcome additivity, elemental processing and blocking in human causality judgements. Q J Exp Psychol: Sect B 2004, 57:361-379.
40. Shepard R.N. Toward a universal law of generalization for psychological science. Science 1987, 237:1317-1323.
41. Tenenbaum J.B., Griffiths T.L. Generalization, similarity, and Bayesian inference. Behav Brain Sci 2001, 24:629-640.
42. Anderson J.R. The adaptive nature of human categorization. Psychol Rev 1991, 98:409-429.
43. Goldwater S., Griffiths T.L., Johnson M. A Bayesian framework for word segmentation: exploring the effects of context. Cognition 2009, 112:21-54.
44. Sanborn A.N., Griffiths T.L., Navarro D.J. Rational approximations to rational models: alternative algorithms for category learning. Psychol Rev 2010, 117:1144-1167.
45. Kemp C., Tenenbaum J.B., Niyogi S., Griffiths T.L. A probabilistic model of theory formation. Cognition 2010, 114:165-196.
46. Collins A.G.E., Frank M.J. Cognitive control over learning: creating, clustering, and generalizing task-set structure. Psychol Rev 2013, 120:190-229.
47. Collins A.G.E., Cavanagh J.F., Frank M.J. Human EEG uncovers latent generalizable rule structure during learning. J Neurosci 2014, 34:4677-4685.
48. Fuhs M.C., Touretzky D.S. Context learning in the rodent hippocampus. Neural Comput 2007, 19:3173-3215.
49. Takahashi Y.K., Roesch M.R., Wilson R.C., Toreson K., O'Donnell P., Niv Y., Schoenbaum G. Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat Neurosci 2011, 14:1590-1597.
50. Wilson R.C., Takahashi Y.K., Schoenbaum G., Niv Y. Orbitofrontal cortex as a cognitive map of task space. Neuron 2014, 81:267-279.
51. Aggleton J.P., Sanderson D.J., Pearce J.M. Structural learning and the hippocampus. Hippocampus 2007, 17:723-734.
52. Ji J., Maren S. Hippocampal involvement in contextual modulation of fear extinction. Hippocampus 2007, 17:749-758.
53. Crossley M.J., Ashby F.G., Maddox W.T. Erasing the engram: the unlearning of procedural skills. J Exp Psychol: Gen 2013, 142:710-741.
54. Bradfield L.A., Bertran-Gonzalez J., Chieng B., Balleine B.W. The thalamostriatal pathway and cholinergic control of goal-directed action: interlacing new with existing learning in the striatum. Neuron 2013, 79:153-166.
55. Schoenbaum G., Stalnaker T.A., Niv Y. How did the chicken cross the road? With her striatal cholinergic interneurons, of course. Neuron 2013, 79:3-6.
56. Aldous D. Exchangeability and related topics. École d'Été de Probabilités de Saint-Flour XIII 1985, 1-198. Springer, Berlin.
57. Griffiths T.L., Ghahramani Z. The Indian buffet process: an introduction and review. J Mach Learn Res 2011, 12:1185-1224.
58. Austerweil J.L., Griffiths T.L. A rational model of the effects of distributional information on feature learning. Cogn Psychol 2011, 63:173-209.
59. Austerweil J.L., Griffiths T.L. A nonparametric Bayesian framework for constructing flexible feature representations. Psychol Rev 2013, 120:817-851.
60. Yildirim I., Jacobs R.A. A rational analysis of the acquisition of multisensory representations. Cogn Sci 2012, 36:305-332.
61. Austerweil J.L. Testing the psychological validity of cluster construction biases. Proceedings of the 36th Annual Conference of the Cognitive Science Society 2014.
62. Sanborn A.N., Griffiths T.L., Shiffrin R.M. Uncovering mental representations with Markov chain Monte Carlo. Cogn Psychol 2010, 60:63-106.