Evaluating the TD model of classical conditioning

Learning and Behavior

Volumn 40, Issue 3, 2012, Pages 305-319

  Ludvig, Elliot A ^a   Sutton, Richard S ^b   Kehoe, E James ^c  

^a Princeton University   (United States)

^b UNIVERSITY OF ALBERTA   (Canada)

^c UNIVERSITY OF NEW SOUTH WALES   (Australia)

Author keywords

Associative learning; Classical conditioning; Reinforcement learning; Timing

Indexed keywords

ALGORITHM; ANIMAL; ARTICLE; COMPUTER SIMULATION; CONDITIONED REFLEX; LEARNING; PSYCHOLOGICAL MODEL; REINFORCEMENT; STATISTICS; TIME;

ALGORITHMS; ANIMALS; ASSOCIATION LEARNING; COMPUTER SIMULATION; CONDITIONING, CLASSICAL; MODELS, PSYCHOLOGICAL; REINFORCEMENT (PSYCHOLOGY); TIME FACTORS;

EID: 84870910490     PISSN: 15434494     EISSN: 15434508     Source Type: Journal    
DOI: 10.3758/s13420-012-0082-6     Document Type: Article

References (54)

1. Amundson, J. C., & Miller, R. R. (2008). CS-US temporal relations in blocking. Learning & Behavior, 36, 92-103.
2. Brown, J., Bullock, D., & Grossberg, S. (1999). How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. Journal of Neuroscience, 19, 10502-10511. (Pubitemid 30228056)
3. Buhusi, C. V., & Schmajuk, N. A. (1999). Timing in simple conditioning and occasion setting: A neural network approach. Behavioural Processes, 45, 33-57. (Pubitemid 29134913)
4. Church, R. M., & Kirkpatrick, K. (2001). Theories of conditioning and timing. In R. R. Mowrer & S. B. Klein (Eds.), Contemporary learning: Theory and applications (pp. 211-253). Hillsdale, NJ: Erlbaum.
5. Cole, S., & McNally, G. P. (2007). Temporal-difference prediction errors and Pavlovian fear conditioning: Role of NMDA and opioid receptors. Behavioral Neuroscience, 121, 1043-1052. (Pubitemid 350051306)
6. Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006). Representation and timing in theories of the dopamine system. Neural Computation, 18, 1637-1677. (Pubitemid 44024733)
7. Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8, 1704-1711. (Pubitemid 41683198)
8. Desmond, J. E., & Moore, J. W. (1988). Adaptive timing in neural networks: The conditioned response. Biological Cybernetics, 58, 405-415.
9. Drew, M. R., Zupan, B., Cooke, A., Couvillon, P. A., & Balsam, P. D. (2005). Temporal control of conditioned responding in goldfish. Journal of Experimental Psychology Animal Behavior Processes, 31, 31-39. (Pubitemid 40176536)
10. Egger, M. D., & Miller, N. E. (1962). Secondary reinforcement in rats as a function of information value and reliability of the stimulus. Journal of Experimental Psychology, 64, 97-104.
11. Frey, P. W., & Sears, R. J. (1978). Model of conditioning incorporating the Rescorla-Wagner associative axiom, a dynamic attention process, and a catastrophe rule. Psychological Review, 85, 321-348.
12. Gallistel, C. R., & King, A. P. (2009). Memory and the computational brain. Medford, MA: Wiley-Blackwell.
13. Grossberg, S., & Schmajuk, N. A. (1989). Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural Networks, 2, 79-102.
14. Honey, R. C., & Hall, G. (1989). Acquired equivalence and distinctiveness of cues. Journal of Experimental Psychology: Animal Behavior Processes, 15, 338-346.
15. Jennings, D. J., Bonardi, C., & Kirkpatrick, K. (2007). Overshadowing and stimulus duration. Journal of Experimental Psychology: Animal Behavior Processes, 33, 464-475. (Pubitemid 350100768)
16. Jennings, D. J., & Kirkpatrick, K. (2006). Interval duration effects on blocking in appetitive conditioning. Behavioural Processes, 71, 318-329. (Pubitemid 43163028)
17. Jin, D. Z., Fujii, N., & Graybiel, A. M. (2009). Neural representation of time in cortico-basal ganglia circuits. Proceedings of the National Academy of Sciences, 106, 19156-19161.
18. Kehoe, E. J., Ludvig, E. A., Dudeney, J. E., Neufeld, J., & Sutton, R. S. (2008). Magnitude and timing of nictitating membrane movements during classical conditioning of the rabbit (Oryctolagus cuniculus). Behavioral Neuroscience, 122, 471-476. (Pubitemid 351672338)
19. Kehoe, E. J., Ludvig, E. A., & Sutton, R. S. (2009a). Magnitude and timing of CRs in delay and trace classical conditioning of the nictitating membrane response of the rabbit (Oryctolagus cuniculus). Behavioral Neuroscience, 123, 1095-1101.
20. Kehoe, E. J., Olsen, K. N., Ludvig, E. A., & Sutton, R. S. (2009b). Scalar timing varies with response magnitude in classical conditioning of the nictitating membrane response of the rabbit (Oryctolagus cuniculus). Behavioral Neuroscience, 123, 212-217.
21. Kehoe, E. J., Schreurs, B. G., & Amodei, N. (1981). Blocking acquisition of the rabbit's nictitating membrane response to serial conditioned stimuli. Learning and Motivation, 12, 92-108.
22. Kehoe, E. J., Schreurs, B. G., & Graham, P. (1987). Temporal primacy overrides prior training in serial compound conditioning of the rabbit's nictitating membrane response. Animal Learning & Behavior, 15, 455-464. (Pubitemid 18000790)
23. Kehoe, E. J., & White, N. E. (2002). Extinction revisited: Similarities between extinction and reductions in US intensity in classical conditioning of the rabbit's nictitating membrane response. Animal Learning & Behavior, 30, 96-111. (Pubitemid 34651254)
24. Ludvig, E. A., Bellemare, M. G., & Pearson, K. G. (2011). A primer on reinforcement learning in the brain: Psychological, computational, and neural perspectives. In E. Alonso & E. Mondragon (Eds.), Computational neuroscience for advancing artificial intelligence: Models, methods and applications (pp. 111-144). Hershey, PA: IGI Global.
25. Ludvig, E. A., & Koop, A. (2008). Learning to generalize through predictive representations: A computational model of mediated conditioning. In From Animals to Animats 10: Proceedings of Simulation of Adaptive Behavior (SAB-08), 342-351.
26. Ludvig, E. A., Mirian, M. S., Sutton, R. S., & Kehoe, E. J. (2012). Associative learning from replayed experience. Manuscript submitted for publication.
27. Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2008). Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural Computation, 20, 3034-3054.
28. Ludvig, E. A., Sutton, R. S., Verbeek, E. L., & Kehoe, E. J. (2009). A computational model of hippocampal function in trace conditioning. Advances in Neural Information Processing Systems (NIPS-08), 21, 993-1000.
29. MacDonald, C. J., Lepage, K. Q., Eden, U. T., & Eichenbaum, H. (2011). Hippocampal "time cells" bridge the gap in memory for discontiguous events. Neuron, 71, 737-749.
30. Machado, A. (1997). Learning the temporal dynamics of behavior. Psychological Review, 104, 241-265. (Pubitemid 127455260)
31. Maia, T. V. (2009). Reinforcement learning, conditioning, and the brain: Successes and challenges. Cognitive, Affective, & Behavioral Neuroscience, 9, 343-364.
32. Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience, 16, 1936-1947. (Pubitemid 26145969)
33. Moore, J. W., & Choi, J. S. (1997). The TD model of classical conditioning: Response topography and brain implementation. In J. W. Donahoe & V. P. Dorsel (Eds.), Neural-network models of cognition, biobehavioral foundations (Advances in Psychology (pp, Vol. 121, pp. 387-405). Amsterdam: North-Holland/Elsevier.
34. Moore, J. W., Desmond, J. E., Berthier, N. E., Blazis, D. E. J., Sutton, R. S., & Barto, A. G. (1986). Simulation of the classically conditioned nictitating membrane response by a neuron-like adaptive element: Response topography, neuronal firing and inter-stimulus intervals. Behavioral Brain Research, 21, 143-154. (Pubitemid 16042818)
35. Niv, Y. (2009). Reinforcement learning in the brain. Journal of Mathematical Psychology, 53, 139-154.
36. Pan, W. X., Schmidt, R., Wickens, J. R., & Hyland, B. I. (2008). Tripartite mechanism of extinction suggested by dopamine neuron activity and temporal difference model. Journal of Neuroscience, 28, 9619-9631.
37. Pearce, J. M. (1987). A model of stimulus generalization for Pavlovian conditioning. Psychological Review, 94, 61-73.
38. Pearce, J. M. (1994). Similarity and discrimination: A selective review and a connectionist model. Psychological Review, 101, 587-607. (Pubitemid 24976818)
39. Pearce, J. M., & Hall, G. (1980). A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychological Review, 87, 532-552.
40. Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II (pp. 64-99). New York: Appleton-Century-Crofts.
41. Roberts, S. (1981). Isolation of an internal clock. Journal of Experimental Psychology: Animal Behavior Processes, 7, 242-268.
42. Schreurs, B. G., & Westbrook, R. F. (1982). The effects of changes in the CS-US interval during compound conditioning upon an otherwise blocked element. Quarterly Journal of Experimental Psychology, 34B, 19-30.
43. Schultz, W. (2006). Behavioral theories and the neurophysiology of reward. Annual Review of Psychology, 57, 87-115. (Pubitemid 43237228)
44. Schultz,W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599. (Pubitemid 27120526)
45. Smith, M. C. (1968). CS-US interval and US intensity in classical conditioning of the rabbit's nictitating membrane response. Journal of Comparative and Physiological Psychology, 66, 679-687.
46. Smith, M. C., Coleman, S. R., & Gormezano, I. (1969). Classical conditioning of the rabbit's nictitating membrane response at backward, simultaneous, and forward CS-US intervals. Journal of Comparative and Physiological Psychology, 69, 226-231.
47. Suri, R. E., & Schultz, W. (1999). A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience, 91, 871-890. (Pubitemid 29225621)
48. Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9-44.
49. Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. International Conference on Machine Learning (ICML), 7, 216-224.
50. Sutton, R. S. (1992). Adapting bias by gradient descent: An incremental version of delta-bar-delta. National Conference on Artificial Intelligence, 10, 171-176.
51. Sutton, R. S., & Barto, A. G. (1981). Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review, 88, 135-171.
52. Sutton, R. S., & Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In M. Gabriel & J. W. Moore (Eds.), Learning and computational neuroscience (pp. 497-537). Cambridge, MA: MIT Press.
53. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
54. Vogel, E. H., Brandon, S. E., & Wagner, A. R. (2003). Stimulus representation in SOP: II. An application to inhibition of delay. Behavioural Processes, 62, 27-48. (Pubitemid 36511421)