The actor-critic learning is behind the matching law: Matching versus optimal behaviors

Neural Computation

Volumn 20, Issue 1, 2008, Pages 227-251

  Sakai, Yutaka ^a   Fukai, Tomoki ^b  

^a TAMAGAWA UNIVERSITY   (Japan)

^b RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ANIMAL; ARTICLE; ARTIFICIAL INTELLIGENCE; BEHAVIOR; BIOLOGICAL MODEL; COMPUTER SIMULATION; DECISION MAKING; HUMAN; LEARNING; PHYSIOLOGY; REINFORCEMENT; REWARD;

ALGORITHMS; ANIMALS; ARTIFICIAL INTELLIGENCE; BEHAVIOR; COMPUTER SIMULATION; DECISION MAKING; HUMANS; LEARNING; MODELS, NEUROLOGICAL; REINFORCEMENT (PSYCHOLOGY); REWARD;

EID: 37749023538     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/neco.2008.20.1.227     Document Type: Article

References (44)

1. Barraclough, D., Conroy, M., & Lee, D. (2004). Prefrontal cortex and decision making in a mixed-strategy game. Nature Neuroscience, 7(4), 404-410.
2. Baum, W. M. (1981). Optimization and the matching law as accounts of instrumental behavior. Journal of the Experimental Analysis of Behavior, 36, 387-402.
3. Baum, W., & Rachlin, H. (1969). Choice as time allocation. Journal of the Experimental Analysis of Behavior, 12, 861-874.
4. Breiter, H. C., Aharon, I., Kahneman, D., Dale, A., & Shizgal, P. (2001). Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron, 30, 619-639.
5. Davison, M., & McCarthy, D. (1987). The matching law: A research review. Mahwah, NJ: Erlbaum.
6. Daw, N. D., & Touretzky, D. S. (2001). Operant behavior suggests attentional gating of dopamine system inputs. Neurocomputing, 38-40, 1161-1167.
7. Daw, N. D., & Touretzky, D. S. (2002). Long-term reward prediction in TD models of the dopamine system. Neural Computation, 14, 2567-2583.
8. Dayan, P., & Abbott, L. (2001). Theoretical Neuroscience. Cambridge, MA: MIT press.
9. Dayan, P., & Balleine, B. W. (2002). Reward, motivation, and reinforcement learning. Neuron, 36, 285-298.
10. DeCarlo, L. T. (1985). Matching and maximizing with variable-time schedules. Journal of the Experimental Analysis of Behavior, 43, 75-81.
11. Doya, K. (2000). Complementary roles of basal ganglia and cerebellum in learning and motor control. Current Opinion in Neurobiology, 10, 732-739.
12. Gallistel, C., Mark, T., King, A., & Latham, P. (2001). The rat approximates an ideal detector of changes in rates of reward: Implications for the law of effect. J. Exp. Psychol. Anim. Behav. Processes, 27, 354-372.
13. Haruno, M., Kuroda, T., Doya, K., Toyama, K., Kimura, M., Samejima, K., Imamizu, H., & Kawato, M. (2004). A neural correlate of reward-based behavioral learning in caudate nucleus: A functional magnetic resonance imaging study of a stochastic decision task. J. Neurosci., 24(7), 1660-1665.
14. Herrnstein, R. J. (1997). The matching law: Papers in psychology and economics. Cambridge, MA: Harvard University Press.
15. Herrnstein, R. J., & Heyman, G. M. (1979). Is matching compatible with reinforcement maximization on concurrent variable interval, variable ratio? Journal of the Experimental Analysis of Behavior, 31, 209-223.
16. Herrnstein, R. J., & Vaughan, W. J. (1980). Melioration and behavioral allocation. In J. Staddon (Ed.), Limits to action: The allocation of individual behavior. New York: Academic Press.
17. Heyman, G. M. (1979). A Markov model description of changeover probabilities on concurrent variable-interval schedules. Journal of the Experimental Analysis of Behavior, 31, 41-51.
18. Heyman, G., & Monaghan, M. (1994). Reinforcer magnitude (sucrose concentration) and the matching law theory of response strength. Journal of the Experimental Analysis of Behavior, 61, 505-516.
19. Houk, J. C., Davis, J. L., & Beiser, D. G. (1994). Models of information processing in the basal ganglia (computational neuroscience). Cambridge, MA: Bradford Books, MIT Press.
20. Houston, A. I., & McNamara, J. (1981). How to maximize reward rate in two variable-interval paradigms. Journal of the Experimental Analysis of Behavior, 35, 367-396.
21. Jacobs, E. A., & Hackenberg, T. D. (1996). Humans' choices in situations of time-based diminishing returns: Effects of fixed-interval duration and progressive-interval step size. Journal of the Experimental Analysis of Behavior, 65, 5-19.
22. Knutson, B., Adams, C. M., Fong, G. W., & Hommer, D. J. (2001). Anticipation of increasing monetary reward selectively recruits nucleus accumbens. J. Neuroscience, 15, 1-5.
23. Mazur, J. (1981). Optimization theory fails to predict performance of pigeons in a two-response situation. Science, 224(4522), 823-825.
24. Mazur, J. E. (2005). Learning and behavior.(6th ed.). Upper Saddle River, NJ: Prentice Hall.
25. McClure, S., Berns, G. S., & Montague, P. (2003). Temporal prediction errors in a passive learning task activate human striatum. Neuron, 38(2), 339-346.
26. Montague, P., & Berns, G. (2002). Neural economics and the biological substrates of valuation. Neuron, 36(2), 265-284.
27. Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neuroscience, 16, 1936-1947.
28. Morris, G., Arkadir, D., Nevet, A., Vaadia, E., & Bergman, H. (2004). Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron, 43, 133-143.
29. Platt, M., & Glimcher, P. (1999). Neural correlates of decision variables in parietal cortex. Nature, 400(6741), 233-238.
30. Rachlin, H., Green, L., Kagel, J., & Battalio, R. (1976). Economic demand theory and psychological studies of choice. In G. Bower (Ed.), The psychology of learning and motivation (Vol. 10, pp. 129-154). New York: Academic Press.
31. Sakagami, T., Hursh, S. R., Christensen, J., & Silberberg, A. (1989). Income maximizing in concurrent interval-ratio schedules. Journal of the Experimental Analysis of Behavior, 52, 41-46.
32. Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005). Representation of actionspecific-reward value in the striatum. Science, 310, 1337-1340.
33. Savastano, H. I., & Fantino, E. (1994). Human choice in concurrent ratio-interval schedules of reinforcement. Journal of the Experimental Analysis of Behavior, 61, 453-463.
34. Schultz, W. (2004). Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology. Current Opinion in Neurobiology, 14, 139-147.
35. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599.
36. Seung, H. (2003). Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron, 40(6), 1063-1073.
37. Silberberg, A., Thomas, J., & Brendzen, N. (1991). Human choice on concurrent variable-interval, variable-ratio schedules. Journal of the Experimental Analysis of Behavior, 56, 575-584.
38. Staddon, J., & Hinson, J. (1983). Optimization: A result or a mechanism? Science, 221, 976-977.
39. Stubbs, D. A., Pliskoff, S. S., & Reid, H. M. (1977). Concurrent schedules: A quantitative relation between changeover behavior and its consequences. Journal of the Experimental Analysis of Behavior, 27, 85-96.
40. Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2004). Matching behavior and the representation of value in the parietal cortex. Science, 304, 1782-1787.
41. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning. Cambridge, MA: MIT press.
42. Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2004). Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nature Neuroscience, 7, 887-893.
43. Vyse, S. A., & Belke, T. W. (1992). Maximizing versus matching on concurrent variable-interval schedules. Journal of the Experimental Analysis of Behavior, 58, 325-334.
44. Wang, X. (2002). Probabilistic decision making by slow reverberation in cortical circuits. Neuron, 36(5), 955-968.