Explicit melioration by a neural diffusion model

Brain Research

Volumn 1299, Issue , 2009, Pages 95-117

  Simen, Patrick ^a   Cohen, Jonathan D ^a  

^a PRINCETON UNIVERSITY   (United States)

Author keywords

Decision making; Drift diffusion; Instrumental conditioning; Matching; Melioration; Neural network; Operant conditioning; Reinforcement learning; Reward rate

Indexed keywords

ARTICLE; ARTIFICIAL NEURAL NETWORK; DECISION MAKING; INSTRUMENTAL CONDITIONING; MATHEMATICAL MODEL; PRIORITY JOURNAL; REINFORCEMENT; REWARD; ANIMAL; BIOLOGICAL MODEL; COMPUTER SIMULATION; NERVE CELL; NERVE CELL PLASTICITY; PHYSIOLOGY; PROBABILITY; PSYCHOLOGICAL MODEL;

ANIMALIA;

ANIMALS; CHOICE BEHAVIOR; COMPUTER SIMULATION; MODELS, NEUROLOGICAL; MODELS, PSYCHOLOGICAL; NEURAL NETWORKS (COMPUTER); NEURONAL PLASTICITY; NEURONS; PROBABILITY; REINFORCEMENT SCHEDULE; REWARD;

EID: 70449709575     PISSN: 00068993     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.brainres.2009.07.017     Document Type: Article

References (89)

1. Alexander G.E., DeLong M.R., and Strick P.L. Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu. Rev. Neurosci. 9 (1986) 357-381
2. Aston-Jones G., and Cohen J.D. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28 (2005) 403-450
3. Barraclough D.J., Conroy M.L., and Lee D. Prefrontal cortex and decision making in a mixed-strategy game. Nat. Neurosci. 7 4 (2004) 404-410
4. Baum W.A. On two types of deviation from the matching law: bias and undermatching. J. Exp. Anal. Behav. 22 1 (July 1974) 231-242
5. Blough D. Reaction time signatures of discriminative processes: differential effects of stimulus similarity and incentive. Learn. Behav. 32 2 (2004) 157-172
6. Bogacz R., Brown E., Moehlis J., Holmes P., and Cohen J.D. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced choice tasks. Psychol. Rev. 113 4 (2006) 700-765
7. Bogacz R., and Gurney K. The basal ganglia and cortex implement optimal decision making between alternative actions. Neural Comput. 19 (2007) 442-477
8. Bogacz R., McClure S.M., Li J., Cohen J.D., and Montague P.R. Short-term memory traces for action bias in human reinforcement learning. Brain Res. 1153 (2007) 111-121
9. Busemeyer J.R., and Townsend J.T. Fundamental derivations from decision field theory. Math. Soc. Sci. 23 3 (1992) 255-282
10. Busemeyer J.R., and Townsend J.T. Decision field theory: a dynamic-cognitive approach to decision making in an uncertain environment. Psychol. Rev. 100 3 (1993) 432-459
11. Bush R.R., and Mosteller F. A mathematical model for simple learning. Psychol. Rev. 58 5 (1951) 313-323
12. Church R.M., Meck W.H., and Gibbon J. Application of scalar timing theory to individual trials. J. Exp. Psychol., Anim. Behav. Processes 20 (1994) 135-155
13. Cohen J.D., Dunbar K., and McClelland J.L. On the control of automatic processes: a parallel distributed processing account of the Stroop effect. Psychol. Rev. 97 3 (1990) 332-361
14. Cohen J.D., McClure S.M., and Yu A.J. Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philos. Trans. R. Soc. B 362 (2007) 933-942
15. Corrado G.S., Sugrue L.P., Seung H.S., and Newsome W.T. Linear-nonlinear-poisson models of primate choice dynamics. J. Exp. Anal. Behav. 84 3 (2005) 581-617
16. Davison M. Interresponse times and the structure of choice. Behavioral Processes 66 (2004) 173-187
17. Davison M., and McCarthy D. The Matching Law: A Research Review (1988), Lawrence Erlbaum Associates, Hillsdale, N.J.
18. Davison M., and Tustin R.D. The relation between the generalized matching law and signal-detection theory. J. Exp. Anal. Behav. 29 (1978) 331-336
19. Daw N.D., O'Doherty J.P., Dayan P., Seymour B., and Dolan R.J. Cortical substrates for exploratory decisions in humans. Nature 441 (2006) 876-879
20. De Villiers P.A., and Herrnstein R.J. Toward a law of response strength. Psychol. Bull. 83 6 (1976) 1131-1153
21. Ditterich J. Stochastic models of decisions about motion direction: behavior and physiology. Neural Netw. 19 (2006) 981-1012
22. Egelman D.M., Person C., and Montague P.R. A computational role for dopamine delivery in human decision making. J. Cogn. Neurosci. 10 5 (1998) 623-630
23. Ferster C.B., and Skinner B.F. Schedules of Reinforcement (1957), Appleton-Century-Crofts, New York
24. Frank M.J. Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making. Neural Netw. 19 8 (2006) 1120-1136
25. Gallistel C.R., Mark T.A., King A., and Latham P. The rat approximates an ideal detector of changes in rates of reward: implications for the law of effect. J. Exp. Psychol., Anim. Behav. Processes 27 (2001) 354-372
26. Gardiner C.W. Handbook of Stochastic Methods. 3rd Edition (2004), Springer-Verlag, New York, NY
27. Gerstner W. Population dynamics of spiking neurons: fast transients, asynchronous states, and locking. Neural Comput. 12 (1999) 43-89
28. Glimcher P.W. Indeterminacy in brain and behavior. Annu. Rev. Psychol. 56 (2005) 25-56
29. Gold J.I., and Shadlen M.N. Neural Comput.s that underlie decisions about sensory stimuli. Trends Cogn. Sci. 5 1 (2001) 10-16
30. Gold J.I., and Shadlen M.N. Banburismus and the brain: decoding the relationship between sensory stimuli, decisions, and reward. Neuron 36 2 (2002) 299-308
31. Green D.M., and Swets J.A. Signal Detection Theory and Psychophysics (1966), Wiley, New York
32. Grossberg S. On the dynamics of operant conditioning. J. Theor. Biol. 33 (1971) 225-255
33. Grossberg S. A psychophysiological theory of reinforcement, drive, motivation and attention. J. Theor. Neurobiol. 1 (1982) 286-369
34. Grossberg S. Nonlinear neural networks principles, mechanisms, and architectures. Neural Netw. 1 (1988) 17-61
35. Grossberg S., and Gutowski W. Neural dynamics of decision making under risk: affective balance and cognitive-emotional interactions. Psychol. Rev. 94 3 (1987) 300-318
36. Herrnstein R.J. Relative and absolute strength of responses as a function of frequency of reinforcement. J. Exp. Anal. Behav. 4 (1961) 267-272
37. Herrnstein R.J. Melioration as behavioral dynamism. In: Commons M.L., Herrnstein R.J., and Rachlin H. (Eds). Quantitative analyses of behavior. Vol. II: Matching and maximizing accounts (1982), Ballinger, Cambridge, MA
38. Herrnstein R.J. The Matching Law: Papers in Psychology and Economics (1997), Harvard University Press, Cambridge, MA
39. Herrnstein R.J., and Prelec D. Melioration: a theory of distributed choice. J. Econ. Perspect. 5 (1991) 137-156
40. Herrnstein R.J., and Prelec D. A theory of addiction. In: Rachlin H., and Laibson D.I. (Eds). The Matching Law: Papers in Psychology and Economics (1997), Russell Sage Foundation and Harvard University Press, Cambridge, MA 160-187 Ch. 9
41. Herrnstein R.J., and Vaughan W.J. Melioration and behavioral allocation. In: Staddon J. (Ed). Limits to Action: The Allocation of Individual Behavior (1980), Academic Press, New York
42. Hertz J., Krogh A., and Palmer R.G. Introduction to the Theory of Neural Comput.. Vol. 1 of Santa Fe Institute Studies in the Sciences of Complexity Lecture Notes (1991), Addison Wesley, Redwood City, CA
43. Killeen P.R. Principles of reinforcement. Behav. Brain Sci. 17 1 (1994) 105-172
44. Lau B., and Glimcher P.W. Dynamic response-by-response models of matching behavior in rhesus monkeys. J. Exp. Anal. Behav. 84 3 (2005) 555-579
45. Lo C.-C., and Wang X.-J. Cortico-basal ganglia circuit mechanism for a decision threshold in reaction time tasks. Nat. Neurosci. 9 7 (2006) 956-963
46. Loewenstein Y., and Seung H.S. Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proc. Natl. Acad. Sci. U. S. A. 103 41 (October 2006) 15224-15229
47. Luce R.D. Response Times: Their Role in Inferring Elementary Mental Organization (1986), Oxford University Press, New York
48. McClure S.M., Gilzenrat M.S., and Cohen J.D. An exploration-exploitation model based on norepinephrine and dopamine activity. In: Weiss Y., Scholkopf B., and Platt J. (Eds). Advances in Neural Information Processing Vol. 18 (2006), MIT Press, Cambridge, MA 867-874
49. McMillen T., and Holmes P. The dynamics of choice among multiple alternatives. J. Math. Psychol. 50 (2006) 30-57
50. Moehlis, J., Brown, E., Bogacz, R., Holmes, P., Cohen, J. D., 2004. Optimizing Reward Rate in Two Alternative Choice Tasks: Mathematical Formalism. Tech. Rep. 04-01, Center for the Study of Brain, Mind and Behavior, Princeton University.
51. Montague P.R., and Berns G.S. Neural economics and the biological substrates of valuation. Neuron 36 (2002) 265-284
52. Nash J. Equilibrium points in n-person games. Proc. Natl. Acad. Sci. U. S. A. 36 (1950) 48-49
53. Neuringer A., Jensen G., and Piff P. Stochastic matching and the voluntary nature of choice. J. Exp. Anal. Behav. 88 (2007) 1-28
54. Nevin J.A., Jenkins P., Whittaker S., and Yarensky P. Reinforcement contingencies and signal detection. J. Exp. Anal. Behav. 37 (1982) 65-79
55. Oppenheim A.V., and Willsky A.S. Signals and Systems. 2nd Edition (1996), Prentice Hall, New York, NY
56. Petrusic W.M., and Jamieson D.G. Relation between probability of preferential choice and time to choose changes with practice. J. Exp. Psychol. Hum. Percept. Perform. 4 3 (1978) 471-482
57. Platt M.L., and Glimcher P.W. Neural correlates of decision variables in parietal cortex. Nature 400 6741 (1999) 233-238
58. Posner M.I. Chronometric Explorations of Mind (1978), Lawrence Erlbaum Associates, Hillsdale, N.J.
59. Ratcliff R. A theory of memory retrieval. Psychol. Rev. 85 (1978) 59-108
60. Ratcliff R., and Tuerlinckx F. Estimating parameters of the diffusion model: approaches to dealing with contaminant reaction times and parameter variability. Psychon. Bull. Rev. 9 3 (2002) 438-481
61. Robinson D.A. Integrating with neurons. Annu. Rev. Neurosci. 12 (1989) 33-45
62. Roe R.M., Busemeyer J.R., and Townsend J.T. Multialternative decision field theory: a dynamic connectionist model of decision making. Psychol. Rev. 108 2 (2001) 370-392
63. Roitman J.D., and Shadlen M.N. Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. J. Neurosci. 22 21 (2002) 9475-9489
64. Rushworth M.F.S., Walton M.E., Kennerley S.W., and Bannerman D.M. Action sets and decisions in the medial prefrontal cortex. Trends Cogn. Sci. 8 9 (2004) 410-417
65. Sakai Y., and Fukai T. The actor-critic learning is behind the matching law: matching versus optimal behaviors. Neural Comput. 20 (2008) 227-251
66. Schall J.D. On building a bridge between brain and behavior. Annu. Rev. Psychol. 55 (2004) 23-50
67. Seo H., and Lee D. Temporal filtering of reward signals in the dorsal anterior cingulate cortex during a mixed-strategy game. J. Neurosci. 27 31 (2007) 8366-8377
68. Servan-Schreiber D., Printz H., and Cohen J.D. A network model of catecholamine effects: gain, signal-to-noise ratio, and behavior. Science 249 (August 1990) 892-895
69. Seung H.S. Amplification, attenuation and integration. In: Arbib M.A. (Ed). The Handbook of Brain Theory and Neural Netw. (2003), MIT Press, Cambridge, MA 94-97
70. Seung H.S., Lee D.D., Reis B.Y., and Tank D.W. The autapse: a simple illustration of short-term analog memory storage by tuned synaptic feedback. J. Comput. Neurosci. 9 (2000) 171-185
71. Shadlen M.N., and Newsome W.T. The variable discharge of cortical neurons: implications for connectivity, computation, and information coding. J. Neurosci. 18 10 (1998) 3870-3896
72. Simen P.A. Ramping, ramping everywhere: an overlooked model of interval timing. COSYNE Abstracts (2008)
73. Simen, P. A., Balci, F., In preparation. Adaptive Interval Timing by a Noisy Integrate-and-Fire Model.
74. Simen P.A., Cohen J.D., and Holmes P. Rapid decision threshold modulation by reward rate in a neural network. Neural Netw. 19 (2006) 1013-1026
75. Smith P.L., and Ratcliff R. Psychology and neurobiology of simple decisions. Trends Neurosci. 27 (2004) 161-168
76. Soltani A., and Wang X.J. A biophysically based neural model of matching law behavior: melioration by stochastic synapses. J. Neurosci. 26 14 (2006) 3731-3744
77. Staddon J.E.R. The New Behaviorism (2001), Psychology Press, Philadelphia, PA
78. Staddon J.E.R., and Higa J. Multiple time scales in simple habituation. Psychol. Rev. 103 (1996) 720-733
79. Sugrue L.P., Corrado G.S., and Newsome W.T. Matching behavior and the representation of value in the parietal cortex. Science 304 (2004) 1782-1787
80. Sutton R.S., and Barto A.G. Reinforcement Learning (1998), MIT Press, Cambridge, MA
81. Usher M., and McClelland J.L. The time course of perceptual choice: the leaky, competing accumulator model. Psychol. Rev. 108 3 (2001) 550-592
82. Vaughan W.J. Melioration, matching, and maximization. J. Exp. Anal. Behav. 36 2 (September 1981) 141-149
83. Vickers D. Evidence for an accumulator model of psychophysical discrimination. Ergonomics 13 (1970) 37-58
84. Von Neumann J.V., and Morgenstern O. Theory of Games and Economic Behavior (1944), Princeton University Press, Princeton, N.J.
85. Wang X.J. Probabilistic decision making by slow reverberation in cortical circuits. Neuron 36 5 (2002) 955-968
86. Williams B.A. Reinforcement, choice, and response strength. In: Atkinson R.C., Herrnstein R.J., Lindzey G., and Luce R.D. (Eds). Stevens' Handbook of Experimental Psychology: Vol. 2. Learning and Cognition Vol. 2 (1988), Wiley, New York 167-244
87. Wilson H.R., and Cowan J.D. Excitatory and inhibitory interactions in localized populations of model neurons. Biophys. J. 12 (1972) 1-24
88. Wong K.F., and Wang X.J. A recurrent network mechanism of time integration in perceptual decisions. J. Neurosci. 26 4 (2006) 1314-1328
89. Yang T., Hanks T., Mazurek M.E., McKinley M., Palmer J., and Shadlen M.N. Incorporating prior probability into decision-making in the face of uncertain reliability of evidence. Society for Neuroscience Abstracts (2005)