A reinforcement learning model of precommitment in decision making

Frontiers in Behavioral Neuroscience

Volumn 4, Issue DEC, 2010, Pages

  Kurth Nelson, Zeb ^a   David Redish, A ^a  

^a UNIVERSITY OF MINNESOTA   (United States)

Author keywords

Addiction; Decision making; Delay discounting; Hyperbolic discounting; Impulsivity; Precommitment; Reinforcement learning

Indexed keywords

EID: 84856116217     PISSN: 16625153     EISSN: None     Source Type: Journal    
DOI: 10.3389/fnbeh.2010.00184     Document Type: Article

References (58)

1. Ainslie, G. (1974). Impulse control in pigeons. J. Exp. Anal. Behav. 21, 485-489.
2. Ainslie, G. (1975). Specious reward: a behavioral theory of impulsiveness and impulse control. Psychol. Bull. 82, 463-496.
3. Ainslie, G. (1992). Picoeconomics. Cambridge: Cambridge University Press.
4. Ainslie, G. (2001). Breakdown of Will. Cambridge: Cambridge University Press.
5. Alexander, W. H., and Brown, J. W. (2010). Hyperbolically discounted temporal difference learning. Neural. Comput. 22, 1511-1527.
6. Bellman, R. (1958). On a routing problem. Q. J. Appl. Math. 16, 87-90.
7. Bernheim, B. D., and Rangel, A. (2004). Addiction and cue-triggered decision processes. Am. Econ. Rev. 94, 1558-1590.
8. Bickel, W. K., Miller, M. L., Yi, R., Kowal, B. P., Lindquist, D. M., and Pitcock, J. A. (2007). Behavioral and neuroeconomics of drug addiction: competing neural systems and temporal discounting processes. Drug Alcohol Depend. 90, S85-S91.
9. Bickel, W. K., Odum, A. L., and Madden, G. J. (1999). Impulsivity and cigarette smoking: delay discounting in current, never, and ex-smokers. Psychopharmacology (Berl.) 146, 447-454.
10. Coffey, S. F., Gudleski, G. D., Saladin, M. E., and Brady, K. T. (2003). Impulsivity and rapid discounting of delayed hypothetical rewards in cocainedependent individuals. Exp. Clin. Psychopharmacol. 11, 18-25.
11. Daw, N. D. (2003). Reinforcement Learning Models of the Dopamine System and Their Behavioral Implications. Ph.D. thesis, Carnegie Mellon University, Pittsburgh.
12. Daw, N. D., and Touretzky, D. S. (2000). Behavioral considerations suggest an average reward TD model of the dopamine system. Neurocomputing 32-33, 679-684.
13. de Wit, H., Enggasser, J. L., and Richards, J. B. (2002). Acute administration of d-amphetamine decreases impulsivity in healthy volunteers. Neuropsychopharmacology 27, 813-825.
14. Dezfouli, A., Piray, P., Keramati, M. M., Ekhtiari, H., Lucas, C., and Mokri, A. (2009). Aneurocomputational model for cocaine addiction. Neural Comput. 21, 2869-2893.
15. Dixon, M. R., Jacobs, E. A., and Sanders, S. (2006). Contextual control of delay discounting by pathological gamblers. J. Appl. Behav. Anal. 39, 413-422.
16. Dom, G., D'haene, P., Hulstijn, W., and Sabbe, B. (2006). Impulsivity in abstinent early-and late-onset alcoholics: differences in self-report measures and a discounting task. Addiction 101, 50-59.
17. Doya, K. (1999). What are the computations of the cerebellum, the basal ganglia, and the cerebral cortex? Neural Netw. 12, 961-974.
18. Dripps, D. A. (1993). Precommitment, prohibition, and the problem of dissent. J. Legal Stud. 22, 255-263.
19. Evenden, J. L. (1999). Varieties of impulsivity. Psychopharmacology 146, 348-361.
20. Fishburn, P. C., and Rubinstein, A. (1982). Time preference. Int. Econ. Rev. 23, 677-694.
21. Frederick, S., Loewenstein, G., and O'Donoghue, T. (2002). Time discounting and time preference: a critical review. J. Econ. Lit. 40, 351-401.
22. Gläscher, J., Daw, N., Dayan, P., and O'Doherty, J. P. (2010). States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585-595.
23. Hinson, J. M., Jameson, T. L., and Whitney, P. (2003). Impulsive decision making and working memory. J. Exp. Psychol. Learn. Mem. Cogn. 29, 298-306.
24. Hollerman, J. R., and Schultz, W. (1998). Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304-309.
25. Isoda, M., and Hikosaka, O. (2007). Switching from automatic to controlled action by monkey medial frontal cortex. Nat. Neurosci. 10, 240-248.
26. Johnson, A., and Redish, A. D. (2007). Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176-12189.
27. Johnson, A., van der Meer, M. A., and Redish, A. D. (2007). Integrating hippocampus and striatum in decisionmaking. Curr. Opin. Neurobiol. 17, 692-697.
28. Kable, J. W., and Glimcher, P. W. (2007). The neural correlates of subjective value during intertemporal choice. Nat. Neurosci. 10, 1625-1633.
29. Kable, J. W., and Glimcher, P. W. (2009). The neurobiology of decision: consensus and controversy. Neuron 63, 733-745.
30. Kendall, P. C., and Wilcox, L. E. (1980). Cognitive-behavioral treatment for impulsivity: concrete versus conceptual training in non-self-controlled problem children. J. Consult. Clin. Psychol. 48, 80-91.
31. Koopmans, T. C. (1960). Stationary ordinal utility and impatience. Econometrica 28, 287-309.
32. Kurth-Nelson, Z., and Redish, A. D. (2009). Temporal-difference reinforcement learning with distributed representations. PLoS One 4:e7362. doi:10.1371/ journal.pone.0007362.
33. Ljungberg, T., Apicella, P., and Schultz, W. (1992). Responses of monkey dopamine neurons during learning of behavioral reactions. J. Neurophysiol. 67, 145-163.
34. Madden, G. J., and Bickel, W. K. (2010). Impulsivity: The Behavioral and Neurological Science of Discounting. Washington, DC: American Psychological Association.
35. Madden, G. J., Petry, N. M., Badger, G. J., and Bickford, W. K. (1997). Impulsive and self-control choices in opioid-dependent patients and non-drug-using control patients: drug and monetary rewards. Exp. Clin. Psychopharmacol. 5, 256-262.
36. Mazur, J. (1987). "An adjusting procedure for studying delayed reinforcement, " in Quantitative Analysis of Behavior: Vol. 5. The Effect of Delay and of Intervening Events on Reinforcement Value, eds M. L. Commons, J. F. Mazur, J. A. Nevin, and H. Rachlin (Hillsdale, NJ: Erlbaum), 55-73.
37. Mobini, S., Chiang, T. J., Al-Ruwaitea, A. S., Ho, M. Y., Bradshaw, C. M., and Szabadi, E. (2000). Effect of central 5-hydroxytryptamine depletion on inter-temporal choice: a quantitative analysis. Psychopharmacology 149, 313-318.
38. Montague, P. R., Dayan, P., and Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936-1947.
39. Nelson, W. M., and Behler, J. J. (1989). Cognitive impulsivity training: the effects of peer teaching. J. Behav. Ther. Exp. Psychiatry 20, 303-309.
40. O'Donoghue, T., and Rabin, M. (1999). Doing it now or later. Am. Econ. Rev. 89, 103-124.
41. Peters, J., and Büchel, C. (2010). Episodic future thinking reduces reward delay discounting through an enhancement of prefrontal-mediotemporal interactions. Neuron 66, 138-148.
42. Rachlin, H., and Green, L. (1972). Commitment, choice, and self-control. J. Exp. Anal. Behav. 17, 15-22.
43. Redish, A. D., Jensen, S., and Johnson, A. (2008). A unified framework for addiction: vulnerabilities in the decision process. Behav. Brain Sci. 31, 415-437.
44. Reynolds, B., Ortengren, A., Richards, J. B., and Wit, H. D. (2006). Dimensions of impulsive behavior: Personality and behavioral measures. Pers. Individ. Dif. 40, 305-315.
45. Rubin, D. C., Hinton, S., and Wenzel, A. (1999). The precise time course of retention. J. Exp. Psychol. Learn. Mem. Cogn. 25, 1161-1176.
46. Rubin, D. C., and Wenzel, A. E. (1996). One hundred years of forgetting: a quantitative description of retention. Psychol. Rev. 103, 734-760.
47. Samejima, K., Ueda, Y., Doya, K., and Kimura, M. (2005). Representation of action-specific reward values in the striatum. Science 310, 1337-1340.
48. Samuelson, P. A. (1937). A note on measurement of utility. Rev. Econ. Stud. 4, 155-161.
49. Schweighofer, N., Bertin, M., Shishida, K., Okamoto, Y., Tanaka, S. C., Yamawaki, S., and Doya, K. (2008). Low-serotonin levels increase delayed reward discounting in humans. J. Neurosci. 28, 4528-4532.
50. Sozou, P. D. (1998). On hyperbolic discounting and uncertain hazard rates. Proc. Biol. Sci. 265, 2015-2020.
51. Strotz, R. H. (1955). Myopia and inconsistency in dynamic utility maximization. Rev. Econ. Stud. 23, 165-180.
52. Sutton, R. S., and Barto, A. G. (1998). Re infor c ement Learning: An Introduction. Cambridge, MA: MIT Press.
53. Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., and Yamawaki, S. (2004). Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat. Neurosci. 7, 887-893.
54. Tanaka, S. C., Schweighofer, N., Asahi, S., Shishida, K., Okamoto, Y., Yamawaki, S., and Doya, K. (2007). Serotonin differentially regulates short-and long-term prediction of rewards in the ventral and dorsal striatum. PLoS One 2:e1333. doi: 10.1371/journal. pone.0001333.
55. Tiffany, S. T. (1990). A cognitive model of drug urges and drug-use behavior: role of automatic and nonautomatic processes. Psychol. Rev. 97, 147-168.
56. Tobler, P. N., O'Doherty, J. P., Dolan, R. J., and Schultz, W. (2007). Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. J. Neurophysiol. 97, 1621-1632.
57. Tsitsiklis, J. N., and Van Roy, B. (1999). Average cost temporal-difference learning. Automatica 35, 1799-1808.
58. Vohs, K. D., Baumeister, R. F., Schmeichel, B. J., Twenge, J. M., Nelson, N. M., and Tice, D. M. (2008). Making choices impairs subsequent self-control: a limited-resource account of decision making, self-regulation, and active initiative. J. Pers. Soc. Psychol. 94, 883-898.