Dont let me do that! - models of precommitment

Frontiers in Neuroscience

Volumn , Issue OCT, 2012, Pages

  Kurth Nelson, Zeb ^a   Redish, A David ^b  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF MINNESOTA   (United States)

Author keywords

Decision making; Discounting function; Neuroeconomics; Precommitment; Temporal diference reinforcement learning

Indexed keywords

CONSENSUS; DECISION MAKING; EXECUTIVE FUNCTION; IMPULSIVENESS; LANGUAGE; LEARNING; MENTAL DISEASE; MENTAL FUNCTION; PRECOMMITMENT; PREDICTIVE VALUE; QUANTITATIVE ANALYSIS; REINFORCEMENT; REVIEW; SENSITIVITY ANALYSIS; STIMULUS RESPONSE; THEORETICAL MODEL;

EID: 84870947380     PISSN: 16624548     EISSN: 1662453X     Source Type: Journal    
DOI: 10.3389/fnins.2012.00138     Document Type: Review

References (67)

1. Ainslie, G. (1974). Impulse control in pigeons. J. Exp. Anal. Behav. 21, 485-489.
2. Ainslie, G. (1992). Picoeconomics: The Strategic Interaction of Successive Motivational States Within the Person. New York, NY: Cambridge University Press.
3. Ainslie, G. (2001). Breakdown of Will. New York, NY: Cambridge University Press.
4. Ainslie, G., and Monterosso, J. R. (2003). Building blocks of self-control: increased tolerance for delay with bundled rewards. J. Exp. Anal. Behav. 79, 37-48.
5. Alexander, W. H., and Brown, J. W. (2010). Hyperbolically discounted temporal difference learning. Neural. Comput. 22, 1511-1527.
6. American Psychiatric Association. (2000). Diagnostic and Statistical Manual of Mental Disorders, 4th Edn. Washington, DC: APA.
7. Ariely, D., and Wertenbroch, K. (2002). Procrastination, deadlines, and performance: self-control by precommitment. Psychol. Sci. 13, 219-224.
8. Baumeister, R. F., Heatherton, T. F., and Tice, D. M. (1994). Losing Control: How and Why People Fail at Self-Regulation. San Diego, CA: Academic Press.
9. Baumeister, R. F., and Tierney, J. (2011). Willpower: Rediscovering the Greatest Human Strength. New York, NY: Penguin Press.
10. Bechara, A., Nader, K., and van der Kooy, D. (1998). A two-separate-motivational-systems hypothesis of opioid addiction. Pharmacol. Biochem. Behav. 59, 1-17.
11. Bellman, R. (1957). Dynamic programming. Princeton: Princeton University Press.
12. Bickel, W. K., Jarmolowicz, D. P., Mueller, E. T., Koffarnus, M. N., and Gatchalian, K. M. (2012). Excessive discounting of delayed reinforcers as a trans-disease process contributing to addiction and other disease-related vulnerabilities: Emerging evidence. Pharmacol. Ther. 134, 287-297.
13. Bickel, W. K., and Marsch, L. A. (2001). Toward a behavioral economic understanding of drug dependence: delay discounting processes. Addiction 96, 73-86.
14. Burks, S. V., Carpenter, J. P., Goette, L., and Rustichini, A. (2009). Cognitive skills affect economic preferences, strategic behavior, and job attachment. Proc. Natl. Acad. Sci. U.S.A. 106, 7745-7750.
15. Chopra, M. P., Landes, R. D., Gatchalian, K. M., Jackson, L. C., Bickel, W. K., Buchhalter, A. R., et al. (2009). Buprenorphine medication versus voucher contingencies in promoting abstinence from opioids and cocaine. Exp. Clin. Psychopharmacol. 17, 226-236.
16. Dalley, J. W., Mar, A. C., Economidou, D., and Robbins, T. W. (2008). Neurobehavioral mechanisms of impulsivity: fronto-striatal systems and functional neurochemistry. Pharmacol. Biochem. Behav. 90, 250-260.
17. Daw, N. (2000). Behavioral considerations suggest an average reward TD model of the dopamine system. Neurocomputing 32, 679-684.
18. Daw, N. D. (2003). Reinforcement Learning Models of the Dopamine System and their Behavioral Implications, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA.
19. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., and Dolan, R. J. (2011). Model-based influences on humans' choices and striatal prediction errors. Neuron 69, 1204-1215.
20. Daw, N. D., Niv, Y., and Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704-1711.
21. Dayan, P., Niv, Y., Seymour, B., and Daw, N. D. (2006). The misbehavior of value and the discipline of the will. Neural. Netw. 19, 1153-1160.
22. Fermin, A., Yoshida, T., Ito, M., Yoshimoto, J., and Doya, K. (2010). Evidence for model-based action planning in a sequential finger movement task. J. Mot. Behav. 42, 371-379.
23. Frederick, S., Loewenstein, G., and O'Donoghue, T. (2002). Time discounting and time preference: a critical review. J. Econ. Lit. 40, 351-401.
24. Glascher, J., Daw, N., Dayan, P., and O'Doherty, J. P. (2010). States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585-595.
25. Glimcher, P. W. (2008). Neuroeconomics: Decision Making and the Brain. San Diego, CA: Academic Press.
26. Haidt, J. (2006). The Happiness Hypothesis: Finding Modern Truth in Ancient Wisdom. New York: Basic Books.
27. Heyman, G. M. (2009). Addiction: A Disorder of Choice. Cambridge, MA: Harvard University Press.
28. Huys, Q. J., Eshel, N., O'Lions, E., Sheridan, L., Dayan, P., and Roiser, J. P. (2012). Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput. Biol. 8, e1002410. doi: 10.1371/journal.pcbi.1002410
29. Jimura, K., Myerson, J., Hilgard, J., Keighley, J., Braver, T. S., and Green, L. (2011). Domain independence and stability in young and older adults' discounting of delayed rewards. Behav. Processes 87, 253-259.
30. Kable, J. W., and Glimcher, P. W. (2009). The neurobiology of decision: consensus and controversy. Neuron 63, 733-745.
31. Kirby, K. N. (2009). One-year temporal stability of delay-discount rates. Psychon. Bull. Rev. 16, 457-462.
32. Kurth-Nelson, Z., Bickel, W., and Redish, A. D. (2012). A theoretical account of cognitive effects in delay discounting. Eur. J. Neurosci. 35, 1052-1064.
33. Kurth-Nelson, Z., and Redish, A. D. (2009). Temporal-difference reinforcement learning with distributed representations. PLoS ONE. 4, e7362. doi: 10.1371/journal. pone.0007362
34. Kurth-Nelson, Z., and Redish, A. D. (2010). A reinforcement learning model of precommitment in decision making. Front. Behav. Neurosci. 4:184. doi: 10.3389/fnbeh.2010.00184
35. Kurth-Nelson, Z., and Redish, A. D. (2012). "Modeling decision-making systems in addiction," in Computational Neuroscience of Drug Addiction, ed. B. Gutkin and S. Ahmed (New York, NY: Springer), 163-188.
36. Kurzban, R. (2010). Why Everyone (else) is A Hypocrite: Evolution and The Modular Mind. Princeton, NJ: Princeton University Press.
37. Ladouceur, R., Blaszczynski, A., and Lalande, D. R. (2012). Pre-commitment in gambling: a review of the empirical evidence. Int. Gambl. Stud. 1-16.
38. Madden, G. J., and Bickel, W. K. (2010). Impulsivity: The Behavioral and Neurological Science of Discounting. Washington, DC: American Psychological Association.
39. Maia, T. V., and Frank, M. J. (2011). From reinforcement learning models to psychiatric and neurological disorders. Nat. Neurosci. 14, 154-162.
40. Mazur, J. E. (1997). Choice, delay, probability, and conditioned reinforcement. Anim. Learn. Behav. 25, 131.
41. McClure, S. M., Laibson, D. I., Loewenstein, G., and Cohen, J. D. (2004). Separate neural systems value immediate and delayed monetary rewards. Science 306, 503-507.
42. Montague, P. R., Dayan, P., and Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936-1947.
43. Montague, P. R., Dolan, R. J., Friston, K. J., and Dayan, P. (2012). Computational psychiatry. Trends Cogn. Sci. (Regul. Ed.) 16, 72-80.
44. Niv, Y., Joel, D., and Dayan, P. (2006). A normative perspective on motivation. Trends Cogn. Sci. (Regul. Ed.) 10, 375-381.
45. O'Doherty, J. P. (2012). Beyond simple reinforcement learning: the computational neurobiology of reward-learning and valuation. Eur. J. Neurosci. 35, 987-990.
46. Ohmura, Y., Takahashi, T., Kitamura, N., and Wehr, P. (2006). Three-month stability of delay and probability discounting measures. Exp. Clin. Psychopharmacol. 14, 318-328.
47. Petry, N. M. (2012). Contingency Management for Substance Abuse Treatment: A Guide to Implementing this Evidence-Based Practice. New York: Routledge.
48. Rachlin, H., and Green, L. (1972). Commitment, choice and self-control. J. Exp. Anal. Behav. 17, 15-22.
49. Rangel, A., Camerer, C., and Montague, P. R. (2008). A framework for studying the neurobiology of value-based decision making. Nat. Rev. Neurosci. 9, 545-556.
50. Redish, A. D., Jensen, S., and Johnson, A. (2008). A unified framework for addiction: vulnerabilities in the decision process. Behav. Brain Sci. 31, 415-437.
51. Rick, S., and Loewenstein, G. (2008). Intangibility in intertemporal choice. Phil. Trans. Roy. Soc. B 363, 3813-3824.
52. Robbins, T. W., Gillan, C. M., Smith, D. G., de Wit, S., and Ersche, K. D. (2012). Neurocognitive endophenotypes of impulsivity and compulsivity: towards dimensional psychiatry. Trends Cogn. Sci. (Regul. Ed.) 16, 81-91.
53. Roesch, M. R., Esber, G. R., Li, J., Daw, N. D., and Schoenbaum, G. (2012). Surprise! neural correlates of pearce-hall and rescorla-wagner coexist within the brain. Eur. J. Neurosci. 35, 1190-1200.
54. Romer, D., Betancourt, L. M., Brodsky, N. L., Giannetta, J. M., Yang, W., and Hurt, H. (2011). Does adolescent risk taking imply weak executive function? A prospective study of relations between working memory performance, impulsivity, and risk taking in early adolescence. Dev. Sci. 14, 1119-1133.
55. Schultz, W. (1998). Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1-27.
56. Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science 275, 1593-1599.
57. Schweighofer, N., Bertin, M., Shishida, K., Okamoto, Y., Tanaka, S. C., and Yamawaki, S. Doya, K. (2008). Low-serotonin levels increase delayed reward discounting in humans. J. Neurosci. 28, 4528-4532.
58. Simon, D. A., and Daw, N. D. (2011). Neural correlates of forward planning in a spatial decision task in humans. J. Neurosci. 31, 5526-5539.
59. Strotz, R. H. (1955). Myopia and inconsistency in dynamic utility maximization. Rev. Econ. Stud. 23, 165-180.
60. Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
61. Takahashi, T., Furukawa, A., Miyakawa, T., Maesato, H., and Higuchi, S. (2007). Two-month stability of hyperbolic discount rates for delayed monetary gains in abstinent inpatient alcoholics. Neuro Endocrinol. Lett. 28, 131-136.
62. Tanaka, S. C., Schweighofer, N., Asahi, S., Shishida, K., Okamoto, Y., and Yamawaki, S. Doya, K. (2007). Serotonin differentially regulates short- and long-term prediction of rewards in the ventral and dorsal striatum. PLoS ONE 2, e1333. doi: 10.1371/ journal.pone.0001333
63. van der Meer, M. A. A., Kurth-Nelson, Z., and Redish, A. D. (2012). Information processing in decision-making systems. Neuroscientist 18, 342-359.
64. Vohs, K. D., and Faber, R. J. (2007). Spent resources: self-regulatory resource availability affects impulse buying. J. Consum. Res. 33, 537-547.
65. Vohs, K. D., Nelson, N. M., Baumeister, R. F., Tice, D. M., Schmeichel, B. J., and Twenge, J. M. (2008). Making choices impairs subsequent self-control: a limited-resource account of decision making, self-regulation, and active initiative. J. Pers. Soc. Psychol. 94, 883-898.
66. Wohl, M. J. A., Lyon, M., Donnelly, C. L., Young, M. M., Matheson, K., and Anisman, H. (2008). Episodic cessation of gambling: a numerically aided phenomenological assessment of why gamblers stop playing in a given session. Int. Gambl. Stud. 8, 249-263.
67. Wunderlich, K., Dayan, P., and Dolan, R. J. (2012). Mapping value based planning and extensively trained choice in the human brain. Nat. Neurosci. 15, 786-791.