Model-based learning protects against forming habits

Cognitive, Affective and Behavioral Neuroscience

Volumn 15, Issue 3, 2015, Pages 523-536

  Gillan, Claire M ^a,b   Otto, A Ross ^a   Phelps, Elizabeth A ^a,c   Daw, Nathaniel D ^a  

^a NEW YORK UNIVERSITY   (United States)

^b UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^c NATHAN S KLINE INSTITUTE FOR PSYCHIATRIC RESEARCH   (United States)

Author keywords

Devaluation; Goal directed; Habit; Model based; Model free; Reinforcement learning

Indexed keywords

ADOLESCENT; ADULT; AGED; DECISION MAKING; EXECUTIVE FUNCTION; FEMALE; HABIT; HUMAN; INSTRUMENTAL CONDITIONING; MALE; MIDDLE AGED; MOTIVATION; PSYCHOLOGIC TEST; PSYCHOLOGICAL MODEL; REINFORCEMENT; YOUNG ADULT;

ADOLESCENT; ADULT; AGED; CONDITIONING, OPERANT; DECISION MAKING; EXECUTIVE FUNCTION; FEMALE; GOALS; HABITS; HUMANS; MALE; MIDDLE AGED; MODELS, PSYCHOLOGICAL; PSYCHOLOGICAL TESTS; REINFORCEMENT (PSYCHOLOGY); YOUNG ADULT;

EID: 84938749786     PISSN: 15307026     EISSN: None     Source Type: Journal    
DOI: 10.3758/s13415-015-0347-6     Document Type: Article

References (47)

1. Adams, C. D. (1982). Variations in the sensitivity of instrumental responding to reinforcer devaluation. Quarterly Journal of Experimental Psychology, 34B, 77–98.
2. Adams, C. D., & Dickinson, A. (1981). Instrumental responding following reinforcer devaluation. Quarterly Journal of Experimental Psychology, 33B, 109–121. doi:10.1080/14640748108400816
3. Akam, T., Dayan, P., & Costa, R. (2013). Multi-step decision tasks for dissociating model-based and model-free learning in rodents. Paper presented at the Cosyne 2013, Salt Lake City, UT.
4. Balleine, B. W., & Dickinson, A. (1998). Goal-directed instrumental action: Contingency and incentive learning and their cortical substrates. Neuropharmacology, 37, 407–419.
5. Balleine, B. W., & O’Doherty, J. P. (2010). Human and rodent homologies in action control: Corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology, 35, 48–69. doi:10.1038/npp.2009.131
6. Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS ONE, 8, e57410. doi:10.1371/journal.pone.0057410
7. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based influences on humans’ choices and striatal prediction errors. Neuron, 69, 1204–1215. doi:10.1016/j.neuron.2011.02.027
8. Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8, 1704–1711. doi:10.1038/nn1560
9. Daw, N. D., & O’Doherty, J. P. (2014). Multiple systems for value learning neuroeconomics: Decision making and the brain (pp. 393–410). Amsterdam, The Netherlands: Elsevier.
10. de Wit, S., Corlett, P. R., Aitken, M. R., Dickinson, A., & Fletcher, P. C. (2009). Differential engagement of the ventromedial prefrontal cortex by goal-directed and habitual behavior toward food pictures in humans. Journal of Neuroscience, 29, 11330–11338. doi:10.1523/JNEUROSCI. 1639-09.2009
11. de Wit, S., Niry, D., Wariyar, R., Aitken, M. R., & Dickinson, A. (2007). Stimulus–outcome interactions during instrumental discrimination learning by rats and humans. Journal of Experimental Psychology: Animal Behavior Processes, 33, 1–11. doi:10.1037/0097-7403.33.1.1
12. de Wit, S., Watson, P., Harsay, H. A., Cohen, M. X., van de Vijver, I., & Ridderinkhof, K. R. (2012). Corticostriatal connectivity underlies individual differences in the balance between habitual and goal-directed action control. Journal of Neuroscience, 32, 12066–12075.
13. Dezfouli, A., & Balleine, B. W. (2013). Actions, action sequences and habits: Evidence that goal-directed and habitual action control are hierarchically organized. PLoS Computational Biology, 9, e1003364.
14. Dias-Ferreira, E., Sousa, J. C., Melo, I., Morgado, P., Mesquita, A. R., Cerqueira, J. J., … Sousa, N. (2009). Chronic stress causes frontostriatal reorganization and affects decision-making. Science, 325, 621–625. doi:10.1126/science.1171203
15. Dickinson, A. (1985). Actions and habits: The development of behavioural autonomy. Philosophical Transactions of the Royal Society B, 308, 67–78.
16. Dickinson, A., & Balleine, B. (1994). Motivational control of goal-directed action. Animal Learning & Behavior, 22, 1–18. doi:10.3758/BF03199951
17. Dickinson, A., Nicholas, D. J., & Adams, C. D. (1983). The effect of the instrumental training contingency on susceptibility to reinforcer devaluation. Quarterly Journal of Experimental Psychology, 35B, 35–51. doi:10.1080/14640748308400912
18. Dickinson, A., Wood, N., & Smith, J. W. (2002). Alcohol seeking by rats: Action or habit? Quarterly Journal of Experimental Psychology, 55B, 331–348. doi:10.1080/0272499024400016
19. Dolan, R. J., & Dayan, P. (2013). Goals and habits in the brain. Neuron, 80, 312–325.
20. Doya, K. (1999). What are the computations in the cerebellum, the basal ganglia, and the cerebral cortex. Neural Networks, 12, 961–974.
21. Eppinger, B., Walter, M., Heekeren, H. R., & Li, S. C. (2013). Of goals and habits: Age-related and individual differences in goal-directed decision-making. Frontiers in Neuroscience, 7, 253. doi:10.3389/fnins.2013.00253
22. Frank, M. J., Rudy, J. W., Levy, W. B., & O’Reilly, R. C. (2005). When logic fails: Implicit transitive inference in humans. Memory & Cognition, 33, 742–750. doi:10.3758/BF03195340
23. Friedel, E., Koch, S. P., Wendt, J., Heinz, A., Deserno, L., & Schlagenhauf, F. (2014). Devaluation and sequential decisions: Linking goal-directed and model-based behaviour. Frontiers in Human Neuroscience, 8, 587. doi:10.3389/fnhum.2014.00587
24. Gillan, C. M., Apergis-Schoute, A. M., Morein-Zamir, S., Urcelay, G. P., Sule, A., Fineberg, N. A., … Robbins, T. W. (2015). Functional neuroimaging of avoidance habits in obsessive-compulsive disorder. American Journal of Psychiatry, 172, 284–293. doi:10.1176/appi.ajp.2014.14040525
25. Gillan, C. M., Morein-Zamir, S., Urcelay, G. P., Sule, A., Voon, V., Apergis-Schoute, A. M., … Robbins, T. W. (2014). Enhanced avoidance habits in obsessive-compulsive disorder. Biological Psychiatry, 75, 631–638. doi:10.1016/j.biopsych.2013.02.002
26. Gillan, C. M., Papmeyer, M., Morein-Zamir, S., Sahakian, B. J., Fineberg, N. A., Robbins, T. W., & de Wit, S. (2011). Disruption in the balance between goal-directed behavior and habit learning in obsessive-compulsive disorder. American Journal of Psychiatry, 168, 718–726. doi:10.1176/appi.ajp.2011.10071062
27. Gillan, C. M., & Robbins, T. W. (2014). Goal-directed learning and obsessive-compulsive disorder. Philosophical Transactions of the Royal Society B, 369, 475. doi:10.1098/rstb.2013.0475
28. Gläscher, J., Daw, N., Dayan, P., & O’Doherty, J. P. (2010). States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66, 585–595.
29. Keramati, M., Dezfouli, A., & Piray, P. (2011). Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Computational Biology, 7, e1002055. doi:10.1371/journal.pcbi.1002055
30. Miller, K., Erlich, J., Kopec, C., Botvinick, M., & Brody, C. (2014). A multi-step decision task elicits planning behavior in rats. Paper presented at Cosyne 2014, Salt Lake City, UT.
31. Otto, A. R., Gershman, S. J., Markman, A. B., & Daw, N. D. (2013a). The curse of planning: Dissecting multiple reinforcement-learning systems by taxing the central executive. Psychological Science, 24, 751–761.
32. Otto, A. R., Raio, C. M., Chiang, A., Phelps, E. A., & Daw, N. D. (2013b). Working-memory capacity protects model-based learning from stress. Proceedings of the National Academy of Sciences, 110, 20941–20946.
33. Otto, A. R., Skatova, A., Madlon-Kay, S., & Daw, N. D. (2015). Cognitive control predicts use of model-based reinforcement learning. Journal of Cognitive Neuroscience, 27, 319–333. doi:10.1162/jocn_a_00709
34. Pezzulo, G., Rigoli, F., & Chersi, F. (2013). The mixed instrumental controller: Using value of information to combine habitual choice and mental simulation. Frontiers in Psychology, 4, 92.
35. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593–1599. doi:10.1126/science.275.5306.1593
36. Schwabe, L., & Wolf, O. T. (2009). Stress prompts habit behavior in humans. Journal of Neuroscience, 29, 7191–7198.
37. Seger, C. A., & Spiering, B. J. (2011). A critical review of habit learning and the basal ganglia. Frontiers in Systems Neuroscience, 5, 66. doi:10.3389/fnsys.2011.00066
38. Simcox, T., & Fiez, J. A. (2014). Collecting response times using Amazon Mechanical Turk and Adobe Flash. Behavior Research Methods, 46, 95–111. doi:10.3758/s13428-013-0345-y
39. Sjoerds, Z., de Wit, S., van den Brink, W., Robbins, T. W., Beekman, A. T., Penninx, B. W. & Veltman, D. J. (2013). Behavioral and neuroimaging evidence for overreliance on habit learning in alcohol-dependent patients. Translational Psychiatry, 3, e337.
40. Sutton, R., & Barto, A. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
41. Tolman, E. C. (1948). Cognitive maps in rats and men. Psychological Review, 55, 189–208. doi:10.1037/h0061626
42. Tricomi, E., Balleine, B. W., & O’Doherty, J. P. (2009). A specific role for posterior dorsolateral striatum in human habit learning. European Journal of Neuroscience, 29, 2225–2232. doi:10.1111/j.1460-9568.2009.06796.x
43. Valentin, V. V., Dickinson, A., & O’Doherty, J. P. (2007). Determining the neural substrates of goal-directed learning in the human brain. Journal of Neuroscience, 27, 4019–4026.
44. Voon, V., Derbyshire, K., Rück, C., Irvine, M. A., Worbe, Y., Enander, J., … Bullmore, E. T. (2014). Disorders of compulsivity: A common bias towards learning habits. Molecular Psychiatry. doi:10.1038/mp.2014.44
45. Wunderlich, K., Smittenaar, P., & Dolan, R. J. (2012). Dopamine enhances model-based over model-free choice behavior. Neuron, 75, 418–424.
46. Yin, H. H., Knowlton, B. J., & Balleine, B. W. (2004). Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. European Journal of Neuroscience, 19, 181–189.
47. Yin, H. H., Ostlund, S. B., Knowlton, B. J., & Balleine, B. W. (2005). The role of the dorsomedial striatum in instrumental conditioning. European Journal of Neuroscience, 22, 513–523. doi:10.1111/j.1460-9568.2005.04218.x