Processing speed enhances model-based over model-free reinforcement learning in the presence of high working memory functioning

Frontiers in Psychology

Volumn 5, Issue DEC, 2014, Pages

  Schad, Daniel J ^a   Jünger, Elisabeth ^b   Sebold, Miriam ^a   Garbusow, Maria ^a   Bernhardt, Nadine ^c   Javadi, Amir Homayoun ^d   Zimmermann, Ulrich S ^b   Smolka, Michael N ^c   Heinz, Andreas ^a   Rapp, Michael A ^a,e   Huys, Quentin J M ^f,g  

^a CHARITÉ UNIVERSITÄTSMEDIZIN BERLIN   (Germany)

^b UNIVERSITY HOSPITAL CARL GUSTAV CARUS   (Germany)

^c DRESDEN UNIVERSITY OF TECHNOLOGY   (Germany)

^d UNIVERSITY COLLEGE LONDON   (United Kingdom)

^e UNIVERSITY OF POTSDAM   (Germany)

^f ETH ZURICH   (Switzerland)

^g UNIVERSITY HOSPITAL OF PSYCHIATRY   (Switzerland)

Author keywords

Cognitive abilities; Decision making; Fluid intelligence; Habitual and goal directed system; Model based and model free learning; Reward

Indexed keywords

EID: 84923385272     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2014.01450     Document Type: Article

References (60)

1. Arbuthnott, K., and Frank, J. (2000). Trail making test, part B as a measure of executive control: validation using a set-switching paradigm. J. Clin. Exp. Neuropsychol. 22, 518-528. doi: 10.1076/1380-3395(200008)22:4;1-0;FT518
2. Army Individual Test Battery. (1944). Manual of Directions and Scoring. Washington, DC: War Department, Adjutant General's Office.
3. Barr, D. J., Levy, R., Scheepers, C., and Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: keep it maximal. J. Mem. Lang. 68, 255-278. doi: 10.1016/j.jml.2012.11.001
4. Bates, D., Maechler, M., and Bolker, B. (2013). Linear mixed-Effects Models Using S4 Classes, [Software] Version: 0.999999-2. Available online at: http://www.R-project.org
5. Benjamini, Y., and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Statist. Soc. B 57, 289-300.
6. Bhandari, A., and Duncan, J. (2014). Goal neglect and knowledge chunking in the construction of novel behaviour. Cognition 130, 11-30. doi: 10.1016/j.cognition.2013.08.013
7. Bickel, W. K., Landes, R. D., Kurth-Nelson, Z., and Redish, A. D. (2014). A quantitative signature of self-control repair rate-dependent effects of successful addiction treatment. Clin. Psychol. Sci. 2, 685-695. doi: 10.1177/2167702614528162
8. Burks, S. V., Carpenter, J. P., Goette, L., and Rustichini, A. (2009). Cognitive skills affect economic preferences, strategic behavior, and job attachment. Proc. Natl. Acad. Sci. U.S.A. 106, 7745-7750. doi: 10.1073/pnas.0812360106
9. Corrigan, J. D., and Hinkeldey, N. S. (1987). Relationships between parts A and B of the trail making test. J. Clin. Psychol. 43, 402-409. doi: 10.1002/1097-4679(198707)43:4<402::AID-JCLP2270430411>3.0.CO;2-E
10. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., and Dolan, R. J. (2011). Model-based influences on humans' choices and striatal prediction errors. Neuron 69, 1204-1215. doi: 10.1016/j.neuron.2011.02.027
11. Daw, N. D., Niv, Y., and Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704-1711. doi: 10.1038/nn1560
12. De Wit, S., Barker, R. A., Dickinson, A. D., and Cools, R. (2011). Habitual versus goal-directed action control in Parkinson disease. J. Cogn. Neurosci. 23, 1218-1229. doi: 10.1162/jocn.2010.21514
13. Dezfouli, A., and Balleine, B. W. (2012). Habits, action sequences and reinforcement learning. Eur. J. Neurosci. 35, 1036-1051. doi: 10.1111/j.1460-9568.2012.08050.x
14. Dezfouli, A., and Balleine, B. W. (2013). Actions, action sequences and habits: evidence that goal-directed and habitual action control are hierarchically organized. PLoS Comput. Biol. 9:e1003364. doi: 10.1371/journal.pcbi.1003364
15. Dolan, R. J., and Dayan, P. (2013). Goals and habits in the brain. Neuron 80, 312-325. doi: 10.1016/j.neuron.2013.09.007
16. Everitt, B. J., and Robbins, T. W. (2005). Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat. Neurosci. 8, 1481-1489. doi: 10.1038/nn1579
17. FitzGerald, T. H. B., Seymour, B., Bach, D. R., and Dolan, R. J. (2010). Differentiable neural substrates for learned and described value and risk. Curr. Biol. 20, 1823-1829. doi: 10.1016/j.cub.2010.08.048
18. Friedel, E., Koch, S. P., Wendt, J., Heinz, A., Deserno, L., and Schlagenhauf, F. (2014). Devaluation and sequential decisions: linking goal-directed and model-based behavior. Front. Hum. Neurosci. 8:587. doi: 10.3389/fnhum.2014.00587
19. Gelman, A., Su, S., Yajima, M., Hill, J., Pittau, M. G., and Kerman, J. (2013). Arm: Data Analysis Using Regression and Multi- Level/ Hierarchical Models. R Package, Version 1.6-09., [Software] Version: 1.6.-09. Avaliable online at: http://cran.r-project.org/web/packages/arm
20. Gillan, C. M., Papmeyer, M., Morein-Zamir, S., Sahakian, B. J., Fineberg, N. A., Robbins, T. W., et al. (2011). Disruption in the balance between goal-directed behavior and habit learning in obsessive-compulsive disorder. Am. J. Psychiatry 168, 718-726. doi: 10.1176/appi.ajp.2011.10071062
21. Horn, J. L., and Cattell, R. B. (1966). Refinement and test of the theory of fluid and crystallized general intelligences. J. Educ. Psychol. 57, 253-270. doi: 10.1037/h0023816
22. Huys, Q. J. M., Eshel, N., O'Nions, E., Sheridan, L., Dayan, P., and Roiser, J. P. (2012). Bonsai trees in your head: how the pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput. Biol. 8:e1002410. doi: 10.1371/journal.pcbi.1002410
23. Huys, Q. J., Tobler, P. T., Hasler, G., and Flagel, S. B. (2014). The role of learning-related dopamine signals in addiction vulnerability. Prog. Brain Res. 211, 31-77.
24. Johnson, A., and Redish, A. D. (2007). Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176-12189. doi: 10.1523/JNEUROSCI.3761-07.2007
25. Johnson, W., and Bouchard, T. J. Jr. (2005). The structure of human intelligence: it is verbal, perceptual, and image rotation (VPR), not fluid and crystallized. Intelligence 33, 393-416. doi: 10.1016/j.intell.2004.12.002
26. Joy, S., Fein, D., and Kaplan, E. (2003). Decoding digit symbol: speed, memory, and visual scanning. Assessment 10, 56-65. doi: 10.1177/0095399702250335
27. Keramati, M., Dezfouli, A., and Piray, P. (2011). Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput. Biol. 7:e1002055. doi: 10.1371/journal.pcbi.1002055
28. Killcross, S., and Coutureau, E. (2003). Coordination of actions and habits in the medial prefrontal cortex of rats. Cereb. Cortex 13, 400-408. doi: 10.1093/cercor/13.4.400
29. Kurth-Nelson, Z., Bickel, W., and Redish, A. D. (2012). A theoretical account of cognitive effects in delay discounting. Eur. J. Neurosci. 35, 1052-1064. doi: 10.1111/j.1460-9568.2012.08058.x
30. Laux, L. F., and Lane, D. M. (1985). Information processing components of substitution test performance. Intelligence 9, 111-136. doi: 10.1016/0160-2896(85)90012-1
31. Lee, S. W., Shimojo, S., and O'Doherty, J. P. (2014). Neural computations underlying arbitration between model-based and model-free learning. Neuron 81, 687-699. doi: 10.1016/j.neuron.2013.11.028
32. Lehrl, S. (2005). Mehrfachwahl-Wortschatz-Intelligenztest MWT-B, 5th Edn. Balingen: Spitta.
33. Li, S.-C., and Lewandowsky, S. (1995). Forward and backward recall: different retrieval processes. J. Exp. Psychol. Learn. Mem. Cogn. 21, 837-847. doi: 10.1037/0278-7393.21.4.837
34. Li, S.-C., Lindenberger, U., Hommel, B., Aschersleben, G., Prinz, W., and Baltes, P. B. (2004). Transformations in the couplings among intellectual abilities and constituent cognitive processes across the life span. Psychol. Sci. 15, 155-163. doi: 10.1111/j.0956-7976.2004.01503003.x
35. O'Keefe, J., and Nadel, L. (1978). The Hippocampus as a Cognitive Map, Vol. 3. Oxford: Clarendon Press.
36. Otto, A. R., Gershman, S. J., Markman, A. B., and Daw, N. D. (2013a). The curse of planning: dissecting multiple reinforcement-learning systems by taxing the central executive. Psychol. Sci. 24, 751-761. doi: 10.1177/0956797612463080
37. Otto, A. R., Raio, C. M., Chiang, A., Phelps, E. A., and Daw, N. D. (2013b). Working-memory capacity protects model-based learning from stress. Proc. Natl. Acad. Sci. U.S.A. 110, 20941-20946. doi: 10.1073/pnas.1312011110
38. Pinheiro, J. C., and Bates, D. M. (2000). Mixed-Effects Models in S and S-Plus. New York, NY: Springer. doi: 10.1007/978-1-4419-0318-1
39. Rangel, A., Camerer, C., and Montague, P. R. (2008). A framework for studying the neurobiology of value-based decision making. Nat. Rev. Neurosci. 9, 545-556. doi: 10.1038/nrn2357
40. R Development Core Team (2013). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. Avaliable online at: http://www.R-project.org/
41. Redish, A. D. (1999). Beyond the Cognitive Map: From Place Cells to Episodic Memory. Cambridge: MIT Press.
42. Redish, A. D., Jensen, S., and Johnson, A. (2008). A unified framework for addiction: vulnerabilities in the decision process. Behav. Brain Sci. 31, 415-437. doi: 10.1017/S0140525X0800472X
43. Reichenberg, A., Gross, R., Weiser, M., Bresnahan, M., Silverman, J., Harlap, S., et al. (2006). Advancing paternal age and autism. Arch. Gen. Psychiatry 63, 1026-1032. doi: 10.1001/archpsyc.63.9.1026
44. Rich, E. L., and Shapiro, M. (2009). Rat prefrontal cortical neurons selectively code strategy switches. J. Neurosci. 29, 7208-7219. doi: 10.1523/JNEUROSCI.6068-08.2009
45. Rolfhus, E. L., and Ackerman, P. L. (1999). Assessing individual differences in knowledge: knowledge, intelligence, and related traits. J. Educ. Psychol. 91:511. doi: 10.1037/0022-0663.91.3.511
46. Salthouse, T. A. (1992). What do adult age differences in the digit symbol substitution test reflect? J. Gerontol. 47, P121-P128. doi: 10.1093/geronj/47.3.P121
47. Sánchez-Cubillo, I., Periáñez, J. A., Adrover-Roig, D., Rodríguez-Sánchez, J. M., Ríos-Lago, M., Tirapu, J., et al. (2009). Construct validity of the trail making test: role of task-switching, working memory, inhibition/interference control, and visuomotor abilities. J. Int. Neuropsychol. Soc. 15, 438-450. doi: 10.1017/S1355617709090626
48. Schlagenhauf, F., Rapp, M. A., Huys, Q. J. M., Beck, A., Wüstenberg, T., Deserno, L., et al. (2013). Ventral striatal prediction error signaling is associated with dopamine synthesis capacity and fluid intelligence. Hum. Brain Mapp. 34, 1490-1499. doi: 10.1002/hbm.22000
49. Schwabe, L., and Wolf, O. T. (2009). Stress prompts habit behavior in humans. J. Neurosci. 29, 7191-7198. doi: 10.1523/JNEUROSCI.0979-09.2009
50. Sebold, M., Deserno, L., Nebe, S., Schad, D. J., Garbusow, M., Hägele, C., et al. (2014). Model-based and model-free decisions in alcohol dependence. Neuropsychobiology 70, 122-131. doi: 10.1159/000362840
51. Sjoerds, Z., Luigjes, J., van den Brink, W., Denys, D., and Yücel, M. (2014). The role of habits and motivation in human drug addiction: a reflection. Front. Psychiatry 5:8. doi: 10.3389/fpsyt.2014.00008
52. Smittenaar, P., FitzGerald, T. H. B., Romei, V., Wright, N. D., and Dolan, R. J. (2013). Disruption of dorsolateral prefrontal cortex decreases model-based in favor of model-free control in humans. Neuron 80, 914-919. doi: 10.1016/j.neuron.2013.08.009
53. Sternberg, R. J. (2012). Intelligence. Wiley Interdiscip. Rev. Cogn. Sci. 3, 501-511. doi: 10.1002/wcs.1193
54. Sutton, R. S., and Barto, A. G. (2009). Reinforcement Learning: an Introduction. Cambridge, MA: MIT Press.
55. Van der Meer, M., Kurth-Nelson, Z., and Redish, A. D. (2012). Information processing in decision-making systems. Neuroscientist 18, 342-359. doi: 10.1177/1073858411435128
56. Waltz, J. A., Frank, M. J., Robinson, B. M., and Gold, J. M. (2007). Selective reinforcement learning deficits in schizophrenia support predictions from computational models of striatal-cortical dysfunction. Biol. Psychiatry 62, 756-764. doi: 10.1016/j.biopsych.2006.09.042
57. Wechsler, D. (1997). WAIS-III, Wechsler Adult Intelligence Scale: Administration and Scoring Manual. San Antonio, TX: The Psychological Corporation.
58. Wunderlich, K., Smittenaar, P., and Dolan, R. J. (2012). Dopamine enhances model-based over model-free choice behavior. Neuron 75, 418-424. doi: 10.1016/j.neuron.2012.03.042
59. Yin, H. H., Knowlton, B. J., and Balleine, B. W. (2004). Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur. J. Neurosci. 19, 181-189. doi: 10.1111/j.1460-9568.2004.03095.x
60. Yin, H. H., Ostlund, S. B., Knowlton, B. J., and Balleine, B. W. (2005). The role of the dorsomedial striatum in instrumental conditioning. Eur. J. Neurosci. 22, 513-523. doi: 10.1111/j.1460-9568.2005.04218.x