Speed/accuracy trade-off between the habitual and the goal-directed processes

PLoS Computational Biology

Volumn 7, Issue 5, 2011, Pages

  Keramati, Mehdi ^a   Dezfouli, Amir ^b   Piray, Payam ^b  

^a SHARIF UNIVERSITY OF TECHNOLOGY   (Iran)

^b UNIVERSITY OF TEHRAN   (Iran)

Author keywords

[No Author keywords available]

Indexed keywords

AMINES; BEHAVIORAL RESEARCH; COMPUTATION THEORY; DECISION THEORY; ECONOMIC AND SOCIAL EFFECTS; NEUROPHYSIOLOGY; REINFORCEMENT LEARNING;

ASSOCIATIVE LEARNING; COMPARATIVE ADVANTAGE; CORTICO-BASAL GANGLIA CIRCUITS; DOPAMINE; GOAL-DIRECTED; INSTRUMENTAL RESPONSE; LEARNING MECHANISM; SENSORIMOTORS; SPEED ACCURACY; TRADE OFF;

DECISION MAKING;

ACCURACY; ADAPTIVE BEHAVIOR; ARTICLE; BEHAVIORAL SCIENCE; BRAIN FUNCTION; CONTROLLED STUDY; DECISION MAKING; DOPAMINERGIC NERVE CELL; HABITUATION; MATHEMATICAL MODEL; MESENCEPHALON; MOTIVATION; OBSERVATIONAL STUDY; REINFORCEMENT; RESPONSE TIME; REWARD; SIGNAL DETECTION; STIMULUS RESPONSE; THEORETICAL STUDY; ALGORITHM; ANIMAL; ANIMAL BEHAVIOR; BIOLOGICAL MODEL; COMPUTER SIMULATION; HUMAN; LEARNING; MAZE TEST; NERVE CELL; PHYSIOLOGY; PROBABILITY; RAT; REPRODUCIBILITY;

DOPAMINE;

ALGORITHMS; ANIMALS; BEHAVIOR, ANIMAL; CHOICE BEHAVIOR; COMPUTER SIMULATION; DECISION MAKING; DOPAMINE; GOALS; HUMANS; LEARNING; MARKOV CHAINS; MAZE LEARNING; MODELS, NEUROLOGICAL; NEURONS; RATS; REINFORCEMENT (PSYCHOLOGY); REPRODUCIBILITY OF RESULTS;

EID: 79958143780     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1002055     Document Type: Article

References (81)

1. Rangel A, Camerer C, Montague PR, (2008) A framework for studying the neurobiology of valuebased decision making. Nat Rev Neurosci 9: 545-556.
2. Dickinson A, Balleine BW, (2002) The role of learning in motivation. In: Gallistel CR, editors. Steven's Handbook of Experimental Psychology: Learning, Motivation, and Emotion New York Wiley pp. 497-533 Volume 3 3rd edition.
3. Adams CD, (1982) Variations in the sensitivity of instrumental responding to reinforcer devaluation. Q J Exp Psychol 34: 77-98.
4. Balleine BW, O'Doherty JP, (2010) Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacol 35: 48-69.
5. Daw ND, Niv Y, Dayan P, (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8: 1704-11.
6. Tolman E, (1938) The determiners of behavior at a choice point. Psychol Rev 45: 1-41.
7. Tolman E, (1939) Prediction of vicarious trial and error by means of the schematic sowbug. Psychol Rev 46: 318-336.
8. Muenzinger K, (1938) Vicarious trial and error at a point of choice. i. a general survey of its relation to learning efficacy. J Genet Psychol 53: 75-86.
9. Brown MF, (1992) Does a cognitive map guide choices in the radial-arm maze? J Exp Psychol Anim Behav Process 18: 55-66.
10. Buckner RL, Carroll DC, (2007) Self-projection and the brain. Trends Cogn Sci 11: 49-57.
11. Redish AD, Jensen S, Johnson A, Kurth-Nelson Z, (2007) Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol Rev 114: 784-805.
12. Hu D, Xu X, Gonzalez-Lima F, (2006) Vicarious trial-and-error behavior and hippocampal cytochrome oxidase activity during y-maze discrimination learning in the rat. Int J Neurosci 116: 265-280.
13. Munn NL, (1950) Handbook of psychological research on the rat. Boston Houghton Mifflin.
14. Redish AD, Jensen S, Johnson A, (2008) A unified framework for addiction: vulnerabilities in the decision process. Behav Brain Sci 31: 415-437 415 discussion 437-487.
15. Sutton RS, Barto AG, (1998) Reinforcement Learning: An Introduction. Cambridge MIT Press.
16. Geist M, Pietquin O, Fricout G, (2009) Kalman temporal differences: the deterministic case. USA Nashville pp. 185-192 In:Proceedings of the 2009 IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL-09).
17. Howard R, (1996) Information value theory. IEEE T Syst Sci Cyb 2: 22-26.
18. Dearden R, Friedman N, Russell S, (1998) Bayesian Q-Learning. pp. 761-768 In:Proceedings of the 15th National Conference on Artificial Intelligence (AAAI).
19. Niv Y, Daw ND, Joel D, Dayan P, (2007) Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl) 191: 507-520.
20. Dickinson A, Balleine B, (1995) Motivational control of instrumental action. Curr Dir Psychol Sci 4: 162-167.
21. Holland PC, (2004) Relations between pavlovian-instrumental transfer and reinforcer devaluation. J Exp Psychol Anim Behav Process 30: 104-117.
22. Killcross S, Coutureau E, (2003) Coordination of actions and habits in the medial prefrontal cortex of rats. Cereb Cortex 13: 400-408.
23. Yin HH, Knowlton BJ, Balleine BW, (2004) Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur J Neurosci 19: 181-189.
24. Colwill RM, Rescorla RA, (1985) Instrumental responding remains sensitive to reinforcer devaluation after extensive training. J Exp Psychol Anim Behav Process 11: 520-536.
25. Colwill RM, Rescorla RA, (1988) The role of response-reinforcer associations increases throughout extended instrumental training. Anim Learn Behav 16: 105-111.
26. Kosaki Y, Dickinson A, (2010) Choice and contingency in the development of behavioral autonomy during instrumental conditioning. J Exp Psychol Anim Behav Process 36: 334-342.
27. Pessiglione M, Czernecki V, Pillon B, Dubois B, Schüpbach M, et al. (2005) An effect of dopamine depletion on decision-making: the temporal coupling of deliberation and execution. J Cog Neurosci 17: 1886-1896.
28. Hick WE, (1952) On the rate of gain of information. Q J Exp Psychol 4: 11-26.
29. Hyman R, (1953) Stimulus information as a determinant of reaction time. J Exp Psychol 45: 188-196.
30. Alluisi E, (1965) Interaction of S-R compatibility and the rate of gain of information. Percept Mot Skills 20: 815-816.
31. Alluisi E, Strain G, Thursmond J, (1964) Stimulus-response compatibility and the rate of gain of information. Psychon Sci 1: 111-112.
32. Broadbent DE, Gregory M, (1965) On the interaction of S-R compatibility with other variables affecting reaction time. Brit J Psychol 56: 61-67.
33. Spigel IM, (1965) Lift reaction time and topographic compatibility of the S-R field. J Gen Psychol 72: 165-172.
34. Mahurin RK, Pirozzolo FJ, (1993) Application of hick's law of response speed in alzheimer and parkinson diseases. Percept Mot Skills 77: 107-113.
35. Vickrey C, Neuringer A, (2000) Pigeon reaction time, hick's law, and intelligence. Psychon Bull Rev 7: 284-291.
36. Mowbray GH, Rhoades MV, (1959) On the reduction of choice reaction-times with practice. Q J Exp Psychol 11: 16-23.
37. Welford AT, (1980) Choice reaction time: basic concepts. In: Welford AT, editors. Reaction Times New York Academic Press pp. 73-128.
38. Yuille A, Geiger D, (1995) Winner-Take-All mechanisms. In: Arbib M, editors. Handbook of Brain Theory and Neural Networks MIT Press.
39. Ellias SA, Grossberg S, (1975) Pattern formation, contrast control, and oscillations in the short term memory of shunting on-center off-surround networks. Biol Cybernetics 20: 69-98.
40. Montague PR, Dayan P, Sejnowski TJ, (1996) A framework for mesencephalic dopamine systems based on predictive hebbian learning. J Neurosci 16: 1936-1947.
41. Schultz W, (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80: 1-27.
42. Lyons M, Robbins TW, (1975) The action of central nervous system stimulant drugs: a general theory concerning amphetamine effects. In: Essman W, Valzelli L, editors. Current developments in psychopharmacology New York Spectrum pp. 80-163 Volume 2.
43. Evenden JL, Robbins TW, (1983) Increased response switching, perseveration and perseverative switching following d-amphetamine in the rat. Psychopharmacology (Berl) 80: 67-73.
44. Taylor JR, Robbins TW, (1984) Enhanced behavioural control by conditioned reinforcers following microinjections of d-amphetamine into the nucleus accumbens. Psychopharmacology (Berl) 84: 405-412.
45. Taylor JR, Robbins TW, (1986) 6-Hydroxydopamine lesions of the nucleus accumbens, but not of the caudate nucleus, attenuate enhanced responding with reward-related stimuli produced by intra-accumbens d-amphetamine. Psychopharmacology (Berl) 90: 390-397.
46. Ljungberg T, Enquist M, (1987) Disruptive effects of low doses of d-amphetamine on the ability of rats to organize behaviour into functional sequences. Psychopharmacology (Berl) 93: 146-151.
47. Jackson DM, Andén NE, Dahlström A, (1975) A functional effect of dopamine in the nucleus accumbens and in some other dopamine-rich parts of the rat brain. Psychopharmacologia 45: 139-149.
48. Carr GD, White NM, (1987) Effects of systemic and intracranial amphetamine injections on behavior in the open field: a detailed analysis. Pharmacol Biochem Behav 27: 113-122.
49. Sokolowski JD, Salamone JD, (1998) The role of accumbens dopamine in lever pressing and response allocation: effects of 6-OHDA injected into core and dorsomedial shell. Pharmacol Biochem Behav 59: 557-566.
50. Aberman JE, Salamone JD, (1999) Nucleus accumbens dopamine depletions make rats more sensitive to high ratio requirements but do not impair primary food reinforcement. Neuroscience 92: 545-552.
51. Salamone JD, Wisniecki A, Carlson BB, Correa M, (2001) Nucleus accumbens dopamine depletions make animals highly sensitive to high fixed ratio requirements but do not impair primary food reinforcement. Neuroscience 105: 863-870.
52. Correa M, Carlson BB, Wisniecki A, Salamone JD, (2002) Nucleus accumbens dopamine and work requirements on interval schedules. Behav Brain Res 137: 179-187.
53. Mingote S, Weber SM, Ishiwari K, Correa M, Salamone JD, (2005) Ratio and time requirements on operant schedules: effort-related effects of nucleus accumbens dopamine depletions. Eur J Neurosci 21: 1749-1757.
54. Mas-Colell A, Whinston MD, Green JR, (1995) Microeconomic Theory. Cambridge Cambridge Univ. Press.
55. Simon H, (1947) Administrative behavior. New York Macmillan.
56. Simon H, (1982) Volume 2, Models of bounded rationality. Cambridge MIT Press.
57. Simon H, (1955) A behavioral model of rational choice. Q J Econ 69: 99-118.
58. Gold JI, Shadlen MN, (2007) The neural basis of decision making. Annu Rev Neurosci 30: 535-574.
59. Gigerenzer G, (2000) Adaptive thinking: Rationality in the real world. Evolution and New York Oxford University Press.
60. Gigerenzer G, Todd PM, Group AR (1999) Simple heuristics that make us smart. New York Oxford University Press 1st edition.
61. Zilberstein S, Russell S, (1995) Approximate reasoning using anytime algorithms. In: Natarajan S, editors. Imprecise and Approximate Computation Springer pp. 43-62 Volume 318 The Kluwer International Series in Engineering and Computer Science.
62. Jueptner M, Frith CD, Brooks DJ, Frackowiak RS, Passingham RE, (1997) Anatomy of motor learning. II. subcortical structures and learning by trial and error. J Neurophysiol 77: 1325-1337.
63. Jueptner M, Stephan KM, Frith CD, Brooks DJ, Frackowiak RS, et al. (1997) Anatomy of motor learning. i. frontal cortex and attention to action. J Neurophysiol 77: 1313-1324.
64. Shah A, Barto AG, (2009) Effect on movement selection of an evolving sensory representation: a multiple controller model of skill acquisition. Brain Res 1299: 55-73.
65. Tassinari H, Hudson TE, Landy MS, (2006) Combining priors and noisy visual cues in a rapid pointing task. J Neurosci 26: 10154-10163.
66. McClure SM, Daw ND, Montague PR, (2003) A computational substrate for incentive salience. Trends Neurosci 26: 423-428.
67. Dickinson A, (1985) Actions and habits: The development of behavioural autonomy. Philos Trans R Soc Lond B Biol Sci 308: 78-67.
68. Balleine B, Garner C, Gonzalez F, Dickinson A, (1995) Motivational control of heterogeneous instrumental chains. J Exp Psychol Anim Behav Process 21: 203-217.
69. Suri RE, (2001) Anticipatory responses of dopamine neurons and cortical neurons reproduced by internal model. Exp Brain Res 140: 234-240.
70. Robbins TW, Everitt BJ, (1992) Functions of dopamine in the dorsal and ventral striatum. Semin Neurosci 4: 119-127.
71. Shiv B, Fedorikhin A, (1999) Heart and mind in conflict: The interplay of affect and cognition in consumer decision making. J Cons Res 26: 278-92.
72. Evans ML, Sherwin RS, (2002) Blood glucose and the brain in diabetes: between a rock and a hard place? Curr Diab Rep 2: 101-102.
73. Haber SN, Fudge JL, McFarland NR, (2000) Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J Neurosci 20: 2369-2382.
74. Haber SN, (2003) The primate basal ganglia: parallel and integrative networks. J Chem Neuroanat 26: 317-330.
75. Belin D, Jonkman S, Dickinson A, Robbins TW, Everitt BJ, (2009) Parallel and interactive learning processes within the basal ganglia: relevance for the understanding of addiction. Behav Brain Res 199: 89-102.
76. Haruno M, Kawato M, (2006) Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning. Neural Netw 19: 1242-1254.
77. Joel D, Niv Y, Ruppin E, (2002) Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw 15: 535-547.
78. Barto AG, Mahadevan S, (2003) Recent advances in hierarchical reinforcement learning. Discrete Event Dyn S 13: 341-379.
79. Julier SJ, Uhlmann J, (2004) Unscented filtering and nonlinear estimation. Proc IEEE 92: 401-422.
80. Daw ND, Touretzky DS, (2002) Long-term reward prediction in TD models of the dopamine system. Neural Comput 14: 2567-2583.
81. Tsitsiklis JN, Roy BV, (1997) Average cost temporal-difference learning. Automatica 35: 1799-1808.