A Biologically Plausible Computational Theory for Value Integration and Action Selection in Decisions with Competing Alternatives

PLoS Computational Biology

Volumn 11, Issue 3, 2015, Pages

  Christopoulos, Vassilios ^a   Bonaiuto, James ^a,b   Andersen, Richard A ^a  

^a California Institute of Technology HHMI   (United States)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMALS; BEHAVIORAL RESEARCH; BRAIN; COMPUTATION THEORY;

ACTION SELECTION; ANIMAL BEHAVIOUR; COMPUTATIONAL THEORIES; DECISION TASK; DECISIONS MAKINGS; DYNAMIC ENVIRONMENTS; HUMAN BEHAVIORS; PSYCHOLOGICAL THEORY; SIMPLE++; VALUE INTEGRATIONS;

DECISION MAKING;

ACTION SELECTION; ARTICLE; BRAIN FUNCTION; COGNITION; CONCEPTUAL FRAMEWORK; DECISION MAKING; DECISION THEORY; HUMAN; LEARNING; MATHEMATICAL COMPUTING; MATHEMATICAL MODEL; NEUROPHYSIOLOGY; NONHUMAN; PREDICTION; PROBLEM SOLVING; REWARD; SENSORIMOTOR CORTEX; SENSORIMOTOR FUNCTION; VALUE INTEGRATION; ANIMAL; BIOLOGICAL MODEL; BIOLOGY; EYE MOVEMENT; PERCEPTION; PHYSIOLOGY; PRIMATE;

ANIMALIA;

ANIMALS; COMPUTATIONAL BIOLOGY; DECISION MAKING; EYE MOVEMENTS; HUMANS; MODELS, NEUROLOGICAL; PERCEPTION; PRIMATES; REWARD;

EID: 84926313819     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1004104     Document Type: Article

References (73)

1. Tversky A, Kahneman D, (1981) The framing of decisions and the psychology of choice. Science. 211: 453–458. doi: 10.1126/science.7455683 7455683
2. Roesch MR, Olson CR, (2004) Neuronal activity related to reward value and motivation in primate frontal cortex. Science. 304: 307–310. doi: 10.1126/science.1093223 15073380
3. Padoa-Schioppa C, Assad J, (2006) Neurons in orbitofrontal cortex encode economic value. Nature. 441: 223–226. doi: 10.1038/nature04676 16633341
4. Padoa-Schioppa C, (2011) Neurobiology of economic choice: a good-based model. Annu Rev Neurosci. 34: 333–359. doi: 10.1146/annurev-neuro-061010-113648 21456961
5. Cai X, Padoa-Schioppa C, (2012) Neuronal encoding of subjective value in dorsal and ventral anterior cingulate cortex. J Neurosci. 32: 3791–3808. doi: 10.1523/JNEUROSCI.3864-11.2012 22423100
6. Basso MA, Wurtz RH, (1998) Modulation of neuronal activity in superior colliculus by changes in target probability. J Neurosci. 18: 7519–7534. 9736670
7. Sugrue LP, Corrado GS, Newsome WT, (2005) Choosing the greater of two goods: neural currencies for valuation and decision making. Nat. Rev. Neurosci. 6: 363–375. doi: 10.1038/nrn1666 15832198
8. Cisek P, (2007) Cortical mechanisms of action selection: the affordance competition hypothesis. Philos Trans R Soc Lond B Biol Sci. 362: 1585–1599. doi: 10.1098/rstb.2007.2054 17428779
9. Hudson TE, Maloney LT, Landy MS, (2007) Movement planning with probabilistic target information. J Neurophysiol. 98: 3034–3046. doi: 10.1152/jn.00858.2007 17898140
10. Chapman CS, Gallivan JP, Wood DK, Milne JL, Culham JC, Goodale MA, (2010) Reaching for the unknown: multiple target encoding and real-time decision-making in a rapid reach task. Cognition. 116: 168–176. doi: 10.1016/j.cognition.2010.04.008 20471007
11. Rangel A, Hare T, (2010) Neural computations associated with goal-directed choice. Curr Opin Neurobiol. 2: 262–270. doi: 10.1016/j.conb.2010.03.001
12. Platt ML, Glimcher PW, (1999) Neural correlates of decision variables in parietal cortex. Nature. 400: 233–238. doi: 10.1038/22268 10421364
13. Gold JI, Shadlen MN, (2007) The neural basis of decision making. Annu Rev Neurosci. 30: 535–574. doi: 10.1146/annurev.neuro.29.051605.113038 17600525
14. Kable JW, Glimcher PW, (2009) The neurobiology of decision: consensus and controversy. Neuron. 63: 733–745. doi: 10.1016/j.neuron.2009.09.003 19778504
15. Cisek P, Kalaska JF, (2010) Neural mechanisms for interacting with a world full of action choices. Annu Rev Neurosci. 33: 269–298. doi: 10.1146/annurev.neuro.051508.135409 20345247
16. Song JH, Nakayama K, (2008) Target selection in visual search as revealed by movement trajectories. Vision Res. 48: 853–861. doi: 10.1016/j.visres.2007.12.015 18262583
17. Song JH, Nakayama K, (2009) Hidden cognitive states revealed in choice reaching tasks. Trends Cogn Sci. 8: 360–366. doi: 10.1016/j.tics.2009.04.009
18. Ghez C, Favilla M, Ghilardi MF, Gordon J, Bermejo R, Pullman S, (1997) Discrete and continuous planning of hand movements and isometric force trajectories. Exp. Brain Res. 115: 217–233. doi: 10.1007/PL00005692 9224851
19. Chapman CS, Gallivan JP, Wood DK, Milne JL, Culham JC, Goodale MA, (2010) Short-term motor plasticity revealed in a visuomotor decision-making task. Behav Brain Res. 214: 130–134. doi: 10.1016/j.bbr.2010.05.012 20472001
20. Gallivan JP, Chapman CS, Wood DK, Milne JL, Ansari D, Culham JC, Goodale MA, (2011) One to four, and nothing more: nonconscious parallel individuation of objects during action planning. Psychol Sci. 22: 803–811. doi: 10.1177/0956797611408733 21562312
21. Gallivan JP, Chapman CS, (2014) Three-dimensional reach trajectories as a probe of real-time decision-making between multiple competing targets. Front Neurosci. 8:215. doi: 10.3389/fnins.2014.00215 25100941
22. Eriksen CW, Schultz DW, (1979) Information processing in visual search: a continuous flow conception and experimental results. Percept Psychophys. 25: 249–263. doi: 10.3758/BF03198804 461085
23. Miller J, (1982) Discrete versus continuous stage models of human information processing: in search of partial output. J Exp Psychol Hum Percept Perform. 8: 273–296. doi: 10.1037/0096-1523.8.2.273 6461722
24. Gratton G, Coles MG, Sirevaag EJ, Eriksen CW, Donchin E, (1988) Pre- and poststimulus activation of response channels: a psychophysiological analysis. J Exp Psychol Hum Percept Perform. 14: 331–344. doi: 10.1037/0096-1523.14.3.331 2971764
25. Erlhagen W, Schöner G, (2002) Dynamic field theory of movement preparation. Psychol Rev. 109: 545–572. doi: 10.1037/0033-295X.109.3.545 12088245
26. Todorov E, Liu D, (2007) Evidence for the flexible sensorimotor strategies predicted by optimal feedback control. J. Neurosci. 27: 9354–9368. doi: 10.1523/JNEUROSCI.1110-06.2007 17728449
27. Cisek P, Kalaska JF, (2005) Neural correlates of reaching decisions in dorsal premotor cortex: specification of multiple direction choices and final selection of action. Neuron. 45: 801–814. doi: 10.1016/j.neuron.2005.01.027 15748854
28. Scherberger H, Andersen RA, (2007) Target selection signals for arm reaching in the posterior parietal cortex. J Neurosci. 27: 2001–2012. doi: 10.1523/JNEUROSCI.4274-06.2007 17314296
29. Cui H, Andersen RA, (2007) Posterior parietal cortex encodes autonomously selected motor plans. Neuron. 56: 552–559. doi: 10.1016/j.neuron.2007.09.031 17988637
30. McPeek RM, Han JH, Keller EL, (2003) Competition between saccade goals in the superior colliculus produces saccade curvature. J Neurophysiol. 89: 2577–2590. doi: 10.1152/jn.00657.2002 12611995
31. Platt ML, Glimcher PW, (1997) Responses of intraparietal neurons to saccadic targets and visual distractors. J Neurophysiol. 78: 1574–1589. 9310444
32. Pastor-Bernier A, Cisek P, (2011) Neural correlates of biased competition in premotor cortex. J Neurosci. 31: 7083–7088. doi: 10.1523/JNEUROSCI.5681-10.2011 21562270
33. Klaes C, Westendorff S, Chakrabarti S, Gail A, (2011) Choosing goals, not rules: deciding among rule-based action plans. Neuron. 70: 536–548. doi: 10.1016/j.neuron.2011.02.053 21555078
34. Tipper SP, Lortie C, Baylis GC, (1992) Selective reaching: evidence for action-centered attention. J Exp Psychol Hum Percept Perform. 18: 891–905. doi: 10.1037/0096-1523.18.4.891 1431753
35. Van der Stigchel S, Meeter M, Theeuwes J, (2006) Eye movement trajectories and what they tell us. Neurosci Biobehav Rev. 30: 666–679. doi: 10.1016/j.neubiorev.2005.12.001 16497377
36. Andersen RA, Cui H, (2009) Intention, action planning, and decision making in parietal-frontal circuits. Neuron. 63: 568–583. doi: 10.1016/j.neuron.2009.08.028 19755101
37. Bernier P-M, Cieslak M, Grafton S, (2012) Effector selection precedes reach planning in the dorsal parieto-frontal cortex. J Neurophysiol. 108: 57–68. doi: 10.1152/jn.00011.2012 22457458
38. Sutton RS, Barto AG, (1998) Reinforcement learning: an introduction. MIT Press, Cambridge, MA.
39. Milstein DM, Dorris MC, (2011) The relationship between saccadic choice and reaction times with manipulations of target value. Front Neurosci. 5:122. doi: 10.3389/fnins.2011.00122 22028681
40. Pasquereau B, Turner RS, (2013) Limited encoding of effort by dopamine neurons in a cost-benefit trade-off task. J Neurosci. 33: 8288–8300. doi: 10.1523/JNEUROSCI.4619-12.2013 23658169
41. Bendiksby MS, Platt ML, (2006) Neural correlates of reward and attention in macaque area LIP. Neuropsychologia. 44: 2411–2420. doi: 10.1016/j.neuropsychologia.2006.04.011 16757005
42. Sugrue LP, Corrado GS, Newsome WT, (2004) Matching behavior and the representation of value in the parietal cortex. Science 304: 1782–1787. doi: 10.1126/science.1094765 15205529
43. Janssen P, Shadlen MN, (2005) A representation of the hazard rate of elapsed time in macaque area LIP. Nat Neurosci. 8: 234–241. doi: 10.1038/nn0305-396b 15657597
44. Seo H, Barraclough DJ, Lee D, (2009) Lateral intraparietal cortex and reinforcement learning during a mixed-strategy game. J Neurosci. 29: 7278–7289. doi: 10.1523/JNEUROSCI.1479-09.2009 19494150
45. Kennerley SW, Walton ME, (2011) Decision making and reward in frontal cortex: complementary evidence from neurophysiological and neuropsychological studies. Behav Neurosci. 125: 297–317. doi: 10.1037/a0023575 21534649
46. Louie K, Glimcher PW, (2012) Efficient coding and the neural representation of value. Ann N Y Acad Sci. 1251: 13–32. doi: 10.1111/j.1749-6632.2012.06496.x 22694213
47. Wilke M, Kagan I, Andersen RA, (2012) Functional imaging reveals rapid reorganization of cortical activity after parietal inactivation in monkeys. Proc Natl Acad Sci U S A 109: 8274–8279. doi: 10.1073/pnas.1204789109 22562793
48. Wardak C, Olivier E, Duhamel JR, (2002) Saccadic target selection deficits after lateral intraparietal area inactivation in monkeys. J Neurosci. 22: 9877–9884. 12427844
49. McPeek RM, Keller EL, (2004) Deficits in saccade target selection after inactivation of superior colliculus. Nat Neurosci. 7: 757–763. doi: 10.1038/nn1269 15195099
50. Wilke M, Kagan I, Andersen RA, (2013) Effects of pulvinar inactivation on spatial decision-making between equal and asymmetric reward options. J Cogn Neurosci. 25: 1270–1283. doi: 10.1162/jocn_a_00399 23574581
51. Usher M, McClelland JL, (2004) Loss aversion and inhibition in dynamical models of multialternative choice. Psychol Rev. 111: 757–769. doi: 10.1037/0033-295X.111.3.757 15250782
52. Trueblood JS, Brown SD, Heathcote A, (2014) The multiattribute linear ballistic accumulator model of context effects in multialternative choice. Psychol Rev. 121: 179–205. doi: 10.1037/a0036137 24730597
53. Roe RM, Busemeyer JR, Townsend JT, (2001) Multialternative decision field theory: a dynamic connectionist model of decision making. Psychol Rev. 108: 370–392. doi: 10.1037/0033-295X.108.2.370 11381834
54. Ratcliff R, (2002) A diffusion model account of response time and accuracy in a brightness discrimination task: fitting real data and failing to fit fake but plausible data. Psychon Bull Rev. 9: 278–291. doi: 10.3758/BF03196283 12120790
55. Usher M, McClelland JL, (2001) The time course of perceptual choice: the leaky, competing accumulator model. Psychol. Rev. 108: 550–592. doi: 10.1037/0033-295X.108.3.550 11488378
56. Mazurek ME, Roitman JD, Ditterich J, Shadlen MN, (2003) A role for neural integrators in perceptual decision making. Cereb Cortex. 13: 1257–1269. doi: 10.1093/cercor/bhg097 14576217
57. Klaes C, Schneegans S, Schöner G, Gail A, (2012) Sensorimotor learning biases choice behavior: a learning neural field model for decision making. PLoS Comput Biol. 8:e1002774. doi: 10.1371/journal.pcbi.1002774 23166483
58. Cisek P, (2006) Integrated neural processes for defining potential actions and deciding between them: a computational model. J Neurosci. 26: 9761–9770. doi: 10.1523/JNEUROSCI.5605-05.2006 16988047
59. Botvinick MM, (2012) Hierarchical reinforcement learning and decision making. Curr Opinion Neurobio. 22: 956–962. doi: 10.1016/j.conb.2012.05.008
60. Hoshi E, (2013) Cortico-basal ganglia networks subserving goal-directed behavior mediated by conditional visuo-goal association. Front Neural Circuits. 7:158. doi: 10.3389/fncir.2013.00158 24155692
61. Watanabe K, Funahashi S, (2014) Neural mechanisms of dual-task interference and cognitive capacity limitation in the prefrontal cortex. Nat Neurosci. 17: 601–611. doi: 10.1038/nn.3667 24584049
62. Watanabe M, (1986) Prefrontal unit activity during delayed conditional Go/No-Go discrimination in the monkey. I. Relation to the stimulus. Brain Res. 382: 1–14.
63. Asaad WF, Rainer G, Miller EK, (2000) Task-specific neural activity in the primate prefrontal cortex. J Neurophysiol. 84: 451–459. 10899218
64. Kusunoki M, Sigala N, Gaffan D, Duncan J, (2009) Detection of fixed and variable targets in the monkey prefrontal cortex. Cereb. Cortex 19: 2522–2534. doi: 10.1093/cercor/bhp005 19193714
65. O’Doherty JP, (2011) Contributions of the ventromedial prefrontal cortex to goal-directed action selection. Ann N Y Acad Sci. 1239, 118–129. doi: 10.1111/j.1749-6632.2011.06290.x 22145881
66. Rudebeck PH, Behrens TE, Kennerley SW, Baxter MG, Buckley MJ, Walton ME, Rushworth MF, (2008) Frontal cortex subregions play distinct roles in choices between actions and stimuli. J Neurosci. 28: 13775–13785. doi: 10.1523/JNEUROSCI.3541-08.2008 19091968
67. Wallis JD, Kennerley SW, (2010) Heterogeneous reward signals in prefrontal cortex. Curr Opin Neurobiol. 20: 191–198. doi: 10.1016/j.conb.2010.02.009 20303739
68. Kennerley SW, Dahmubed AF, Lara AH, Wallis JD, (2009) Neurons in the frontal lobe encode the value of multiple decision variables. J Cogn Neurosci. 21: 1162–1178. doi: 10.1162/jocn.2009.21100 18752411
69. Christopoulos VN, Schrater PR, (2011) An optimal feedback control framework for grasping objects with position uncertainty. Neural Comput. 23: 2511–2536. doi: 10.1162/NECO_a_00180 21732861
70. Chen-Harris H, Joiner WM, Ethier V, Zee DS, Shadmehr R, (2008) Adaptive control of saccades via internal feedback. J Neurosci. 28: 2804–2813. doi: 10.1523/JNEUROSCI.5300-07.2008 18337410
71. Goodwin GC, Seron MM, de Dona JA, (2005) Constrained control and estimation: an optimisation approach. Springer, London, UK.
72. Talmi D, Dayan P, Kiebel SJ, Frith CD, Dolan RJ, (2009) How humans integrate the prospects of pain and reward during choice. J Neurosci. 29: 14617–14626. doi: 10.1523/JNEUROSCI.2026-09.2009 19923294
73. Croxson PL, Walton ME, O’Reilly JX, Behrens TE, Rushworth MF, (2009) Effort-based cost-benefit valuation and the human brain. J Neurosci. 29: 4531–4541. doi: 10.1523/JNEUROSCI.4515-08.2009 19357278