Physiological and behavioral signatures of reflective exploratory choice

Cognitive, Affective and Behavioral Neuroscience

Volumn 14, Issue 4, 2014, Pages 1167-1183

  Otto, A Ross ^a   Knox, W Bradley ^b   Markman, Arthur B ^c   Love, Bradley C ^d  

^a NEW YORK UNIVERSITY   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c UNIVERSITY OF TEXAS AT AUSTIN   (United States)

^d UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Arousal; Decision making; Emotion; Reinforcement learning; Reward

Indexed keywords

AROUSAL; DECISION MAKING; ELECTRODERMAL RESPONSE; EMOTION; EXPLORATORY BEHAVIOR; FEMALE; HUMAN; MALE; PHYSIOLOGY; PROBABILITY; PSYCHOLOGICAL MODEL; STUDENT; UNIVERSITY;

AROUSAL; CHOICE BEHAVIOR; EMOTIONS; EXPLORATORY BEHAVIOR; FEMALE; GALVANIC SKIN RESPONSE; HUMANS; MALE; MODELS, PSYCHOLOGICAL; PROBABILITY; STUDENTS; UNIVERSITIES;

EID: 84896425358     PISSN: 15307026     EISSN: None     Source Type: Journal    
DOI: 10.3758/s13415-014-0260-4     Document Type: Article

References (69)

1. Badre, D., Doll, B. B., Long, N. M., & Frank, M. J. (2012). Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration. Neuron, 73(3), 595–607.
2. Bechara, A., Damasio, H., Damasio, A. R., & Lee, G. P. (1999). Different contributions of the human amygdala and ventromedial prefrontal cortex to decision-making. Journal of Neuroscience, 19(13), 5473–5481.
3. Bechara, A., Tranel, D., Damasio, H., & Damasio, A. R. (1996). Failure to respond autonomically to anticipated future outcomes following damage to prefrontal cortex. Cerebral Cortex, 6(2), 215–225.
4. Benedek, M., & Kaernbach, C. (2010). A continuous measure of phasic electrodermal activity. Journal of Neuroscience Methods, 190(1), 80–91.
5. Blanco, N. J., Otto, A. R., Maddox, W. T., Beevers, C. G., & Love, B. C. (2013). The influence of depression symptoms on exploratory decision-making. Cognition, 129(3), 563–568.
6. Botvinick, M. M., & Rosen, Z. B. (2009). Anticipation of cognitive demand during decision-making. Psychological Research Psychologische Forschung, 73(6), 835–842.
7. Busemeyer, J. R., & Stout, J. C. (2002). A contribution of cognitive decision models to clinical assessment: Decomposing performance on the Bechara gambling task. Psychological Assessment, 14(3), 253–262.
8. Camille, N., Coricelli, G., Sallet, J., Pradat-Diehl, P., Duhamel, J.-R., & Sirigu, A. (2004). The involvement of the orbitofrontal cortex in the experience of regret. Science, 304(5674), 1167–1170.
9. Cassandra, A., Littman, M. L., & Zhang, N. L. (1997). Incremental pruning: A simple, fast, exact method for partially observable Markov decision processes. In Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence, 54–61.
10. Cohen, J. D., McClure, S. M., & Yu, A. J. (2007). Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philosophical Transactions of the Royal Society, B: Biological Sciences, 362(1481), 933–942.
11. Critchley, H. D. (2005). Neural mechanisms of autonomic, affective, and cognitive integration. The Journal of Comparative Neurology, 493(1), 154–166.
12. Critchley, H. D., Elliott, R., Mathias, C. J., & Dolan, R. J. (2000). Neural activity relating to generation and representation of galvanic skin conductance responses: A functional magnetic resonance imaging study. Journal of Neuroscience, 20(8), 3033–3040.
13. Critchley, H. D., Mathias, C. J., & Dolan, R. J. (2001). Neural activity in the human brain relating to uncertainty and arousal during anticipation. Neuron, 29(2), 537–545.
14. Critchley, H. D., Tang, J., Glaser, D., Butterworth, B., & Dolan, R. J. (2005). Anterior cingulate activity during error and autonomic response. NeuroImage, 27(4), 885–895.
15. Damasio, A. (1994). Descartes’ error: Emotion, reason, and the human brain. New York: Putnam.
16. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based influences on humans’ choices and striatal prediction errors. Neuron, 69(6), 1204–1215.
17. Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8(12), 1704–1711.
18. Daw, N. D., O’Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature, 441, 876–879.
19. Dolan, R. J. (2002). Emotion, cognition, and behavior. Science, 298(5596), 1191–1194.
20. Dunn, B. D., Dalgleish, T., & Lawrence, A. D. (2006). The somatic marker hypothesis: A critical evaluation. Neuroscience & Biobehavioral Reviews, 30(2), 239–271.
21. Fellows, L. K. (2007). Advances in understanding ventromedial prefrontal function the accountant joins the executive. Neurology, 68(13), 991–995.
22. Fellows, L. K., & Farah, M. J. (2005). Different underlying impairments in decision-making following ventromedial and dorsolateral frontal lobe damage in humans. Cerebral Cortex, 15(1), 58–63.
23. Figner, B., Mackinlay, R. J., Wilkening, F., & Weber, E. U. (2009). Affective and deliberative processes in risky choice: Age differences in risk taking in the Columbia Card Task. Journal of Experimental Psychology Learning, Memory, and Cognition, 35(3), 709–730.
24. Foerde, K., Knowlton, B. J., & Poldrack, R. A. (2006). Modulation of competing memory systems by distraction. Proceedings of the National Academy of Sciences, 103(31), 11778–11783.
25. Gershman, S. J., Markman, A. B., & Otto, A. R. (2014). Retrospective revaluation in sequential decision making: A tale of two systems. Journal of Experimental Psychology: General, 143(1), 182–194.
26. Gilzenrat, M., Nieuwenhuis, S., Jepma, M., & Cohen, J. (2010). Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function. Cognitive, Affective, & Behavioral Neuroscience, 10(2), 252–269.
27. Gläscher, J., Daw, N., Dayan, P., & O’Doherty, J. P. (2010). States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66(4), 585–595.
28. Hajcak, G., McDonald, N., & Simons, R. F. (2003). To err is autonomic: Error-related brain potentials, ANS activity, and post-error compensatory behavior. Psychophysiology, 40(6), 895–903.
29. Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006). The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. Journal of Neuroscience, 26(32), 8360–8367.
30. Hare, T. A., Camerer, C. F., & Rangel, A. (2009). Self-control in decision-making involves modulation of the vmPFC valuation system. Science, 324(5927), 646–648.
31. Humphries, M. D., Khamassi, M., & Gurney, K. (2012). Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia. Frontiers in Decision Neuroscience, 6, 9.
32. Jepma, M., & Nieuwenhuis, S. (2011). Pupil diameter predicts changes in the exploration–exploitation trade-off: Evidence for the adaptive gain theory. Journal of Cognitive Neuroscience, 23(7), 1587–1596.
33. Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1–2), 99–134.
34. Keramati, M., Dezfouli, A., & Piray, P. (2011). Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Computational Biology, 7(5), e1002055.
35. Knox, W. B., Otto, A. R., Stone, P. H., & Love, B. C. (2012). The nature of belief-directed exploratory choice by human decision-makers. Frontiers in Psychology, 2, 398.
36. Lang, P. J., Greenwald, M. K., Bradley, M. M., & Hamm, A. O. (1993). Looking at pictures: Affective, facial, visceral, and behavioral reactions. Psychophysiology, 30(3), 261–273.
37. Ledoux, J. (1996). The emotional brain: The mysterious underpinnings of emotional life. New York: Simon and Schuster.
38. Loewenstein, G. F., Weber, E. U., Hsee, C. K., & Welch, N. (2001). Risk as feelings. Psychological Bulletin, 127(2), 267–286.
39. Lovibond, P. F. (2003). Causal beliefs and conditioned responses: Retrospective revaluation induced by experience and by instruction. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29(1), 97–106.
40. Martin, L. N., & Delgado, M. R. (2011). The influence of emotion regulation on decision-making under risk. Journal of Cognitive Neuroscience, 23(9), 2569–2581.
41. Mellers, B. A., Schwartz, A., Ho, K., & Ritov, I. (1997). Decision affect theory: Emotional reactions to the outcomes of risky options. Psychological Science, 8(6), 423–429.
42. Mitchell, D. G. V. (2011). The nexus between decision making and emotion regulation: A review of convergent neurocognitive substrates. Behavioural Brain Research, 217(1), 215–231.
43. Nagai, Y., Critchley, H., Featherstone, E., Trimble, M., & Dolan, R. (2004). Activity in ventromedial prefrontal cortex covaries with sympathetic skin conductance level: A physiological account of a “default mode” of brain function. NeuroImage, 22(1), 243–251.
44. Nassar, M. R., Rumsey, K. M., Wilson, R. C., Parikh, K., Heasly, B., & Gold, J. I. (2012). Rational regulation of learning dynamics by pupil-linked arousal systems. Nature Neuroscience, 15(7), 1040–1046.
45. Nicolle, A., Klein-Flügge, M. C., Hunt, L. T., Vlaev, I., Dolan, R. J., & Behrens, T. E. J. (2012). An agent independent axis for executed and modeled choice in medial prefrontal cortex. Neuron, 75(6), 1114–1121.
46. Öhman, A., & Soares, J. J. F. (1994). “Unconscious anxiety”: Phobic responses to masked stimuli. Journal of Abnormal Psychology, 103(2), 231–240.
47. Olsson, A., & Phelps, E. A. (2004). Learned fear of “unseen” faces after pavlovian, observational, and instructed fear. Psychological Science, 15(12), 822–828.
48. Otto, A. R., Gershman, S. J., Markman, A. B., & Daw, N. D. (2013a). The curse of planning dissecting multiple reinforcement-learning systems by taxing the central executive. Psychological Science, 24(5), 751–761.
49. Otto, A. R., Markman, A. B., Gureckis, T. M., & Love, B. C. (2010). Regulatory fit and systematic exploration in a dynamic decision-making environment. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(3), 797–804.
50. Otto, A. R., Raio, C. M., Chiang, A., Phelps, E. A., & Daw, N. D. (2013b). Working-memory capacity protects model-based learning from stress. Proceedings of the National Academy of Sciences, 110(52), 20941–20946.
51. Otto, A. R., Taylor, E. G., & Markman, A. B. (2011). There are at least two kinds of probability matching: Evidence from a secondary task. Cognition, 118(2), 274–279.
52. Preuschoff, K., Hart, B., & Einhäuser, W. (2011). Pupil dilation signals surprise: Evidence for noradrenaline’s role in decision making. Frontiers in Decision Neuroscience, 5, 115.
53. Rolls, E. T. (1999). The brain and emotion. Oxford: Oxford University Press.
54. Rushworth, M. F. S., Noonan, M. P., Boorman, E. D., Walton, M. E., & Behrens, T. E. (2011). Frontal cortex and reward-guided learning and decision-making. Neuron, 70(6), 1054–1069.
55. Schonberg, T., Fox, C. R., & Poldrack, R. A. (2011). Mind the gap: Bridging economic and naturalistic risk-taking with cognitive neuroscience. Trends in Cognitive Sciences, 15(1), 11–19.
56. Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464.
57. Sokol-Hessner, P., Hsu, M., Curley, N. G., Delgado, M. R., Camerer, C. F., & Phelps, E. A. (2009). Thinking like a trader selectively reduces individuals’ loss aversion. Proceedings of the National Academy of Sciences, 106(13), 5035–5040.
58. Studer, B., & Clark, L. (2011). Place your bets: Psychophysiological correlates of decision-making under risk. Cognitive, Affective, & Behavioral Neuroscience, 11(2), 144–158.
59. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning. Cambridge, MA: MIT Press.
60. Suzuki, A., Hirota, A., Takasawa, N., & Shigemasu, K. (2003). Application of the somatic marker hypothesis to individual differences in decision making. Biological Psychology, 65(1), 81–88.
61. Tomb, I., Hauser, M., Deldin, P., & Caramazza, A. (2002). Do somatic markers mediate decisions on the gambling task? Nature Neuroscience, 5(11), 1103–1104.
62. Tversky, A., & Kahneman, D. (1992). Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5(4), 297–323.
63. Whitney, P., Hinson, J. M., Wirick, A., & Holben, H. (2007). Somatic responses in behavioral inhibition. Cognitive, Affective, & Behavioral Neuroscience, 7(1), 37–43.
64. Worthy, D. A., Maddox, W. T., & Markman, A. B. (2007). Regulatory fit effects in a choice task. Psychonomic Bulletin & Review, 14(6), 1125–1132.
65. Worthy, D. A., Hawthorne, M. J., & Otto, A. R. (2013). Heterogeneity of strategy use in the Iowa gambling task: A comparison of win-stay/lose-shift and reinforcement learning models. Psychonomic Bulletin & Review, 20(2), 364–371.
66. Wunderlich, K., Dayan, P., & Dolan, R. J. (2012). Mapping value based planning and extensively trained choice in the human brain. Nature Neuroscience. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/22406551
67. Yechiam, E., & Busemeyer, J. R. (2005). Comparison of basic assumptions embedded in learning models for experience-based decision making. Psychonomic Bulletin & Review, 12(3), 387–402.
68. Zajonc, R. B. (1984). On the primacy of affect. American Psychologist, 39(2), 117–123.
69. Zeithamova, D., & Maddox, W. T. (2006). Dual-task interference in perceptual category learning. Memory & Cognition, 34, 387–398.