A computational framework for the study of confidence in humans and animals

Philosophical Transactions of the Royal Society B: Biological Sciences

Volumn 367, Issue 1594, 2012, Pages 1322-1337

  Kepecs, Adam ^a   Mainen, Zachary F ^b  

^a Howard Hughes Medical Institute Cold Spring Harbor Laboratory Cold Spring Harbor   (United States)

^b CHAMPALIMAUD CENTRE FOR THE UNKNOWN   (Portugal)

Author keywords

Bayesian; Metacognition; Model based; Post decision wagering; Psychophysics; Uncertainty

Indexed keywords

ANIMAL; BAYESIAN ANALYSIS; COGNITION; DECISION MAKING; HUMAN BEHAVIOR; NEUROLOGY; PSYCHOLOGY; RODENT; UNCERTAINTY ANALYSIS;

ANIMALIA; RATTUS;

EID: 84859551502     PISSN: 09628436     EISSN: 14712970     Source Type: Journal    
DOI: 10.1098/rstb.2012.0037     Document Type: Review

References (146)

1. Flavell, J. H. 1979 Metacognition and cognitive monitoring: a new area of cognitive-developmental inquiry. Am. Psychol. 34, 906-911. (doi:10.1037/0003-066X. 34.10.906)
2. Metcalfe, J. & Shimamura, A. P. (eds) 1994 Metacognition: knowing about knowing. Cambridge, MA: MIT Press.
3. Terrace, H. S. & Metcalfe, J. (eds) 2005 The missing link in cognition: origins of self-reflective consciousness. New York, NY: Oxford University Press.
4. Metcalfe, J. 2008 Evolution of metacognition. In Handbook of metamemory and memory (eds J. Dunlosky & R. J. Bjork), pp. 29-46. Hove, UK: Psychology Press.
5. Smith, J. D., Shields, W. E. & Washburn, D. A. 2003 The comparative psychology of uncertainty monitoring and metacognition. Behav. Brain Sci. 26, 317-339; discussion 340-373.
6. Smith, J. D., Couchman, J. J. & Beren, J. 2012 The highs and lows of theoretical interpretation in animal- metacognition research. Phil. Trans. R. Soc. B 367, 1297-1309. (doi:10.1098/rstb.2011.0366)
7. Cox, D. R. 2006 Principles of statistical inference. Cambridge, UK: Cambridge University Press.
8. Zemel, R. S., Dayan, P. & Pouget, A. 1998 Probabilistic interpretation of population codes. Neural Comput. 10, 403-430. (doi:10.1162/089976698300017818)
9. Rao, R. P. N., Olshausen, B. A. & Lewicki, M. S. 2002 Probabilistic models of the brain: perception and neural function. Cambridge, MA: MIT Press.
10. Knill, D. C. & Pouget, A. 2004 The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 27, 712-719. (doi:10.1016/j.tins. 2004.10.007)
11. Ma, W. J., Beck, J. M., Latham, P. E. & Pouget, A. 2006 Bayesian inference with probabilistic population codes. Nat. Neurosci. 9, 1432-1438. (doi:10.1038/nn1790)
12. Rao, R. P. 2004 Bayesian computation in recurrent neural circuits. Neural Comput. 16, 1-38. (doi:10. 1162/08997660460733976)
13. Higham, P. A. 2007 No special K! A signal detection framework for the strategic regulation of memory accuracy. J. Exp. Psychol. Gen. 136, 1-22. (doi:10.1037/0096-3445.136.1.1)
14. Johnson, D. M. 1939 Confidence and speed in the two- category judgment. Thesis, Columbia University, NY.
15. Festinger, L. 1943 Studies in decision: I. Decision-time, relative frequency of judgment and subjective confidence as related to physical stimulus difference. J Exp. Psychol. 32, 291-306. (doi:10.1037/h0056685)
16. Baranski, J. V. & Petrusic, W. M. 1994 The calibration and resolution of confidence in perceptual judgments. Percept. Psychophys. 55,412-428. (doi:10.3758/BF03205299)
17. Fleming, S. M., Weil, R. S., Nagy, Z., Dolan, R. J. & Rees, G. 2010 Relating introspective accuracy to individual differences in brain structure. Science 329, 1541-1543. (doi:10.1126/science.1191883)
18. Yokoyama, O. et al. 2010 Right frontopolar cortex activity correlates with reliability of retrospective rating of confidence in short-term recognition memory performance. Neurosci. Res. 68, 199-206. (doi:10. 1016/j.neures.2010.07.2041)
19. Gigerenzer, G., Hoffrage, U. & Kleinbolting, H. 1991 Probabilistic mental models: a Brunswikian theory of confidence. Psychol. Rev. 98, 506-528. (doi:10.1037/ 0033-295X.98.4.506)
20. Klayman, J., Soll, J. B., Gonzalez-Vallejo, C. & Barlas, S. 1999 Overconfidence: it depends on how, what, and whom you ask. Organ. Behav. Hum. Decis. Process. 79, 216-247. (doi:10.1006/obhd.1999.2847)
21. Finn, B. 2008 Framing effects on metacognitive monitoring and control. Mem. Cognit. 36, 813-821. (doi:10.3758/MC.36.4.813)
22. Angell, F. 1907 On judgments of 'like' in discrimination experiments. Am. J. Psychol. 18, 253-260. (doi:10. 2307/1412416)
23. Watson, C. S., Kellogg, S. C., Kawanishi, D. T. & Lucas, P. A. 1973 The uncertain response in detection-oriented psychophysics. J. Exp. Psychol. 99, 180-185. (doi:10.1037/h0034736)
24. Woodworth, R. S. 1938 Experimental psychology. New York, NY: Henry Holt and Company.
25. Peirce, C. S. & Jastrow, J. 1885 On small differences of sensation. Mem. Natl Acad. Sci. 3, 73-83.
26. George, S. S. 1917 Attitude in relation to the psychophysical judgment. Am. J. Psychol. 28, 1-37. (doi:10. 2307/1412939)
27. Smith, J. D., Schull, J., Strote, J., McGee, K., Egnor, R. & Erb, L. 1995 The uncertain response in the bottlenosed dolphin (Tursiops truncatus). J. Exp. Psychol. Gen. 124, 391-408. (doi:10.1037/0096-3445.124.4.391)
28. Shields, W. E., Smith, J. D. & Washburn, D. A. 1997 Uncertain responses by humans and rhesus monkeys (Macaca mulatta) in a psychophysical same-different task. J. Exp. Psychol. Gen. 126, 147-164. (doi:10. 1037/0096-3445.126.2.147)
29. Smith, J. D., Shields, W. E., Schull, J. & Washburn, D. A. 1997 The uncertain response in humans and animals. Cognition 62, 75-97. (doi:10.1016/S0010- 0277(96)00726-3)
30. Shields, W. E., Smith, J. D., Guttmannova, K. & Washburn, D. A. 2005 Confidence judgments by humans and rhesus monkeys. J. Gen. Psychol. 132, 165-186.
31. Beran, M. J., Smith, J. D., Redford, J. S. & Washburn, D. A. 2006 Rhesus macaques (Macaca mulatta) monitor uncertainty during numerosity judgments. J. Exp. Psychol. Anim. Behav. Process. 32, 111-119. (doi:10.1037/0097-7403.32.2.111)
32. Sole, L. M., Shettleworth, S. J. & Bennett, P. J. 2003 Uncertainty in pigeons. Psychon. Bull. Rev. 10, 738-745. (doi:10.3758/BF03196540)
33. Smith, J. D., Beran, M. J., Redford, J. S. & Washburn, A. 2006 Dissociating uncertainty responses and reinforcement signals in the comparative study of uncertainty monitoring. J. Exp. Psychol. Gen. 135, 282-297. (doi:10.1037/0096-3445.135.2.282)
34. Sutton, R. S. & Barto, A. G. 1998 Reinforcement learning: an introduction. Cambridge, MA: MIT Press.
35. Hampton, R. R. 2001 Rhesus monkeys know when they remember. Proc. Natl Acad. Sci. USA 98, 5359-5362. (doi:10.1073/pnas.071600998)
36. Kiani, R. & Shadlen, M. N. 2009 Representation of confidence associated with a decision by neurons in the parietal cortex. Science 324, 759-764. (doi:10. 1126/science.1169405)
37. Inman, A. & Shettleworth, S. J. 1999 Detecting metamemory in nonverbal subjects: a test with pigeon. J. Exp. Psychol. Anim. Behav. Process. 25, 389-395. (doi:10.1037/0097-7403.25.3.389)
38. Teller, S. A. 1989 Metamemory in the pigeon: prediction of performance on a delayed matching to sample task. Thesis, Reed College, Portland, OR.
39. Sutton, J. E. & Shettleworth, S. J. 2008 Memory without awareness: pigeons do not show metamemory in delayed matching to sample. J. Exp. Psychol. Anim. Behav. Process. 34, 266-282. (doi:10.1037/0097-7403. 34.2.266)
40. Foote, A. L. & Crystal, J. D. 2007 Metacognition in the rat. Curr. Biol. 17, 551-555. (doi:10.1016/j.cub.2007. 01.061)
41. Persaud, N., McLeod, P. & Cowey, A. 2007 Postdecision wagering objectively measures awareness. Nat. Neurosci. 10, 257-261. (doi:10.1038/nn1840)
42. Persaud, N. & McLeod, P. 2008 Wagering demonstrates subconscious processing in a binary exclusion task. Conscious. Cogn. 17, 565-575. (doi:10.1016/j.concog.2007. 05.003)
43. Sahraie, A., Weiskrantz, L. & Barbur, J. L. 1998 Awareness and confidence ratings in motion perception without geniculo-striate projection. Behav. Brain Res. 96, 71-77. (doi:10.1016/S0166-4328(97)00194-0)
44. Rajaram, S., Hamilton, M. & Bolton, A. 2002 Distinguishing states of awareness from confidence during retrieval: evidence from amnesia. Cogn. Affect. Behav. Neurosci. 2, 227-235. (doi:10.3758/ CABN.2.3.227)
45. Clifford, C. W., Arabzadeh, E. & Harris, J. A. 2008 Getting technical about awareness. Trends Cogn. Sci. 12, 54-58. (doi:10.1016/j.tics.2007.11.009)
46. Fleming, S. M. & Dolan, R. J. 2010 Effects of loss aversion on post-decision wagering: implications for measures of awareness. Conscious. Cogn. 19, 352-363. (doi:10.1016/j.concog.2009.11.002)
47. Schurger, A. & Sher, S. 2008 Awareness, loss aversion, and post-decision wagering. Trends Cogn. Sci. 12, 209-210. (doi:10.1016/j.tics.2008.02.012)
48. Middlebrooks, P. G. & Sommer, M. A. 2011 Metacognition in monkeys during an oculomotor task. J. Exp. Psychol. Learn Mem. Cogn. 37, 325-337. (doi:10. 1037/a0021611)
49. Kepecs, A., Uchida, N., Zariwala, H. A. & Mainen, Z. F. 2008 Neural correlates, computation and behavioural impact of decision confidence. Nature 455, 227-231. (doi:10.1038/nature07200)
50. Call, J. & Carpenter, M. 2001 Do apes and children know what they have seen? Anim. Cogn. 3, 207-220. (doi:10.1007/s100710100078)
51. Hampton, R. R., Zivin, A. & Murray, E. A. 2004 Rhesus monkeys (Macaca mulatta) discriminate between knowing and not knowing and collect information as needed before acting. Anim. Cogn. 7, 239-246. (doi:10.1007/s10071-004-0215-1)
52. Radecki, C. M. & Jaccard, J. 1995 Perceptions of knowledge, actual knowledge, and information search behavior. J. Exp. Soc. Psychol. 31, 107-138. (doi:10. 1006/jesp.1995.1006)
53. Basile, B. M., Hampton, R. R., Suomi, S. J. & Murray, A. 2009 An assessment of memory awareness in tufted capuchin monkeys (Cebus apella). Anim. Cogn. 12, 169-180. (doi:10.1007/s10071-008-0180-1)
54. Brauer, J., Call, J. & Tomasello, M. 2007 Chimpanzees really know what others can see in a competitive situation. Anim. Cogn. 10, 439-448. (doi:10.1007/ s10071-007-0088-1)
55. Suda-King, C. 2008 Do orangutans (Pongo pygmaeus) know when they do not remember? Anim. Cogn. 11, 21-42. (doi:10.1007/s10071-007-0082-7)
56. Brauer, J., Call, J. & Tomasello, M. 2004 Visual perspective taking in dogs (Canis familiaris) in the presence of barriers. Appl. Anim. Behav. Sci. 88, 299317. (doi:10.1016/j.applanim.2004.03.004)
57. Brauer, J., Kaminski, J., Riedel, J., Call, J. & Tomasello, M. 2006 Making inferences about the location of hidden food: social dog, causal ape. J. Comp. Psychol. 120, 38-47. (doi:10.1037/0735-7036.120.1.38)
58. Bromberg-Martin, E. S. & Hikosaka, O. 2009 Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron 63, 119-126. (doi:10.1016/j.neuron.2009.06.009)
59. Shettleworth, S. J. & Sutton, J. E. 2003 Metacognition in animals: it's all in the methods. Behav. Brain Sci. 23, 353-354.
60. Smith, J. D., Beran, M. J., Couchman, J. J. & Coutinho, M. V. 2008 The comparative study of metacognition: sharper paradigms, safer inferences. Psychon. Bull. Rev. 15, 679-691. (doi:10.3758/PBR.15.4.679)
61. Hampton, R. R. 2009 Multiple demonstrations of metacognition in nonhumans: converging evidence or multiple mechanisms? Comp. Cogn. Behav. Rev. 4, 17-28.
62. Schwartz, B. L. & Metcalfe, J. 1994 Methodological problems and pitfalls in the study of human metacognition. In Metacognition: knowing about knowing (eds J. Metcalfe & A. Shimamura), pp. 93-113. Cambridge, MA: MIT Press.
63. Kornell, N. 2009 Metacognition in humans and animals. Curr. Dir. Psychol. Sci. 18, 11-15. (doi:10.1111/ j.1467-8721.2009.01597.x)
64. Washburn, D. A., Smith, J. D. & Shields, W. E. 2006 Rhesus monkeys (Macaca mulatta) immediately generalize the uncertain response. J. Exp. Psychol. Anim. Behav. Process. 32,185-189. (doi:10.1037/0097-7403.32.2.185)
65. Kornell, N., Son, L. K. & Terrace, H. S. 2007 Transfer of metacognitive skills and hint seeking in monkeys. Psychol. Sci. 18, 64-71. (doi:10.1111/j.1467-9280. 2007.01850.x)
66. Metcalfe, J. 2004 Drawing the line on metacognition. Behav. Brain Sci. 26, 350-351.
67. Glimcher, P. W. 2011 Foundations of neuroeconomic analysis. New York, NY: Oxford University Press.
68. Geeold, J. I. & Shadlen, M. N. 2001 Neural computations that underlie decisions about sensory stimuli. Trends Cogn. Sci. 5, 10-16. (doi:10.1016/S1364-6613 (00)01567-9)
69. Dayan, P. & Daw, N. D. 2008 Decision theory, reinforcement learning, and the brain. Cogn. Affect Behav. Neurosci. 8, 429-453. (doi:10.3758/CABN.8.4.429)
70. Juslin, P. & Olsson, H. 1997 Thurstonian and Brunswikian origins of uncertainty in judgment: a sampling model of confidence in sensory discrimination. Psychol. Rev. 104,344-366. (doi:10.1037/0033-295X.104.2.344)
71. Vickers, D. & Pietsch, A. 2001 Decision making and memory: a critique of Juslin and Olsson's (1997) sampling model of sensory discrimination. Psychol. Rev. 108, 789-804. (doi:10.1037/0033-295X.108.4.789)
72. Knight, F. 1921 Risk, ambiguity, and profit. Boston, MA: Houghton, Mifflin.
73. Hsu, M., Bhatt, M., Adolphs, R., Tranel, D. & Camerer, C. F. 2005 Neural systems responding to degrees of uncertainty in human decision-making. Science, 310, 1680-1683. (doi:10.1126/science.1115327)
74. Glimcher, P. W. 2003 Decisions, uncertainty, and the brain: the science of neuroeconomics. Cambridge, MA: MIT Press.
75. McCoy, A. N. & Platt, M. L. 2005 Risk-sensitive neurons in macaque posterior cingulate cortex. Nat. Neurosci. 8, 1220-1227. (doi:10.1038/nn1523)
76. Platt, M. L. & Huettel, S. A. 2008 Risky business: the neuroeconomics of decision making under uncertainty. Nat. Neurosci. 11, 398-403. (doi:10.1038/nn2062)
77. Green, D. M. & Swets, J. A. 1966 Signal detection theory and psychophysics. New York, NY: Wiley.
78. Parker, A. J. & Newsome, W. T. 1998 Sense and the single neuron: probing the physiology of perception. Annu. Rev. Neurosci. 21, 227-277. (doi:10.1146/ annurev.neuro.21.1.227)
79. Vickers, D. 1970 Evidence for an accumulator model of psychophysical discrimination. Ergonomics 13, 37-58. (doi:10.1080/00140137008931117)
80. Moreno-Bote, R. 2010 Decision confidence and uncertainty in diffusion models with partially correlated neuronal integrators. Neural Comput. 22, 1786-1811. (doi:10.1162/neco.2010.12-08-930)
81. Rolls, E. T., Grabenhorst, F. & Deco, G. 2010 Decision-making, errors, and confidence in the brain. J. Neurophysiol. 104, 2359-2374. (doi:10.1152/jn. 00571.2010)
82. Insabato, A., Pannunzi, M., Rolls, E. T. & Deco, G. 2010 Confidence-related decision making. J. Neurophysiol. 104, 539-547. (doi:10.1152/jn.01068.2009)
83. Sollich, P. 2002 Bayesian methods for support vector machines: evidence and predictive class probabilities. Mach. Learn. 46, 21-52. (doi:10.1023/ A:1012489924661)
84. Mazurek, M. E., Roitman, J. D., Ditterich, J. & Shadlen, M. N. 2003 A role for neural integrators in perceptual decision making. Cereb. Cortex 13, 1257-1269. (doi:10.1093/cercor/bhg097)
85. Bogacz, R., Brown, E., Moehlis, J., Holmes, P. & Cohen, J. D. 2006 The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychol. Rev. 113, 700-765. (doi:10.1037/0033-295X.113.4.700)
86. Vickers, D. & Packer, J. 1982 Effects of alternating set for speed or accuracy on response time, accuracy and confidence in a unidimensional discrimination task. Acta Psychol. (Amst). 50, 179-197. (doi:10.1016/ 0001-6918(82)90006-3)
87. Tong, S. & Koller, D. 2002 Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 2, 45-66.
88. Shenoy, P., Kepecs, A. & Rao, R. 2009 Active updating of decision boundaries in rats can be explained using Bayesian classifiers. In Computational and systems neuroscience. Salt Lake City, UT: Frontiers in Systems Neuroscience.
89. Rolls, E. T., Grabenhorst, F. & Deco, G. 2010 Choice, difficulty, and confidence in the brain. Neuroimage 53, 694-706. (doi:10.1016/j.neuroimage.2010.06.073)
90. Rolls, E. T. & Deco, G. 2010 The noisy brain: stochastic dynamics as a principle of brain function. New York, NY: Oxford University Press.
91. Wang, X. J. 2002 Probabilistic decision making by slow reverberation in cortical circuits. Neuron 36, 955-968. (doi:10.1016/S0896-6273(02)01092-9)
92. Timmermans, B., Schilbach, L., Pasquali, A. & Cleeremans, A. 2012 Higher-order thoughts in action: consciousness as an unconscious re-description process. Phil. Trans. R. Soc. B 367, 1412-1423. (doi:10.1098/rstb.2011.0421)
93. Juslin, P. & Olsson, H. 1999 Computational models of subjective probability calibration. In Judgment and decision making: neo-Brunswikian and process-tracing approaches (eds P. Juslin & H. Montgomery), pp. 6795. Hove, UK: Psychology Press.
94. Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. 2006 Cortical substrates for exploratory decisions in humans. Nature 441, 876-879. (doi:10. 1038/nature04766)
95. Cohen, J. D., McClure, S. M. & Yu, A.J. 2007 Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Phil. Trans. R. Soc. B 362, 933-942. (doi:10.1098/rstb.2007.2098)
96. Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J. & Frith, C. D. 2006 Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442, 1042-1045. (doi:10.1038/nature05051)
97. Tobler, P. N., Christopoulos, G. I., O'Doherty, J. P., Dolan, R. J. & Schultz, W. 2009 Risk-dependent reward value signal in human prefrontal cortex. Proc. Natl Acad. Sci. USA 106, 7185-7190. (doi:10.1073/ pnas.0809599106)
98. Corrado, G. & Doya, K. 2007 Understanding neural coding through the model-based analysis of decision making. J. Neurosci. 27, 8178. (doi:10.1523/JNEUR- OSCI.1590-07.2007)
99. Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J. & Frackowiak, R. S. 2004 Temporal difference models describe higher-order learning in humans. Nature 429, 664-667. (doi:10.1038/nature02581)
100. Sugrue, L. P., Corrado, G. S. & Newsome, W. T. 2004 Matching behavior and the representation of value in the parietal cortex. Science 304, 1782-1787. (doi:10. 1126/science.1094765)
101. Barraclough, D. J., Conroy, M. L. & Lee, D. 2004 Prefrontal cortex and decision making in a mixed- strategy game. Nat. Neurosci. 7, 404-410. (doi:10. 1038/nn1209)
102. Lau, B. & Glimcher, P. W. 2005 Dynamic response-by- response models of matching behavior in rhesus monkeys. J. Exp. Anal. Behav. 84, 555-579. (doi:10. 1901/jeab.2005.110-04)
103. Rolls, E. T. & Grabenhorst, F. 2008 The orbitofrontal cortex and beyond: from affect to decision-making. Prog. Neurobiol. 86, 216-244. (doi:10.1016/j.pneuro bio.2008.09.001)
104. Schoenbaum, G. & Roesch, M. 2005 Orbitofrontal cortex, associative learning, and expectancies. Neuron 47, 633-636. (doi:10.1016/j.neuron.2005.07.018)
105. Elliott, R., Dolan, R. J. & Frith, C. D. 2000 Dissociable functions in the medial and lateral orbitofrontal cortex: evidence from human neuroimaging studies. Cereb. Cortex 10, 308-317. (doi:10.1093/cercor/10.3.308)
106. Wallis, J. D. 2007 Orbitofrontal cortex and its contribution to decision-making. Annu. Rev. Neurosci. 30, 31-56. (doi:10.1146/annurev.neuro.30.051606.094334)
107. Mainen, Z. F. & Kepecs, A. 2009 Neural representation of behavioral outcomes in the orbitofrontal cortex. Curr. Opin. Neurobiol. 19, 84-91. (doi:10.1016/j.conb. 2009.03.010)
108. Maunsell, J. H. 2004 Neuronal representations of cognitive state: reward or attention? Trends Cogn. Sci. 8, 261-265. (doi:10.1016/j.tics.2004.04.003)
109. Sanders, J. & Kepecs, A. 2010 How certain are you? Explicit and implicit measures of decision confidence. In Proc. 6th Annu. Conf. on Neuroeconomics, Evanston, IL.
110. Kahneman, D., Slovic, P. & Tversky, A. 1982 Judgment under uncertainty: heuristics and biases. New York, NY: Cambridge University Press.
111. Kaelbling, L. P., Littman, M. L. & Cassandra, A. R. 1998 Planning and acting in partially observable stochastic domains. Artif. Intell. 101, 99-134. (doi:10. 1016/S0004-3702(98)00023-X)
112. Volz, K. G., Schubotz, R. I. & von Cramon, D. Y. 2005 Variants of uncertainty in decision-making and their neural correlates. Brain Res. Bull. 67, 403-412. (doi:10.1016/j.brainresbull.2005.06.011)
113. Yu, A. J. & Dayan, P. 2005 Uncertainty, neuromodulation, and attention. Neuron 46, 681-692. (doi:10. 1016/j.neuron.2005.04.026)
114. Daw, N. D., Niv, Y. & Dayan, P. 2005 Uncertainty- based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704-1711. (doi:10.1038/nn1560)
115. Averbeck, B. B., Sohn, J. W. & Lee, D. 2006 Activity in prefrontal cortex during dynamic selection of action sequences. Nat. Neurosci. 9, 276-282. (doi:10.1038/ nn1634)
116. Fleming, S. M., Dolan, R. J. & Frith, C. D. 2012 Metacognition: computation, biology and function. Phil. Trans. R. Soc. B 367, 1280-1286. (doi:10.1098/rstb. 2012.0021)
117. Stephens, D. W. & Krebs, J. R. 1986 Foraging theory. Princeton, NJ: Princeton University Press.
118. Bernstein, C., Kacelnik, A. & Krebs, J. R. 1988 Individual decisions and the distribution of predators in a patchy environment. J. Anim. Ecol. 57, 1007-1026. (doi:10.2307/5108)
119. Hayden, B. Y., Pearson, J. M. & Platt, M. L. 2011 Neuronal basis of sequential foraging decisions in a patchy environment. Nat. Neurosci. 14, 933-939. (doi:10.1038/nn.2856)
120. Kamil, A. C., Misthal, R. L. & Stephens, D. W. 1993 Failure of simple optimal foraging models to predict residence time when patch quality is uncertain. Behav. Ecol. 4, 350-363. (doi:10.1093/beheco/4.4.350)
121. Nelson, T. O. & Narens, L. 1990 Metamemory: a theoretical framework and new findings. Psychol. Learn. Motiv. 26, 125-173. (doi:10.1016/S0079-7421 (08)60053-5)
122. Son, L. K. & Metcalfe, J. 2000 Metacognitive and control strategies in study-time allocation. J. Exp. Psychol. Learn Mem. Cogn. 26, 204-221. (doi:10.1037/0278- 7393.26.1.204)
123. Son, L. K. 2004 Spacing one's study: evidence for a metacognitive control strategy. J. Exp. Psychol. Learn. Mem. Cogn. 30, 601-604. (doi:10.1037/0278-7393. 30.3.601)
124. Son, L. K. & Sethi, R. 2006 Metacognitive control and optimal learning. Cogn. Sci. A Multidiscip. J. 30, 759-774. (doi:10.1207/s15516709cog0000_74)
125. Schohn, G. & Cohn, D. 2000 Less is more: active learning with support vector machines. In Proc. 17th Intl Conf. Machine Learning, Stanford, CA, June 29-July 2, 2000, pp. 839 -846. San Francisco, CA: Morgan Kaufmann.
126. Sutton, R. S. 1992 Gain adaptation beats least squares. In Proc. 7th Yale Workshop on Adaptive and Learning Systems, 20-22 May, pp. 161-166. New Haven, CT: Yale University.
127. Dayan, P. & Kakade, S. 2001 Explaining away in weight space. In Advances in Neural Information Processing Systems 13 (eds T. K. Leen, T. G. Dietterich & V. Tresp), pp. 451-457. Cambridge, MA: MIT Press.
128. Pearce, J. M. & Hall, G. 1980 A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 87, 532-552. (doi:10.1037/0033-295X.87.6.532)
129. Courville, A. C., Daw, N. D. & Touretzky, D. S. 2006 Bayesian theories of conditioning in a changing world. Trends Cogn. Sci. 10, 294-300. (doi:10.1016/j.tics. 2006.05.004)
130. Li, J., Schiller, D., Schoenbaum, G., Phelps, E. A. & Daw, N. D. 2011 Differential roles of human striatum and amygdala in associative learning. Nat. Neurosci. 14, 1250-1252. (doi:10.1038/nn.2904)
131. Dearden, R., Friedman, N. & Russell, S. 1998 Bayesian Q-learning. In Proc. 15th National Conf. on Artificial Intelligence, pp. 761-768. Menlo Park, CA: AAAI.
132. Strens, M. 2000 A Bayesian framework for reinforcement learning. In Proc. 17th Intl Conf. on Machine Learning, Stanford, CA, pp. 943-950.
133. Bromberg-Martin, E. S. & Hikosaka, O. 2011 Lateral habenula neurons signal errors in the prediction of reward information. Nat. Neurosci. 14, 1209-1216. (doi:10.1038/nn.2902)
134. Rushworth, M. F. & Behrens, T. E. 2008 Choice, uncertainty and value in prefrontal and cingulate cortex. Nat. Neurosci. 11, 389-397. (doi:10.1038/ nn2066)
135. Sallet, J. & Rushworth, M. F. 2009 Should I stay or should I go: genetic bases for uncertainty-driven exploration. Nat. Neurosci. 12, 963-965. (doi:10. 1038/nn0809-963)
136. Frank, M. J., Doll, B. B., Oas-Terpstra, J. & Moreno, 2009 Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat. Neurosci. 12, 1062-1068. (doi:10. 1038/nn.2342)
137. Ernst, M. O. & Bulthoff, H. H. 2004 Merging the senses into a robust percept. Trends Cogn. Sci. 8, 162-169. (doi:10.1016/j.tics.2004.02.002)
138. Kording, K. P. & Wolpert, D. M. 2004 Bayesian integration in sensorimotor learning. Nature 427, 244-247. (doi:10.1038/nature02169)
139. Graf, E. W., Warren, P. A. & Maloney, L. T. 2005 Explicit estimation of visual uncertainty in human motion processing. Vis. Res. 45, 3050-3059. (doi:10.1016/j. visres.2005.08.007)
140. Kording, K. P. & Wolpert, D. M. 2006 Bayesian decision theory in sensorimotor control. Trends Cogn. Sci. 10, 319-326. (doi:10.1016/j.tics.2006.05.003)
141. Deneve, S. 2008 Bayesian spiking neurons I: inference. Neural Comput. 20, 91-117. (doi:10.1162/neco.2008. 20.1.91)
142. Bahrami, B., Olsen, K., Bang, D., Roepstroff, A., Rees, & Frith, C. 2012 What failure in collective decisionmaking tells us about metacognition. Phil. Trans. R. Soc. B 367, 1350-1365. (doi:10.1098/rstb.2011.0420)
143. Montague, P. R. & Berns, G. S. 2002 Neural economics and the biological substrates of valuation. Neuron 36, 265-284. (doi:10.1016/S0896-6273(02)00974-1)
144. Glimcher, P. W., Dorris, M. C. & Bayer, H. M. 2005 Physiological utility theory and the neuroeconomics of choice. Games Econ. Behav. 52, 213-256. (doi:10. 1016/j.geb.2004.06.011)
145. Sugrue, L. P., Corrado, G. S. & Newsome, W. T. 2005 Choosing the greater of two goods: neural currencies for valuation and decision making. Nat. Rev. Neurosci. 6, 363-375. (doi:10.1038/nrn1666)
146. Padoa-Schioppa, C. & Assad, J. A. 2006 Neurons in the orbitofrontal cortex encode economic value. Nature 441, 223-226. (doi:10.1038/nature04676)