Thermodynamics as a theory of decision-making with information-processing costs

Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences

Volumn 469, Issue 2153, 2013, Pages

  Ortega, Pedro A ^a,b   Braun, Daniel A ^a,b  

^a MAX PLANCK INSTITUTE FOR BIOLOGICAL CYBERNETICS   (Germany)

^b MAX PLANCK INSTITUTE FOR INTELLIGENT SYSTEMS   (Germany)

Author keywords

Bounded rationality; Decision making; Information processing

Indexed keywords

COSTS; DATA PROCESSING; DECISION MAKING; EXPERIMENTS; FREE ENERGY; MACHINERY; OPTIMIZATION; PHYSICS; THERMODYNAMICS; VARIATIONAL TECHNIQUES;

BOUNDED RATIONALITY; DECISION-MAKING FRAMEWORKS; DECISION-MAKING PROBLEM; EXPECTED UTILITY THEORY; HUMAN DECISION-MAKING; THERMODYNAMIC FREE-ENERGY; THERMODYNAMIC SYSTEMS; VARIATIONAL PRINCIPLES;

BEHAVIORAL RESEARCH;

EID: 84877280273     PISSN: 13645021     EISSN: 14712946     Source Type: Journal    
DOI: 10.1098/rspa.2012.0683     Document Type: Article

References (92)

1. Gigerenzer G, Todd PM, ABC Research Group. 1999 Simple heuristics that make us smart. New York, NY: Oxford University Press.
2. Gladwell M. 2005 Blink: the power of thinking without thinking. New York, NY: Little, Brown and Company.
3. Niv Y, Daw ND, Joel D, Dayan P. 2007 Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology 191, 507-520. (doi:10.1007/s00213-006-0502-4).
4. Daw ND. 2012 Model-based reinforcement learning as cognitive search: neurocomputational theories. In Cognitive search: evolution algorithms and the brain. Boston, MA: MIT Press.
5. Von Neumann J, Morgenstern O. 1944 Theory of games and economic behavior. Princeton, NJ: Princeton University Press.
6. Savage LJ. 1954 The foundations of statistics. New York, NY:Wiley.
7. Fishburn P. 1982 The foundations of expected utility. Dordrecht, The Netherlands: D. Reidel Publishing.
8. Simon HA. 1956 Rational choice and the structure of the environment. Psychol. Rev. 63, 129-38. (doi:10.1037/h0042769).
9. Simon H. 1972 Theories of bounded rationality. In Decision and organization (eds CB Radner, R Radner), pp. 161-176. Amsterdam, The Netherlands: North-Holland.
10. Simon H. 1984 Models of bounded rationality. Cambridge, MA: MIT Press.
11. Lipman B. 1995 Information processing and bounded rationality: a survey. Canad. J. Econ. 28, 42-67. (doi:10.2307/136022).
12. Russell SJ. 1995 Rationality and intelligence. In Proc. 14th Int. Joint Conf. on Artificial Intelligence (ed. C Mellish), pp. 950-957. San Francisco, CA: Morgan Kaufmann.
13. Russell SJ, Subramanian D. 1995 Provably bounded-optimal agents. J. Artif. Intell. Res. 3, 575-609.
14. Aumann RJ. 1997 Rationality and bounded rationality. Games Econ. Behav. 21, 2-14. (doi:10.1006/game.1997.0585).
15. Rubinstein A. 1998 Modeling bounded rationality. Cambridge, MA: MIT Press.
16. Gigerenzer G, Selten R. 2001 Bounded rationality: the adaptive toolbox. Cambridge, MA: MIT Press.
17. Kahneman D. 2003 Maps of bounded rationality: psychology for behavioral economics. Am. Econ. Rev. 93, 1449-1475. (doi:10.1257/000282803322655392).
18. Spiegler R. 2011 Bounded rationality and industrial organization. Oxford, UK: Oxford University Press.
19. Camerer C. 2003 Behavioral game theory: experiments in strategic interaction. Princeton, NJ: Princeton University Press.
20. McKelvey RD, Palfrey TR. 1995 Quantal response equilibria for normal form games. Games Econ. Behav. 10, 6-38. (doi:10.1006/game.1995.1023).
21. Mckelvey R, Palfrey TR. 1998 Quantal response equilibria for extensive form games. Exp. Econ. 1, 9-41. (doi:10.1007/BF01426213).
22. Wolpert DH. 2004 Information theory-the bridge connecting bounded rational game theory and statistical physics. In Complex engineering systems (eds D Braha, Y Bar-Yam). Cambridge, MA: Perseus Books.
23. Wolpert DH, Harré M, Bertschinger N, Olbrich E, Jost J. 2012 Hysteresis effects of changing parameters of noncooperative games. Phys. Rev. E 85, 036102. (doi:10.1103/PhysRevE.85. 036102).
24. Luce RD. 2000 Utility of gains and losses: measurement-theoretical and experimental approaches. Mahwah, NJ: Erlbaum.
25. Train KE. 2009 Discrete choice methods with simulation, 2nd edn. Cambridge, UK: Cambridge University Press.
26. Luce RD. 1959 Individual choice behavior. Oxford, UK: Wiley.
27. McFadden D. 1974 Conditional logit analysis of qualitative choice behavior. In Frontiers in econometrics (ed. P Zarembka), pp. 105-142. New York, NY: Academic Press.
28. Meginnis JR. 1976 A new class of symmetric utility rules for gambles, subjective marginal probability functions, and a generalized Bayes' rule. In Proc. American Statistical Association, Business and Economic Statistics Section, pp. 471-476.
29. Fudenberg D, Kreps D. 1993 Learning mixed equilibria. Games Econ. Behav. 5, 320-367. (doi:10.1006/game.1993.1021).
30. Sutton RS, Barto AG. 1998 Reinforcement learning: an introduction. Cambridge, MA: MIT Press.
31. Callen HB. 1985 Thermodynamics and an introduction to thermostatistics. New York, NY:Wiley.
32. Ortega PA, Braun DA. 2010 A conversion between utility and information. In Proc. 3rd Conf. on Artificial General Intelligence, Lugano, Switzerland, pp. 115-120. Atlantis Press.
33. Ortega PA, Braun DA. 2011 Information, utility and bounded rationality. Lecture Notes on Artificial Intelligence, vol. 6830, pp. 269-274. Berlin, Germany: Springer.
34. Ortega PA. 2011 A unified framework for resource-bounded autonomous agents interacting with unknown environments. Department of Engineering, University of Cambridge, UK.
35. Kahneman D, Tversky A. 1979 Prospect theory: an analysis of decision under risk. Econometrica 47, 263-291. (doi:10.2307/1914185).
36. Friston K. 2009 The free-energy principle: a rough guide to the brain? Trends Cogn. Sci. 13, 293-301. (doi:10.1016/j.tics.2009.04.005).
37. Friston K. 2010 The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127-138. (doi:10.1038/nrn2787).
38. Bishop CM. 2006 Pattern recognition and machine learning. Berlin, Germany: Springer.
39. Landauer R. 1961 Irreversibility and heat generation in the computing process. IBM J. Res. Dev. 5, 183-191. (doi:10.1147/rd.53.0183).
40. Tribus M, McIrvine EC. 1971 Energy and information. Scient. Am. 225, 179-188. (doi:10.1038/scientificamerican0971-179).
41. Bennett CH. 1973 Logical reversibility of computation. IBM J. Res. Dev. 17, 525-532. (doi:10.1147/rd.176.0525).
42. Bennett CH. 1982 The thermodynamics of computation-a review. Int. J. Theoret. Phys. 21, 905-940. (doi:10.1007/BF02084158).
43. Feynman RP. 1996 The Feynman lectures on computation. Reading, MA: Addison-Wesley.
44. Gumbel EJ. 1958 Statistics of extremes. New York, NY: Columbia University Press.
45. Bogacz R, Brown E, Moehlis J, Holmes P, Cohen JD. 2006 The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychol. Rev. 113, 700-765. (doi:10.1037/0033-295X.113.4.700).
46. Ortega PA, Braun DA. 2012 Free energy and the generalized optimality equations for sequential decision making. In European Workshop for Reinforcement Learning, Edinburgh, UK.
47. Kappen HJ. 2005 A linear theory for control of non-linear stochastic systems. Phys. Rev. Lett. 95, 200201. (doi:10.1103/PhysRevLett.95.200201).
48. Todorov E. 2006 Linearly solvable Markov decision problems. In Advances in neural information processing systems, Vancouver, Canada, vol. 19, pp. 1369-1376.
49. Todorov E. 2009 Efficient computation of optimal actions. Proc. Natl Acad. Sci. USA 106, 11478- 11483. (doi:10.1073/pnas.0710743106).
50. Kappen HJ, Gómez V, Opper M. 2012 Optimal control as a graphical model inference problem. Mach. Learn. 87, 159-182. (doi:10.1007/S10994-012-5278- 7).
51. Braun DA, Ortega PA. 2011 Path integral control and bounded rationality. In IEEE Symp. on Adaptive Dynamic Programming and Reinforcement Learning, Paris, France, pp. 202-209.
52. Bellman RE. 1957 Dynamic programming. Princeton, NJ: Princeton University Press.
53. Whittle P. 1990 Risk-sensitive optimal control. New York, NY:Wiley.
54. Ba̧sar T, Bernhard P. 1991 H-infinity optimal control and related minimax-design problems: a dynamic game approach. Boston, MA: Birkhäuser.
55. Hansen LP, Sargent TJ. 2008 Robustness. Princeton, NJ: Princeton University Press.
56. Jacobson D. 1973 Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games. IEEE Trans. Automat. Control 18, 124-131. (doi:10.1109/TAC.1973.1100265).
57. Glover K, Boyle J. 1988 State-space formulae for all stabilizing controllers that satisfy an H-norm bound and relations to risk sensitivity. Syst. Control Lett. 11, 167-172. (doi:10.1016/0167-6911(88)90055-2).
58. van den Broek JL, Wiegerinck WAJJ, Kappen HJ. 2010 Risk-sensitive path integral control. In UAI, Catalina Island, CA, vol. 6, pp. 1-8.
59. Ellsberg D. 1961 Risk, ambiguity and the savage axioms. Q. J. Econ. 75, 643-669. (doi:10.2307/1884324).
60. Maccheroni F, Marinacci M, Rustichini A. 2006 Ambiguity aversion, robustness, and the variational representation of preferences. Econometrica 74, 1447-1498. (doi:10.1111/j.1468-0262.2006.00716.x).
61. Allais M. 1953 Le comportment de l'homme rationnel devant la risque: critique des postulats et axiomes de l'ecole Americaine. Econometrica 21, 503-546. (doi:10.2307/1907921).
62. Kahnemann D, Tversky A. 2002 Choices, values and frames. Cambridge, MA: Cambridge University Press.
63. Wu S, Delgado MR, Maloney LT. 2009 Economic decision-making compared with an equivalent motor task. Proc. Natl Acad. Sci. USA 106, 6088-6093. (doi:10.1073/pnas. 0900102106).
64. Tversky A, Kahneman D. 1992 Advances in prospect theory: cumulative representation of uncertainty. J. Risk Uncertain. 5, 297-323. (doi:10.1007/BF00122574).
65. Hong CS. 1983 A generalization of the quasilinear mean with application to the measurement of income inequality and decision theory resolving the Allais paradox. Econometrica 51, 1065- 1092. (doi:10.2307/1912052).
66. Nagumo M. 1930 Über eine Klasse der Mittelwerte. Japan J. Math. 7, 71-79.
67. Kolmogorov A. 1930 Sur la notion de la moyenne. Rend. Accad. Lincei 12, 388-391.
68. de Finetti B. 1931 Sul concetto di media. Giornale dell' istituto italiano degli attuari 2, 369-396.
69. Britten K, Shadlen MN, Newsome WT, Movshon JA. 1992 The analysis of visual motion: a comparison of neuronal and psychophysical performance. J. Neurosci. 12, 4745-4767.
70. Trommershauser J, Maloney LT, Landy MS. 2003 Statistical decision theory and trade-offs in the control of motor response. Spat. Vis. 16, 255-275. (doi:10.1163/156856803322467527).
71. Todorov E. 2004 Optimality principles in sensorimotor control. Nat. Neurosci. 7, 907-915. (doi:10.1038/nn1309).
72. Wolpert DM, Landy MS. 2012 Motor control is decision-making. Curr. Opin. Neurobiol. 22, 996-1003. (doi:10.1016/j.conb.2012.05.003).
73. Nagengast AJ, Braun DA, Wolpert DM. 2010 Risk-sensitive optimal feedback control accounts for sensorimotor behavior under uncertainty. PLoS Comput. Biol. 6, e1000857. (doi:10.1371/journal.pcbi.1000857).
74. Nagengast AJ, Braun DA, Wolpert DM. 2011 Risk-sensitivity and the mean-variance tradeoff: decision making in sensorimotor control. Proc. R. Soc. B 278, 2325-2332. (doi:10.1098/rspb.2010.2518).
75. Nagengast AJ, Braun DA, Wolpert DM. 2011 Risk sensitivity in a motor task with speed- accuracy trade-off. J. Neurophysiol. 105, 2668-2674. (doi:10.1152/jn.00804.2010).
76. Braun DA, Nagengast AJ, Wolpert DM. 2011 Risk-sensitivity in sensorimotor control. Front. Hum. Neurosci. 5, 1. (doi:10.3389/fnhum.2011.00001).
77. Grau-Moya J, Ortega PA, Braun DA. 2012 Risk-sensitivity in bayesian sensorimotor integration. PLoS Comput. Biol. 8, e1002698. (doi:10.1371/journal. pcbi.1002698).
78. Wolpert D, Jamison J, Newth D, Harré M. 2011 Strategic choice of preferences: the persona model. BE J. Theoret. Econ. 11, 18.
79. Peters J, Mülling K, Altun Y. 2010 Relative entropy policy search. In 24th AAAI Conf. Artificial Intelligence, Atlanta, GA.
80. Theodorou E, Buchli J, Schaal S. 2010 A generalized path integral approach to reinforcement learning. J. Mach. Learn. Res. 11, 3137-3181.
81. Rawlik K, Toussaint M, Vijayakumar S. 2012 On stochastic optimal control and reinforcement learning by approximate inference. In Proc. Robotics: Science and Systems, Sydney, Australia.
82. Saridis G. 1988 Entropy formulation for optimal and adaptive control. IEEE Trans. Automat. Control 33, 713-721. (doi:10.1109/9.1287).
83. Still S. 2009 An information-theoretic approach to interactive learning. Europhys. Lett. 85, 28005. (doi:10.1209/0295-5075/85/28005).
84. Stone LD. 1998 Theory of optimal search. New York, NY: Academic Press.
85. Jaynes ET. 1985 Entropy and search theory. In Maximum entropy and Bayesian methods in inverse problems (eds CR Smith, WT Grandy). Dordrecht, The Netherlands: D. Reidel.
86. Tishby N, Polani D. 2011 Information theory of decisions and actions. In Perception-reasonaction cycle: models, algorithms and systems (ed. VT Hussain). Berlin, Germany: Springer.
87. Vitanyi PMB. 2005 Time, space, and energy in reversible computing. In Proc. 2nd ACM Conf. on Computing Frontiers, Ischia, Italy, pp. 435-444.
88. Rieskamp J. 2008 The probabilistic nature of preferential choice. J. Exp. Psychol.: Learn. Mem. Cogn. 34, 1446-1465. (doi:10.1037/a0013646).
89. Gläscher J, Daw N, Dayan P, O'Doherty JP. 2010 States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585-595. (doi:10.1016/j.neuron. 2010.04.016).
90. McDannald MA, Takahashi YK, Lopatina N, Pietras BW, Jones JL, Schoenbaum G. 2012 Model-based learning and the contribution of the orbitofrontal cortex to the model-free world. Eur. J. Neurosci. 35, 991-996. (doi:10.1111/j.1460-9568. 2011.07982.x).
91. MacKay DJC. 2003 Information theory, inference, and learning algorithms. Cambridge, UK: Cambridge University Press.
92. Shahlizi CR. 2009 Dynamics of Bayesian updating with dependent data and misspecified models. Electron. J. Statist. 3, 1039-1074. (doi:10.1214/09-EJS485) .