Melioration as rational choice: Sequential decision making in uncertain environments

Psychological Review

Volumn 120, Issue 1, 2013, Pages 139-154

  Sims, Chris R ^a   Neth, Hansjörg ^b   Jacobs, Robert A ^a   Gray, Wayne D ^c  

^a UNIVERSITY OF ROCHESTER   (United States)

^b MAX PLANCK INSTITUTE FOR HUMAN DEVELOPMENT   (Germany)

^c RENSSELAER POLYTECHNIC INSTITUTE   (United States)

Author keywords

Bayesian modeling; Melioration; Rational analysis; Sequential decision making

Indexed keywords

ARTICLE; BAYES THEOREM; DECISION MAKING; ENVIRONMENT; HUMAN; LEARNING; LOGIC; PHYSIOLOGY; PSYCHOLOGICAL MODEL; RISK; UNCERTAINTY;

BAYES THEOREM; CHOICE BEHAVIOR; DECISION MAKING; ENVIRONMENT; HUMANS; LEARNING; LOGIC; MODELS, PSYCHOLOGICAL; RISK; UNCERTAINTY;

EID: 84881118309     PISSN: 0033295X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0030850     Document Type: Article

References (73)

1. Acuña, D. E., & Schrater, P. (2010). Structure learning in human sequential decision-making. PLoS Computational Biology, 6(12), Article e1001003. doi:10.1371/journal.pcbi.1001003
2. Anderson, J. R. (1990). The adaptive character of thought. Hillsdale, NJ: Erlbaum.
3. Baum, W. M. (1979). Matching, undermatching, and overmatching in studies of choice. Journal of the Experimental Analysis of Behavior, 32, 269-281. doi:10.1901/jeab.1979.32-269
4. Blei, D. M., & Jordan, M. I. (2006). Variational inference for Dirichlet process mixtures. Bayesian Analysis, 1, 121-143. doi:10.1214/06-BA104
5. Camerer, C., & Ho, T.-H. (1999). Experience-weighted attraction learning in normal form games. Econometrica, 67, 827-874. doi:10.1111/1468-0262.00054
6. Chung, S.-H., & Herrnstein, R. J. (1967). Choice and delay of reinforcement. Journal of the Experimental Analysis of Behavior, 10, 67-74. doi:10.1901/jeab.1967.10-67
7. Davison, M., & McCarthy, D. (1988). The matching law: A research review. Hillsdale, NJ: Erlbaum.
8. Daw, N. D., & Frank, M. J. (2009). Reinforcement learning and higher level cognition: Introduction to special issue. Cognition, 113, 259-261. doi:10.1016/j.cognition.2009.09.005
9. Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006, June 15). Cortical substrates for exploratory decisions in humans. Nature, 441, 876-879. doi:10.1038/nature04766
10. Dawkins, R. (1999). The extended phenotype: The long reach of the gene (rev. ed.). Oxford, England: Oxford University Press.
11. Erev, I., & Barron, G. (2005). On adaptation, maximization, and reinforcement learning among cognitive strategies. Psychological Review, 112, 912-931. doi:10.1037/0033-295X.112.4.912
12. Ferguson, S. (1973). A Bayesian analysis of some nonparametric problems. Annals of Statistics, 1, 209-230. doi:10.1214/aos/1176342360
13. Friedman, D. (1988). Monty Hall's three doors: Construction and deconstruction of a choice anomaly. American Economic Review, 88, 933-946.
14. Friedman, M., & Savage, L. J. (1948). The utility analysis of choices involving risk. Journal of Political Economy, 56, 279-304. doi:10.1086/256692
15. Fu, W.-T., & Anderson, J. R. (2006). From recurrent choice to skill learning: A reinforcement-learning model. Journal of Experimental Psychology: General, 135, 184-206. doi:10.1037/0096-3445.135.2.184
16. Gallistel, C. R. (2005). Deconstructing the law of effect. Games and Economic Behavior, 52, 410-423. doi:10.1016/j.geb.2004.06.012
17. Gigerenzer, G., & Brighton, H. (2009). Homo heuristicus: Why biased minds make better inferences. Topics in Cognitive Science, 1, 107-143. doi:10.1111/j.1756-8765.2008.01006.x
18. Gigerenzer, G., Hertwig, R., & Pachur, T. (Eds.). (2011). Heuristics: The foundations of adaptive behavior. New York, NY: Oxford University Press.
19. Gilks, W. R., Richardson, S., & Spiegelhalter, D. J. (1998). Markov chain Monte Carlo in practice. London, England: Chapman & Hall.
20. Gold, J. I., & Shadlen, M. N. (2007). The neural basis of decision making. Annual Review of Neuroscience, 30, 535-574. doi:10.1146/annurev.neuro.29.051605.113038
21. Goldstein, D. G., & Gigerenzer, G. (2002). Models of ecological rationality: The recognition heuristic. Psychological Review, 109, 75-90. doi: 10.1037/0033-295X.109.1.75
22. Goldwater, S., Griffiths, T. L., & Johnson, M. (2009). A Bayesian framework for word segmentation: Exploring the effects of context. Cognition, 112, 21-54. doi:10.1016/j.cognition.2009.03.008
23. Gray, J. R. (1999). A bias toward short-term thinking in threat-related negative emotional states. Personality and Social Psychology Bulletin, 25, 65-75. doi:10.1177/0146167299025001006
24. Gray, W. D., & Fu, W. T. (2004). Soft constraints in interactive behavior: The case of ignoring perfect knowledge in-the-world for imperfect knowledge in-the-head. Cognitive Science, 28, 359-382.
25. Gureckis, T. M., & Love, B. C. (2009a). Learning in noise: Dynamic decision-making in a variable environment. Journal of Mathematical Psychology, 53, 180-193. doi:10.1016/j.jmp.2009.02.004
26. Gureckis, T. M., & Love, B. C. (2009b). Short-term gains, long-term pains: How cues about state aid learning in dynamic environments. Cognition, 113, 293-313. doi:10.1016/j.cognition.2009.03.013
27. Herrnstein, R. J. (1961). Relative and absolute strength of response as a function of frequency of reinforcement. Journal of the Experimental Analysis of Behavior, 4, 267-272. doi:10.1901/jeab.1961.4-267
28. Herrnstein, R. J. (1979). Derivatives of matching. Psychological Review, 86, 486-495. doi:10.1037/0033-295X.86.5.486
29. Herrnstein, R. J. (1981). Self-control as response strength. In C. M. Bradshaw, E. Szabadi, & C. F. Lowe (Eds.), Recent developments in the quantification of steady-state operant behavior (pp. 3-20). Amsterdam, the Netherlands: Elsevier.
30. Herrnstein, R. J. (1982). Melioration as behavioral dynamism. Quantitative Analyses of Behavior, 2, 433-458.
31. Herrnstein, R. J. (1991). Experiments on stable suboptimality in individual behavior. Learning and Adaptive Economic Behavior, 81, 360-364.
32. Herrnstein, R. J., Loewenstein, G. F., Prelec, D., & Vaughan, W., Jr. (1993). Utility maximization and melioration: Internalities in individual choice. Journal of Behavioral Decision Making, 6, 149-185. doi: 10.1002/bdm.3960060302
33. Herrnstein, R. J., & Prelec, D. (1991). Melioration: A theory of distributed choice. Journal of Economic Perspectives, 5, 137-156. doi:10.1257/jep.5.3.137
34. Herrnstein, R. J., & Prelec, D. (1992). A theory of addiction. In J. T. Elster & G. Loewenstein (Eds.), Choice over time (pp. 331-360). New York, NY: Russell Sage Foundation.
35. Herrnstein, R. J., & Vaughan, W. (1980). Melioration and behavioral allocation. In J. E. R. Staddon (Ed.), Limits to action: The allocation of individual behavior (pp. 143-175). New York, NY: Academic Press.
36. Heyman, G. M. (1996). Resolving the contradictions of addiction. Behavioral and Brain Sciences, 19, 561-610. doi:10.1017/S0140525X00042990
37. Heyman, G. M., & Dunn, B. (2002). Decision biases and persistent illicit drug use: An experimental study of distributed choice and addiction. Drug and Alcohol Dependence, 67, 193-203. doi:10.1016/S0376-8716(02)00071-6
38. Kalish, M. L., Griffiths, T. L., & Lewandowsky, S. (2007). Iterated learning: Intergenerational knowledge transmission reveals inductive biases. Psychonomic Bulletin & Review, 14, 288-294. doi:10.3758/BF03194066
39. Katsikopoulos, K. V., Schooler, L. J., & Hertwig, R. (2010). The robust beauty of ordinary information. Psychological Review, 117, 1259-1266. doi:10.1037/a0020418
40. Knight, F. (1921). Risk, uncertainty, and profit. Boston, MA: Houghton Mifflin. Kudadjie-Gyamfi, E., & Rachlin, H. (1996). Temporal patterning in choice among delayed outcomes. Organizational Behavior and Human Decision Processes, 65, 61-67. doi:10.1006/obhd.1996.0005
41. Kudadjie-Gyamfi, E., & Rachlin, H. (2002). Rule-governed versus contingency-governed behavior in a self-control task: Effects of changes in contingencies. Behavioural Processes, 57, 29-35. doi:10.1016/S0376-6357(01)00205-4
42. Lee, D., Seo, H., & Jung, M. W. (2012). Neural basis of reinforcement learning and decision making. Annual Review of Neuroscience, 35, 287-308. doi:10.1146/annurev-neuro-062111-150512
43. Marewski, J. N., & Schooler, L. J. (2011). Cognitive niches: An ecological model of strategy selection. Psychological Review, 118, 393-437. doi: 10.1037/a0024143
44. Neal, R. M. (2000). Markov chain sampling methods for Dirichlet process mixture models. Journal of Computational and Graphical Statistics, 9, 249-265.
45. Neth, H., Sims, C. R., & Gray, W. D. (2006). Melioration dominates maximization: Stable suboptimal performance despite global feedback. In R. Sun & N. Miyake (Eds.), Proceedings of the 28th Annual Meeting of the Cognitive Science Society (pp. 627-632). Hillsdale, NJ: Erlbaum.
46. Otto, A. R., Gureckis, T. M., Markman, A. B., & Love, B. C. (2009). Navigating through abstract decision spaces: Evaluating the role of state generalization in a dynamic decision-making task. Psychonomic Bulletin & Review, 16, 957-963. doi:10.3758/PBR.16.5.957
47. Otto, A. R., Markman, A. B., & Love, B. C. (2012). Taking more, now: The optimality of impulsive choice hinges on environment structure. Social Psychological and Personality Science, 3, 131-138. doi:10.1177/1948550611411311
48. Rachlin, H., & Laibson, D. I. (Eds.). (1997). The matching law: Papers on psychology and economics by Richard Herrnstein. New York, NY: Russell Sage Foundation.
49. Redish, A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007). Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling. Psychological Review, 114, 784-805. doi:10.1037/0033-295X.114.3.784
50. Rieskamp, J., & Otto, P. E. (2006). SSL: A theory of how people learn to select strategies. Journal of Experimental Psychology: General, 135, 207-236. doi:10.1037/0096-3445.135.2.207
51. Robbins, H. (1952). Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58, 527-535. doi: 10.1090/S0002-9904-1952-09620-8
52. Sakai, Y., & Fukai, T. (2008). When does reward maximization lead to matching law? PLoS ONE, 3(11), Article e3795. doi:10.1371/journal.pone.0003795
53. Sanborn, A. N., Griffiths, T. L., & Navarro, D. J. (2010). Rational approximations to rational models: Alternative algorithms for category learning. Psychological Review, 117, 1144-1167. doi:10.1037/a0020511
54. Savage, L. J. (1954). The foundations of statistics. New York, NY: Wiley. Shafir, E., & LeBoeuf, R. A. (2002). Rationality. Annual Review of Psychology, 53, 491-517. doi:10.1146/annurev.psych.53.100901.135213
55. Simon, H. A. (1955). A behavioral model of rational choice. Quarterly Journal of Economics, 69, 99-118. doi:10.2307/1884852
56. Staddon, J. E. R. (1992). Rationality, melioration, and law-of-effect models for choice. Psychological Science, 3, 136-141. doi:10.1111/j.1467-9280.1992.tb00013.x
57. Stevens, J. R., Volstorf, J., Schooler, L. J., & Rieskamp, J. (2011). Forgetting constrains the emergence of cooperative decision strategies. Frontiers in Psychology, 1, Article 235. doi:10.3389/fpsyg.2010.00235
58. Steyvers, M., Lee, M. D., & Wagenmakers, E.-J. (2009). A Bayesian analysis of human decision-making on bandit problems. Journal of Mathematical Psychology, 53, 168-179. doi:10.1016/j.jmp.2008.11.002
59. Stillwell, D. J., & Tunney, R. J. (2009). Melioration behavior in the Harvard game is reduced by simplifying decision outcomes. Quarterly Journal of Experimental Psychology, 62, 2252-2261. doi:10.1080/17470210902765999
60. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
61. Thorndike, E. L. (1911). Animal intelligence: Experimental studies. New York, NY: Macmillan. doi:10.5962/bhl.title.55072
62. Todd, P. M., Gigerenzer, G., & the ABC Research Group. (2012). Ecological rationality: Intelligence in the world. New York, NY: Oxford University Press. doi:10.1093/acprof:oso/9780195315448.001.0001
63. Tunney, R. J., & Shanks, D. R. (2002). A re-examination of melioration and rational choice. Journal of Behavioral Decision Making, 15, 291-311. doi:10.1002/bdm.415
64. Tversky, A., & Kahneman, D. (1981, January 30). The framing of decisions and the psychology of choice. Science, 211, 453-458. doi:10.1126/science.7455683
65. Vaughan, W. (1981). Melioration, matching, and maximization. Journal of the Experimental Analysis of Behavior, 36, 141-149. doi:10.1901/jeab.1981.36-141
66. Vaughan, W., & Herrnstein, R. J. (1987). Stability, melioration, and natural selection. In L. Green & J. H. Kagel (Eds.), Advances in behavioral economics (Vol. 1, pp. 185-215). Norwood, NJ: Ablex Publishing. Von Neumann, J., & Morgenstern, O. (1944). Theory of games and economic behavior. Princeton, NJ: Princeton University Press.
67. Warry, C. J., Remington, B., & Sonuga-Barke, J. S. (1999). When more means less: Factors affecting human self-control in a local versus global choice paradigm. Learning and Motivation, 30, 53-73. doi:10.1006/lmot.1998.1018
68. Wilson, J. Q., & Herrnstein, R. J. (1985). Crime and human nature. New York, NY: Simon & Schuster.
69. Wilson, R. C., & Niv, Y. (2011). Inferring relevance in a changing world. Frontiers in Human Neuroscience, 5, Article 189. doi:10.3389/fnhum.2011.00189
70. Worthy, D. A., Otto, A. R., & Maddox, W. T. (2012). Working-memory load and temporal myopia in dynamic decision making. Journal of Experimental Psychology: Learning, Memory, and Cognition, 38, 1640-1658. doi:10.1037/a0028146
71. Yakushijin, R., & Jacobs, R. A. (2011). Are people successful at learning sequences of actions on a perceptual matching task? Cognitive Science, 35, 939-962. doi:10.1111/j.1551-6709.2011.01176.x
72. Yechiam, E., & Busemeyer, J. R. (2005). Comparison of basic assumptions embedded in learning models for experience-based decision making. Psychonomic Bulletin & Review, 12, 387-402. doi:10.3758/BF03193783
73. Yechiam, E., Erev, I., Yehene, V., & Gopher, D. (2003). Melioration and the transition from touch-typing training to everyday use. Human Factors, 45, 671-684. doi:10.1518/hfes.45.4.671.27085