Modeling human decision making in generalized gaussian multiarmed bandits

Proceedings of the IEEE

Volumn 102, Issue 4, 2014, Pages 544-571

  Reverdy, Paul B ^a   Srivastava, Vaibhav ^a   Leonard, Naomi Ehrich ^a  

^a Princeton University   (United States)

Author keywords

Adaptive control; human decision making; machine learning; multiarmed bandit

Indexed keywords

BAYESIAN NETWORKS; BEHAVIORAL RESEARCH; INFERENCE ENGINES; LEARNING SYSTEMS; PROBABILITY; STOCHASTIC SYSTEMS;

ADAPTIVE CONTROL; CORRELATION STRUCTURE; DETERMINISTIC ALGORITHMS; GENERALIZED GAUSSIAN; HUMAN DECISION MAKING; MULTI ARMED BANDIT; MULTI-ARMED BANDIT PROBLEM; OPTIMAL PERFORMANCE;

DECISION MAKING;

EID: 84897532572     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2014.2307024     Document Type: Article

References (52)

1. F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis, "Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers," IEEE Control Syst. Mag., vol. 32, no. 6, pp. 76-105, Dec. 2012.
2. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 1998.
3. R. Bellman, Dynamic Programming. Princeton, NJ, USA: Princeton Univ. Press, 1957.
4. L. P. Kaelbling, M. L. Littman, and A. W. Moore, "Reinforcement learning: A survey," J. Artif. Intell. Res., vol. 4, pp. 237-285, 1996.
5. C. J. C. H. Watkins and P. Dayan, "Q-learning," Mach. Learn., vol. 8, no. 3-4, pp. 279-292, 1992.
6. P. Auer and R. Ortner, "Logarithmic online regret bounds for undiscounted reinforcement learning," in Advances in Neural Information Processing Systems 19, B. Schölkopf, J. Platt, and T. Hoffmanb, Eds. Cambridge, MA, USA: MIT Press, 2007, pp. 49-56.
7. J. D. Cohen, S. M. McClure, and A. J. Yu, "Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration," Philosoph. Trans. Roy. Soc. B, Biol. Sci., vol. 362, no. 1481, pp. 933-942, 2007.
8. J. Gittins, K. Glazebrook, and R. Weber, Multi-Armed Bandit Allocation Indices, 2nd ed. New York, NY, USA: Wiley, 2011.
9. J. C. Gittins, "Bandit processes and dynamic allocation indices," J. Roy. Stat. Soc. B (Methodological), vol. 41, no. 2, pp. 148-177, 1979.
10. T. L. Lai and H. Robbins, "Asymptotically efficient adaptive allocation rules," Adv. Appl. Math., vol. 6, no. 1, pp. 4-22, 1985.
11. W. R. Thompson, "On the likelihood that one unknown probability exceeds another in view of the evidence of two samples," Biometrika, vol. 25, no. 3/4, pp. 285-294, 1933.
12. H. Robbins, "Some aspects of the sequential design of experiments," Bull. Amer. Math. Soc., vol. 58, pp. 527-535, 1952.
13. M. Babaioff, Y. Sharma, and A. Slivkins, "Characterizing truthful multi-armed bandit mechanisms," in Proc. 10th ACM Conf. Electron. Commerce, Stanford, CA, USA, Jul. 2009, pp. 79-88.
14. F. Radlinski, R. Kleinberg, and T. Joachims, "Learning diverse rankings with multi-armed bandits," in Proc. 25th Int. Conf. Mach. Learn., Helsinki, Finland, Jul. 2008, pp. 784-791.
15. J. L. Ny, M. Dahleh, and E. Feron, "Multi-UAV dynamic routing with partial observations using restless bandit allocation indices," in Proc. Amer. Control Conf., Seattle, Washington, USA, Jun. 2008, pp. 4220-4225.
16. B. P. McCall and J. J. McCall, "A sequential study of migration and job search," J. Labor Econ., vol. 5, no. 4, pp. 452-476, 1987.
17. M. Y. Cheung, J. Leighton, and F. S. Hover, "Autonomous mobile acoustic relay positioning as a multi-armed bandit with switching costs," in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., Tokyo, Japan, Nov. 2013, pp. 3368-3373.
18. J. R. Krebs, A. Kacelnik, and P. Taylorm, "Test of optimal sampling by foraging great tits," Nature, vol. 275, no. 5675, pp. 27-31, 1978.
19. R. Agrawal, "Sample mean based index policies with O regret for the multi-armed bandit problem," Adv. Appl. Probab., vol. 27, no. 4, pp. 1054-1078, 1995.
20. P. Auer, N. Cesa-Bianchi, and P. Fischer, "Finite-time analysis of the multiarmed bandit problem," Mach. Learn., vol. 47, no. 2, pp. 235-256, 2002.
21. S. Bubeck and N. Cesa-Bianchi, "Regret analysis of stochastic and nonstochastic multi-armed bandit problems," Mach. Learn., vol. 5, no. 1, pp. 1-122, 2012.
22. J.-Y. Audibert, R. Munos, and C. Szepesvári, "Exploration- exploitation tradeoff using variance estimates in multi-armed bandits," Theor. Comput. Sci., vol. 410, no. 19, pp. 1876-1902, 2009.
23. N. Cesa-Bianchi and P. Fischer, "Finite-time regret bounds for the multiarmed bandit problem," in Proc. 15th Int. Conf. Mach. Learn., Madison, WI, USA, Jul. 1998, pp. 100-108.
24. A. Garivier and O. Cappé, "The KL-UCB algorithm for bounded stochastic bandits and beyond," in Proc. Conf. Comput. Learn. Theory, 2011, pp. 359-376.
25. R. Dearden, N. Friedman, and S. Russell, "Bayesian Q-learning," in Proc. 15th Nat. Conf. Artif. Intell., 1998, pp. 761-768.
26. N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger, "Information- theoretic regret bounds for Gaussian process optimization in the bandit setting," IEEE Trans. Inf. Theory, vol. 58, no. 5, pp. 3250-3265, May 2012.
27. S. Agrawal and N. Goyal, "Analysis of Thompson sampling for the multi-armed bandit problem," in Proc. 25th Annu. Conf. Learn. Theory, S.Mannor, N. Srebro, and R. C. Williamson, Eds., 2012, pp. 391-3926.
28. E. Kaufmann, O. Cappé, and A. Garivier, "On Bayesian upper confidence bounds for bandit problems," in Proc. Int. Conf. Artif. Intell. Stat., La Palma, Canary Islands, Spain, Apr. 2012, pp. 592-600.
29. R. Agrawal, M. V. Hedge, and D. Teneketzis, "Asymptotically efficient adaptive allocation rules for the multi-armed bandit problem with switching cost," IEEE Trans. Autom. Control, vol. 33, no. AC-10, pp. 899-906, Oct. 1988.
30. J. S. Banks and R. K. Sundaram, "Switching costs and the Gittins index," Econometrica, J. Econometric Soc., vol. 62, no. 3, pp. 687-694, 1994.
31. M. Asawa and D. Teneketzis, "Multi-armed bandits with switching penalties," IEEE Trans. Autom. Control, vol. 41, no. 3, pp. 328-348, Mar. 1996.
32. T. Jun, "A survey on the bandit problem with switching costs," De Economist, vol. 152, no. 4, pp. 513-541, 2004.
33. R. Kleinberg, A. Niculescu-Mizil, and Y. Sharma, "Regret bounds for sleeping experts and bandits," Mach. Learn., vol. 80, no. 2-3, pp. 245-272, 2010.
34. D. Acuña and P. Schrater, "Bayesian modeling of human sequential decision-making on the multi-armed bandit problem," in Proc. 30th Annu. Conf. Cogn. Sci. Soc., B. C. Love, K. McRae, and V. M. Sloutsky, Eds., Washington, DC, USA, Jul. 2008, pp. 2065-2070.
35. D. E. Acuña and P. Schrater, "Structure learning in human sequential decision-making," PLoS Comput. Biol., vol. 6, no. 12, 2010, e1001003.
36. M. Steyvers, M. D. Lee, and E. Wagenmakers, "A Bayesian analysis of human decision-making on bandit problems," J. Math. Psychol., vol. 53, no. 3, pp. 168-179, 2009.
37. M. D. Lee, S. Zhang, M. Munro, and M. Steyvers, "Psychological models of human and optimal performance in bandit problems," Cogn. Syst. Res., vol. 12, no. 2, pp. 164-174, 2011.
38. S. Zhang and A. J. Yu, "Cheap but clever: Human active learning in a bandit setting," in Proc. 35th Annu. Conf. Cogn. Sci. Soc., Berlin, Germany, Aug. 2013, pp. 1647-1652.
39. R. C. Wilson, A. Geana, J. M. White, E. A. Ludvig, and J. D. Cohen, "Why the grass is greener on the other side: Behavioral evidence for an ambiguity bonus in human exploratory decision-making," in Proc. Neurosci. Abstr., Washington, DC, USA, Nov. 2011, Program No. 830.10.
40. D. Tomlin, A. Nedic, R. C. Wilson, P. Holmes, and J. D. Cohen, "Group foraging task reveals separable influences of individual experience and social information," in Proc. Neurosci. Abstr., New Orleans, LA, USA, Oct. 2012, Program No. 596.12.
41. P. Reverdy, R. C. Wilson, P. Holmes, and N. E. Leonard, "Towards optimization of a human-inspired heuristic for solving explore-exploit problems," in Proc. IEEE Conf. Decision Control, Maui, HI, USA, Dec. 2012, pp. 2820-2825.
42. V. Srivastava, P. Reverdy, and N. E. Leonard, "On optimal foraging and multi-armed bandits," in Proc. 51st Annu. Allerton Conf. Commun. Control Comput., Monticello, IL, USA, 2013, pp. 494-499.
43. A. Krause and C. E. Guestrin, "Near-optimal nonmyopic value of information in graphical models," in Proc. 21st Conf. Uncertainty Artif. Intell., Edinburgh, Scotland, Jul. 2005, pp. 324-331.
44. P. Fan, "New inequalities of Mill's ratio and its application to the inverse Q-function approximation," Dec. 2012. [Online]. Available: http://arxiv.org/pdf/1212.4899.pdf.
45. D. Bertsimas and J. N. Tsitsiklis, "Simulated annealing," Stat. Sci., vol. 8, no. 1, pp. 10-15, 1993.
46. D. Mitra, F. Romeo, and A. Sangiovanni-Vincentelli, "Convergence and finite-time behavior of simulated annealing," Adv. Appl. Probab., vol. 18, no. 3, pp. 747-771, 1986.
47. S. Kirkpatrick, C. D. Gelatt, Jr., and M. P. Vecchi, "Optimization by simulated annealing," Science, vol. 220, no. 4598, pp. 671-680, 1983.
48. S. M. Kay, Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory. Englewood Cliffs, NJ, USA: Prentice-Hall, 1993.
49. J. Sherman and W. J. Morrison, "Adjustment of an inverse matrix corresponding to a change in one element of a given matrix," Ann. Math. Stat., vol. 21, no. 1, pp. 124-127, 1950.
50. M. Buhrmester, T. Kwang, and S. D. Gosling, "Amazon's mechanical Turk: A new source of inexpensive, yet high-quality, data?" Perspectives Psychol. Sci., vol. 6, no. 1, pp. 3-5, 2011, DOI: 10.1177/1745691610393980.
51. N. E. Leonard, D. A. Paley, F. Lekien, R. Sepulchre, D. M. Fratantoni, and R. E. Davis, "Collective motion, sensor networks, and ocean sampling," Proc. IEEE, vol. 95, no. 1, pp. 48-74, Jan. 2007.
52. K. Friston, P. Schwartenbeck, T. Fitzgerald, M. Moutoussis, T. Behrens, and R. J. Dolan, "The anatomy of choice: Active inference and agency," Front. Human Neurosci., vol. 7, 2013, DOI: 10.3389/fnhum.2013.00598.