Parameter estimation in softmax decision-making models with linear objective functions

IEEE Transactions on Automation Science and Engineering

Volumn 13, Issue 1, 2016, Pages 54-67

  Reverdy, Paul ^a   Leonard, Naomi Ehrich ^b  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b Princeton University   (United States)

Author keywords

Automation; Decision making; Estimation

Indexed keywords

AUTOMATION; DECISION MAKING; ESTIMATION; MAXIMUM LIKELIHOOD; MAXIMUM LIKELIHOOD ESTIMATION; STOCHASTIC MODELS; STOCHASTIC SYSTEMS;

ASYMPTOTIC DISTRIBUTIONS; DECISION MAKING MODELS; HUMAN DECISION MAKING; LIKELIHOOD FUNCTIONS; LINEAR OBJECTIVE FUNCTIONS; NONLINEAR OBJECTIVE FUNCTIONS; PARAMETER ESTIMATION PROBLEMS; STATISTICALLY SIGNIFICANT DIFFERENCE;

PARAMETER ESTIMATION;

EID: 85006480147     PISSN: 15455955     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASE.2015.2499244     Document Type: Article

References (38)

1. D. Böhning, "Multinomial logistic regression algorithm," Ann. Inst. Statist. Math., vol. 44, no. 1, pp. 197-200, 1992.
2. C. G. Broyden, "The convergence of a class of double-rank minimization algorithms," IMA J. Appl. Math., vol. 6, no. 1, pp. 76-90, 1970.
3. M. Buhrmester, T. Kwang, and S. D. Gosling, "Amazon's mechanical turk: A new source of inexpensive, yet high-quality, data?," Perspectives on Psychological Sci., vol. 6, no. 1, pp. 3-5, 2011.
4. J. D. Cohen, S. M. McClure, and A. J. Yu, "Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration," Philosph. Trans. Roy. Soc. B: Bio. Sci., vol. 362, no. 1481, pp. 933-942, 2007.
5. J. A. Cooper, M. A. Gorlick, T. Denny, D. A. Worthy, C. G. Beevers, and W. T. Maddox, "Training attention improves decision making in individuals with elevated self-reported depressive symptoms," Cognitive, Affective Behavioral Neurosci., vol. 14, no. 2, pp. 729-741, June, 2014.
6. N. D. Daw, "Trial-by-trial data analysis using computational models," Decision Making, Affect, and Learning: Attention and Performance XXIII, vol. 23, pp. 3-38, 2011.
7. N. D. Daw, J. P. O'Docherty, P. Dayan, B. Seymour, and R. J. Dolan, "Cortical substrates for exploratory decisions in humans," Nature, vol. 441, no. 7095, pp. 876-879, 2006.
8. R. Fletcher, "A new approach to variable metric algorithms," The Comput. J., vol. 13, no. 3, pp. 317-322, 1970.
9. K. Gimpel and N. A. Smith, "Softmax-margin training for structured log-linear models," Carnegie Mellon Univ., Pittsburgh, PA, USA, Tech. Rep. CMU-LTI-10-008, 2010.
10. J. Gläscher, N. Daw, P. Dayan, and J. P. O'Doherty, "States versus rewards: Dissociable neural prediction error signals underlying modelbased and model-free reinforcement learning," Neuron, vol. 66, no. 4, pp. 585-595, 2010.
11. A. S. Goldberger, A Course in Econometrics. Cambridge, MA, USA: Harvard Univ. Press, 1991.
12. D. Goldfarb, "A family of variable-metric methods derived by variational means," Math. Comput., vol. 24, no. 109, pp. 23-26, 1970.
13. E. Kaufmann, O. Cappé, and A. Garivier, "On Bayesian upper confidence bounds for bandit problems," in Proc. Int. Conf. Artif. Intell. Statist., La Palma, Canary Islands, Spain, Apr. 2012, pp. 592-600.
14. S. Kay, Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory. Englewood Cliffs, NJ, USA: Prentice-Hall, 1993.
15. B. Krishnapuram, L. Carin, M. A. T. Figueiredo, and A. J. Hartemink, "Sparse multinomial logistic regression: Fast algorithms and generalization bounds," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 6, pp. 957-968, Jun. 2005.
16. B. Lau and P. W. Glimcher, "Dynamic response-by-response models of matching behavior in rhesus monkeys," J. Experimental Anal. Behavior, vol. 84, no. 3, pp. 555-579, Nov. 2005.
17. D. McFadden, "Conditional logit analysis of qualitative choice behavior," in Frontiers Econometrics, P. Zarembka, Ed. New York: Academic Press, 1974, pp. 105-142.
18. D. Mitra, F. Romeo, and A. Sangiovanni-Vincentelli, "Convergence and finite-time behavior of simulated annealing," Adv. Appl. Probability, vol. 18, no. 3, pp. 747-771, 1986.
19. P. R. Montague, B. King-Casas, and J. D. Cohen, "Imaging valuation models in human choice," Annu. Rev. Neurosci., vol. 29, pp. 417-448, 2006.
20. M. R. Nassar and J. I. Gold, "A healthy fear of the unknown: Perspectives on the interpretation of parameter fits from computational models in neuroscience," PLoS Comput. Bio., vol. 9, no. 4, 2013, e1003015.
21. A. Nedic, D. Tomlin, P. Holmes, D. A. Prentice, and J. D. Cohen, "A decision task in a social context: Human experiments, models, and analyses of behavioral data," Proc. IEEE, vol. 100, no. 3, pp. 713-733, 2012.
22. W. K. Newey and D. McFadden, "Large sample estimation and hypothesis testing," in Handbook of Econometrics, R. F. Engle and D. L. McFadden, Eds. Philadelphia, PA, USA: Elsevier, 1994, vol. 4, ch. 36, pp. 2111-2245.
23. A. Y. Ng and S. J. Russell, "Algorithms for inverse reinforcement learning," in Proc. Int. Conf. Mach. Learn., 2000, pp. 663-670.
24. V. Ramanujam and H. Balakrishnan, "Estimation of maximum-likelihood discrete-choice models of the runway configuration selection process," in Proc. Amer. Control Conf., 2011, pp. 2160-2167.
25. P. Reverdy, "Human-inspired algorithms for search: a framework for human-machine multi-armed bandit problems," Ph.D. dissertation, Dept. Mech. Aerosp. Eng., Princeton Univ., Princeton, NJ, USA, 2014.
26. P. Reverdy, V. Srivastava, and N. E. Leonard, "Modeling human decision-making in generalized Gaussian multi-armed bandits," Proc. IEEE, vol. 102, no. 4, pp. 544-571, 2014.
27. H. Robbins, "Some aspects of the sequential design of experiments," Bull. Amer. Math. Soc., vol. 58, pp. 527-535, 1952.
28. S. Russell, "Learning agents for uncertain environments," in Proc. 11th ACM Annu. Conf. Comput. Learn. Theory, 1998, pp. 101-103.
29. K. Samejima, Y. Ueda, K. Doya, and M. Kimura, "Representation of action-specific reward values in the striatum," Science, vol. 310, no. 5752, pp. 1337-1340, 2005.
30. D. F. Shanno, "Conditioning of quasi-Newton methods for function minimization," Math. Comput., vol. 24, no. 111, pp. 647-656, 1970.
31. V. Srivastava, P. Reverdy, and N. E. Leonard, "Correlated multiarmed bandit problem: Bayesian algorithms and regret analysis," arXiv:1507. 01160v2, 2015.
32. A. R. Stewart, M. Cao, A. Nedic, D. Tomlin, and N. E. Leonard, "Towards human-robot teams: Model-based analysis of human decision making in two-alternative choice tasks with social feedback," Proc. IEEE, vol. 100, no. 3, pp. 751-775, 2012.
33. R. S. Sutton and A. G. Barto, Introduction to Reinforcement Learning. Cambridge, MA, USA: MIT Press, 1998.
34. The Mathworks, Inc., Fminunc 2015. [Online]. Available: http://www. mathworks.com/help/optim/ug/fminunc.html
35. C. J. C. H. Watkins and P. Dayan, "-learning," Mach. Learn., vol. 8, no. 3-4, pp. 279-292, 1992.
36. B. L. Welch, "The generalization of "Student's" problem when several different population variances are involved," Biometrika, vol. 34, no. 1-2, pp. 28-35, 1947.
37. R. C. Wilson, A. Geana, J. M. White, E. A. Ludvig, and J. D. Cohen, "Humans use directed and random exploration to solve the exploreexploit dilemma," J. Experimental Psychology: Gen., vol. 143, no. 6, pp. 2074-2081, 2014.
38. R. C. Wilson and Y. Niv, "Is model fitting necessary for model-based fMRI?," in Proc. Multi-Disciplinary Conf. Reinforcement Learn. Decision Making, 2013, p. S41.