The empirical bayes envelope and regret minimization in competitive Markov decision processes

Mathematics of Operations Research

Volumn 28, Issue 2, 2003, Pages 327-345

  Mannor, Shie ^a   Shimkin, Nahum ^b  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

Approachability; Bayes envelope; Controlled Markov processes; Regret minimization; Stochastic games

Indexed keywords

DECISION MAKING; GAME THEORY; MATRIX ALGEBRA; STOCHASTIC PROGRAMMING;

EMPIRICAL BAYES ENVELOPE (EBE);

MARKOV PROCESSES;

EID: 0038386340     PISSN: 0364765X     EISSN: None     Source Type: Journal    
DOI: 10.1287/moor.28.2.327.14483     Document Type: Article

References (31)

1. Auer, P., N. Ceas-Bianchi, Y. Freund, R. E. Schapire. 1995. Gambling in a rigged casino: The adversarial multiarmed bandit problem. Proc. 36th Annual Sympos. on Foundations of Comput. Sci. IEEE Computer Society Press, Los Alamitos, CA, 322-331.
2. Bertsekas, D., J. Tsitsiklis. 1995. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA.
3. Blackwell, D. 1956a. An analog of the minimax theorem for vector payoffs. Pacific J. Math. 6(1) 1-8.
4. _. 1956b. Controlled random walks. Proc. Internat. Congress of Mathematicians, 1954, Vol. III. North-Holland, Amsterdam, The Netherlands, 336-338.
5. Feder, M., N. Merhav. 1998. Universal prediction. IEEE Trans. on Inform. Theory 44(6) 2124-2147.
6. Filar, J., K. Vrieze. 1996. Competitive Markov Decision Processes. Springer-Verlag, New York.
7. Flesch, J., F. Thuijsman, O. J. Vrieze. 1998. Simplifying optimal strategies in stochastic games. SIAM J. Control Optim. 36(4) 1331-1347.
8. Freund, Y., R. Schapire. 1999. Adaptive game playing using multiplicative weights. Games and Econom. Behavior 29 79-103.
9. Fudenberg, D., D. Levine. 1995. Universal consistency and cautious fictitious play. J. Econom. Dynamics and Control 19 1065-1990.
10. _, _. 1998. The Theory of Learning in Games. MIT Press, Cambridge, MA.
11. Hannan, J. 1957. Approximation to Bayes risk in repeated play. M. Dresher, A. W. Tucker, P. Wolde, eds. Contribution to the Theory of Games, III. Princeton University Press, Princeton, NJ, 97-139.
12. Hart, S., A. Mas-Colell. 2000. A simple adaptive procedure leading to correlated equilibrium. Econometrica 68 1127-1150.
13. _, _. 2001. A general class of adaptive strategies. J. Econom. Theory 98 26-54.
14. Kaelbling, L., M. Littman, A. Moore. 1996. Reinforcement learning - A survey. J. Artificial Intelligence Res. 4 237-285.
15. Kumar, P. R., P. Varaiya. 1986. Stochastic Systems: Estimation, Identification and Adaptive Control. Prentice Hall, Englewood Cliffs, NJ.
16. Lapidoth, A., P. Narayan. 1998. Reliable communication under channel uncertainty. IEEE Trans. on Inform. Theory 44 2148-2177.
17. Lehrer, E. 1998. Approachability in infinite dimensional spaces and an application: A universal algorithm for generating extended normal numbers. Preprint, May.
18. Mannor, S., N. Shimkin. 2000a. The empirical Bayes envelope approach to regret minimization in stochastic games. Technical report No. EE-1262, Faculty of Electrical Engineering, Technion, Haifa, Israel.
19. _, _. 2000b. Generalized approachability results for stochastic games with a single communicating state. Technical report No. EE-1263, Faculty of Electrical Engineering, Technion, Haifa, Israel.
20. Mertens, J., A. Neyman. 1981. Stochastic games. Internat. J. Game Theory 10(2) 53-66.
21. _, S. Sorin, S. Zamir. 1994. Repeated games. CORE Reprint Nos. Discussion Papers 9420, 9421, and 9422. Center for Operations Research and Economics, Universite Catholique de Louvain, Louvain, Belgium.
22. Milman, E. 2000. Uniform properties of stochastic games and approachability. Unpublished master's thesis, Tel Aviv University, Tel Aviv, Israel.
23. Pated, S. 1997. Stochastic shortest path games. Unpublished Ph.D., Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA.
24. Puterman, M. 1994. Markov Decision Processes. Wiley-Interscience, New York.
25. Rockafellar, R. 1970. Convex Analysis. Princeton University Press, Princeton, NJ.
26. Rustichini, A. 1999. Minimizing regret: The general case. Games and Econom. Behavior 29 224-243.
27. Shimkin, N., A. Shwartz. 1993. Guaranteed performance regions in Markovian systems with competing decision makers. IEEE Trans. on Automatic Control 38(1) 84-95.
28. Spinat, X. 1999. An approachability condition for general sets. Technical Report No. 496, Ecole Polytechnique, Paris, France.
29. Stoer, J., C. Witzgall. 1970. Convexity and Optimization in Finite Dimensions, Vol. I. Springer-Verlag, New York.
30. Vohra, R., D. K. Levine, D. Foster, eds. 1999. Special issue on learning in games. Games and Econom. Behavior 29(1); see entire issue.
31. Vovk, V. 1998. A game of prediction with expert advice. J. Comput. Systems Sci. 56(2) 153-173.