Q-learning algorithms for constrained Markov decision processes with randomized monotone policies: Application to MIMO transmission control

IEEE Transactions on Signal Processing

Volumn 55, Issue 5 II, 2007, Pages 2170-2181

  Djonin, Dejan V ^a   Krishnamurthy, Vikram ^b  

^a Dyaptive Systems   (Canada)

^b UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

Constrained Markov decision process (CMDP); Delay constraints; Monotone policies; Q learning; Randomizedpolicies; Reinforcementlearning; Supermodularity; Transmission scheduling; V BLAST

Indexed keywords

CONVERGENCE OF NUMERICAL METHODS; LEARNING ALGORITHMS; MARKOV PROCESSES; OPTIMIZATION; QUALITY OF SERVICE; SCHEDULING ALGORITHMS; STOCHASTIC CONTROL SYSTEMS; TIME VARYING SYSTEMS;

CONSTRAINED MARKOV DECISION PROCESS; DELAY CONSTRAINTS; MONOTONE POLICIES; TRANSMISSION SYSTEMS;

DATA COMMUNICATION SYSTEMS;

EID: 34247898874     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSP.2007.893228     Document Type: Article

References (26)

1. D. V. Djonin and V. Krishnamurthy, "V-BLASt power and rate control under delay constraints in Markovian fading channels - Optimality of randomized monotonic policies," IEEE Trans. Signal Process., 2007, accepted for publication.
2. B. E. Collins and R. Cruz, "Transmission policies for time varying channels with average delay constraints," in Proc. Allerton Conf. Commun., Contr. Comput., Sep. 1999, pp. 709-717.
3. A. K. Karmokar, D. V. Djonin, and V. K. Bhargava, "Optimal and suboptimal packet scheduling over time-varying fading channels," IEEE Trans. Wireless Commun., vol. 5, no. 2, pp. 446-457, Feb. 2006.
4. M. J. Hossain, D. V. Djonin, and V. K. Bhargava, "Delay limited optimal and suboptimal power and bit loading algorithms for OFDM systems over correlated fading," in Proc. GLOBECOM 2005, St. Louis, 2005, pp. 3448-3453.
5. D.P. Bertsekas and J. Tsitsildis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
6. G. J. Foschini and M. J. Gans, "On limits of wireless communication in a fading environment when using multiple antennas," Wireless Pers. Commun., vol. 6, pp. 311-335, Mar. 1998.
7. G. J. Foschini, "Layered space-time architecture for wireless communication in a fading environment when using multielement antennas," Bell. Labs. Tech. J., pp. 41-59, Oct. 1996.
8. S. Chung, H. C. Howard, and A. Lozano, "Low complexity algorithm for rate quantization in extended V-BLAST," in Proc. IEEE VTC' 2001, 2001, pp. 910-914.
9. H. Zhang, L. Dai, S. Zhou, and Y. Yao, "Low complexity per-antenna rate and power control approach for closed-loop V-BLAST," IEEE Trans. Commun., vol. 51, no. 11, pp. 1783-1787, Nov. 2003.
10. V. Krishnamurthy, X. Wang, and G. Yin, "Spreading code optimization and adaptation in CDMA via discrete stochastic approximation," IEEE Trans. Inf. Theory, vol. 50, no. 9, pp. 1927-1949, Sep. 2004.
11. A. Barto, R. Sutton, and C. Anderson, "Neuron-like elements that can solve difficult learning control problems," IEEE Trans. Syst, Man, Cybern., vol. SMC-13, pp. 834-846, 1983.
12. P. Marbach and J. N. Tsitsiklis, "Simulation-based optimization of Markov reward processes," IEEE Trans. Autom. Contr., vol. 42, no. 2, pp. 191-209, Feb. 2001.
13. E. Altman, Constrained MDPes: Stochastic Modeling. London, U.K.: Chapman and Hall, 1999.
14. M. L. Putterman, Markov Decision Procsses: Discrete Stochastic Dynammic Programming. New York: Wiley, 1994.
15. D. P. Bertsekas, Dynamic Programming and Optimal Control. Belmont, MA: Athena Scientific, 1996, vol. 2.
16. S. T. Chung and A. J. Goldsmith, "Degrees of freedom in adaptive modulation: A unified view," IEEE Trans. Commun., vol. 49, no. 9, pp. 1561-1571, Sep. 2001.
17. D. M. Topkis, Supermodularity and Complementarity. Princeton, NJ: Princeton Univ. Press, 1998.
18. E. Altman, B. Gaujal, and A. Hordijk. Discrete-Event Control of Stochastic Networks: Multimodularity and Regularity. Berlin, Germany: Springer-Verlag, 2003.
19. D. P. Bertsekas, Nonlinear Programming, 2nd ed. Belmont, MA: Athena Scientific, 1999.
20. S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge Univ. Press, 2003.
21. D. G. Luenberger, Optimization by Vector Space Methods. New York: Wiley, 1969.
22. H. Kushner and D. Clark, Stochastic Approximation Methods for Constrained and Unconstrained Systems. New York: Springer-Verlag, 1978.
23. H. Kushner and G. Yin, Stochastic Approximation Algorithms and Applications, 1st ed. New York: Springer-Verlag, 1997.
24. F. J. Beutler and K. W. Ross, "Optimal policies for controlled Markov chains with a constraint," J. Math. Anal. Appl., vol. 112, pp. 236-252, 1985.
25. F. Vazquez Abad and V. Krishnamurthy, "Constrained stochastic approximation algorithms for adaptive control of constrained markov decision processes," in Proc. 42nd IEEE Conf. Decision Contr., 2003, pp. 2823-2828.
26. V. Krishnamurthy, F. Vazquez Abad, and K. Martin, "Implementation of gradient estimation to a constrained Markov decision problem," in Proc. 42nd IEEE Conf. Decision Contr., 2003, pp. 4841-4846