Semi-Markov adaptive critic heuristics with application to airline revenue management

Journal of Control Theory and Applications

Volumn 9, Issue 3, 2011, Pages 421-430

  Kulkarni, Ketaki ^a   Gosavi, Abhijit ^a   Murray, Susan ^a   Grantham, Katie ^a  

^a Architectural and Environmental Engineering   (United States)

Author keywords

Actor critics; Adaptive critics; Approximate dynamic programming; Reinforcement learning; Semi Markov

Indexed keywords

ACTOR CRITIC; ADAPTIVE CRITIC; AIRLINE INDUSTRY; AIRLINE TICKETS; APPROXIMATE DYNAMIC PROGRAMMING; AVERAGE REWARD; AVERAGE REWARD CRITERIA; CLASSICAL METHODS; CONVERGENCE ANALYSIS; CURSE OF DIMENSIONALITY; DISCOUNTED REWARD; MAINTENANCE MANAGEMENT; MARKOV DECISION PROCESSES; NUMERICAL RESULTS; OPTIMAL SOLUTIONS; REAL-WORLD PROBLEM; REVENUE MANAGEMENT; SEMI-MARKOV; SEMI-MARKOV DECISION PROCESS; TIME SPENT;

AIR TRANSPORTATION; ALGORITHMS; COMMERCE; CONVERGENCE OF NUMERICAL METHODS; HEURISTIC PROGRAMMING; MAINTENANCE; MANAGEMENT; MARKOV PROCESSES; RANDOM VARIABLES; REAL VARIABLES; REINFORCEMENT LEARNING; SUPPLY CHAIN MANAGEMENT;

DYNAMIC PROGRAMMING;

EID: 79960466561     PISSN: 16726340     EISSN: 10008152     Source Type: Journal    
DOI: 10.1007/s11768-011-0161-9     Document Type: Article

References (37)

1. R. Bellman. The theory of dynamic programming. Proceedings of the National Academy of Sciences, 1952, 38(8): 716-719.
2. P. Abbeel, A. Coates, M. Quigley, et al. An application of reinforcement learning to aerobatic helicopter fight. Advances in Neural Information Processing Systems 19, Cambridge: MIT Press, 2006: 1-8.
3. G. Tesaru. T. gammon. A self-teaching backgammon program, achieves master-level play. Neural Computation, 1994, 6(2): 215-219.
4. N. J. V. Eck, M. W. Wezel. Application of reinforcement learning to the game of othello. Computers and Operations Research, 2008, 35(6): 1999-2017.
5. R. E. Bellman. Dynamic Programming. Princeton: Princeton University Press, 1957.
6. R. Howard. Dynamic Programming and Markov Processes. Cambridge: MIT Press, 1960.
7. D. P. Bertsekas, J. N. Tsitsiklis. Neuro-dynamic Programming. Belmont: Athena Scientific, 1996.
8. R. Sutton, A. G. Barto. Reinforcement Learning: An Introduction. Cambridge: MIT Press, 1998.
9. A. Gosavi. Simulation-based Optimization: Parametric Optimization Techniques and Reinforcement Learning. Boston: Kluwer Academic, 2003.
10. A. Gosavi. Adaptive critics for airline revenue management. Proceedings of the Production and Operations Management Society, Dallas, TX, 2007: 7-58.
11. S. Ross. Applied Probability Models with Optimization Applications. New York: Dover Publications, 1992.
12. D. P. Bertsekas. Dynamic Programming and Optimal Control. 2nd ed. Belmont: Athena Scientific, 2000.
13. P. J. Werbos. Building and understanding adaptive systems: a statistical/numerical approach to factory automation and brain research. IEEE Transactions on Systems, Man, and Cybernetics, 1987, 17(1): 7-20.
14. G. Venayagamoorthy, R. Harley, D. Wunsch. Comparison of heuristic dynamic programming and dual heuristic programming adaptive critics for neuro-control of a turbogenerator. IEEE Transactions on Neural Networks, 2002, 13(3): 764-773.
15. A. G. Barto, R. S. Sutton, C. W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. Artificial Neural Networks, Piscataway: New York, 1990: 81-93.
16. V. R. Konda, V. S. Borkar. Actor-critic type learning algorithms for Markov decision processes. SIAM Journal on Control and Optimization, 1999, 38(1): 94-123.
17. C. J. Watkins. Learning from Delayed Rewards. Ph. D. thesis. Cambridge, U. K.: Kings College, 1989.
18. G. A. Rummery, M. Niranjan. On-line Q-learning Using Connectionist Systems. Report CUED/F-INFENG/TR 166. Cambridge: Engineering Department, Cambridge University, 1994.
19. F. L. Lewis, D. Vrabie. Reinforcement learning and adaptive dynamic programming for feedback control. IEEE Circuits and Systems Magazine, 2009, 9(3): 32-50.
20. S. N. Balakrishnan, J. Ding, F. L. Lewis. Issues on stability of adp feedback controllers for dynamical systems. IEEE Transactions on Systems, Man and Cybernetics (Special Issue on ADP/RL), 2008, 38(4): 913-917.
21. C. Szepesvari. Algorithms for Reinforcement Learning (Synthesis Lectures on Artificial Intelligence and Machine Learning). San Rafael, CA: Morgan & Claypool Publishers, 2010.
22. A. Gosavi. Reinforcement learning: a tutorial survey and recent advances. INFORMS Journal on Computing, 2009, 21(2): 178-192.
23. T. Dierks, S. Jagannathan. Optimal control of ane nonlinear discretetime systems. Proceedings of the IEEE Mediterranean Conference on Control and Automation, New York: IEEE, 2009: 1390-1395.
24. D. Liu, X. Xiong, Y. Zhang. Action-dependent adaptive critic designs. Proceedings of INNS-IEEE International Joint Conference Neural Networks, New York: IEEE, 2001: 990-995.
25. J. Baxter, P. Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence, 2001, 15(1): 319-350.
26. S. Singh, V. Tadic, A. Doucet. A policy-gradient method for semi-Markov decision processes with application to call admission control. European Journal of Operational Research, 2007, 178(3): 808-818.
27. H. J. Kushner, D. S. Clark. Stochastic Approximation Methods for Constrained and Unconstrained Systems. New York: Springer-Verlag, 1978.
28. V. S. Borkar. Stochastic approximation with two-time scales. Systems & Control Letters, 1997, 29(5): 291-294.
29. J. Abounadi, D. P. Bertsekas, V. Borkar. Learning algorithms for Markov decision processes with average cost. SIAM Journal of Control and Optimization, 2001, 40(3): 681-698.
30. K. Littlewood. Forecasting and control of passenger bookings. Journal of Revenue and Pricing Management, 2005, 4(2): 111-123.
31. P. P. Belobaba. Application of a probabilistic decision model to airline seat inventory control. Operations Research, 1989, 37(2): 183-197.
32. J. I. McGill, G. J. van Ryzin. Revenue management: Research overview and prospects. Transportation Science, 1999, 33(2): 233-256.
33. W. C. Chiang, J. C. H. Chen, X. Xu. An overview of research on revenue management: current issues and future research. International Journal of Revenue Management, 2007, 1(1): 97-128.
34. K. Talluri, G. van Ryzin. The Theory and Practice of Revenue Management. Boston: Kluwer Academic, 2004.
35. R. Phillips. Pricing and Revenue Optimization. Stanford: Stanford Business Books, 2005.
36. A. Gosavi, N. Bandla, T. K. Das. A reinforcement learning approach to a single leg airline revenue management problem with multiple fare classes and overbooking. IIE Transactions, 2002, 34(9): 729-742.
37. A. Gosavi. A reinforcement learning algorithm based on policy iteration for average reward: empirical results with yield management and convergence analysis. Machine Learning, 2004, 55(1): 5-29.