Reinforcement learning for resource allocation in LEO satellite networks

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

Volumn 37, Issue 3, 2007, Pages 515-527

  Usaha, Wipawee ^a   Barria, Javier A ^b  

^a SURANAREE UNIVERSITY OF TECHNOLOGY   (Thailand)

^b IMPERIAL COLLEGE LONDON   (United Kingdom)

Author keywords

Call admission control (CAC); Low Earth orbit (LEO) satellite network; Reinforcement learning (RL); Routing; Temporal difference (TD) learning

Indexed keywords

COMPUTATIONAL COMPLEXITY; CONGESTION CONTROL (COMMUNICATION); DECISION MAKING; DYNAMIC PROGRAMMING; MARKOV PROCESSES; RESOURCE ALLOCATION; ROUTING ALGORITHMS;

LOW EARTH ORBIT (LEO) SATELLITE NETWORKS; TEMPORAL-DIFFERENCE LEARNING;

REINFORCEMENT LEARNING;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; AUTOMATED PATTERN RECOGNITION; COMPUTER NETWORK; DECISION SUPPORT SYSTEM; METHODOLOGY; RESOURCE ALLOCATION; SIGNAL PROCESSING; SPACE FLIGHT;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER COMMUNICATION NETWORKS; DECISION SUPPORT TECHNIQUES; PATTERN RECOGNITION, AUTOMATED; RESOURCE ALLOCATION; SIGNAL PROCESSING, COMPUTER-ASSISTED; SPACECRAFT;

EID: 34249107342     PISSN: 10834419     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCB.2006.886173     Document Type: Article

References (41)

1. J. A. Boyan and M. L. Littman, "Packet routing in dynamically changing networks: A reinforcement learning approach," in Advances in Neural Information Processing Systems 6. San Mateo, CA: Morgan Kaufmann, 1994.
2. J. Carlstrom, "Reinforcement learning for admission control and routing," Ph.D. dissertation, Uppsala Univ., Uppsala, Sweden, 2000.
3. O. Ercetin, S. Krishnamurthy, S. Dao, and L. Tassiulas, "Provision of guaranteed services in broadband LEO satellite networks," Comput. Netw., vol. 39, no. 1, pp. 61-77, May 2002.
4. L. Franck and G. Maral, "Signalling for inter-satellite link routing in broadband non-GEO satellite systems," Comput. Netw., vol. 39, no. 1, pp. 79-92, May 2002.
5. E. Gelenbe, R. Lent, and A. Nunez, "Self-aware networks and QoS," Proc. IEEE, vol. 92, no. 9, pp. 1478-1489, Sep. 2004.
6. I. Gragopoulos, E. Papapetrou, and F. Pavlidou, "Performance study of adaptive routing algorithms for LEO satellite constellations under self-similar and Poisson traffic," Space Commun., vol. 16, no. 1, pp. 15-22, Jan. 2000.
7. T. C.-K. Hui and C.-K. Tham, "Adaptive provisioning of differentiated services networks based on reinforcement learning," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 33, no. 4, pp. 492-501, Nov. 2003.
8. B. W. Kim, S. L. Min, H. S. Yang, and C. S. Kim, "Predictive call admission control scheme for low Earth orbit satellite networks," IEEE Trans. Veh. Technol., vol. 49, no. 6, pp. 2320-2335, Nov. 2000.
9. V. R. Konda, "Actor-critic algorithms," Ph.D. dissertation, MIT, Cambridge, MA, 2002.
10. V. R. Konda and J. N. Tsitsiklis, "On actor-critic algorithms," SIAM J. Control Optim., vol. 42, no. 4, pp. 1143-1166, Aug. 2003.
11. P. Marbach, O. Mihatsch, and J. N. Tsitsiklis, "Call admission control and routing in integrated services networks using Neuro-dynamic programming," IEEE J. Sel. Areas Commun., vol. 18, no. 2, pp. 197-208, Feb. 2000.
12. L. Peshkin and V. Savova, "Reinforcement learning for adaptive routing," in Proc. Int. Joint Conf. Neural Netw., 2002, pp. 1825-1830.
13. S. Singh and D. P. Bertsekas, "Reinforcement learning for dynamic channel allocation in cellular telephone systems," in Advances in Neural Information Processing Systems 10. Cambridge, MA: MIT Press, 1997, pp. 974-980.
14. H. Tong and T. X. Brown, "Adaptive call admission control under quality-of-service constraints: A reinforcement learning solution," IEEE J. Sel. Areas Commun., vol. 18, no. 2, pp. 209-220, Feb. 2000.
15. N. Tao, J. Baxter, and L. Weaver, "A multi-agent, policy-gradient approach to network routing," in Proc. 18th Int. Mach. Learn. Conf. 2001, pp. 553-560.
16. G. Tesauro, "Practical issues in temporal difference learning," Mach. Learn., vol. 8, no. 3/4, pp. 257-277, May 1992.
17. P. Tam, J. Lui, H. W. Chan, C. Sze, and C. N. Sze, "An optimized routing scheme and a channel reservation strategy for a low Earth orbit satellite system," in Proc. IEEE VTC, Sep. 1999, vol. 5, pp. 2870-2874.
18. H. Uzunalioglu, I. F. Akyilidiz, and M. Bender, "A routing algorithm for connection-oriented low Earth orbit (LEO) satellite networks with dynamic connectivity," Wireless Netw., vol. 6, no. 3, pp. 181-190, May 2000.
19. W. Usaha and J. Barria, "Markov decision theory framework for resource allocation in LEO satellite constellations," Proc. Inst. Electr. Eng. - Commun., vol. 149, no. 5, pp. 270-276, Oct. 2002.
20. D. P. Bertsekas, Dynamic Programming and Optimal Control. Belmont, MA: Athena Scientific, 1995.
21. W. Usaha, "Resource allocation in networks with dynamic topology," Ph.D. dissertation, Imperial College London, London, U.K., 2004.
22. H. Uzunalioglu, "Probabilistic routing protocol for low Earth orbit satellite networks," in Proc. ICC, Jun. 1998, vol. 1, pp. 89-93.
23. M. Werner, C. Delluchi, H. Vogel, G. Maral, and J. De Ridder, "ATM-based routing in LEO/MEO satellite networks with intersatellite links," IEEE J. Sel. Areas Commun., vol. 15, no. 1, pp. 69-82, Jan. 1997.
24. M. Werner, "A dynamic routing concept for ATM-based satellite personal communication network," IEEE J. Sel. Areas Commun., vol. 15, no. 8, pp. 1636-1648, Oct. 1997.
25. M. Werner, C. Mayer, G. Maral, and M. Holzbock, "A neural network base approach to distributed adaptive routing of LEO intersatellite link traffic," in Proc. IEEE VTC, May 1998, vol. 2, pp. 1498-1502.
26. W. Zhang and T. G. Dietterich, "High performance job-shop scheduling with a time delay TD(λ) network," in Advances in Neural Information Processing Systems 8. Cambridge, MA: MIT Press, 1996, pp. 1024-1030.
27. J. Abounadi, D. P. Bertsekas, and V. S. Borkar, "Learning algorithms for Markov decision processes," Lab. Inf. Decision Syst., MIT, Cambridge, MA, Rep. LIDS-P-2434, 1998.
28. A. G. Barto, R. S. Sutton, and C. W. Anderson, "Neuronlike adaptive elements that can solve difficult learning control problems," IEEE Trans. Syst., Man, Cybern., vol. SMC-13, no. 5, pp. 834-846, Sep./Oct. 1983.
29. J. Baxter and P. L. Bartlett, "Infinite-horizon policy gradient estimation," J. Artif. Intell. Res., vol. 15, no. 4, pp. 319-350, Jul.-Dec. 2001.
30. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
31. E. Greensmith, P. L. Bartlett, and J. Baxter, "Variance reduction techniques for gradient estimates in reinforcement learning," in Advances in Neural Information Processing Systems 14. Vancouver, BC, Canada: MIT Press, 2001.
32. T. Jaakkola, S. P. Singh, and M. I. Jordan, "Reinforcement learning algorithm for partially observable Markov decision problems," in Advances in Neural Information Processing Systems 7. San Mateo, CA: Morgan Kaufmann, 1995, pp. 345-352.
33. V. R. Konda and V. S. Borkar, "Actor-critic like learning algorithms for Markov decision processes," SIAM J. Control Optim., vol. 38, no. 1, pp. 94-123, 1999.
34. S. Mahadevan, "Average reward reinforcement learning: Foundations, algorithms and empirical results," Mach. Learn., vol. 22, no. 1-3, pp. 159-196, 1996.
35. P. Marbach and J. N. Tsitsiklis, "Simulation-based optimization of Markov reward processes," IEEE Trans. Autom. Control, vol. 46, no. 2, pp. 191-209, Feb. 2001.
36. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction Cambridge, MA: MIT Press, 1998.
37. J. N. Tsitsiklis and B. Van Roy, "An analysis of temporal-difference learning with function approximation," IEEE Trans. Autom. Control vol. 42, no. 5, pp. 674-690, May 1997.
38. J. N. Tsitsiklis, "Average cost temporal-difference learning," Automatica, vol. 35, no. 11, pp. 1799-1808, Nov. 1999.
39. L. Weaver and N. Tao, "The optimal reward baseline for gradient based reinforcement learning," in Proc. 17th Conf. Uncertain. Artif. Intell., Seattle, WA, Aug. 2001, pp. 538-545.
40. R. J. William, " Simple statistical gradient-following algorithms for connectionist reinforcement learning," Mach. Learn., vol. 8, no. 3/4 pp. 229-256, May 1992.
41. W. Usaha and J. Barria, "A reinforcement learning ticket-based probing path discovery scheme for MANETs," Ad Hoc Netw., vol. 2, no. 3, pp. 319-334, 2004.