Heuristically accelerated reinforcement learning for dynamic secondary spectrum sharing

IEEE Access

Volumn 3, Issue , 2015, Pages 2771-2783

  Morozs, Nils ^a   Clarke, Tim ^a   Grace, David ^a  

^a UNIVERSITY OF YORK   (United Kingdom)

Author keywords

Dynamic spectrum access; Heuristically accelerated reinforcement learning; Spectrum sharing

Indexed keywords

CELLULAR RADIO SYSTEMS; CELLULAR TELEPHONE SYSTEMS; COGNITIVE RADIO; COGNITIVE SYSTEMS; HEURISTIC ALGORITHMS; INFORMATION MANAGEMENT; INTERFERENCE SUPPRESSION; LONG TERM EVOLUTION (LTE); MOBILE TELECOMMUNICATION SYSTEMS; QUALITY OF SERVICE; TELECOMMUNICATION NETWORKS; WIRELESS TELECOMMUNICATION SYSTEMS;

DYNAMIC SPECTRUM ACCESS; EXTERNAL INFORMATIONS; REINFORCEMENT LEARNING SOLUTION; SECONDARY SPECTRUMS; SPECTRUM MANAGEMENT; SPECTRUM SHARING; SYSTEM LEVEL SIMULATION; WIRELESS COMMUNICATIONS;

REINFORCEMENT LEARNING;

EID: 84961093604     PISSN: None     EISSN: 21693536     Source Type: Journal    
DOI: 10.1109/ACCESS.2015.2507158     Document Type: Article

References (36)

1. H. Sun, A. Nallanathan, C.-X. Wang, and Y. Chen, "Wideband spectrum sensing for cognitive radio networks: A survey," IEEE Wireless Commun., vol. 20, no. 2, pp. 74-81, Apr. 2013.
2. M. Guizani, B. Khalfi, M. Ben Ghorbel, and B. Hamdaoui, "Largescale cognitive cellular systems: Resource management overview," IEEE Commun. Mag., vol. 53, no. 5, pp. 44-51, May 2015.
3. C. Ghosh, S. Roy, and D. Cavalcanti, "Coexistence challenges for heterogeneous cognitive wireless networks in TV white spaces," IEEE Wireless Commun., vol. 18, no. 4, pp. 22-31, Aug. 2011.
4. D. Gurney, G. Buchwald, L. Ecklund, S. L. Kuffner, and J. Grosspietsch, "Geo-location database techniques for incumbent protection in the TV white space," in Proc. IEEE Symp. New Frontiers Dyn. Spectr. Access Netw. (DySPAN), Oct. 2008, pp. 1-9.
5. M. Matinmikko, H. Okkonen, M. Palola, S. Yrjola, P. Ahokangas, and M. Mustonen, "Spectrum sharing using licensed shared access: The concept and its workflow for LTE-advanced networks," IEEE Wireless Commun., vol. 21, no. 2, pp. 72-79, Apr. 2014.
6. S. Hamouda, M. Zitoun, and S. Tabbane, "Winfiwin relationship between macrocell and femtocells for spectrum sharing in LTE-A," IET Commun., vol. 8, no. 7, pp. 1109-1116, May 2014.
7. G. Alnwaimi, T. Zahir, S. Vahid, and K. Moessner, "Machine learning based knowledge acquisition on spectrum usage for LTE femtocells," in Proc. IEEE Veh. Technol. Conf. (VTC-Fall), Sep. 2013, pp. 1-6.
8. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 1998.
9. T. Jiang, D. Grace, and P. D. Mitchell, "Efficient exploration in reinforcement learning-based cognitive radio spectrum sharing," IET Commun., vol. 5, no. 10, pp. 1309-1317, 2011.
10. M. Bennis, S. M. Perlaza, P. Blasco, Z. Han, and H. V. Poor, "Selforganization in small cell networks: A reinforcement learning approach," IEEE Trans. Wireless Commun., vol. 12, no. 7, pp. 3202-3212, Jul. 2013.
11. N. Morozs, T. Clarke, and D. Grace, "Distributed heuristically accelerated Q-learning for robust cognitive spectrum management in LTE cellular systems," IEEE Trans. Mobile Comput., to be published.
12. X. Chen, Z. Zhao, and H. Zhang, "Stochastic power adaptation with multiagent reinforcement learning for cognitive wireless mesh networks," IEEE Trans. Mobile Comput., vol. 12, no. 11, pp. 2155-2166, Nov. 2013.
13. C. Watkins, "Learning from delayed rewards," Ph.D. dissertation, King's College Univ. Cambridge, Cambridge, U.K.,1989.
14. J. Nie and S. Haykin, "A Q-learning-based dynamic channel assignment technique for mobile communication systems," IEEE Trans. Veh. Technol., vol. 48, no. 5, pp. 1676-1687, Sep. 1999.
15. R. Valcarce et al., "Airborne base stations for emergency and temporary events," in Proc. Int. Conf. Pers. Satellite Services, 2013, pp. 13-25.
16. R. A. C. Bianchi, M. F. Martins, C. H. C. Ribeiro, and A. H. R. Costa, "Heuristically-accelerated multiagent reinforcement learning," IEEE Trans. Cybern., vol. 44, no. 2, pp. 252-265, Feb. 2014.
17. R. A. C. Bianchi and R. L. de Màntaras, "Case-based multiagent reinforcement learning: Cases as heuristics for selection of actions," in Proc. Eur. Conf. Artif. Intell. (ECAI), 2010, pp. 355-360.
18. J. Lunden, S. R. Kulkarni, V. Koivunen, and H. V. Poor, "Multiagent reinforcement learning based spectrum sensing policies for cognitive radio networks," IEEE J. Sel. Topics Signal Process., vol. 7, no. 5, pp. 858-868, Oct. 2013.
19. S. Chen, R. Vuyyuru, O. Altintas, and A. M. Wyglinski, "On optimizing vehicular dynamic spectrum access networks: Automation and learning in mobile wireless environments," in Proc. IEEE Veh. Netw. Conf. (VNC), Nov. 2011, pp. 39-46.
20. L. Reynaud et al., "FP7-ICT-2011-8-318632-ABSOLUTE/D2.1 use cases definition and scenarios description," 2014.
21. N. Morozs, D. Grace, and T. Clarke, "Distributed Q-learning based dynamic spectrum access in high capacity density cognitive cellular systems using secondary LTE spectrum sharing," in Proc. Int. Symp. Wireless Pers. Multimedia Commun. (WPMC), Sep. 2014, pp. 462-467.
22. C. Claus and C. Boutilier, "The dynamics of reinforcement learning in cooperative multiagent systems," in Proc. 15th Nat./10th Conf. Artif. Intell./Innov. Appl. Artif. Intell., 1998, pp. 746-752.
23. N. Morozs, T. Clarke, and D. Grace, "A novel adaptive call admission control scheme for distributed reinforcement learning based dynamic spectrum access in cellular networks," in Proc. Int. Symp. Wireless Commun. Syst. (ISWCS), Aug. 2013, pp. 1-5.
24. N. Morozs, T. Clarke, D. Grace, and Q. Zhao, "Distributed Q-learning based dynamic spectrum management in cognitive cellular systems: Choosing the right learning rate," in Proc. IEEE Int. Symp. Comput. Commun. (ISCC), Jun. 2014, pp. 1-6.
25. M. Bowling and M. Veloso, "Multiagent learning using a variable learning rate," Artif. Intell., vol. 136, no. 2, pp. 215-250, Apr. 2002.
26. LTE; Evolved Universal Terrestrial Radio Access (E-UTRA); Physical Layer Procedures (3GPP TS 36.213 Version 11.5.0 Release 11), 3GPP, Dec. 2013.
27. S. Sesia, M. Baker, and I. Toufik, LTE-The UMTS Long Term Evolution: From Theory to Practice. New York, NY, USA: Wiley, 2011.
28. R. K. McLean et al., "An architecture for coexistence with multiple users in frequency hopping cognitive radio networks," IEEE J. Sel. Areas Commun., vol. 32, no. 3, pp. 563-571, Mar. 2014.
29. R. O. Schmidt, "Multiple emitter location and signal parameter estimation," IEEE Trans. Antennas Propag., vol. 34, no. 3, pp. 276-280, Mar. 1986.
30. G. Ross, N. M. Adams, D. K. Tasoulis, and D. J. Hand, "Exponentially weighted moving average charts for detecting concept drift," Pattern Recognit. Lett., vol. 33, pp. 191-198, Jan. 2012.
31. A. Ghasemi and E. S. Sousa, "Spectrum sensing in cognitive radio networks: Requirements, challenges and design trade-offs," IEEE Commun. Mag., vol. 46, no. 4, pp. 32-39, Apr. 2008.
32. P. Kyösti et al., "IST-4-027756 WINNER II deliverable D1.1.2: WINNER II channel models," 2008.
33. Technical Specification Group Radio Access Network; Evolved Universal Terrestrial Radio Access (E-UTRA); Further Advancements for E-UTRA Physical Layer Aspects (3GPP TR 36.814 Version 9.0.0 Release 9), 3GPP, Dec. 2010.
34. LTE; Evolved Universal Terrestrial Radio Access (E-UTRA); Medium Access Control (MAC) Protocol Specification (3GPP TS 36.321 Version 11.4.0 Release 11), 3GPP, Jan. 2014.
35. LTE; Evolved Universal Terrestrial Radio Access (E-UTRA); Radio Frequency (RF) System Scenarios (3GPP TR 36.952 Version 11.0.0 Release 11), 3GPP, Dec. 2012.
36. R. McGill, J.W. Tukey, andW. A. Larsen, "Variations of box plots," Amer. Statist., vol. 32, no. 1, pp. 12-16, 1978.