Design and optimisation of a (FA)Q-learning-based HTTP adaptive streaming client

Connection Science

Volumn 26, Issue 1, 2014, Pages 25-43

  Claeys, Maxim ^a   Latré, Steven ^b   Famaey, Jeroen ^a   Wu, Tingyao ^c   Van Leekwijck, Werner ^c   De Turck, Filip ^a  

^a GHENT UNIVERSITY   (Belgium)

^b UNIVERSITY OF ANTWERP   (Belgium)

^c LUCENT TECHNOLOGIES   (United States)

Author keywords

agent systems; HTTP adaptive streaming; quality of experience; reinforcement learning

Indexed keywords

ALGORITHMS; OPTIMIZATION; QUALITY OF SERVICE; REINFORCEMENT LEARNING;

ADAPTIVE STREAMING; ADAPTIVE VIDEO STREAMING; AGENT SYSTEMS; DETERMINISTIC ALGORITHMS; DETERMINISTIC HEURISTICS; NETWORK CONFIGURATION; NETWORK ENVIRONMENTS; QUALITY OF EXPERIENCE (QOE);

HTTP;

EID: 84896133978     PISSN: 09540091     EISSN: 13600494     Source Type: Journal    
DOI: 10.1080/09540091.2014.885273     Document Type: Article

References (33)

1. Adzic, V., Kalva, H., & Furht, B. (2011). Optimized adaptive HTTP streaming for mobile devices. Applications of Digital Image Processing, XXXIV, 81350T.
2. Akhshabi, S., Begen, A. C., & Dovrolis, C. (2011). An experimental evaluation of rate-adaptation algorithms in adaptive streaming over HTTP. In A. C. Begen & K. Mayer-Patel (Eds.), Proceedings of the second annual ACM conference on multimedia systems (pp. 157-168). New York, NY: ACM.
3. Bagnasco, R., & Serrat, J. (2009). Multi-agent reinforcement learning in network management. In R. Sadre & A. Pras (Eds.), Scalability of networks and services (Vol. 5637, pp. 199-202). Berlin: Springer.
4. Bouten, N., Famaey, J., Latré, S., Huysegems, R., De Vleeschauwer, B., & Van Leekwijck, W. (2012). QoE optimization through in-network quality adaptation for HTTP adaptive streaming. In D. Mehdi (Ed.), 2012 8th international conference on network and service management (CNSM) (pp. 336-342). Washington, DC: IEEE.
5. Cao, J. (2011). Using reinforcement learning for agent-based network fault diagnosis system. In M.Q.-H. Max (Ed.), 2011 IEEE international conference on information and automation (ICIA) (pp. 750-754). Washington, DC: IEEE.
6. Charvillat, V., & Grigoras, R. (2007). Reinforcement learning for dynamic multimedia adaptation. Journal of Network and Computer Applications, 30(3), 1034-1058. doi: 10.1016/j.jnca.2005.12.010
7. Claeys, M., Latré, S., Famaey, J., Wu, T., Van Leekwijck, W., & De Turck, F. (2013). Design of a Q-learning-based client quality selection algorithm for HTTP adaptive video streaming. In M. Gini & O. Shehory (Eds.), 2013 workshop on adaptive and learning agents (ALA) (pp. 1-8). Sydney: Curran Associates.
8. De Vriendt, J., De Vleeschauwer, D., & Robinson, D. (2013). Model for estimating QoE of video delivered using HTTP adaptive streaming. In A. Clemm & P. Demeester (Eds.), 1st IFIP/IEEE workshop on QoE centric management, QCMAN 2013 (pp. 1288-1293). Washington, DC: IEEE.
9. Famaey, J., Latré, S., Bouten, N., Meerssche, W. Van de, De Vleeschauwer, B., & Van Leekwijck, W. (2013). On the merits of SVC-based HTTP adaptive streaming. In A. Clemm & P. Demeester (Eds.), 2013 12th IFIP/IEEE international symposium on integrated network management (IM) (pp. 1-8). Washington, DC: IEEE.
10. Fei, Y., Wong, V. W. S., & Leung, V. C. M. (2006). Efficient QoS provisioning for adaptive multimedia in mobile communication networks by reinforcement learning. Mobile Networks and Applications, 11(1), pp. 101-110. doi: 10.1007/s11036-005-4464-2
11. Jarnikov, D., & Özçelebi, T. (2010). Client intelligence for adaptive streaming solutions. In 2010 IEEE international conference on multimedia and expo (ICME) (pp. 1499-1504). Amsterdam, The Netherlands: Elsevier.
12. Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, pp. 237-285.
13. Kaisers, M., & Tuyls, K. (2010). Frequency adjusted multi-agent Q-learning. In M. Luck & S. Sen (Eds.), Proceedings of the 9th international conference on autonomous agents and multiagent systems: Volume 1 - Volume 1 (pp. 309-316). Sydney: Curran Associates.
14. Lee, M., Marconett, D., Ye, X., & Yoo, S. (2007). Cognitive network management with reinforcement learning for wireless mesh networks. In D. Medhi, J. Marcos Nogueira, T. Pfeifer, S. Felix Wu (Eds.), IP operations and management (Vol. 4786, pp. 168-179). Berlin: Springer.
15. Liu, C., Bouazizi, I., & Gabbouj, M. (2011). Parallel adaptive HTTP media streaming. In H. Wang (Ed.), Proceedings of the 20th international conference on computer communications and networks (ICCCN) (pp. 1-6). Washington, DC: IEEE.
16. Lohmar, T., Einarsson, T., Frojdh, P., Gabin, F., & Kampmann, M. (2011). Dynamic adaptive HTTP streaming of live content. In M. Gerla & E. Mingozzi (Eds.), 2011 IEEE international symposium on a world of wireless, mobile and multimedia networks (WoWMoM) (pp. 1-8). Washington, DC: IEEE.
17. Mastronarde, N., & van der Schaar, M. (2011). Fast reinforcement learning for energy-efficient wireless communication. IEEE Transactions on Signal Processing, 59(12), 6262-6266. doi: 10.1109/TSP.2011.2165211
18. McClary, D. W., Syrotiuk, V. R., & Lecuire, V. (2008). Adaptive audio streaming in mobile ad hoc networks using neural networks. Ad Hoc Networks, 6(4), 524-538. doi: 10.1016/j.adhoc.2007.04.005
19. Menkovski, V., & Liotta, A. (2013). Intelligent control for adaptive video streaming. In P. Friedrich (Ed.), 2013 IEEE international conference on consumer electronics (ICCE) (pp. 127-128). Washington, DC: IEEE.
20. Mok, R., Chan, E., & Chang, R. (2011). Measuring the quality of experience of HTTP video streaming. In V. Wade & S. Samuel (Eds.), 2011 IFIP/IEEE international symposium on integrated network management (IM) (pp. 485-492). Washington, DC: IEEE.
21. Niyato, D., & Hossain, E. (2006). A radio resource management framework for IEEE 802.16-based OFDM/TDD wireless mesh networks. In L. Yenel (Ed.), IEEE international conference on communications, 2006. ICC '06 (Vol. 9, pp. 3911-3916). Washington, DC: IEEE.
22. Ouferhat, N., & Mellouk, A. (2007). A QoS scheduler packets for wireless sensor networks. In M. S. Obaidat (Ed.), IEEE/ACS international conference on computer systems and applications, 2007. AICCSA '07 (pp. 211-216). Washington, DC: IEEE.
23. Parakh, S., & Jagannatham, A. (2012). Game theory based dynamic bit-rate adaptation for H.264 scalable video transmission in 4G wireless systems. In V. Sharma, L. Tong& A. Chockalingam (Eds.), 2012 international conference on signal processing and communications (SPCOM) (pp. 1-5). Washington, DC: IEEE.
24. Riiser, H., Vigmostad, P., Griwodz, C., & Halvorsen, P. (2011). Bitrate and video quality planning for mobile streaming scenarios using a GPS-based bandwidth lookup service. In I. Cheng, G. Fernandez, & H. Wang (Eds.), 2011 IEEE international conference on multimedia and expo (ICME) (pp. 1-6). Washington, DC: IEEE.
25. Stockhammer, T. (2011). Dynamic adaptive streaming over HTTP: Standards and design principles. In K. Mayer-Patel & A. C. Begen (Eds.), Proceedings of the second annual ACM conference on multimedia systems (pp. 133-144). New York, NY: ACM.
26. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (adaptive computation and machine learning). Hardcover: The MIT Press.
27. Tesauro, G., Jong, N. K., Das, R., & Bennani, M. N. (2007). On the use of hybrid reinforcement learning for autonomic resource allocation. Cluster Computing, 10(3), 287-299. doi: 10.1007/s10586-007-0035-6
28. Tokic, M. (2010). Adaptive ε-greedy exploration in reinforcement learning based on value differences. In R. Dillmann, J. Beyerer, U. D. Hanebeck & T. Schultz (Eds.), Proceedings of the 33rd annual German conference on advances in artificial intelligence (pp. 203-210). Berlin: Springer-Verlag.
29. Tokic, M., & Palm, G. (2011). Value-difference based exploration: Adaptive control between epsilon-greedy and softmax. In J. Bach & S. Edelkamp (Eds.), Proceedings of the 34th annual German conference on advances in artificial intelligence (pp. 335-346). Berlin: Springer-Verlag.
30. Trestian, R., Moldovan, A.-N., Ormond, O., & Muntean, G.-M. (2012). Energy consumption analysis of video streaming to android mobile devices. In D. Zuckerman & T. Moore (Eds.), Network operations and management symposium (NOMS), 2012 IEEE (pp. 444-452). Washington, DC: IEEE.
31. Vengerov, D. (2007). A reinforcement learning approach to dynamic resource allocation. Engineering Applications of Artificial Intelligence, 20(3), 383-390. doi: 10.1016/j.engappai.2006.06.019
32. Watkins, C. (1989). Learning from delayed rewards (Unpublished doctoral dissertation). University of Cambridge, England.
33. Xiang, S., Cai, L., & Pan, J. (2012). Adaptive scalable video streaming in wireless networks. In M. Claypool (Ed.), Proceedings of the 3rd multimedia systems conference (pp. 167-172). New York, NY: ACM.