Autonomic multi-policy optimization in pervasive systems: Overview and evaluation

ACM Transactions on Autonomous and Adaptive Systems

Volumn 7, Issue 1, 2012, Pages

  Dusparic, Ivana ^a   Cahill, Vinny ^a  

^a University of Limerick   (Ireland)

Author keywords

Autonomic computing; Decentralized systems; Reinforcement learning

Indexed keywords

AGENT-BASED OPTIMIZATION; AUTONOMIC COMPUTING; COLLABORATION BETWEEN AGENTS; DECENTRALIZED SYSTEM; LARGE-SCALE PERVASIVE SYSTEMS; LEARNING-BASED ALGORITHMS; PERVASIVE SYSTEMS; TRAFFIC LOADS; URBAN TRAFFIC CONTROL;

ADAPTIVE CONTROL SYSTEMS; REINFORCEMENT LEARNING;

OPTIMIZATION;

EID: 84862074365     PISSN: 15564665     EISSN: 15564703     Source Type: Journal    
DOI: 10.1145/2168260.2168271     Document Type: Article

References (38)

1. ABDULHAI, B., PRINGLE, R., AND KARAKOULAS, G. 2003. Reinforcement learning for the true adaptive traffic signal control. J. Trans. Engin. 129, 3, 278-285. (Pubitemid 36594889)
2. BAZZAN, A. L. 2005. A distributed approach for coordination of traffic signal agents. Auton. Agents Multi-Agent Syst. 10, 1, 131-164.
3. BERNSTEIN, D. S., ZILBERSTEIN, S., AND IMMERMAN, N. 2000. The complexity of decentralized control of markov decision processes. In Mathematics of Operations Research.
4. CUAYAHUITL, H., RENALS, S., LEMON, O., AND SHIMODAIRA, H. 2006. Learning multi-goal dialogue strategies using reinforcement learning with reduced state-action spaces. Int. J. Game Theory, 547-565.
5. DA SILVA, B. C., BASSO, E. W., BAZZAN, A. L. C., AND ENGEL, P. M. 2006. Dealing with non-stationary environments using context detection. In Proceedings of the 23rd International Conference on Machine Learning (ICML'06). ACM, New York, 217-224.
6. DOWLING, J. 2005. The decentralised coordination of self-adaptive components for autonomic distributed systems. Ph.D. thesis, Trinity College Dublin.
7. DOWLING, J., CUNNINGHAM, R., CURRAN, E., AND CAHILL, V. 2006. Building autonomic systems using collaborative reinforcement learning. Knowl. Engin. Rev. 21, 3, 231-238. (Pubitemid 44610665)
8. DUSPARIC, I. AND CAHILL, V. 2009a. Distributed W-Learning: Multi-Policy optimization in self-organizing systems. In 3rd IEEE International Conference on Self-Adaptive and Self-Organizing Systems.
9. DUSPARIC, I. AND CAHILL, V. 2009b. Using reinforcement learning for multi-policy optimization in decentralized autonomic systems - An experimental evaluation. In Proceedings of the 6th International Conference on Autonomic and Trusted Computing, W. Reif, G. Wang, and J. Indulska, Eds. Lecture Notes in Computer Science, vol. 5586. Springer, 105-119.
10. FEBBRARO, A. D., GIGLIO, D., AND SACCO, N. 2004. Urban traffic control structure based on hybrid petri nets. IEEE Trans. Intell. Trans. Syst. 5, 4, 224-237.
11. GUESTRIN, C., LAGOUDAKIS, M., AND PARR, R. 2002. Coordinated reinforcement learning. In Proceedings of the ICML-2002 the 19th International Conference on Machine Learning. 227-234.
12. HOAR, R., PENNER, J., AND JACOB, C. 2002. Evolutionary swarm traffic: If ant roads had traffic lights. In (CEC'02) Proceedings of the Evolutionary Computation (CEC '02). Proceedings of the 2002 Congress. IEEE Computer Society, Washington, DC, 1910-1915.
13. HUMPHRYS, M. 1996a. Action selection methods using reinforcement learning. In Proceedings of the 4th International Conference on Simulation of Adaptive Behavior. MIT Press, 135-144.
14. HUMPHRYS, M. 1996b. Action selection methods using reinforcement learning. Ph.D. thesis, University of Cambridge.
15. KALYANAKRISHNAN, S. AND STONE, P. 2007. Batch reinforcement learning in a complex domain. In Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems. ACM, New York, 650-657.
16. KEPHART, J. O. AND CHESS, D. M. 2003. The vision of autonomic computing. Comput. 36, 1, 41-50.
17. KOK, J. R., 'T HOEN, P. J., BAKKER, B., AND VLASSIS, N. 2005. Utile coordination: Learning interdependencies among cooperative agents. In Proceedings of the IEEE Symposium on Computational Intelligence and Games (CIG). 29-36.
18. LITTMAN, M. L., RAVI, N., FENSON, E., AND HOWARD, R. 2004. Reinforcement learning for autonomic network repair. In Proceedings of the 1st International Conference on Autonomic Computing (ICAC'04). IEEE Computer Society, Washington, DC, 284-285.
19. MELO, F. AND VELOSO, M. 2009. Learning of coordination: Exploiting sparse interactions in multiagent systems. In Proceedings of the 8th International Conference on Autonomous Agents and Multi-Agent Systems.
20. OLIVEIRA, E. AND DUARTE, N. 2005. Making way for emergency vehicles. In Proceedings of the European Simulation and Modelling Conference. 128-135.
21. PEREZ, J., GERMAIN-RENAUD, C., KEGL, B., AND LOOMIS, C. 2008. Grid differentiated services: A reinforcement learning approach. In Proceedings of the 8th IEEE International Symposium on Cluster Computing and the Grid (CCGRID '08). IEEE Computer Society, Washington, DC, 287-294.
22. PROTHMANN, H., ROCHNER, F., TOMFORDE, S., BRANKE, J., MÜLLER-SCHLOER, C., AND SCHMECK, H. 2008. Organic control of traffic lights. In Proceedings of the 5th International Conference on Autonomic and Trusted Computing (ATC '08). Springer, 219-233.
23. REYNOLDS, V., CAHILL, V., AND SENART, A. 2006. Requirements for an ubiquitous computing simulation and emulation environment. In Proceedings of the InterSense '06 Conference. ACM, New York.
24. RICHTER, S. 2006. Learning traffic control - Towards practical traffic control using policy gradients. Tech. rep., Albert-Ludwigs-Universitat Freiburg.
25. RICHTER, S., ABERDEEN, D., AND YU, J. 2007. Natural actor-critic for road traffic optimisation. Adv. Neural Inf. Process. Syst. 19. The MIT Press, Cambridge, MA.
26. SALKHAM, A. AND CAHILL, V. 2010. Soilse: A decentralized approach to optimization of fluctuating urban traffic using reinforcement learning. In 13th International IEEE Conference on Intelligent Transportation System (ITSC '10).
27. SALKHAM, A., CUNNINGHAM, R., GARG, A., AND CAHILL, V. 2008. A collaborative reinforcement learning approach to urban traffic control optimization. In Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT). Vol. 2. 560-566.
28. SCHNEIDER, J., WONG, W.-K., MOORE, A., AND RIEDMILLER, M. 1999. Distributed value functions. In Proceedings of the 16th International Conference on Machine Learning. Morgan Kaufmann, 371-378.
29. SUTON, R. S. AND BARTO, A. G. 1998. Reinforcement Learning: An Introduction. A Bradford Book. The MIT Press, Cambridge, MA.
30. SYCARA, K. 1998. Multiagent systems. AI Mag. 19, 2.
31. TAN, M. 1993. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the 10th International Conference on Machine Learning. Morgan Kaufmann, 330-337.
32. TESAURO, G. 2007. Reinforcement learning in autonomic computing: A manifesto and case studies. IEEE Internet Comput. 11, 1, 22-30. (Pubitemid 46335538)
33. TESAURO, G., CHESS, D. M., WALSH, W. E., DAS, R., SEGAL, A., WHALLEY, I., KEPHART, J. O., AND WHITE, S. R. 2004. A multi-agent systems approach to autonomic computing. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems. 464-471.
34. TESAURO, G., DAS, R., WALSH, W. E., AND KEPHART, J. O. 2005. Utility-Function-Driven resource allocation in autonomic systems. In Proceedings of the International Conference on Autonomic Computing. 342-343. (Pubitemid 43959647)
35. TESAURO, G., JONG, N. K., DAS, R., AND BENNANI, M. N. 2006. A hybrid reinforcement learning approach to autonomic resource allocation. In Proceedings of the IEEE International Conference on Autonomic Computing (ICAC '06). IEEE Computer Society, Washington, DC, 65-73. (Pubitemid 46666907)
36. WATKINS, C. J. C. H. AND DAYAN, P. 1992. Technical note: Q-learning. Mach. Learn. 8, 3, 279-292.
37. WIERING, M., VAN VEENEN, J., VREEKEN, J., AND KOOPMAN, A. 2004. Intelligent traffic light control. Tech. rep., Institute of Information and Computing Sciences, Utrecht University.
38. YANG, Z., CHEN, X., TANG, Y., AND SUN, J. 2005. Intelligent cooperation control of urban traffic networks. In Proceedings of the International Conference on Machine Learning and Cybernetics. 1482-1486. (Pubitemid 41734160)