Modeling of route planning system based on Q value-based dynamic programming with multi-agent reinforcement learning algorithms

Engineering Applications of Artificial Intelligence

Volumn 29, Issue , 2014, Pages 163-177

  Zolfpour Arokhlo, Mortaza ^a   Selamat, Ali ^b   Mohd Hashim, Siti Zaiton ^b   Afkhami, Hossein ^a  

^a ISLAMIC AZAD UNIVERSITY   (Iran)

^b UNIVERSITI TEKNOLOGI MALAYSIA   (Malaysia)

Author keywords

Multi agent reinforcement learning (MARL); Multi agent system (MAS); Q learning; Route planning system (RPS); Traffic congestion

Indexed keywords

BOLTZMANN EQUATION; DYNAMIC PROGRAMMING; FERTILIZERS; INTELLIGENT AGENTS; INTELLIGENT COMPUTING; LEARNING SYSTEMS; MOTOR TRANSPORTATION; MULTI AGENT SYSTEMS; REINFORCEMENT LEARNING; ROADS AND STREETS; TRAFFIC CONGESTION; TRAVEL TIME; VEHICLES;

BOLTZMANN DISTRIBUTION; CASE-STUDIES; LEARNING ABILITIES; MULTI-AGENT REINFORCEMENT LEARNING; Q-LEARNING; ROAD NETWORK; ROUTE PLANNING SYSTEM; ROUTING VEHICLES;

LEARNING ALGORITHMS;

EID: 84894026611     PISSN: 09521976     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.engappai.2014.01.001     Document Type: Article

References (84)

1. J.L. Adler, and V.J. Blue A cooperative multi-agent transportation management and route guidance system J. Transp. Res. Part C Emerg. Technol. 10 5-6 2002 433 454
2. J. Adler, G. Satapathy, V. Manikonda, B. Bowles, and V. Blue A multi-agent approach to cooperative traffic management and route guidance Transp. Res. Part B Methodol. 39 4 2005 297 318
3. Akchurina, N.; 2010. Multi-Agent Reinforcement Learning Algorithms (Ph.D. Thesis). University of Paderborn, pp. 1-182.
4. Almejalli, K.; Dahal, K.; Hossain, A.; 2009. An intelligent multi-agent approach for road traffic management systems. In: 2009 IEEE International Conference on Control Applications/International Symposium on Intelligent Control, vol. 1-3, July 08-10, Petersburg, Russia, pp. 825-830.
5. I. Arel, C. Liu, T. Urbanik, and A.G. Kohls Reinforcement learning based multi-agent system for network traffic signal control Intell. Transp. Syst. 4 2010 128 135
6. Awad, E.; 2011. Learning to Share: A Study of Multi-agent Learning in Transportation Systems (Master's Thesis). Computing and Information Science Program, Masdar Institute of Science and Technology.
7. Bakker, B.; Kester, L.; 2006. DOAS 2006 Project: Reinforcement Learning of Traffic Light Controllers Adapting to Accidents.
8. Bakker, B.; Steingrover, M.; Schouten, R.; Nijhuis, E.; Kester, L.; 2005. Cooperative multiagent reinforcement learning of traffic lights. In: 16th European Conference on Machine Learning (ECML-05) on Workshop on Cooperative Multi-Agent Learning, Porto, Portugal.
9. Balaji, P.G.; Sachdeva, G.; Srinivasan, D.; Tham, C.K.; 2007. Multi-agent system based urban traffic management. In: 2007 IEEE Congress on Evolutionary Computation, Sep 25-28, Singapore, Singapore, vol. 1-10, pp. 1740-1747.
10. Barbucha, D.; 2011. An agent-based guided local search for the capacited vehicle routing problem. In: Proceedings of the 5th KES International Conference on Agent and Multi-Agent Systems: Technologies and Applications, KES-AMSTA 11, Springer-Verlag, Berlin, Heidelberg, ISBN 978-3-642-21999-3, pp. 476-485.
11. Busoniu, L.; De Schutter, B.; Babuska, R.; 2005. Learning and coordination in dynamic multiagent systems. Delft Center for Systems and Control, Delft University of Technology, no. 05-019, Delft, The Netherlands.
12. Bussoniu, L.; Babuska, R.; De Schutter, B.; 2010. Multi-agent reinforcement learning: an overview. In: Innovations in Multi-Agent Systems and Applications - 1. Series of Studies in Computational Intelligence, vol. 310, Springer, Berlin, Germany, pp. 183-22 (Chapter 7).
13. Camiel, E.; 2011. Road Safety Strategic Plan 2008-2020. Ministry of Transport, Public Works and Water Management.
14. M. Chabrol, D. Sarramia, and N. Tchernev Urban traffic systems modelling methodology Int. J. Prod. Econ. 99 1-2 2006 156 176
15. Zhou, Changfeng, Yan, L.; Yuejin, T.; Liangcai, L.; 2006. Dynamic Vehicle Routing and Scheduling with Variable Travel Times in Intelligent Transportation System, pp. 8707-8711.
16. Chang-Qing, C.; Zhao-Sheng, Y.; 2007. Study urban traffic management based multi-agent system. In: Proceedings of the Sixth International Conference on Machine Learning and Cybernetics, IEEE, Hong Kong, pp. 25-29.
17. B. Chen, and H. Cheng A review of the applications of agent technology in traffic and transportation systems Trans. Intell. Transp. Syst. 11 2 2010 485 497
18. B. Chen, H.H. Cheng, and J. Palen Integrating mobile agent technology with multi-agent systems for distributed traffic detection and management systems Transp. Res. Part C 17 2009 1 10
19. M.A. Chowdhury, and A.W. Sadek Fundamentals of Intelligent Transportation Systems Planning 2003 Boston Artech House
20. Chuan-hua, Z.; Qing-song, L.; 2004. Route selecting in the light of the theory of multimode transportation based on multi-agent. In: 2004 IEEE International Confixence on Systems, Man and Cybernetics, vol. 4, pp. 4023-4027.
21. R. Claes, T. Holvoet, and D. Weyns A decentralized approach for anticipatory vehicle routing using delegate multiagent systems IEEE Trans. Intell. Transp. Syst. 12 2 2011 364 373
22. Dangelmaier, W.; Klopper, B.; Rungerer, N.; Aufenanger, M.; 2008. Aspects of Agent Based Planning in the Demand Driven Railcab Scenario. Dynamics in Logistics, pp. 171-178.
23. Delling, D.; 2009. Engineering and Augmenting Route Planning Algorithms (Ph.D.Thesis). University of Karlsruhe.
24. Demircan, S.; Aydin, M.; Durduran, S.S.; 2008. Route optimization with Q-learning. In: Proceedings of the 8th Conference on Applied Computer Science (ACS08), World Scientific and Engineering Academy and Society (WSEAS), Stevens Point, Wisconsin, USA, pp. 416-419.
25. B. Dominik, S. Katharina, and T. Axel Multi-agent-based transport planning in the newspaper industry Int. J. Prod. Econ. 131 1 2011 146 157
26. B. Dong, K. Li, M. Liao, and H. Wang The route choice model under the traffic information guide environment based on game theory J. Beihua Univ. (Nat. Sci.) 8 1 2007 88 91
27. Ferdowsizadeh-Naeeni, A.; 2004. Advanced Multi-Agent Fuzzy Reinforcement Learning (Master Thesis). Dalarna University, Sweden.
28. Flinsenberg, I.; 2004. Route Planning Algorithms for Car Navigation (Ph.D. Thesis), Technische Universiteit Eindhoven.
29. Gehrke, J.; Wojtusiak, J.; 2008. Traffic prediction for agent route planning. In: Proceedings of the 8th International Conference on Computational Science, Part III, ICCS 08, vol. 5103, Springer-Verlag, Berlin, Heidelberg, ISBN 978-3-540-69388-8, pp. 692-701.
30. Geisberger, R.; 2011. Advanced Route Planning in Transportation Networks (Ph.D. thesis). Karlsruhe Institute of Technology, Karlsruhe, pp. 1-227.
31. M. Isabel, R.T. Vicente, L.A. Garcia, and J.J. Martinez A Rule-Based Multi-agent System for Local Traffic Management. Intelligent Data Engineering and Automated Learning vol. 5788 2009 Burgos Spain 502 509
32. L. Jie, and F. Meng-yin Research on route planning and map-matching in vehicle GPS/dead-reckoning/electronic map integrated navigation system IEEE Intell. Transp. Syst. 2 2003 1639 1643
33. M. Ji, X. Yu, Y. Yong, X. Nan, and W. Yu Collision-avoiding aware routing based on real-time hybrid traffic informations J. Adv. Mater. Res. 396-398 2012 2511 2514
34. Jones, K.; 2010. A Trust Based Approach to Mobile Multi-Agent System Security (Ph.D. Thesis). The Faculty of Technology, De Montfort University, Leicester.
35. Khanjary, M.; Hashemi, S.; 2012. Route Guidance Systems: Review and Classification. In: 6th IEEE/ACM Euro American Conference on Telematics and Information Systems (EATIS), May, ACM, New York, NY, USA, pp. 269-275.
36. T.S. Kiam, and D. Lee Performance of multiagent taxi dispatch on extended-runtime taxi availability a simulation study IEEE Trans. Intell. Transp. Syst. 11 March 2010 231 236
37. Kosicek, M.; Tesar, R.; Darena, F.; Malo, R.; Motycka, A.; 2012. Route planning module as a part of supply chain management system. Acta Universitatis Agriculturae et Silviculturae Mendelianae Brunensis, vol. LX (2), pp. 135-142
38. Kovalchuk, Y.; 2009. A Multi-Agent Decision Support System for Supply Chain Management (Ph.D. Thesis). School of Computer Science and Electronic Engineering, University of Essex, July.
39. Kuyer, L.; Whiteson, S.; Bakker, B.; Vlassis, N.; 2008. Multiagent reinforcement learning for urban traffic control using coordination graphs. In: Proceedings of the 19th European Conference on Machine Learning, pp. 656-671.
40. Lauer, M.; Riedmiller, M.; 2000. An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In: Proceedings 17th International Conference on Machine Learning (ICML-00), Stanford University, US, pp. 535-542.
41. Lecce, D.; Amato, V.; 2008. Multi agent negotiation for a decision support system in route planning. In: Proceedings of the International Conference on Computational Intelligence for Modelling Control & Automation, pp. 458-463.
42. Ligtenberg, A.; 2006. Exploring the Use of Multi-Agent Systems for Interactive Multi-Actor Spatial Planning (Ph.D. Thesis). Wageningen University.
43. B. Lucian, B. Robert, D. Bart, and E. Damien Reinforcement Learning and Dynamic Programming Using Function Approximators 2010 CRC Press Boca Raton, Florida.
44. Marcelo, C.; Cristina, B.; Aura, C.; 2010. A Multi-agent System for Dynamic Path Planning. In: BWSS Brazilian Workshop on Social Simulation-SBC, Joint Conference of XX Brazilian Symposium on Artificial Intelligence-SBIA. FEI University Campus, Sao Bernardo do Campo, Brazil.
45. Michael, W.; 2009. An Introduction to MultiAgent Systems. May, second ed. John Wiley and Sons. ISBN-13: 978-0470519462.
46. Negenborn, R.; De Schutter, B.; Hellendoorn, J.; 2008a. Multi-Agent Model Predictive Control For Transportation Networks: Serial Versus Parallel Schemes. Engineering Applications of Artificial Intelligence. Elsevier 21(3), pp. 353-366
47. C.D. Paternina-Arboleda, and T.K. Das A multi-agent reinforcement learning approach to obtaining dynamic control policies for stochastic lot scheduling problem Simul. Model. Pract. Theory 13 5 2005 389 406
48. Pellazar, M.S.; 1998. Vehicle route planning with constraints using genetic algorithms. In: Proceedings of the IEEE 1998 National Aerospace and Electronics Conference (NAECON), 13-17 July, pp. 392-399.
49. M.A. Riedmiller, A.W. Moore, and J.G. Schneider Reinforcement learning for cooperating and communicating reactive agents in electrical power grids M. Hannebauer, J. Wendler, E. Pagello, Balancing Reactivity and Social Deliberation in Multi-Agent Systems 2000 Springer 137 149
50. V. Robu, H. Noot, H.L. Poutre, and W.J. van Schijndel A multi-agent platform for auction-based allocation of loads in transportation logistics Expert Syst. Appl. 38 4 2011 3483 3491
51. Rothkrantz, L.; 2009. Dynamic routing using the network of car drivers. In: EATIS 09: Proceedings of the 2009 Euro American Conference on Telematics and Information Systems, New York, NY, USA, ACM, ISBN 978-1-60558-398-3, pp. 1-8.
52. L. Ruimin, and S. Qixin Study on integration of urban traffic control and route guidance based on multi-agent technology IEEE Intell. Transp. Syst. 2 2003 1740 1744
53. Ryan, J.; Donald, C.; 2008. Divide and Conquer Evolutionary TSP Solution for Vehicle Path Planning. In: Proceedings of IEEE Congress on Evolutionary Computation. Hong Kong, June 2008, EC0172, pp. 676-681.
54. Schmitt, E.; Jula, H.; 2006. Vehicle route guidance systems: classification and comparison. In: Intelligent Transportation Systems Conference, pp. 242-247.
55. Schultes, D.; 2008. Route Planning in Road Networks (Ph.D. Thesis). Fakultat fur Informatik, Universitat Karlsruhe (TH).
56. Seow, K.; Dang, N.; Lee, D.; 2007. Towards an Automated Multiagent Taxi-Dispatch System. In: IEEE International Conference, 2007, pp. 1045-1050.
57. Shanqing, Y.; Jing, Z.; Bing, L.; Mabu, S.; Hirasawa, K.; 2012. Q value-based dynamic programming with SARSA learning for real time route guidance in large scale road networks. In: The 2012 International Joint Conference on Neural Networks (IJCNN), vol. 10-15, pp. 1-7.
58. Shi, A.; Na, C.; Chun-b., H.; 2008. Simulation and analysis of route guidance strategy based on a multi-agent-game approach. In: Management Science and Engineering, September 2008, pp. 140-146.
59. Shimohara, X.; Hu, J.; Yangsheng, X.; Song, J.; 2005. A simulation study on agent-network based route guidance system. In: 2005 IEEE Intelligent Transportation Systems Conference (ITSC).
60. Y. Shoham, and K. Leyton-Brown Multiagent Systems Algorithmic, Game Theoretic and Logical Foundations 2008 Cambridge University Press 2008, ISBN 978-0-521-89943-7, pp. I-XX, 1-483.
61. S. Srinivasan, D. Kumar, and V. Jaglan Multi-agent system supply chain management in steel pipe manufacturing Int. J. Comput. Sci. Issues 7 4 2010 30 34
62. Stephan, W.; Yunhui, W.; 2008. Towards a conceptual model of talking to a route planner. In: The 8th International Symposium on Web and Wireless Geographical Information Systems (W2GIS 08), Springer-Verlag, Berlin, Heidelberg, pp. 107-123.
63. Suzuki, N.; Araki, D.; Higashide, A.; Suzuki, T.; 1995. Geographical route planning based on uncertain knowledge. In: Proceedings of Seventh International Conference on Tools with Artificial Intelligence, 5-8 November, pp. 434-441.
64. Tampere, C.; Immers, B.; Stada, J.; Janssens, B.; 2008. Multi-agent control in road Traffic management. In: 7th International Conference Environmental Engineering, vol. 1-3, Lithuania, pp. 1052-1061.
65. Tesauro, G.; Jong, N.K.; Das, R.; Bennani, N.; 2006. A hybrid reinforcement learning approach to autonomic resource allocation. In: ICAC 06, Dublin, Ireland, pp. 65-73.
66. Torrey, L.; 2010. Crowd simulation via multi-agent reinforcement learning. In: Proceedings of the Sixth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE), 2010, Stanford, California, USA, The AAAI Press.
67. Tu, H.; 2008. Monitoring Travel Time Reliability on Freeways. Transportation and Planning (Ph.D. Thesis), Delft University of Technology, Delft.
68. M.S. Uppin Multi agent system model of supply chain for information sharing Contemp. Eng. Sci. 3 1 2010 1 16
69. A. Vasan, and P. Slobodan Optimization of water distribution network design using differential evolution J. Water Resour. Plan. Manag. 136 2 2010 279 287
70. Vasirani, M.; Ossowski, S.; 2009. A market-inspired approach to reservation based urban road traffic management. In: AAMAS 09: Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems, Richland, pp. 617-624.
71. Vlassis, N.; 2007. A concise introduction to multiagent systems and distributed artificial intelligence. In: Synthesis Lectures in Artificial Intelligence and Machine Learning, Morgan & Claypool Publishers.
72. P. Volf, D. Sislak, and M. Pechoucek Large-scale high-fidelity agent based simulation in air traffic domain Cybern. Syst. 42 2011 502 525
73. Watanabe, T.; Lijiao, C.; 2011. Vehicle routing based on traffic cost at intersection. In: Shea, J.; Nguyen, N.; Crockett, K.; Howlett, R.; Jain, L. (Eds.) Agent and Multi-Agent Systems: Technologies and Applications. Lecture Notes in Computer Science, vol. 6682. Springer, Berlin, Heidelberg, ISBN 978-3-642-21999-3, pp. 592-601.
74. Wedde, H.; Lehnhoff, S.; van Bonn, B.; Bay, Z.; Becker, S.; Bottcher, S.; Brunner, C.; Buscher, A.; Furst, T.; Lazarescu, A.; Rotaru, E.; Senge, S.; Steinbach, B.; Yilmaz, F.; Zimmermann, T.; 2007. A novel class of multi-agent algorithms for highly dynamic transport planning inspired by honey bee behavior. In: IEEE Conference on Emerging Technologies and Factory Automation, IEEE, Greece, pp. 1157-1164.
75. Weyns, D.; Holvoet, T.; Helleboogh, A.; 2007. Anticipatory vehicle routing using delegate multi-agent systems. In IEEE Intelligent Transportation System Conference, USA, pp. 87-93.
76. Wilson, A.; Fern, A.; Tadepalli, P.; 2010. Bayesian role discovery for multi-agent reinforcement learning. In: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, vol. 1, pp. 1587-1588.
77. J. Wu, S. Jin, H. Ji, and T. Srikanthan Algorithm for time-dependent shortest safe path on transportation networks Procedia CS 4 2011 958 966
78. D. Yu, Z. Yang, Y. Wang, and J. Sun Urban road traffic control system and its coordinate optimization based on multi-agent system J. Jilin Univ. (Eng. Technol. Ed.) 36 1 2006 113 118
79. Yunhui, W.; Stephan, W.; 2007. Agent Behaviour in Peer-to-Peer Shared Ride Trip Planning (Master's Thesis). Department of Geomatics, The University of Melbourne, Australia.
80. K. Zafar, and A.R. Baig Optimization of route planning and exploration using multi agent system Multimed. Tools Appl. 56 2 2012 245 265
81. Zegeye, K.G.; 2010. A Dynamic Prediction of Travel Time for Transit Vehicles in Brazil Using GPS Data. Department of Civil Engineering and Management, University of Twente, Enschede, The Netherlands.
82. Zi, Z.; Jian-Min, X.; 2005. A dynamic route guidance arithmetic based on reinforcement learning. In: Proceedings of the Fourth International Conference on Machine Learning and Cybernetics, pp. 3607-3611.
83. M. Zolfpour-Arokhlo, A. Selamat, and S.Z. Mohd-Hashim Multi-agent reinforcement learning for route guidance system Int. J. Adv. Comput. Technol. 3 6 2011 224 232
84. L. Zou, J. Xu, and L. Zhu Application of genetic algorithm in dynamic route guidance system J. Transp. Syst. Eng. Inf. Technol. 7 3 2007 45 48