Cooperative adaptive cruise control: A reinforcement learning approach

IEEE Transactions on Intelligent Transportation Systems

Volumn 12, Issue 4, 2011, Pages 1248-1260

  Desjardins, Charles ^a   Chaib Draa, Brahim ^b  

^a INSTITUTE FOR RESEARCH IN IMMUNOLOGY AND CANCER   (Canada)

^b UNIVERSITÉ LAVAL   (Canada)

Author keywords

Autonomous vehicle control; cooperative adaptive cruise control (CACC); neural networks; policy gradient algorithms; reinforcement learning (RL)

Indexed keywords

ADAPTIVE CRUISE CONTROL; AUTONOMOUS VEHICLE CONTROL; AUTONOMOUS VEHICLES; COMPUTING TECHNOLOGY; CONSTANT SPEED; CONTROL POLICY; CONTROL TASK; COOPERATIVE ADAPTIVE CRUISE CONTROL; CURRENT SPEED; DESIGN APPROACHES; DRIVER-ASSISTANCE SYSTEMS; FOLLOWING VEHICLE; FUNCTION APPROXIMATION TECHNIQUES; GRADIENT-DESCENT; INTER VEHICLE COMMUNICATIONS; MACHINE LEARNING TECHNIQUES; REINFORCEMENT LEARNING APPROACH; SAFE DISTANCE; VEHICLE-TO-VEHICLE COMMUNICATION;

APPROXIMATION ALGORITHMS; BEHAVIORAL RESEARCH; COMMUNICATION; CONTROL SYSTEM SYNTHESIS; CONTROLLERS; INTELLIGENT VEHICLE HIGHWAY SYSTEMS; LEARNING ALGORITHMS; NEURAL NETWORKS; REINFORCEMENT; REINFORCEMENT LEARNING; VEHICLES;

ADAPTIVE CONTROL SYSTEMS;

EID: 82455188002     PISSN: 15249050     EISSN: None     Source Type: Journal    
DOI: 10.1109/TITS.2011.2157145     Document Type: Article

References (58)

1. J. Piao and M. McDonald, "Advanced driver assistance systems from autonomous to cooperative approach,"Transp. Rev., vol. 28, no. 5, pp. 659-684, Sep. 2008.
2. Standard Specification for Telecommunications and Information Exchange Between Roadside and Vehicle Systems-5-GHz Band Dedicated Short-Range Communications (DSRC) Medium Access Control (MAC) and Physical Layer (PHY) Specifications, ASTM E2213-03, 2003.
3. J. Peters and S. Schaal, "Reinforcement learning of motor skills with policy gradients,"Neural Netw., vol. 21, no. 4, pp. 682-697, May 2008.
4. P. Fancher, Z. Bareket, and R. Ervin, "Human-centered design of an ACC with braking and forward-crash-warning system,"Vehicle Syst. Dyn., vol. 36, no. 2, pp. 203-223, 2001.
5. D. Augello, "Description of three PROMETHEUS demonstrators having potential safety effects,"in Proc. 13th Int. Tech. Conf. Exp. Safety Vehicles, 1993, pp. 4-7.
6. C. Bonnet and H. Fritz, "Fuel consumption reduction experienced by two promote chauffeur trucks in electronic tow bar operation,"in Proc. 7th World Congr. Intell. Transp. Syst., 2000.
7. E. Commission, CVIS: Cooperative Vehicle Infrastructure Systems.[Online]. Available: http://www.cvisproject.org/en/home.htm
8. E. Commission, COOPERS: Cooperative Systems for Intelligent Road Safety. [Online]. Available: http://www.coopers-ip.eu/
9. S. Tsugawa, "An introduction to Demo 2000: The cooperative driving scenario,"IEEE Intell. Syst., vol. 15, no. 4, pp. 78-79, Jul. 2000.
10. K. Petty and W. Mahoney, "Enhancing road weather information through vehicle infrastructure integration,"Transp. Res. Rec., no. 2015, pp. 132- 140, 2007.
11. U. Hofmann, A. Rieder, and E. Dickmanns, "Radar and vision data fusion for hybrid adaptive cruise control on highways,"Mach. Vis. Appl., vol. 14, no. 1, pp. 42-49, Apr. 2003.
12. L. Davis, "Effect of adaptive cruise control systems on traffic flow,"Phys. Rev. E: Stat. Phys. Plasmas Fluids Relat. Interdiscip. Top., vol. 69, no. 6, p. 066 110, Jun. 2004.
13. J. E. Naranjo, C. Gonzàlez, J. Reviejo, R. Garcia, and T. de Pedro, "Adaptive fuzzy control for intervehicle gap keeping,"IEEE Trans. Intell. Transp. Syst., vol. 4, no. 2, pp. 132-142, Sep. 2003.
14. J. E. Naranjo, C. Gonzàlez, T. de Pedro, R. Garcia, J. Alonso, M. A. Sotelo, and D. Fernandez, "Autopia architecture for automatic driving and maneuvering,"in Proc. IEEE ITSC, Toronto, ON, Canada, Sep. 2006, pp. 1220-1225.
15. J. E. Naranjo, C. Gonzàlez, R. Garcia, and T. de Pedro, "ACC +stop&go maneuvers with throttle and brake fuzzy control,"IEEE Trans. Intell. Transp. Syst., vol. 7, no. 2, pp. 213-225, Jun. 2006.
16. L. Xiao and F. Gao, "A comprehensive review of the development of adaptive cruise control systems,"Vehicle Syst. Dyn., vol. 48, no. 10, pp. 1167-1192, 2010.
17. D. De Bruin, J. Kroon, R. Van Klaveren, and M. Nelisse, "Design and test of a cooperative adaptive cruise control system,"in Proc. IEEE Intell. Vehicles Symp., 2004, pp. 392-396.
18. G. Naus, R. Vugts, J. Ploeg, M. Van de Molengraft, and M. Steinbuch, "Towards on-the-road implementation of cooperative adaptive cruise control,"in Proc. 16th World Congr. Exhib. Intell. Transp. Syst. Serv., Stockholm, Sweden, 2009.
19. M. van Eenennaam, W. Wolterink, G. Karagiannis, and G. Heijenk, "Exploring the solution space of beaconing in VANETs,"in Proc. IEEE VNC, 2010, pp. 1-8.
20. B. Van Arem, C. Van Driel, and R. Visser, "The impact of cooperative adaptive cruise control on traffic-flow characteristics,"IEEE Trans. Intell. Transp. Syst., vol. 7, no. 4, pp. 429-436, Dec. 2006.
21. D. Pomerleau, "Neural network vision for robot driving,"in The Handbook of Brain Theory and Neural Networks, M. Arbib, Ed. Cambridge, MA: MIT Press, 1995.
22. G. Yu and I. Sethi, "Road following with continuous learning,"in Proc. Intell. Vehicles Symp., 1995, pp. 412-417.
23. F. Lewis and D. Vrabie, "Reinforcement learning and adaptive dynamic programming for feedback control,"IEEE Circuits Syst. Mag., vol. 9, no. 3, pp. 32-50, Third Quarter, 2009.
24. S. Oh, J. Lee, and D. Choi, "A new reinforcement learning vehicle control architecture for vision-based road following,"IEEE Trans. Veh. Technol., vol. 49, no. 3, pp. 997-1005, May 2000.
25. D. Moriarty and P. Langley, "Distributed learning of lane-selection strategies for traffic management,"Daimler-Benz Res. Technol. Center, Palo Alto, CA, Tech. Rep. 98-2, 1998.
26. J. R. Forbes, "Reinforcement learning for autonomous vehicles,"Ph.D. dissertation, Univ. California, Berkeley, 2002.
27. L. Ng, C. Clark, and J. Huissoon, "Reinforcement learning of dynamic collaborative driving-Part I: Longitudinal adaptive control,"Int. J. Vehicle Inf. Commun. Syst., vol. 1, no. 3, pp. 208-228, 2008.
28. J. Laumonier, "Méthodes d'apprentissage dans le cadre de la coordination multiagent: Application au transport intelligent,"Ph.D. dissertation, Univ. Laval, Québec, QC, Canada, 2008.
29. C. Desjardins, P.-L. Grégoire, J. Laumônier, and B. Chaib-draa, "Architecture and design of a multilayered cooperative cruise control system,"in Proc. SAE World Congr., Detroit, MI, Apr. 2007.
30. C. Desjardins, J. Laumônier, and B. Chaib-draa, "Learning agents for collaborative driving,"in Multiagent Architectures for Traffic and Transportation Engineering. Hershey, PA: IGI-Global, 2008.
31. T. Strobel, A. Servel, C. Coue, and T. Tatschke, Compendium on Sensors-State of the Art of Sensors and Sensor Data Fusion for Automotive Preventive Safety Applications, PReVENT Consortium, Tech. Rep., 2004. [Online]. Available: http://www.prevent-ip.org/download/deliverables/ProFusion/PR-13400- IPD-040531-v10-Compendium-on-Sensors.pdf
32. T. Strobel and A. Servel, Sensor Data Sheets-State of the Art of Sensors and Sensor Data Fusion for Automotive Preventive Safety Applications, PReVENT Consortium, Tech. Rep., 2004. [Online]. Available: http://www.prevent-ip.org/ download/deliverables/ProFusion/PR-13400-IPD- 040531-v10-Sensor-Data-Sheets.pdf
33. W. R. Bishop, Intelligent Vehicle Technology and Trends. Norwood, MA: Artech House, 2005.
34. Y. Günter and H. P. Gromann, "Usage of wireless LAN for intervehicle communication,"in Proc. IEEE Conf. Intell. Transp. Syst., 2005, pp. 296-301.
35. P. A. Ehlert, "The agent approach to tactical driving in autonomous vehicle and traffic simulation,"M.S. thesis, Knowl. Based Syst. Group, Delft Univ. Technol., Delft, The Netherlands, 2001.
36. U. Kiencke and L. Nielsen, Automotive Control Systems: For Engine, Driveline and Vehicle. New York: Springer-Verlag, 2000.
37. X. Huppé, "Guidance et commande longitudinale d'un train de voitures adaptées aux conditions routières et climatiques canadiennes,"M.S. thesis, Univ. Sherbrooke, Sherbrooke, QC, Canada, 2004.
38. S. Hallé, "Automated highway systems: Platoons of vehicles viewed as a multiagent system,"M.S. thesis, Univ. Laval, Québec, QC, Canada, 2005.
39. H. Kimura, M. Yamamura, and S. Kobayashi, "Reinforcement learning in POMDPS with function approximation,"in Proc. 14th ICML, 1997, pp. 152-160.
40. J. A. Bagnell and J. G. Schneider, "Autonomous helicopter control using reinforcement learning policy search methods,"in Proc. IEEE ICRA, Seoul, Korea, May 2001, pp. 1615-1620.
41. T. Field, "Policy-gradient learning for motor control,"M.S. thesis, Victoria Univ. Wellington, Wellington, New Zealand, 2005.
42. N. Kohl and P. Stone, "Policy-gradient reinforcement learning for fast quadrupedal locomotion,"in Proc. IEEE ICRA, New Orleans, LA, May 2004, pp. 2619-2624.
43. A. El-Fakdi, M. Carreras, and P. Ridao, "Direct gradient-based reinforcement learning for robot behavior learning,"in Proc. 2nd ICINCO, Sep. 2005, pp. 225-231.
44. J. Baxter and P. Bartlett, "Infinite-horizon policy-gradient estimation,"J. Artif. Intell. Res., vol. 15, no. 4, pp. 319-350, Jul. 2001.
45. J. Baxter, P. Bartlett, and L. Weaver, "Experiments with infinitehorizon policy-gradient estimation,"J. Artif. Intell. Res., vol. 15, no. 1, pp. 351-381, Jul. 2001.
46. J. Peters and S. Schaal, "Policy-gradient methods for robotics,"in Proc. IEEE/RSJ Int. Conf. IROS, Los Alamitos, CA, Oct. 2006, pp. 2219-2225.
47. R. J. Williams, "Simple statistical gradient-following algorithms for connectionist reinforcement learning,"Mach. Learn., vol. 8, no. 3/4, pp. 229-256, May 1992.
48. P. Marbach, "Simulation-based optimization of Markov decision processes,"Ph.D. dissertation, Lab. Inf. Decision Syst., Mass. Inst. Technol., Cambridge, MA, Sep., 1998.
49. H. Kimura,M. Yamamura, and S. Kobayashi, "Reinforcement learning by stochastic hill climbing on discounted reward,"in Proc. 12th ICML, 1995, pp. 295-303.
50. P. Bartlett and J. Baxter, "Stochastic optimization of controlled partially observable Markov decision processes,"in Proc. 39th IEEE Conf. Decision Control, 2000, vol. 1, pp. 124-129.
51. R. Hallouzi, V. Verdult, H. Hellendorn, P. L. Morsink, and J. Ploeg, "Communication based longitudinal vehicle control using an extended Kalman filter,"in Proc. IFAC Symp. Adv. Automot. Control, Salerno, Italy, 2004.
52. D. de Bruin, J. Kroon, R. van Klaveren, and M. Nelisse, "Design and test of a cooperative adaptive cruise control,"in Proc. IEEE Intell. Vehicles Symp., Parma, Italy, Jun. 2004, pp. 392-396.
53. Z. Bareket, P. S. Fancher, H. Peng, K. Lee, and C. A. Assaf, "Methodology for assessing adaptive cruise control behavior,"IEEE Trans. Intell. Transp. Syst., vol. 4, no. 3, pp. 123-131, Sep. 2003.
54. M. Richardson, D. Corrigan, P. King, I. Smith, P. Barber, and K. J. Burnham, "Application of a control system simulation in the automotive industry,"in Proc. IEE Semin. Tools Simul. Model., London, U.K., 2000, pp. 8/1-8/13.
55. A. Y. Ng, D. Harada, and S. J. Russell, "Policy invariance under reward transformations: Theory and application to reward shaping,"in Proc. 16th ICML, 1999, pp. 278-287.
56. G. V. Cybenko, "Approximation by superpositions of a sigmoidal function,"Math. Control, Signals Syst., vol. 2, no. 4, pp. 303-314, Dec. 1989.
57. R. Schubert, K. Schulze, and G.Wanielik, "Situation assessment for automatic lane-change maneuvers,"IEEE Trans. Intell. Transp. Syst., vol. 11, no. 3, pp. 607-616, Sep. 2010.
58. J. Laumônier and B. Chaib-draa, "Partial local friendQ multiagent learning: Application to team automobile coordination problem,"in Proc. Can. AI, Lecture Notes in Artificial Intelligence, Jun. 2006, pp. 361-372.