A game theory-reinforcement learning (GT-RL) method to develop optimal operation policies for multi-operator reservoir systems

Journal of Hydrology

Volumn 519, Issue PA, 2014, Pages 732-742

  Madani, Kaveh ^a   Hooshyar, Milad ^b  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b University of Central Florida   (United States)

Author keywords

Conflict resolution; Evolutionary algorithm; Game theory; Optimization; Reinforcement learning; Reservoir operation

Indexed keywords

GAME THEORY; OPTIMIZATION; REINFORCEMENT LEARNING;

CONFLICT RESOLUTION; MULTI OPERATORS; OPTIMAL OPERATION POLICY; RESERVOIR OPERATION; RESERVOIR SYSTEMS;

EVOLUTIONARY ALGORITHMS;

ALGORITHM; GAME THEORY; OPTIMIZATION; RESERVOIR;

EID: 84906765896     PISSN: 00221694     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jhydrol.2014.07.061     Document Type: Article

References (49)

1. Arnold E., Tatjewsli P., Wolochowisc P. Two methods for large-scale nonlinear optimization and their comparison on a case study of hydropower optimization. J. Optim. Theory Appl. 1994, 81(2):221-224.
2. Asgari S., Afshar A., Madani K. A cooperative game theoretic framework for joint resource management in construction. J. Constr. Eng. Manage. 2014, 140(3):04013066. 10.1061/(ASCE)CO.1943-7862.0000818.
3. Bergez, J., Eigenraam, M., Garica, F., 2001. Comparison between dynamic programming and reinforcement learning: a case study on maize irrigation management. In: Proceedings of the 3rd European Conference on Information Technology in Agriculture, Montpellier, Fr, June 18-20, pp. 343-348.
4. Castelletti A., Galelli S., Restelli M. Tree-based reinforcement learning for optimal water reservoir operation. Water Resour. Res. 2010, 46(9):W09507.
5. Dinar A. Scale and equity in water resource development: a Nash bargaining model. Nat. Resour. Model. 2001, 14(4):477-494.
6. Dinar A., Howitt R.E. Mechanisms for allocation of environmental control cost: empirical tests of acceptability and stability. J. Environ. Manage. 1997, 49:183-203.
7. Gately D. Sharing the gains from regional cooperation: a game theoretic application to planning investment in electric power. Int. Econ. Rev. 1974, 15(1):195-208.
8. Gosavi A. Simulation-based Optimization: Parametric Optimization Techniques and Reinforcement Learning 2003, Norwell, Massachusetts, USA.
9. Harsanyi J. A bargaining model for the cooperative n-person game. Contributions to the Theory of Games 4 1959, 324-356. Princeton University Press, Princeton, New Jersey.
10. Hiew, K., 1987. Optimization Algorithms for Large Scale Multi-Reservoir Hydropower Systems. PhD Dissertation. Department of Civil Engineering, Colorado State University, USA.
11. Imen, S., Madani, K., Chang, N.-B., 2012. Bringing environmental benefits into Caspian Sea negotiations for resources allocation: cooperative game theory insights. In: Proceedings of the World Environmental and Water Resources Congress 2012. ASCE, Albuquerque, New Mexico, pp. 2264-2271. doi:10.1061/9780784412312.228.
12. Kelman J., Stedinger J.R., Cooper L.A. Sampling stochastic dynamic programming applied to reservoir operation. Water Resour. Res. 1990, 26(3):447-454.
13. Labadie J.W. Optimal operation of multireservoir systems. J. Water Resour. Plann. Manage. 2004, 130(2):93-111.
14. Lee J.H., Labadie J.W. Stochastic optimization of multireservoir systems via reinforcement learning. Water Resour. Res. 2007, 43(11):W11408.
15. Loehman E., Orlando J., Tschirhart J., Winstion A. Cost allocation for a regional wastewater treatment system. Water Resour. Res. 1979, 15:193-202.
16. Loucks D., Dorfman P. An evaluation of some linear decision rules in chance-constrained models for reservoir planning and operation. Water Resour. Res. 1975, 11(6):777-782.
17. Lund J., Ferreira I. Operating rule optimization for Missouri River reservoir system. J. Water Resour. Plann. Manage. 1996, 122(4):287-295.
18. Madani K. Game theory and water resources. J. Hydrol. 2010, 381(3-4):225-238.
19. Madani K. Hydropower licensing and climate change: insights from cooperative game theory. Adv. Water Resour. 2011, 34(2):174-183.
20. Madani K., Dinar A. Cooperative institutions for sustainable management of common pool resources: Application to Groundwater. Water Resour. Res. 2012, 48(9). 10.1029/2011WR010849.
21. Madani K., Gue´gan M., Uvo C. Climate change impacts on high-elevation hydroelectricity in California. J. Hydrol. 2014, 510(14):153-163.
22. Madani K., Read L., Shalikarian L. Voting under uncertainty: a stochastic framework for analyzing group decision making problems. Water Resour. Manage 2014, 28(7):1839-1856.
23. Madani K., Zarezadeh M., Morid S. A new framework for resolving conflicts over transboundary rivers using bankruptcy methods. Hydrol. Earth Syst. Sci. 2014, 18:3055-3068. 10.5194/hess-18-1-2014.
24. Mahootchi, M., 2009. Storage System Management Using Reinforcement Learning Techniques and Nonlinear Models. PhD Dissertation. University of Waterloo, Canada.
25. Mahootchi M., Tizhoosh H.R., Ponnambalam K. Opposition-based reinforcement learning in the management of water resources. Proc. IEEE Symp. Approx. Dyn. Program. Reinf. Learn. 2007, 217-224.
26. Mousavi S.J., Karamouz M. Computational improvement for dynamic programming models by diagnosing infeasible storage combinations. Adv. Water Resour. 2003, 26(8):851-859.
27. Nakayama M., Suzuki M. The cost assignment of cooperative water resource development: a game theoretic approach. Manage. Sci. 1976, 22(10):1081-1086.
28. Nash J. Two-person cooperative games. Econometrica 1953, 21:128-140.
29. Needham J., Watkins D., Lund J., Nanda S. Linear programming for flood control in the Iowa and Des Moines Rivers. J. Water Resour. Plann. Manage. 2000, 126(3):118-127.
30. Parrachino I., Dinar A., Patrone F. Cooperative game theory and its application to natural, environmental, and water resource issues: 3. Application to water resources. The World Bank 2006.
31. Ratner A., Yaron D. Regional cooperation in the use of irrigation water, efficiency and game theory analysis of income distribution. Agric. Econ. 1990, 4(1).
32. Read L., Madani K., Inanloo B. Optimality versus stabiloty in water resource allocation negotiations. J. Environ. Manage. 2014, 133:343-354. 10.1016/j.jenvman.2013.11.045.
33. Robbins H., Monro S. A stochastic approximation method. Annu. Math. Stat. 1951, 22(3):400.
34. Sadegh M., Mahjouri N., Kerachian R. Optimal inter-basin water allocation using crisp and fuzzy Shapley games. Water Resour. Manage. 2010, 24(10):2291-2310.
35. Schmeidler D. The nucleolus of a characteristic function game. SIAM J. Appl. Math. 1969, 17:1163-1170.
36. Shapley L.S. A value for n-person games. Annu. Math. Stud. 1953, 28:307-318.
37. Shapley L., Shubik M. A method for evaluating the distribution of power in a committee system. Am. Polit. Sci. Rev. 1954, 48:787-792.
38. Sheikhmohammady, M., Madani, K., 2008. Bargaining over the Caspian Sea - the largest lake on the earth. In: Proceeding of the 2008 World Environmental and Water Resources Congress.
39. Straffin P.D., Heaney J.P. Game theory and the Tennessee Valley Authority. Int. J. Game Theory 1981, 10:35-43.
40. Sutten R., Barto A. Reinforcement Learning: An Introduction 2000, MIT Press, Cambridge, MA.
41. Szidarovszky F., Duckstein L., Bogardi I. Multiobjective management of mining under water hazard by game theory. Eur. J. Oper. Res. 1984, 15(2):251-258.
42. Teasley R., McKinney D. Calculating the benefits of transboundary river basin cooperation: Syr Darya Basin. J. Water Resour. Plann. Manage. 2011, 137(6):481-490.
43. Tejada-Guibert J.A., Stedinger J.R. Optimization of value of CVP'S hydropower production. J. Water Resour. Plann. Manage. 1990, 116(1):52-70.
44. Thomas H., Watermeyer P. Mathematical Models: A Stochastic Sequential Approach 1962, Harvard University Press, Cambridge, MA.
45. Wang L., Fang L., Hipel K. Basin-wide cooperative water resources allocation. Eur. J. Oper. Res. 2008, 190(3):798-817.
46. Watkins, C., 1989. Learning from Delayed Rewards. PhD Thesis. University of Cambridge, England.
47. Watkins C., Dayan P. Q-learning. Mach. Learn. 1992, 8:279-292.
48. Willis R., Finneya B.A., Chu W. Monte Carlo optimization for reservoir operation. Water Resour. Res. 1984, 20(9):1177-1182.
49. Young H.P., Okada N., Hashimoto N. Cost allocation in water resources development. Water Resour. Res. 1982, 18(3):463-475.