Curses, tradeoffs, and scalable management: Advancing evolutionary multiobjective direct policy search to improve water reservoir operations

Journal of Water Resources Planning and Management

Volumn 142, Issue 2, 2016, Pages

  Giuliani, Matteo ^a   Castelletti, Andrea ^a,b   Pianosi, Francesca ^c   Mason, Emanuele ^a   Reed, Patrick M ^d  

^a POLITECNICO DI MILANO   (Italy)

^b ETH ZURICH   (Switzerland)

^c UNIVERSITY OF BRISTOL   (United Kingdom)

^d Department of Obstetrics Gynecology   (United States)

Author keywords

Direct policy search; Multiobjective evolutionary algorithm; Water management

Indexed keywords

BENCHMARKING; BINS; COMPLEX NETWORKS; DYNAMIC PROGRAMMING; EVOLUTIONARY ALGORITHMS; FLOODS; MULTIOBJECTIVE OPTIMIZATION; NEURAL NETWORKS; OPTIMIZATION; QUALITY CONTROL; RADIAL BASIS FUNCTION NETWORKS; RELIABILITY ANALYSIS; RESERVOIR MANAGEMENT; STOCHASTIC SYSTEMS; WATER MANAGEMENT;

CONFLICTING OBJECTIVES; DIRECT POLICY SEARCH; HYDRO POWER PRODUCTION; MULTI OBJECTIVE EVOLUTIONARY ALGORITHMS; RADIAL BASIS FUNCTIONS; RELATIVE PERFORMANCE; SIMULATION-BASED OPTIMIZATIONS; STOCHASTIC DYNAMIC PROGRAMMING;

RESERVOIRS (WATER);

ALGORITHM; ARTIFICIAL NEURAL NETWORK; HYDROELECTRIC POWER PLANT; OPTIMIZATION; RESERVOIR; WATER MANAGEMENT;

HOA BINH RESERVOIR; VIET NAM;

EID: 84955276734     PISSN: 07339496     EISSN: None     Source Type: Journal    
DOI: 10.1061/(ASCE)WR.1943-5452.0000570     Document Type: Article

References (96)

1. Ansar, A., Flyvbjerg, B., Budzier, A., and Lunn, D. (2014). "Should we build more large dams? The actual costs of hydropower megaproject development." Energy Policy, 69, 43-56.
2. Baxter, J., Bartlett, P., and Weaver, L. (2001). "Experiments with infinite-horizon, policy-gradient estimation." J. Artif. Intell. Res., 15, 351-381.
3. Bellman, R. (1957). Dynamic programming, Princeton University Press, Princeton, NJ.
4. Bertsekas, D. (1976). Dynamic programming and stochastic control, Academic Press, New York.
5. Bertsekas, D. (2005). "Dynamic programming and suboptimal control: A survey from ADP to MPC." Eur. J. Control, 11(4-5), 310-334.
6. Bertsekas, D., and Tsitsiklis, J. (1996). Neuro-dynamic programming, Athena Scientific, Belmont, MA.
7. Biglarbeigi, P., Giuliani, M., and Castelletti, A. (2014). "Many-objective direct policy search in the Dez and Karoun multireservoir system, Iran." Proc., World Environmental and Water Resources Congress (ASCE EWRI 2014), Portland, OR.
8. Brill, E., Flach, J., Hopkins, L., and Ranjithan, S. (1990). "MGA: A decision support system for complex, incompletely defined problems." IEEE Trans. Syst. Man Cybern., 20(4), 745-757.
9. Buhmann, M. (2003). Radial basis functions: Theory and implementations, Cambridge University Press, Cambridge, U.K.
10. Busoniu, L., Babuska, R., De Schutter, B., and Ernst, D. (2010). Reinforcement learning and dynamic programming using function approximators, CRC Press, New York.
11. Busoniu, L., Ernst, D., De Schutter, B., and Babuska, R. (2011). "Cross-entropy optimization of control policies with adaptive basis functions." IEEE Trans. Syst. Man Cybern. Part B Cybern., 41(1), 196-209.
12. Castelletti, A., de Rigo, D., Rizzoli, A., Soncini-Sessa, R., and Weber, E. (2007). "Neuro-dynamic programming for designing water reservoir network management policies." Control Eng. Pract., 15(8), 1031-1038.
13. Castelletti, A., Galelli, S., Restelli, M., and Soncini-Sessa, R. (2010). "Tree-based reinforcement learning for optimal water reservoir operation." Water Resour. Res., 46(9), W09507.
14. Castelletti, A., Pianosi, F., Quach, X., and Soncini-Sessa, R. (2012). "Assessing water reservoirs management and development in northern Vietnam." Hydrol. Earth Syst. Sci., 16(1), 189-199.
15. Castelletti, A., Pianosi, F., and Restelli, M. (2013). "A multiobjective reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single run." Water Resour. Res., 49, 3476-3486.
16. Castelletti, A., Pianosi, F., and Soncini-Sessa, R. (2008). "Water reservoir control under economic, social and environmental constraints." Automatica, 44(6), 1595-1607.
17. Castelletti, A., Pianosi, F., and Soncini-Sessa, R. (2012a). "Stochastic and robust control of water resource systems: Concepts, methods and applications." System identification, environmental modelling, and control system design, Springer, London, 383-401.
18. Castelletti, A., Pianosi, F., Quach, X., and Soncini-Sessa, R. (2012b). "Assessing water reservoirs management and development in northern Vietnam." Hydrol. Earth Syst. Sci., 16(1), 189-199.
19. Celeste, A., and Billib, M. (2009). "Evaluation of stochastic reservoir operation optimization models." Adv. Water Resour., 32(9), 1429-1443.
20. Chankong, V., and Haimes, Y. (1983). Multiobjective decision making: Theory and methodology, North-Holland, New York.
21. Chen, T., and Chen, H. (1995). "Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems." IEEE Trans. Neural Networks, 6(4), 911-917.
22. Clark, E. (1950). "New York control curves." J. Am. Water Works Assoc., 42(9), 823-827.
23. Clark, E. (1956). "Impounding reservoirs." J. Am. Water Works Assoc., 48(4), 349-354.
24. Coello Coello, C., Lamont, G., and Veldhuizen, D. V. (2007). Evolutionary algorithms for solving multiobjective problems (genetic algorithms and evolutionary computation), 2nd Ed., Springer, New York.
25. Cohon, J. L., and Marks, D. (1975). "A review and evaluation of multiobjective programing techniques." Water Resour. Res., 11(2), 208-220.
26. Cui, L., and Kuczera, G. (2005). "Optimizing water supply headworks operating rules under stochastic inputs: Assessment of genetic algorithm performance." Water Resour. Res., 41, W05016.
27. Cybenko, G. (1989). "Approximation by superpositions of a sigmoidal function." Math. Control Signals Syst., 2(4), 303-314.
28. Dariane, A., and Momtahen, S. (2009). "Optimization of multireservoir systems operation using modified direct search genetic algorithm." J. Water Resour. Plann. Manage., 10.1061/(ASCE)0733-9496(2009)135:3(141), 141-148.
29. Deb, K. (2001). Multiobjective optimization using evolutionary algorithms, Wiley, Hoboken, NJ.
30. Deisenroth, M., Neumann, G., and Peters, J. (2011). "A survey on policy search for robotics." Found. Trends Rob., 2(1-2), 1-142.
31. de Rigo, D., Castelletti, A., Rizzoli, A., Soncini-Sessa, R., and Weber, E. (2005). "A selective improvement technique for fastening neurodynamic programming in water resources network management." Proc., 16th IFAC World Congress, Prague, Czech Republic.
32. Desreumaux, Q., Côté, P., and Leconte, R. (2014). "Role of hydrologic information in stochastic dynamic programming: A case study of the Kemano hydropower system in British Columbia." Can. J. Civ. Eng., 41(9), 839-844.
33. Draper, A., and Lund, J. (2004). "Optimal hedging and carryover storage value." J. Water Resour. Plann. Manage., 10.1061/(ASCE)0733-9496 (2004)130:1(83), 83-87.
34. Esogbue, A. (1989). "Dynamic programming and water resources: Origins and interconnections." Dynamic programming for optimal water resources systems analysis, Prentice-Hall, Englewood Cliffs, NJ.
35. Faber, B., and Stedinger, J. (2001). "Reservoir optimization using sampling SDP with ensemble streamflow prediction (ESP) forecasts." J. Hydrol., 249(1), 113-133.
36. Fleming, P., Purshouse, R., and Lygoe, R. (2005). "Many-objective optimization: An engineering design perspective." Proc., 3rd Int. Conf. on Evolutionary Multi-Criterion Optimization, Guanajuato, Mexico, 14-32.
37. Funahashi, K. (1989). "On the approximate realization of continuous mappings by neural networks." Neural Networks, 2(3), 183-192.
38. Gaggero, M., Gnecco, G., and Sanguineti, M. (2014). "Suboptimal policies for stochastic N-stage optimization: Accuracy analysis and a case study from optimal consumption." Models and methods in economics and management science, F. E. Ouardighi and K. Kogan, eds., Springer, New York, 27-50.
39. Gass, S., and Saaty, T. (1955). "Parametric objective function-Part II." Oper. Res., 3, 316-319.
40. Giuliani, M., Galelli, S., and Soncini-Sessa, R. (2014a). "A dimensionality reduction approach for many-objective Markov decision processes: Application to a water reservoir operation problem." Environ. Modell. Software, 57, 101-114.
41. Giuliani, M., Herman, J., Castelletti, A., and Reed, P. (2014b). "Manyobjective reservoir policy identification and refinement to reduce policy inertia and myopia in water management." Water Resour. Res., 50(4), 3355-3377.
42. Giuliani, M., Mason, E., Castelletti, A., Pianosi, F., and Soncini-Sessa, R. (2014c). "Universal approximators for direct policy search in multipurpose water reservoir management: A comparative analysis." Proc., 19th IFAC World Congress, Cape Town, South Africa.
43. Gleick, P., and Palaniappan, M. (2010). "Peak water limits to freshwater withdrawal and use." Proc. Natl. Acad. Sci. U.S.A., 107(25), 11155-11162.
44. Guariso, G., Rinaldi, S., and Soncini-Sessa, R. (1986). "The management of Lake Como: A multiobjective analysis." Water Resour. Res., 22(2), 109-120.
45. Guo, X., Hu, T., Zeng, X., and Li, X. (2013). "Extension of parametric rule with the hedging rule for managing multireservoir system during droughts." J. Water Resour. Plann. Manage., 10.1061/(ASCE)WR.1943-5452.0000241, 139-148.
46. Hadka, D., and Reed, P. (2012). "Diagnostic assessment of search controls and failure modes in many-objective evolutionary optimization." Evol. Comput., 20(3), 423-452.
47. Hadka, D., and Reed, P. (2013). "Borg: An auto-adaptive manyobjective evolutionary computing framework." Evol. Comput., 21(2), 231-259.
48. Haimes, Y., Lasdon, L., and Wismer, D. (1971). "On a bicriterion formulation of the problems of integrated system identification and system optimization." IEEE Trans. Syst. Man Cybern., 1(3), 296-297.
49. Hall, W., and Buras, N. (1961). "The dynamic programming approach to water-resources development." J. Geophys. Res., 66(2), 517-520.
50. Heidrich-Meisner, V., and Igel, C. (2008). "Variable metric reinforcement learning methods applied to the noisy mountain car problem." Recent advances in reinforcement learning, Springer, Berlin, 136-150.
51. Hejazi, M., and Cai, X. (2009). "Input variable selection for water resources systems using a modified minimum redundancy maximum relevance (mMRMR) algorithm." Adv. Water Resour., 32(4), 582-593.
52. Herman, J., Reed, P., Zeff, H., and Characklis, G. (2015). "How should robustness be defined for water systems planning under change?" J. Water Resour. Plann. Manage., 10.1061/(ASCE)WR.1943-5452.0000509, 04015012.
53. Hornik, K., Stinchcombe, M., and White, H. (1989). "Multilayer feedforward networks are universal approximators." Neural Networks, 2(5), 359-366.
54. Kasprzyk, J., Reed, P., Kirsch, B., and Characklis, G. (2009). "Managing population and drought risks using many-objective water portfolio planning under uncertainty." Water Resour. Res., 45(12).
55. Knowles, J., and Corne, D. (2002). "On metrics for comparing non-dominated sets." Proc., 2002 World Congress on Computational Intelligence (WCCI), IEEE Computer Society, 711-716.
56. Koutsoyiannis, D., and Economou, A. (2003). "Evaluation of the parameterization-simulation-optimization approach for the control of reservoir systems." Water Resour. Res., 39(6).
57. Labadie, J. (2004). "Optimal operation of multireservoir systems: State-of-the-art review." J. Water Resour. Plann. Manage., 10.1061/(ASCE)0733-9496(2004)130:2(93), 93-111.
58. Loucks, D., and Sigvaldason, O. (1982). "Multiple-reservoir operation in North America." The operation of multiple reservoir systems, Z. Kaczmarck and J. Kindler, eds., IIASA Collaborative Proceedings Series, 1-103.
59. Loucks, D., van Beek, E., Stedinger, J., Dijkman, J., and Villars, M. (2005). Water resources systems planning and management: An introduction to methods, models and applications, UNESCO, Paris.
60. Lund, J., and Guzman, J. (1999). "Derived operating rules for reservoirs in series or in parallel." J. Water Resour. Plann. Manage., 10.1061/(ASCE)0733-9496(1999)125:3(143), 143-153.
61. Maass, A., Hufschmidt, M., Dorfman, R., Thomas, H., Jr., Marglin, S., and Fair, G. (1962). Design of water-resource systems, Harvard University Press, Cambridge, MA.
62. Maier, H., et al. (2014). "Evolutionary algorithms and other metaheuristics in water resources: Current status, research challenges and future directions." Environ. Modell. Software, 62, 271-299.
63. Maier, H., and Dandy, G. (2000). "Neural networks for the prediction and forecasting of water resources variables: A review of modelling issues and applications." Environ. Modell. Software, 15(1), 101-124.
64. Marbach, P., and Tsitsiklis, J. (2001). "Simulation-based optimization of Markov reward processes." IEEE Trans. Autom. Control, 46(2), 191-209.
65. Mayne, D. Q., Rawlings, J. B., Rao, C. V., and Scokaert, P. O. (2000). "Constrained model predictive control: Stability and optimality." Automatica, 36(6), 789-814.
66. McDonald, R. I., et al. (2011). "Urban growth, climate change, and freshwater availability." Proc. Natl. Acad. Sci., 108(15), 6312-6317.
67. Mhaskar, H., and Micchelli, C. (1992). "Approximation by superposition of sigmoidal and radial basis functions." Adv. Appl. Math., 13(3), 350-373.
68. Momtahen, S., and Dariane, A. (2007). "Direct search approaches using genetic algorithms for optimization of water reservoir operating policies." J. Water Resour. Plann. Manage., 10.1061/(ASCE)0733-9496 (2007)133:3(202), 202-209.
69. Moriarty, D., Schultz, A., and Grefenstette, J. (1999). "Evolutionary algorithms for reinforcement learning." J. Artif. Intell. Res., 11, 199-229.
70. Nalbantis, I., and Koutsoyiannis, D. (1997). "A parametric rule for planning and management of multiple-reservoir systems." Water Resour. Res., 33(9), 2165-2177.
71. Nicklow, J., et al. (2010). "State of the art for genetic algorithms and beyond in water resources planning and management." J. Water Resour. Plann. Manage., 10.1061/(ASCE)WR.1943-5452.0000053, 412-432.
72. Oliveira, R., and Loucks, D. P. (1997). "Operating rules for multi reservoir systems." Water Resour. Res., 33(4), 839-852.
73. Park, J., and Sandberg, I. (1991). "Universal approximation using radial-basis-function networks." Neural Comput., 3(2), 246-257.
74. Peters, J., and Schaal, S. (2008). "Reinforcement learning of motor skills with policy gradients." Neural Networks, 21(4), 682-697.
75. Pianosi, F., Quach, X., and Soncini-Sessa, R. (2011). "Artificial neural networks and multi objective genetic algorithms for water resources management: An application to the Hoa Binh reservoir in Vietnam." Proc., 18th IFAC World Congress, Milan, Italy.
76. Piccardi, C., and Soncini-Sessa, R. (1991). "Stochastic dynamic programming for reservoir optimal control: Dense discretization and inflow correlation assumption made possible by parallel computing." Water Resour. Res., 27(5), 729-741.
77. Powell,W. (2007). Approximate dynamic programming: Solving the curses of dimensionality, Wiley, Hoboken, NJ.
78. Reed, P., Hadka, D., Herman, J., Kasprzyk, J., and Kollat, J. (2013). "Evolutionary multiobjective optimization in water resources: The past, present, and future." Adv. Water Resour., 51, 438-456.
79. Reed, P. M., and Kollat, J. B. (2013). "Visual analytics clarify the scalability and effectiveness of massively parallel many-objective optimization: A groundwater monitoring design example." Adv. Water Resour., 56, 1-13.
80. ReVelle, C., and McGarity, A. E. (1997). Design and operation of civil and environmental engineering systems, Wiley, Hoboken, NJ.
81. Rippl, W. (1883). "The capacity of storage reservoirs for water supply." Minutes Proc., 71, 270-278.
82. Rosenstein, M., and Barto, A. (2001). "Robot weightlifting by direct policy search." Int. Joint Conf. on Artificial Intelligence, Citeseer, 839-846.
83. Sehnke, F., Osendorfer, C., Rückstieß, T., Graves, A., Peters, J., and Schmidhuber, J. (2010). "Parameter-exploring policy gradients." Neural Networks, 23(4), 551-559.
84. Soncini-Sessa, R., Castelletti, A., and Weber, E. (2007). Integrated and participatory water resources management: Theory, Elsevier, Amsterdam, Netherlands.
85. Sutton, R., McAllester, D., Singh, S., and Mansour, Y. (2000). "Policy gradient methods for reinforcement learning with function approximation." Adv. Neural Inf. Process. Syst., 12, 1057-1063.
86. Tejada-Guibert, J., Johnson, S., and Stedinger, J. (1995). "The value of hydrologic information in stochastic dynamic programming models of a multireservoir system." Water Resour. Res., 31(10), 2571-2579.
87. Tikk, D., Kóczy, L., and Gedeon, T. (2003). "A survey on universal approximation and its limits in soft computing techniques." Int. J. Approximate Reasoning, 33(2), 185-202.
88. Tsitsiklis, J., and Van Roy, B. (1996). "Feature-based methods for large scale dynamic programming." Mach. Learn., 22, 59-94.
89. Tu, M., Hsu, N., and Yeh, W. (2003). "Optimization of reservoir management and operation with hedging rules." J. Water Resour. Plann. Manage., 10.1061/(ASCE)0733-9496(2003)129:2(86), 86-97.
90. U.S. Army Corps of Engineers. (1977). "Reservoir system analysis for conservation, hydrologic engineering methods for water resources development." Hydrologic Engineering Center, Davis, CA.
91. Whiteson, S., and Stone, P. (2006). "Evolutionary function approximation for reinforcement learning." J. Mach. Learn. Res., 7, 877-917.
92. Whitley, D., Dominic, S., Das, R., and Anderson, C. (1994). Genetic reinforcement learning for neurocontrol problems, Springer, New York.
93. Woodruff, M., Reed, P., and Simpson, T. (2013). "Many objective visual analytics: Rethinking the design of complex engineered systems." Struct. Multidiscip. Optim., 48(1), 201-219.
94. Yeh, W. (1985). "Reservoir management and operations models: A state of the art review." Water Resour. Res., 21(12), 1797-1818.
95. Zitzler, E., Thiele, L., Laumanns, M., Fonseca, C., and da Fonseca, V. (2003). "Performance assessment of multiobjective optimizers: An analysis and review." IEEE Trans. Evol. Comput., 7(2), 117-132.
96. Zoppoli, R., Sanguineti, M., and Parisini, T. (2002). "Approximating networks and extended ritz method for the solution of functional optimization problems." J. Optim. Theory Appl., 112(2), 403-440.