Confronting Deep Uncertainties in Risk Analysis

Risk Analysis

Volumn 32, Issue 10, 2012, Pages 1607-1629

  Cox, Louis Anthony ^a  

^a UNIVERSITY OF COLORADO   (United States)

Author keywords

AdaBoost; Deep uncertainty; Low regret online decisions; Markov decision process; Model ensemble methods; POMDP; Reinforcement learning; Robust decision making; Robust optimization; Robust risk analysis; SARSA

Indexed keywords

DEEP UNCERTAINTY; ENSEMBLE METHODS; MARKOV DECISION PROCESSES; ONLINE DECISIONS; POMDP; ROBUST DECISIONS; ROBUST OPTIMIZATION; SARSA;

ADAPTIVE BOOSTING; CLIMATE CHANGE; DECISION MAKING; MARKOV PROCESSES; REINFORCEMENT LEARNING; UNCERTAINTY ANALYSIS;

RISK MANAGEMENT;

ALGORITHM; ARTIFICIAL INTELLIGENCE; CLIMATE CHANGE; DECISION MAKING; DISEASE CONTROL; LEARNING; MARKOV CHAIN; OPTIMIZATION; POLICY MAKING; PREDICTION; PROBABILITY; RISK ASSESSMENT; SENSITIVITY ANALYSIS; UNCERTAINTY ANALYSIS;

ANIMAL; ARTICLE; BAYES THEOREM; CLIMATE CHANGE; DECISION MAKING; DECISION THEORY; ECOSYSTEM; FISHERY; FORECASTING; HUMAN; INFECTION CONTROL; LEARNING; PROBABILITY; RENEWABLE ENERGY; RISK; RISK MANAGEMENT; THEORETICAL MODEL; UNCERTAINTY;

ANIMALS; BAYES THEOREM; CLIMATE CHANGE; DECISION MAKING; DECISION THEORY; ECOSYSTEM; FISHERIES; FORECASTING; HUMANS; INFECTION CONTROL; LEARNING; MARKOV CHAINS; MODELS, THEORETICAL; PROBABILITY; RENEWABLE ENERGY; RISK; RISK MANAGEMENT; UNCERTAINTY;

EID: 84866990641     PISSN: 02724332     EISSN: 15396924     Source Type: Journal    
DOI: 10.1111/j.1539-6924.2012.01792.x     Document Type: Article

References (93)

1. Alagoz O, Hsu H, Schaefer AJ, Roberts MS. Markov decision processes: A tool for sequential decision making under uncertainty. Medical Decision Making, 2010; 30(4):474-483.
2. Balaji PG, German X, Srinivasan D. Urban traffic signal control using reinforcement learning agents. Intelligent Transport Systems, 2010; 4(3):177-188.
3. Ben-Haim Y. Information-Gap Decision Theory. San Diego, CA : Academic Press, 2001.
4. Ben-Tal A, Bertsimas D, Brown DB A soft robust model for optimization under ambiguity. Operations Research, 2010; 58(4, Part 2 of 2): 1220-1234.
5. Ben-Tal A, El Ghaoui L, Nemirovski A. Robust Optimization. Princeton, NJ : Princeton University Press, 2009.
6. Bertsimas D, Brown DB, Caramanis C. Theory and applications of robust optimization. SIAM Review, 2011; 53(3):464-501.
7. Bertsimas D, Brown DB Constructing uncertainty sets for robust linear optimization. Operations Research, 2009; 57(6):1483-1495.
8. Blum A, Mansour Y. From external to internal regret. Journal of Machine Learning Research, 2007; 8:1307-1324.
9. Bolton RJ, Hand DJ. Statistical fraud detection: A review. Statistical Science, 1999; 17(3):235-255.
10. Bryant B, Lempert RJ. Thinking inside the box: A participatory, computer assisted approach to scenario discovery. Technology Forecasting and Social Change, 2010; 77(1):34-49.
11. Buckley JJ. Stochastic dominance: An approach to decision making under risk. Risk Analysis, 1986; 6(1):35-41.
12. Burton R. On Being Certain: Believing You Are Right Even When You're Not. New York, St. Martin's Press, 2008.
13. Busoniu L, Babuska R, Schutter BD. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics Part C: Applications and Reviews, 2008; 38(2): 156-172.
14. Cai C, Liao X, Cari L. Learning to explore and exploit in POMDPs. Pp. 198-206 in Advances in Neural Information Processing Systems 22, 2009. Available at: http://people.ee.duke.edu/~lcarin/LearnE2_NIPS09_22_FINAL.pdf, Accessed January 8, 2012.
15. Carpenter TE, O'Brien JM, Hagerman AD, McCarl BA. Epidemic and economic impacts of delayed detection of foot-and-mouth disease: A case study of a simulated outbreak in California. Journal of Veterinary Diagnostic Investigation, 2011; 23(1):26-33. Available at: http://www.ncbi.nlm.nih.gov/pubmed/21217024, Accessed January 8, 2012.
16. Cesa-Bianchi N, Lugosi, G. Prediction, Learning, and Games. New York : Cambridge University Press, 2006.
17. Chades I, Bouteiller B. Solving multiagent Markov decision processes: A forest management example. In MODSIM 2005 International Congress on Modelling and Simulation. Modelling: Simulation Society of Australia: New Zealand, 2005.
18. Chen Y, Chen Y. Combining incremental hidden Markov model and Adaboost algorithm for anomaly intrusion detection. In Proceedings of the ACM SIGKDD Workshop on Cybersecurity and Intelligence Informatics (Paris, France, June 28 - 28, 2009). Pp. 3-9 in Chen H, Dacier M, Moens M, Paass G, Yang C C (eds). CSI-KDD '09. ACM, New York, 2009. DOI: http://doi.acm.org/10.1145/1599272.1599276.
19. Churchman CW. Wicked problems. Management Science, 1967; 14(4):B141-B142.
20. Condorcet, NC de. Essai sur l'Application de l'Analyse a la Probabilite des Decisions Rendues a la Pluralite des voix. Paris, 1785.
21. Cortés EA, Gámez M, Rubio NG. Multiclass corporate failure prediction by Adaboost.M1. International Advances in Economic Research, 2007; 13(3):301-312.
22. Dalamagkidis D, Kolokotsa D, Kalaitzakis K, Stavrakakis GS. Reinforcement learning for energy conservation and comfort in buildings. Building and Environment, 2007; 42:2686-2698. Available at: http://www.tuc.gr/fileadmin/users_data/elci/Kalaitzakis/J.38.pdf, Accessed January 8, 2012.
23. Das TK, Savachkin AA, Zhu Y. A large scale simulation model of pandemic influenza outbreaks for development of dynamic mitigation strategies. IIE Transactions, 2007; 40(9):893-905. Available at: http://www.eng.usf.edu/~das/papers/das_r1.pdf, Accessed January 8, 2012.
24. Dickens L, Broda K, Russo A. The dynamics of multi-agent reinforcement learning. In Frontiers in Artificial Intelligence and Applications Volume 2010; 215: Proceedings of the 2010 conference on ECAI 2010: 19th European Conference on Artificial Intelligence, Edited by Helder Coelho, Rudi Studer, Michael Wooldridge. Available at: http://www.doc.ic.ac.uk/~lwd03/ecai2010.pdf.
25. Ermon S, Conrad J, Gomes C, Seman B. Risk-sensitive policies for sustainable renewable resource allocation. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI), July 2011. Available at: http://ijcai.org/papers11/contents.php, Accessed January 8, 2012.
26. Ernst D, Stan G-B, Gongalves J, Wehenkel L. Clinical data based optimal STI strategies for HIV: A reinforcement learning approach. Pp. 667-672 in Proceedings of the 45th IEEE Conference on Decision and Control. San Diego, CA, December 13-15, 2006. Available at: http://www.montefiore.ulg.ac.be/~stan/CDC_2006.pdf, Accessed January 8, 2012.
27. Fan W, Stolfo S, Zhang J, Chan P. Adacost: Misclassification cost-sensitive boosting. Pp. 97-105 in Proceedings of the 16th International Conference on Machine Learning Bled, Slovenia, 1999.
28. Fiege J, McCurdy B, Potrebko P, Champion H, Cull A. PARETO: A novel evolutionary optimization approach to multiobjective IMRTs planning. Medical Physics, 2011; 38(9):5217-5229.
29. Forsell, Garcia F, Sabbadin R. Reinforcement learning for spatial processes. World IMACS/MODSIM Congress, Cairns, July 13-17, 2009, Australia Available at: http://www.mssanz.org.au/modsim09/C1/forsell.pdf, Accessed January 8, 2012.
30. Fredriksson A, Forsgren A, Hårdemark B. Minimax optimization for handling range and setup uncertainties in proton therapy. Medical Physics, 2011; 38(3):1672-1684.
31. Ge L, Mourits MC, Kristensen AR, Huirne RB. A modelling approach to support dynamic decision-making in the control of FMD epidemics. Preventive Veterinary Medicine, 2010; 95(3-4):167-174. Available at: http://www.ncbi.nlm.nih.gov/pubmed/20471708s, Accessed January 8, 2012.
32. Geibel P, Wysotzk F. Risk-sensitive reinforcement learning applied to control under constraint. Journal of Artificial Intelligence Research, 2005; 24:81-108.
33. Gilboa I, Schmeidler D. Maxmin expected utility with a non-unique prior. Journal of Mathematical Economics, 1989; 18:141-153.
34. Green CS, Benson C, Kersten D, Schrater P. Alterations in choice behavior by manipulations of world model. Proceedings of the National Academy of Sciences of the United States of America, 2010; 107(37):16401-16406
35. Gregoire PL, Desjardins C, Laumonier J, Chaib-draa B. Urban traffic control based on learning agents. In Intelligent Transportation Systems Conference, 2007. ITSC 2007 IEEE: 916-921. Seattle, WA DOI: 10.1109/ITSC.2007.4357719.
36. Hauskrecht M, Fraser H. Planning treatment of ischemic heart disease with partially observable Markov decision processes. Artificial Intelligence in Medicine, 2000; 18(3):221-244. Available at: http://www.ncbi.nlm.nih.gov/pubmed/10675716, http://veryoldwww.cs.pitt.edu/~milos/research/AIMJ-2000.pdf, Accessed January 8, 2012.
37. Hansen LP, Sargent TJ. Robust control and model uncertainty. American Economic Review, 2001; 91:60-66.
38. Hansen LP, Sargent TJ. Robustness. Princeton, NJ ; University Press Princeton, 2008.
39. Harford T. Adapt: Why Success Always Starts with Failure. New York : Farra, Straus and Giroux, 2011.
40. Hazen E, Seshadhri C. Efficient learning algorithms for changing environments. In ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning. Available at: http://ie.technion.ac.il/~ehazan/papers/adap-icml2009.pdf, 2007, Accessed January 8, 2012.
41. Hoeting JA, Madigan D, Raftery AE, Volinsky CT. Bayesian model averaging: A tutorial. Statistical Science, 1999; 14(4):382-401. Available at: http://mpdc.mae.cornell.edu/Courses/UQ/2676803.pdf, Accessed January 8, 2012.
42. Hrdlicka J, Klema J. Schizophrenia prediction with the adaboost algorithm. Studies in Health Technology and Informatics, 2011; 169:574-578.
43. Hu W, Hu W, Maybank S. AdaBoost-based algorithm for network intrusion detection. IEEE Transactions on Systems, Man, and Cybernetics-Part B, 2008; 38(2):577-583.
44. Hutter M, Poland J. Adaptive online prediction by following the perturbed leader. Journal of Machine Learning Research, 2005; 6:639-660. Available at: http://jmlr.csail.mit.edu/papers/volume6/hutter05a/hutter05a.pdf, Accessed January 8, 2012.
45. Inaniwa T, Kanematsu N, Furukawa T, Hasegawa A. A robust algorithm of intensity modulated proton therapy for critical tissue sparing and target coverage. Physics in Medicine and Biology, 2011; 56(15):4749-4770. Available at: http://www.ncbi.nlm.nih.gov/pubmed/21753233, Accessed January 8, 2012.
46. Itoh H, Nakamura K Partially observable Markov decision processes with imprecise parameters. Artificial Intelligence, 2007; 171(8-9):453-490.
47. Izadi MT, Buckeridge DL. Optimizing anthrax outbreak detection using reinforcement learning. In IAAI'07 Proceedings of the 19th National Conference on Innovative Applications of Artificial Intelligence, Volume 2. AAAI Press. Available at: http://www.aaai.org/Papers/AAAI/2007/AAAI07--286.pdf, Accessed January 8, 2012.
48. Jaksch T, Ortner R, Auer P. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 2010; 11:1563-1600.
49. Jung J, Liu CC, Tanimoto S, Vittal V. Adaptation in load shedding under vulnerable operating conditions. IEEE Transactions on Power Systems, 2002; 17:1199-1205.
50. Kahnt T, Park SQ, Cohen MX, Beck A, Heinz A, Wrase J. Dorsal striatal-midbrain connectivity in humans predicts how reinforcements are used to guide decisions. Journal of Cognitive Neuroscience, 2009; 21(7):1332-1345.
51. Kaplan S, Garrick, B J. On the quantitative definition of risk. Risk Analysis, 1981; 1(1):11-27. Available at: http://josiah.berkeley.edu/2007Fall/NE275/CourseReader/3.pdf
52. Koop G, Tole L. Measuring the health effects of air pollution: To what extent can we really say that people are dying from bad air? Journal of Environmental Economics and Management, 2004; 47:30-54. Available at: http://citeseerx.ist.psu.edu/viewdoc/summary?DOI: 10.1.1.164.6048, Accessed January 8, 2012.
53. Kuyer L, Whiteson S, Bakker B, Vlassis N. Multiagent reinforcement learning for urban traffic control using coordination graphs. Pp. 656-671 in ECML 2008: Proceedings of the Nineteenth European Conference on Machine Learning, Antwe. p. Belgium, 2008.
54. Laeven R, Stadje MA. Entropy coherent and entropy convex measures of risk. Tilburg University Center Discussion Paper, 2011--031. Available at: http://arno.uvt.nl/show.cgi?fid=114115, Accessed January 8, 2012.
55. Lee EK, Chen CH, Pietz F, Benecke B. Disease propagation analysis and mitigation strategies for effective mass dispensing. AMIA Annual Symposium Proceedings, 2010; 2010:427-431. Available at: http://www.ncbi.nlm.nih.gov/pubmed/21347014.
56. Lempert RJ, Collins MT. Managing the risk of uncertain threshold response: Comparison of robust, optimum, and precautionary approaches. Risk Analysis, 2007; 27(4):1009-1026.
57. Lempert R, Kalra N. Managing Climate Risks in Developing Countries with Robust Decision Making. Washington, DC : World Resources Report, 2008. Available at: http://www.worldresourcesreport.org/files/wrr/papers/wrr_lempert_and_kalra_uncertainty.pdf, Accessed January 8, 2012.
58. Lizotte DJ, Gunter L, Laber E, Murphy SA. Missing data and uncertainty in batch reinforcement learning, NIPS-08 Workshop on Model Uncertainty and Risk in RL, 2008. Available at: http://www.cs.uwaterloo.ca/~ppoupart/nips08-workshop/nips08-workshop-schedule.html, Accessed January 8, 2012.
59. Lu F, Boritz JE, Covvey HD. Adaptive fraud detection using Benford's law. In Advances in Artificial Intelligence: 19th Conference of the Canadian Society for Computational Studies of Intelligence, 2006. Available at: http://bit.csc.lsu.edu/~jianhua/petrov.pdf, Accessed January 8, 2012.
60. Maccheroni F, Marinacci M, Rustichini A. Ambiguity aversion, robustness, and the variational representation of preferences. Econometrica, 2006; 74:1447-1498.
61. Makridakis S, Hibon M. The M3-Competition: Results, conclusions and implications. International Journal of Forecasting, 2000; 16:451-476. Available at: http://www.forecastingprinciples.com/files/pdf/Makridakia-The%20M3%20Competition.pdf, Accessed January 8, 2012.
62. Marchau VAWJ, Walker WE, van Wee GP. Dynamic adaptive transport policies for handling deep uncertainty. Technological Forecasting and Social Change, 2010; 77(6):940-950.
63. Masnadi-Shirazi H, Vasconcelos N. Asymmetric boosting. Pp. 609-619 in Proceedings of the 24th International Conference on Machine Learning, Corvallis, OR, 2007.
64. McDonald-Madden E, Chadès I, McCarthy MA, Linkie M, Possingham HP. Allocating conservation resources between areas where persistence of a species is uncertain. Journal of Applied Ecology, 2011; 21(3):844-858. Available at: http://www.ncbi.nlm.nih.gov/pubmed/21639049, Accessed January 8, 2012.
65. Molinaro AM, Simon R, Pfeiffer RM. Prediction error estimation: A comparison of resampling methods. Bioinformatics, 2005; 21(15):3301-3307.
66. Morra JH, Tu Z, Apostolova LG, Green AE, Toga AW, Thompson PM. Comparison of AdaBoost and support vector machines for detecting Alzheimer's disease through automated hippocampal segmentation. IEEE Transactions on Medical Imaging, 2010; 29(1):30-43.
67. Ni Y, Liu Z-Q. Bounded-parameter partially observable Markov decision processes. In Proceedings of the Eighteenth International Conference on Automated Planning and Scheduling, 2008. Available at: http://www.aaai.org/Press/Proceedings/icaps08.php, Accessed January 8, 2012.
68. Niua B, Jinb Y, Lua WC, Li GZ. Predicting toxic action mechanisms of phenols using AdaBoost Learner. Chemometrics and Intelligent Laboratory Systems, 2009; 96(1):43-48.
69. Osada H, Fujita S. CHQ: A multi-agent reinforcement learning scheme IEICE Transactions on Information and Systems, 2005; 88-D(s): 1004-1011.
70. Perkins TJ, Barto AG. Lyapunov design for safe reinforcement learning. Journal of Machine Learning Research, 2002; 3:803-883. Available at: http://jmlr.csail.mit.edu/papers/volume3/perkins02a/perkins02a.pdf, Accessed January 8, 2012.
71. Regan K, Boutilier C. Regret-based reward elicitation for Markov decision processes. NIPS-08 Workshop on Model Uncertainty and Risk in RL. Available at: http://www.cs.uwaterloo.ca/~ppoupart/nips08-workshop/nips08-workshop-schedule.html, 2008, Accessed January 8, 2012.
72. Rittel H, Webber M. (1973). Dilemmas in a general theory of planning. Policy Sciences 4:155-169. [Reprinted in N. Cross (ed). Developments in Design Methodology, J. Wiley & Sons, Chichester, 1984] Available at: http://www.uctc.net/mwebber/RittelWebberDilemmasGeneral_Theory_of_Planning.pdf, Accessed January 8, 2012.
73. Ross S, Pineau J, Chaib-draa B, Kreitmann P. POMDPs: A new perspective on the explore-exploit tradeoff in partially observable domains. Journal of Machine Learning Research, 2011; 12:1729-1770.
74. Savio A, García-Sebastián M, Graña M, Villanúa J. Results of an Adaboost approach on Alzheimer's disease detection on MRI. Bioinspired Applications in Artificial and Natural Computation Lecture Notes in Computer Science, 2009, 5602:2009;114-123. Available at: http://www.ehu.es/ccwintco/uploads/1/11/GarciaSebastianSavio-VBM_SPM_SVM-IWINAC2009_v2.pdf, Accessed January 8, 2012.
75. Schaefer AJ, Bailey MD, Shechter SM, Roberts MS. Modeling medical treatment using Markov decision processes. Pp. 593-612 in Handbook of Operations Research/Management Science Applications in Health Care. Boston, MA : Kluwer Academic Publishers, 2004. Available at: http://www.ie.pitt.edu/~schaefer/Papers/MDPMedTreatment.pdf, Accessed January 8, 2012.
76. Schönberg T, Daw ND, Joel D, O'Doherty JP. Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. Journal of Neuroscience, 2007; 27(47):12860-12867.
77. Srinivasan J, Gadgil S. Asian brown cloud-Fact and fantasy. Current Science, 2002; 83:586-592.
78. Su Q, Lu W, Niu B, Liu X. Classification of the toxicity of some organic compounds to tadpoles (Rana Temporaria) through integrating multiple classifiers. Molecular Informatics, 2011; 30(8):672-675.
79. Sutton RS, Barto AG. Reinforcement Learning: An Introduction. Cambridge, MA : MIT Press, 2005. Available at: http://rlai.cs.ualberta.ca/~sutton/book/ebook/the-book.html, Accessed January 8, 2012.
80. Svetnik V, Wang T, Tong C, Liaw A, Sheridan RP, Song Q. Boosting: An ensemble learning tool for compound classification and QSAR modeling. Journal of Chemical Information and Modeling, 2005; 45(3):786-799. Available at: http://www.ncbi.nlm.nih.gov/pubmed/15921468, Accessed January 8, 2012.
81. Szepesvari C. Reinforcement Learning Algorithms. Morgan & Claypool Publishers, 2010.
82. Tan C, Chen H, Xia C. Early prediction of lung cancer based on the combination of trace element analysis in urine and an Adaboost algorithm. Journal of Pharmaceutical and Biomedical Analysis, 2009; 49(3):746-752.
83. Walker WE, Marchau VAWJ, Swanson D. Addressing deep uncertainty using adaptive policies introduction to section 2. Technological Forecasting and Social Change, 2010; 77(6):917-923.
84. Waltman L, van Eck NJ. Robust evolutionary algorithm design for socio-economic simulation: Some comments. Computational Economics, 2009; 33:103-105. Available at: http://repub.eur.nl/res/pub/18660/RobustEvolutionary_2008.pdf, Accessed January 8, 2012.
85. Wang Y, Xie Q, Ammari A. Deriving a near-optimal power management policy using model-free reinforcement learning and Bayesian classification DAC '11 Proceedings of the 48th Design Automation Conference. New York : ACM, 2011.
86. Weick KE, Sutcliffe KM. Managing the Unexpected: Resilient Performance in an Age of Uncertainty, 2nd ed. New York: John Wiley & Sons, 2007.
87. Xu X, Sun Y, Huang Z. Defending DDoS attacks using hidden Markov models and cooperative reinforcement learning. In Proceedings, PAISI'07 Proceedings of the 2007 Pacific Asia Conference on Intelligence and Security Informatics. Heidelberg, Germany : Springer-Verlag Berlin, 2007.
88. Yousefpour R, Hanewinkel M. Modelling of forest conversion planning with an adaptive simulation-optimization approach and simultaneous consideration of the values of timber, carbon and biodiversity. Ecological Economics, 2009; 68(6):1711-1722.
89. Ye D, Zhang M, Sutato D. A hybrid multiagent framework with Q-learning for power grid systems restoration. IEEE Transactions on Power Systems, 2011; 26(4):2434-2441.
90. Yu JY, Mannor S, Shimkin N). Markov decision processes with arbitrary reward processes. Mathematics of Operations Research, 2009; 34(3):737-757.
91. Zhao Y, Kosorok MR, Zeng D. Reinforcement learning design for cancer clinical trials. Statistical Medicine, 2009; 28(26):3294-3315. Available at: http://www.ncbi.nlm.nih.gov/pubmed/19750510, Accessed January 8, 2012.
92. Zhou B, Chan KW, Yu T. Q-Learning Approach for Hierarchical AGC Scheme of Interconnected Power Grids Energy Procedia, 2011; 12: 43-52.
93. Zhou L, Kin Keung Lai. Adaboosting neural networks for credit scoring. Pp. 875-884 in Advances in Intelligent and Soft Computing, Vol. 56, 2009. DOI: 10.1007/978-3-642-01216-7_93.