Context matters: Using reinforcement learning to develop human-readable, state-dependent outbreak response policies

Philosophical Transactions of the Royal Society B: Biological Sciences

Volumn 374, Issue 1776, 2019, Pages

  Probert, W J M ^a   Lakkur, S ^b   Fonnesbeck, C J ^b   Shea, K ^c   Runge, M C ^d   Tildesley, M J ^e   Ferrari, M J ^c  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b VANDERBILT UNIVERSITY   (United States)

^c PENNSYLVANIA STATE UNIVERSITY   (United States)

^d PATUXENT WILDLIFE RESEARCH CENTER   (United States)

^e UNIVERSITY OF WARWICK   (United Kingdom)

Author keywords

FMD; Machine learning; Optimal control; Outbreak response; Reinforcement learning; Vaccination

Indexed keywords

DECISION MAKING; DISEASE CONTROL; EPIDEMIC; EPIDEMIOLOGY; FOOT AND MOUTH DISEASE; HEALTH POLICY; HEURISTICS; LEARNING; MACHINE LEARNING; POPULATION OUTBREAK; VACCINATION;

ANIMALIA;

EID: 85065400140     PISSN: 09628436     EISSN: 14712970     Source Type: Journal    
DOI: 10.1098/rstb.2018.0277     Document Type: Article

References (46)

1. Ong JBS, Chen MI-C, Cook AR, Lee HC, Lee VJ, Lin RTP, Tambyah PA, Goh LG. 2010 Real-time epidemic monitoring and forecasting of H1N1-2009 using influenza-like illness from general practice and family doctor clinics in Singapore. PLoS ONE 5, e10036. (doi:10.1371/journal.pone.0010036)
2. Reis J, Shaman J. 2016 Retrospective parameter estimation and forecast of respiratory syncytial virus in the United States. PLoS Comput. Biol. 12, e1005133. (doi:10.1371/journal.pcbi.1005133)
3. Shaman J, Karspeck A. 2012 Forecasting seasonal outbreaks of influenza. Proc. Natl Acad. Sci. USA 109, 20 425-20 430. (doi:10.1073/pnas. 1208772109)
4. Drake JM, Kaul RB, Alexander LW, O'Regan SM, Kramer AM, Pulliam JT, Ferrari MJ, Park AW. 2015 Ebola cases and health system demand in Liberia. PLoS Biol. 13, e1002056. (doi:10.1371/journal.pbio. 1002056)
5. Keeling MJ et al. 2001 Dynamics of the 2001 UK foot and mouth epidemic: stochastic dispersal in a heterogeneous landscape. Science 294, 813-817. (doi:10.1126/science.1065973)
6. Ferguson NM, Donnelly CA, Anderson RM. 2001 The foot-and-mouth epidemic in Great Britain: pattern of spread and impact of interventions. Science 292, 1155-1160. (doi:10.1126/science.1061020)
7. Burke DS, Epstein JM, Cummings DAT, Parker JI, Cline KC, Singa RM, Chakravarty S. 2006 Individual-based computational modeling of smallpox epidemic control strategies. Acad. Emerg. Med. 13, 1142-1149. (doi:10.1197/j.aem.2006.07.017)
8. Tildesley MJ, Savill NJ, Shaw DJ, Deardon R, Brooks SP, Woolhouse MEJ, Grenfell BT, Keeling MJ. 2006 Optimal reactive vaccination strategies for a foot- and-mouth outbreak in the UK. Nature 440, 83-86. (doi:10.1038/nature04324)
9. Medlock J, Galvani AP. 2009 Optimizing influenza vaccine distribution. Science 325, 1705-1708. (doi:10.1126/science.1175570)
10. Bradbury NV, Probert WJM, Shea K, Runge MC, Fonnesbeck CJ, Keeling MJ, Ferrari MJ, Tildesley MJ. 2017 Quantifying the value of perfect information in emergency vaccination campaigns. PLoS Comput. Biol. 13, e1005318. (doi:10.1371/journal.pcbi. 1005318)
11. Ferguson NM, Cummings DAT, Cauchemez S, Fraser C, Riley S, Meeyai A, Iamsirithaworn S, Burke DS. 2005 Strategies for containing an emerging influenza pandemic in Southeast Asia. Nature 437, 209-214. (doi:10.1038/nature04017)
12. Magori K, Legros M, Puente ME, Focks DA, Scott TW, Lloyd AL, Gould F. 2009 Skeeter Buster: a stochastic, spatially explicit modeling tool for studying Aedes aegypti population replacement and population suppression strategies. PLoS Negl. Trop. Dis. 3, e508. (doi:10.1371/journal.pntd.0000508)
13. Roche SE et al. 2015 Evaluating vaccination strategies to control foot-and-mouth disease: a model comparison study. Epidemiol. Infect. 143, 1256-1275. (doi:10.1017/S0950268814001927)
14. Bellman R. 1957 Dynamic programming. Princeton, NJ: Princeton University Press.
15. Puterman ML. 2014 Markov decision processes: discrete stochastic dynamic programming. Hoboken, NJ: John Wiley & Sons.
16. Tesauro G. 1995 Temporal difference learning and TD-Gammon. Commun. ACM 38, 58-68. (doi:10. 1145/203330.203343)
17. Stone P, Sutton RS. 2001 Scaling reinforcement learning toward RoboCup soccer. In Proc. 18th Int. Conf. Machine Learning (ICML-2001), pp. 537-544. San Francisco, CA: Morgan Kaufmann.
18. Simonsen L, Gog JR, Olson D, Viboud C. 2016 Infectious disease surveillance in the big data era: towards faster and locally relevant systems. J. Infect. Dis. 214, S380-S385. (doi:10.1093/infdis/jiw376)
19. Leung P, Tran LT. 2000 Predicting shrimp disease occurrence: artificial neural networks vs. logistic regression. Aquaculture 187, 35-49. (doi:10.1016/ S0044-8486(00)00300-8)
20. Eames KTD, Keeling MJ. 2003 Contact tracing and disease control. Proc. R. Soc. Lond. B 270, 2565-2571. (doi:10.1098/rspb.2003.2554)
21. Fraser C, Riley S, Anderson RM, Ferguson NM. 2004 Factors that make an infectious disease outbreak controllable. Proc. Natl Acad. Sci. USA 101, 6146-6151. (doi:10.1073/pnas.0307506101)
22. Kiss IZ, Green DM, Kao RR. 2006 Infectious disease control using contact tracing in random and scale-free networks. J. R. Soc. Interface 3, 55-62. (doi:10. 1098/rsif.2005.0079)
23. Ludkovski M, Niemi J. 2010 Optimal dynamic policies for influenza management. Stat. Commun. Infect. Dis. 2, 5. (doi:10.2202/1948-4690.1020)
24. Zhu J-M, Wang L, Liu J-B. 2016 Eradication of Ebola based on dynamic programming. Comput. Math. Methods Med. 2016, 1-9. (doi:10.1155/2016/1580917)
25. Patel R, Longini IM, Halloran ME. 2005 Finding optimal vaccination strategies for pandemic influenza using genetic algorithms. J. Theor. Biol. 234, 201-212. (doi:10.1016/j.jtbi.2004.11.032)
26. Sutton R, Barto A. 1998 Reinforcement learning: an introduction. Cambridge, MA: MIT Press.
27. Mnih V et al. 2015 Human-level control through deep reinforcement learning. Nature 518, 529-533. (doi:10.1038/nature14236)
28. Anschel O, Baram N, Shimkin N. 2017 Averaged-DQN: variance reduction and stabilization for deep reinforcement learning. In Proc. 34th Int. Conf. Machine Learning, 6-11 August, Sydney, NSW, vol. 70, pp. 176-185.
29. Silver D et al. 2016 Mastering the game of Go with deep neural networks and tree search. Nature 529, 484-489. (doi:10.1038/nature16961)
30. Lin L-J. 1992 Reinforcement learning for robots using neural networks. PhD Thesis, Carnegie Mellon University, Pittsburgh, PA, USA
31. Anderson I. (chair) 2002 Foot and mouth disease 2001: lessons to be learned inquiry report. London, UK: Stationery Office.
32. Knight-Jones TJD, Rushton J. 2013 The economic impacts of foot and mouth disease-what are they, how big are they and where do they occur? Prev. Vet. Med. 112, 161-173. (doi:10.1016/j.prevetmed. 2013.07.013)
33. Tildesley MJ, Deardon R, Savill NJ, Bessell PR, Brooks SP, Woolhouse MEJ, Grenfell BT, Keeling MJ. 2008 Accuracy of models for the 2001 foot-and-mouth epidemic. Proc. R. Soc. Lond. B 275, 1459-1468. (doi:10.1098/rspb.2008.0006)
34. Ong HY, Chavez K, Hong A. 2015 Distributed deep Q-learning. arXiv 1508.04184. https://arxiv.org/abs/1508.04186.
35. Probert WJM et al. 2016 Decision-making for foot- and-mouth disease control: objectives matter. Epidemics 15, 10-19. (doi:10.1016/j.epidem.2015. 11.002)
36. Probert WJM et al. 2018 Real-time decision-making during emergency disease outbreaks. PLoS Comput. Biol. 14, e1006202. (doi:10.1371/journal.pcbi. 1006202)
37. de Klerk PF. 2002 Carcass disposal: lessons from The Netherlands after the foot and mouth disease outbreak of 2001. Rev. Off. Int. Epizoot. 21, 789-796. (doi:10.20506/rst.21.3.1376)
38. Sanson RL, Dubé C, Cork SC, Frederickson R, Morley C. 2014 Simulation modelling of a hypothetical introduction of foot-and-mouth disease into Alberta. Prev. Vet. Med. 114, 151-163. (doi:10. 1016/j.prevetmed.2014.03.005)
39. Samek W, Wiegand T, Müller K-R. 2017 Explainable artificial intelligence: understanding, visualizing and interpreting deep learning models. arXiv 1708.0829. https://arxiv.org/abs/1708.08296.
40. Tildesley MJ, House TA, Bruhn MC, Curry RJ, O'Neil M, Allpress JLE, Smith G, Keeling MJ. 2010 Impact of spatial clustering on disease transmission and optimal control. Proc. Natl Acad. Sci. USA 107, 1041-1046. (doi:10.1073/pnas.0909047107)
41. UK Government. 2018 Contingency plan for exotic notifiable diseases of animals in England 2018. https://www.gov.uk/government/publications/ contingency-plan-for-exotic-notifiable-diseases-of-animals-in-england.
42. Apfelbaum JL et al. 2013 Practice guidelines for management of the difficult airway: an updated report by the American Society of Anesthesiologists Task Force on Management of the Difficult Airway. Anesthesiology 118, 251-270. (doi:10.1097/ALN. 0b013e31827773b2)
43. Teach RL, Shortliffe EH. 1981 An analysis of physician attitudes regarding computer-based clinical consultation systems. Comput. Biomed. Res. 14, 542-558. (doi:10.1016/0010-4809(81)90012-4)
44. Chadès I, Carwardine J, Martin T, Nicol S, Buffet O. 2012 MOMDP: a solution for modelling adaptive management problems. In Proc. 26th AAAI Conf. Artificial Intelligence, pp. 267-273.
45. Shea K, Tildesley MJ, Runge MC, Fonnesbeck CJ, Ferrari MJ. 2014 Adaptive management and the value of information: learning via intervention in epidemiology. PLoS Biol. 12, e1001970. (doi:10.1371/journal.pbio. 1001970)
46. European Union. 2003 Council Directive 2003/85/EC of 29 September 2003 on Community measures for the control of foot-and-mouth disease repealing Directive 85/511/ EEC and Decisions 89/531/EEC and 91/665/EEC and amending Directive 92/46/EEC. https://eur-lex.europa.eu/legal-content/en/ALL/ ?uri=CELEX:32003L0085.