Reinforcement learning for closed-loop propofol anesthesia: A study in human volunteers

Journal of Machine Learning Research

Volumn 15, Issue , 2014, Pages 655-696

  Moore, Brett L ^a   Pyeatt, Larry D ^b   Kulkarni, Vivekanand ^c   Panousis, Periklis ^c   Padrez, Kevin ^c   Doufas, Anthony G ^c  

^a Texas Tech University   (United States)

^b South Dakota School of Mines and Technology   (United States)

^c STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

Anesthesia; Bispectral index; Closed loop control; Hypnosis; Propofol; Reinforcement learning

Indexed keywords

CLOSED LOOP CONTROL SYSTEMS; DRUG DELIVERY; REINFORCEMENT LEARNING;

ANESTHESIA; BISPECTRAL INDEX; CLOSED-LOOP CONTROL; HYPNOSIS; PROPOFOL;

ANESTHESIOLOGY;

EID: 84897075697     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (91)

1. A R Absalom and G N C Kenny. Closed-loop control of propofol anaesthesia using Bispectral IndexTM: Performance assessment in patients receiving computer-controlled propofol and manually controlled remifentanil infusions for minor surgery. Brit J Anaesth, 90(6):737-41, 2003. (Pubitemid 36700776)
2. A R Absalom, N Sutclié, and G N C Kenny. Closed-loop control of anesthesia using Bispectral IndexTM: Performance assessment in patients undergoing major orthopedic surgery under combined general and regional anesthesia. Anesthesiology, 96(1):67-73, Jan 2002. (Pubitemid 34032484)
3. M E Ausems, C C Hug, Jr, D R Stanski, and A G Burm. Plasma concentrations of alfentanil required to supplement nitrous oxide anesthesia for general surgery. Anesthesiology, 65 (4):362-73, Oct 1986. (Pubitemid 16031798)
4. M S Avidan, L Zhang, B A Burnside, K J Finkel, A C Searleman, J A Selvidge, L Saager, M S Turner, S Rao, M Bottros, C Hantler, E Jacobsohn, and A S Evers. Anesthesia awareness and the bispectral index. New Eng J Med, 11(358):1097-1108, Mar 2008. (Pubitemid 351398484)
5. J M Bailey, C T Mora, and S L Shafer. Pharmacokinetics of propofol in adult patients undergoing coronory revascularization. Anesthesiology, 84:1288-97, 1996. (Pubitemid 26192035)
6. L Baird. Residual algorithms: Reinforcement learning with function approximation. In Proc. 12th International Conference on Machine Learning, pages 30-37. Morgan Kaufmann, 1995.
7. L Barvais, I Rausin, J B Glen, S C Hunter, D D'Hulster, F Cantraine, and A d'Hollander. Administration of propofol by target-controlled infusion in patients undergoing coronary artery surgery. J Cardiothorac Vasc Anesth, 10(7):877-83, Dec 1996. (Pubitemid 26422481)
8. H R Berenji and P Kehdkar. Learning and tuning fuzzy logic controllers through reinforcements. IEEE Transactions on Neural Networks, 3(5):724-740, 1992.
9. M J Bloom, A Bekker, C V Seshagiri, and S D Greenwald. Changes in BIS variability reect changes in remifentanil infusion during spinal surgery. Presented at the American Society of Anesthesiologists Annual Meeting, Oct 2008.
10. A Bonarini, A Lazaric, F Montrone, and M Restelli. Reinforcement distribution in fuzzy Q-learning. Fuzzy Sets and Systems, 160(10):1420-1443, 2009.
11. J Boyan and A W Moore. Generalization in reinforcement learning: Safely approximating the value function. In G Tesauro, D S Touretzky, and T K Leen, editors, Advances in Neural Information Processing Systems 7, pages 369-376, Cambridge, MA, 1995. The MIT Press.
12. E Brown, R Lydic, and N Schiff. General anesthesia, sleep, and coma. N Engl J Med, 363 (27):2638-50, Dec 2010.
13. A Carregal, A Lorenzo, J A Taboada, and J L Barreiro. Intraoperative control of mean arterial pressure and heart rate with alfentanyl with fuzzy logic. Rev Esp Anestesiol Reanim, 47(3):108-113, Mar 2000. (Pubitemid 30189495)
14. D Chapman and L P Kaelbling. Input generalization in delayed reinforcement learning: An algorithm and performance comparisons. In Proceedings of the International Joint Conference on Artificial Intelligence, Sydney, Australia, 1991.
15. S Ching, B M Westover, M Liberman, J J Chemali, J Kenny, K Solt, P L Purdon, and E N Brown. Real-time closed-loop control in a rodent model of medically induced coma using burst suppression. Anesthesiology, 119(4):848-860, Oct 2013.
16. A Dahaba. Different conditions that could result in the bispectral index indicating an incorrect hypnotic state. Anesth Analg, 101(3):765-73, Sep 2005. (Pubitemid 41208509)
17. P Dayan. The convergence of TD(-) for general -. Machine Learning, 8:341-362, 1992.
18. T De Smet, M M R F Struys, M M Neckebroek, K Van den Hauwe, S Bonte, and E P Mortier. The accuracy and clinical feasibility of a new Bayesian-based closed-loop control system for propofol administration using the bispectral index as a controlled variable. Anesth Analg, 107:1200-1210, 2008.
19. A G Doufas, M Bakhshandeh, A R Bjorksten, S L Shafer, and D I Sessler. Induction speed is not a determinant of propofol pharmacodynamics. Anesthesiology, 101:1112-21, 2004. (Pubitemid 39430790)
20. D Ernst, G B Stan, J Goncalves, and LWehenkel. Clinical data based optimal STI strategies for HIV; a reinforcement learning approach. In Machine Learning Conference of Belgium and The Netherlands (Benelearn), pages 65-72, 2006.
21. D Ernst, M Glavic, F Capitanescu, and L Wehenkel. Reinforcement learning versus model predictive control: A comparison on a power system problem. Trans Sys Man Cyber Part B, 39(2):517-529, 2009. ISSN 1083-4419.
22. V Esmaeili, A Assareh, M B Shamsollahi, M H Moradi, and N M Areán. Estimating the depth of anesthesia using fuzzy soft computation applied to EEG features. Intell Data Anal, 12(4):393-407, 2008.
23. S P Fitzgibbon, D M Powers, K J Pope, and C R Clark. Removal of EEG noise and artifact using blind source separation. J Clin Neurophysiol, 24(3):232-43, Jun 2007. (Pubitemid 46871870)
24. A E Gaweda, M K Muezzinoglu, A A Jacobs, G R Aronoff, and M E Brier. Model predictive control with reinforcement learning for drug delivery in renal anemia management. Conf Proc IEEE Eng Med Biol Soc, 1:5177-80, 2006. (Pubitemid 46521081)
25. A Gentilini, C Frei, A H Glattfelder, M Morari, and T Schnider. Identification and targeting policies for computer controlled infusion pumps. Crit Rev Biomed Eng, 28(1&2):179-185, 2000.
26. E Gepts. Pharmacokinetic concepts for TCI anaesthesia. Anaesthesia, 53(Suppl 1):4-12, Apr 1998. (Pubitemid 28258620)
27. P S Glass, M Bloom, L Kearse, C Rosow, P Sebel, and P Manberg. Bispectral analysis measures sedation and memory effects of propofol, midazolam, isourane, and alfentanil in healthy volunteers. Anesthesiology, 86(4):836-847, Apr 1997. (Pubitemid 27174724)
28. P Y Glorennec and L Joué. Fuzzy Q-learning. In Proceedings of Fuzz-IEEE'97, Sixth International Conference on Fuzzy Systems, volume 3, pages 659-662, 1997.
29. S D Greenwald and C E Rosow. BIS and EMG variability increase before somatic responses during surgery. Presented at the American Society of Anesthesiologists Annual Meeting, Oct 2006.
30. A Guez, R D Vincent, M Avoli, and J Pineau. Adaptive treatment of epilepsy via batchmode reinforcement learning. In IAAI'08: Proceedings of the 20th national conference on Innovative Applications of Artificial Intelligence, pages 1671-1678. AAAI Press, 2008. ISBN 978-1-57735-368-3.
31. V Gullapalli. Learning control under extreme uncertainty. In Stephen José Hanson, Jack D. Cowan, and C. Lee Giles, editors, Advances in Neural Information Processing Systems, volume 5, pages 327-334. Morgan Kaufmann, San Mateo, CA, 1993. URL citeseer.nj.nec.com/312133.html.
32. J O Hahn, G A Dumont, and J M Ansermino. Closed-loop anesthetic drug concentration estimation using clinical-effect feedback. IEEE Trans Biomed Eng, 58(1):3-6, Jan 2011.
33. T M Hemmerling, S Charabati, C Zaouter, C Minardi, and P A Mathieu. A randomized controlled trial demonstrates that a novel closed-loop propofol system performs better hypnosis control than manual administration. Can J Anaesth, 57(8):725-735, Aug 2010.
34. C Hu, W S Lovejoy, and S L Shafer. Comparison of some control strategies for threecompartment PK/PD models. Journal of Pharmacokinetics and Biopharmaceutics, 22 (6):525-550, 1994.
35. L P Kaelbling, M L Littman, and A W Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237-285, 1996. (Pubitemid 126646155)
36. L P Kaelbling, M L Littman, and A R Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101:99-134, 1998. (Pubitemid 128387390)
37. L A Kearse Jr., P Manberg, N Chamoun, F deBros, and A Zaslavsky. Bispectral analysis of the electroencephalogram correlates with patient movement to skin incision during propofol/nitrous oxide anesthesia. Anesthesiology, 81(6):1365-70, Dec 1994. (Pubitemid 24381427)
38. G N C Kenny and H Mantzaridis. Closed-loop control of propofol anaesthesia. Brit J Anaesth, 83(2):223-8, Aug 1999. (Pubitemid 29361130)
39. K Leslie, A R Absalom, and G N C Kenny. Closed loop control of sedation for colonoscopy using the Bispectral Index. Anaesthesia, 57(7):690-709, Jul 2002.
40. M Lindholm, S Träff, F Granath, S D Greenwald, A Ekbom, C Lennmarken, and R H Sandin. Mortality within 2 years after surgery in relation to low intraoperative bispectral index values and preexisting malignant disease. Anesth Analg, 108(2):508-512, Feb 2009.
41. M Littman. Markov games as a framework for multi-agent reinforcement learning. In Proceedings of the 11th International Conference on Machine Learning (ML-94), pages 157-163, New Brunswick, NJ, 1994. Morgan Kaufmann.
42. N Liu, T Chazot, A Genty, A Landais, A Restoux, K McGee, P A Laloë, B Trillat, L Barvais, and M Fischler. Titration of propofol for anesthetic induction and maintenance guided by the bispectral index: Closed-loop versus manual control: A prospective, randomized, multicenter study. Anesthesiology, 104(4):686-695, April 2006.
43. N Liu, M Le Guen, F Benabbes-Lambert, T Chazot, B Trillat, D I Sessler, and M Fischler. Feasibility of closed-loop titration of propofol and remifentanil guided by the spectral M-Entropy monitor. Anesthesiology, 116(2):286-295, Feb 2012.
44. N Liu, O Pruszkowski, J E Leroy, T Chazot, B Trillat, A Colchen, F Gonin, and M Fischler. Automatic administration of propofol and remifentanil guided by the bispectral index during rigid bronchoscopic procedures: A randomized trial. Can J Anaesth, 60(9):881-887, Sep 2013.
45. S Mahadevan. Average reward reinforcement learning: Foundations, algorithms, and empirical results. Machine Learning, 22(1-3):159-195, 1996. (Pubitemid 126724366)
46. J D Martín-Guerrero, F Gomez, E Soria-Olivas, J Schmidhuber, M Climente-Martí, and N V Jiménez-Torres. A reinforcement learning approach for individualizing erythropoietin dosages in hemodialysis patients. Expert Syst Appl, 36(6):9737-9742, Aug 2009.
47. D M Mathews, L Clark, J Johansen, E Matute, and C V Seshagiri. Increases in electroencephalogram and electromyogram variability are associated with an increased incidence of intraoperative somatic response. Anesth Analg, 114(4):759-70, Apr 2012.
48. B L Moore. Intelligent control of closed-loop sedation in simulated ICU patients. Master's thesis, Texas Tech University, 2003.
49. B L Moore, E D Sinzinger, T M Quasny, and L D Pyeatt. Intelligent control of closed-loop sedation in simulated ICU patients. In FLAIRS 2004. AAAI Press, 2004.
50. B L Moore, L D Pyeatt, and A G Doufas. Fuzzy control for closed-loop, patient-specific hypnosis in intraoperative patients: A simulation study. In Conf Proc IEEE Eng Med Biol Soc, volume 1, 2009.
51. B L Moore, A G Doufas, and L D Pyeatt. Reinforcement learning: A novel method for optimal control of propofol-induced hypnosis. Anesth Analg, 112(2):360-367, Feb 2011a.
52. B L Moore, T M Quasny, and A G Doufas. Reinforcement learning versus proportionalintegral-derivative control of hypnosis in a simulated intraoperative patient. Anesth Analg, 112(2):350-359, Feb 2011b.
53. E P Mortier, M M R F Struys, T De Smet, Y D I Versichelen, and G Rolly. Closedloop controlled administration of propofol using bispectral analysis. Anaesthesia, 53(8): 749-754, Aug 1998. (Pubitemid 28391380)
54. P Myles, K Leslie, J McNeil, A Forbes, and M Chan. Bispectral index monitoring to prevent awareness during anaesthesia: the B-Aware randomised controlled trial. Lancet, 363(9423):1757-1763, 2000. (Pubitemid 38698378)
55. D Nieuwenhuijs, E L Coleman, N J Douglas, G B Drummond, and A Dahan. Bispectral index values and spectral edge frequency at different stages of physiologic sleep. Anesth Analg, 94(1):125-129, Jan 2002. (Pubitemid 34038732)
56. D A O'Hara, D K Bogen, and A Noordergraaf. The use of computers for controlling the delivery of anesthesia. Anesthesiology, 77(3):563-81, Sep 1992.
57. K T Olkkola, H Schwilden, and C Apélstaedt. Model-based adaptive closed-loop feedback control of atracurium-induced neuromuscular blockade. Acta Anaesth Scand, 35(5):420-3, Jul 1991.
58. S Omatu, M Khalid, and R Yusof. Neuro-control and its Applications, chapter 4, pages 152-160. Advances in Industrial Control. Springer, 1996.
59. S Pilge, R Zanner, G Schneider, J Blum, M Kreuzer, and E F Kochs. Analysis of cerebral state, bispectral, and narcotrend indices. Anesthesiology, 104(3):488-494, Mar 2006.
60. I J Rampil. A primer for EEG signal processing in anesthesia. Anesthesiology, 89(4): 980-1002, Oct 1997.
61. M Renna, T Wigmore, A Mofeez, and C Gillbe. Biasing effect of the electromyogram on BIS: A controlled study during high-dose fentanyl induction. J Clin Monit Comput, 17 (6):377-81, Aug 2002. (Pubitemid 36874005)
62. A E Rigby-Jones and J R Sneyd. Pharmacokinetics and pharmacodynamics: Is there anything new? Anaesthesia, 67(1):5-11, Jan 2012.
63. H Röpcke, M Knen-Bergmann, M Cuhls, T Bouillon, and A Hoeft. Propofol and remifentanil pharmacodynamic interaction during orthopedic surgical procedures as measured by effects on bispectral index. J Clin Anesth, 13(3):198-207, May 2001a. (Pubitemid 32463879)
64. H Röpcke, B Rehberg, M Koenen-Bergmann, T Bouillon, J Bruhn, and A Hoeft. Surgical stimulation shifts EEG concentration-response relationship of desurane. Anesthesiology, 94(3):255-113, Mar 2001b.
65. S Russel and P Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall, 2nd edition, 2002.
66. F Sahba, H R Tizhoosh, and M M A Salama. Application of reinforcement learning for segmentation of transrectal ultrasound images. BMC Med Imaging, 8(8), 2008.
67. T Sakai, A Matsuki, P F White, and A H Giesecke. Use of an EEG-bispectral closed-loop delivery system for administering propofol. Acta Anesth Scand, 44:1007-1010, 2000.
68. R H Sandin, G Enlund, P Samuelsson, and C Lennmarken. Awareness during anaesthesia: A prospective case study. Lancet, 355(9205):707-711, 2000. (Pubitemid 30107626)
69. J Schaublin, M Derighetti, P Feigenwinter, S Petersen-Felix, and A M Zbinden. Fuzzy logic control of mechanical ventilation during anaesthesia. Brit J Anaesth, 77(5):636-41, Nov 1996. (Pubitemid 26385795)
70. T Schnider, C F Minto, P L Gambus, C Andresen, D B Goodale, S L Shafer, and E J Youngs. The inuence of method of administration and covariates on the pharmacokinetics of propofol in adult volunteers. Anesthesiology, 88(5):1170-1182, May 1998. (Pubitemid 28216859)
71. T W Schnider, C F Minto, S L Shafer, P L Gambus, C Andresen, D B Goodale, and E J Youngs. The inuence of age on propofol pharmacodynamics. Anesthesiology, 90(6): 1502-16, Jun 1999.
72. H Schwilden, J Schüttler, and H Stoeckel. Closed-loop feedback control of methohexital anesthesia by quantitative EEG analysis in humans. Anesthesiology, 67(3):341-7, Sep 1987. (Pubitemid 18077414)
73. P S Sebel, T A Bowdle, M M Ghoneim, I J Rampil, R E Padilla, T J Gan, and K B Domino. The incidence of awareness during anesthesia: A multicenter United States study. Anesth Analg, 99:833-839, 2004. (Pubitemid 39100377)
74. F S Servin. TCI compared with manually controlled infusion of propofol: A multicentre study. Anaesthesia, 53(Suppl 1):82-86, Apr 1998. (Pubitemid 28258634)
75. M M Shanechi, J J Chemali, M Liberman M, K Solt, and E N Brown. A brain-machine interface for control of medically-induced coma. PLOS Compu Biol, 9(10):1-17, Oct 2013.
76. J C Sigl and N G Chamoun. An introduction to bispectral analyis for the electroencephalogram. J Clin Monitor, 10(6):392-404, November 1994.
77. J W Sleigh, J Andrzejowski, A Steyn-Ross, and M Steyn-Ross. The bispectral index: A measure of depth of sleep? Anesth Analg, 88(3):659-661, Mar 1999. (Pubitemid 29109038)
78. M M R F Struys, T De Smet, S D Greenwald, A R Abasalom, S Bingé, and E P Mortier. Closed-loop controlled administration of propofol using bispectral analysis. Anesthesiology, 95(1):6-17, Jul 2001.
79. M M R F Struys, T De Smet, S D Greenwald, A R Absalom, S Bingé, and E P Mortier. Performance evaluation of two published closed-loop control systems using bispectral index monitoring: A simulation study. Anesthesiology, 100(3):640-7, Mar 2004. (Pubitemid 38268884)
80. M M R F Struys, M J Coppens, N De Neve, E P Mortier, A G Doufas, J F P Van Bocxlaer, and S L Shafer. Inuence of administration rate on propofol plasma-effect site equilibration. Anesthesiology, 07(3):386-396, Sept 2007. (Pubitemid 47329536)
81. R Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Touretzky, Mozer, and Hasselmo, editors, Advances in Neural Information Processing Systems, volume 8, pages 1038-1044. The MIT Press, 1996.
82. R S Sutton and A G Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
83. G Tesauro. Temporal difference learning of backgammon strategy. In Proceedings of the International Conference on Machine Learning, pages 451-457. Morgan Kaufmann, 1992.
84. D R Theil, T E Stanley, 3rd, W D White, D Goodman, P S Glass, S A Bai, J R Jacobs, and J G Reves. Midazolam and fentanyl continuous infusion anesthesia for cardiac surgery: A comparison of computer-assisted versus manual infusion systems. J Cardiothorac Vasc Anesth, 7(3):300-6, Jun 1993. (Pubitemid 23171975)
85. J N Tsitsiklas and B Van Roy. An analysis of temporal difference learning with function approximation. Technical Report LIDS-P-2322, Massachusetts Institute of Technology, 1996.
86. C Vanlersberghe and F Camu. Modern Anesthetics (Handbook of Experimental Pharmacology), volume 182, chapter Propofol, pages 227-252. Springer, 2008.
87. J R Varvel, D L Donoho, and S L Shafer. Measuring the predictive performance of computercontrolled infusion pumps. J Pharmacokinet Biopharm, 20:63-94, Feb 1992.
88. C J C HWatkins. Learning from Delayed Rewards. PhD dissertation, Cambridge University, Computer Science Department, 1989.
89. W Wood. Variability of human drug response. Anesthesiology, 71(4):631-634, Nov 1989.
90. L Zadeh. Fuzzy sets. Information and Control, 8:338-353, 1965.
91. Y Zhao, M R Kosorok, and D Zeng. Reinforcement learning design for cancer clinical trials. Stat Med, 28(26):3294-315, Nov 2009.