An MDP model-based reinforcement learning approach for production station ramp-up optimization: Q-learning analysis

IEEE Transactions on Systems, Man, and Cybernetics: Systems

Volumn 44, Issue 9, 2014, Pages 1125-1138

  Doltsinis, Stefanos ^a   Ferreira, Pedro ^a   Lohse, Niels ^a  

^a UNIVERSITY OF NOTTINGHAM   (United Kingdom)

Author keywords

Decision making; learning systems; manufacturing automation; Markov processes

Indexed keywords

DECISION MAKING; LEARNING ALGORITHMS; LEARNING SYSTEMS; MANUFACTURE; MARKOV PROCESSES;

AUTOMATED PRODUCTIONS; CAUSE-AND-EFFECT RELATIONSHIPS; HUMAN DECISION MAKING; MANUFACTURING AUTOMATION; MARKOV DECISION PROCESSES; MODEL-BASED REINFORCEMENT LEARNING; Q-LEARNING ALGORITHMS; SEQUENCE OF ACTIONS;

REINFORCEMENT LEARNING;

EID: 84906513698     PISSN: 21682216     EISSN: 21682232     Source Type: Journal    
DOI: 10.1109/TSMC.2013.2294155     Document Type: Article

References (34)

1. S. Fj̈allstr̈om, K. S̈afsten, U. Harlin, and J. Stahre, "Information enabling production ramp-up," J. Manuf. Technol. Manag., vol. 20, no. 2, pp. 178-196, 2009.
2. C. Terwiesch and R. E. Bohn, "Learning and process improvement during production ramp-up," Int. J. Prod. Econ., vol. 70, no. 1, pp. 1-19, 2001. (Pubitemid 32160564)
3. D. Ceglarek, W. Huang, S. Zhou, Y. Ding, R. Kumar, and Y. Zhou, "Time-based competition in multistage manufacturing: Stream-of-variation analysis (SOVA) methodology-Review," Int. J. Flexible Manuf. Syst., vol. 16, no. 1, pp. 11-44, 2004.
4. C. Terwiesch and Y. Xu, "The copy-exactly ramp-up strategy: Tradingoff learning with process change," IEEE Trans. Eng. Manag., vol. 51, no. 1, pp. 70-84, Feb. 2004.
5. S. C. Doltsinis and N. Lohse, "A model-free reinforcement learning approach using Monte Carlo method for production ramp-up policy improvement: A copy exactly test case," in Proc. 14th IFAC Symp. Inf. Control Problems Manuf., 2012, pp. 1628-1634.
6. J. E. Carrillo and C. Gaimon, "Improving manufacturing performance through process change and knowledge creation," Manag. Sci., vol. 46, no. 2, pp. 265-288, 2000. (Pubitemid 30594356)
7. K. Mannar and D. Ceglarek, "Continuous failure diagnosis for assembly systems using rough set approach," CIRP Ann. Manuf. Technol., vol. 53, no. 1, pp. 39-42, 2004.
8. R. Oates, D. Scrimieri, and S. Ratchev, "Accelerated ramp-up of assembly systems through self-learning," in Proc. IPAS, vol. 371 AICT. Feb. 2012, pp. 175-182.
9. C. J. McDonald, "Copy EXACTLY! A paradigm shift in technology transfer method," in Proc. IEEE Adv. Semicond. Manuf. Conf. Workshop, Sep. 1997, pp. 414-417.
10. M. Haller, A. Peikert, and J. Thoma, "Cycle time management during production ramp-up," Robot. Comput. Integr. Manuf., vol. 19, nos. 1-2, pp. 183-188, 2003.
11. C. H. Glock, M. Y. Jaber, and S. Zolfaghari, "Production planning for a ramp-up process with learning in production and growth in demand," Int. J. Prod. Res., vol. 50, no. 20, pp. 5707-5718, 2012.
12. P. D. Ball, S. Roberts, A. Natalicchio, and C. Scorzafave, "Modelling production ramp-up of engineering products," Proc. Inst. Mech. Eng. B, J. Eng. Manuf., vol. 225, no. 6, pp. 959-971, Jun. 2011.
13. J. Vits, L. Gelders, and L. Pintelon, "Production process changes: A dynamic programming approach to manage effective capacity and experience," Int. J. Prod. Econ., vol. 104, no. 2, pp. 473-481, 2006. (Pubitemid 44403055)
14. H. Winkler, M. Heins, and P. Nyhuis, "A controlling system based on cause-effect relationships for the ramp-up of production systems," Prod. Eng., vol. 1, no. 1, pp. 103-111, 2007.
15. C. M. Bishop, Pattern Recognition and Machine Learning, vol. 4. New York, NY, USA: Springer, 2006.
16. L. Monostori, "AI and machine learning techniques for managing complexity, changes and uncertainties in manufacturing," Eng. Applicat. Artif. Intell., vol. 16, no. 4, pp. 277-291, 2003.
17. L. Monostori, A. Markus, H. Van Brussel, and E. Westk̈ampfer, "Machine learning approaches to manufacturing," CIRP Ann. Manuf. Technol., vol. 45, no. 2, pp. 675-681, 1996.
18. D. Pham and A. Afify, "Machine-learning techniques and their applications in manufacturing," Proc. Inst. Mech. Eng. B, J. E. Manuf., vol. 219, no. 5, pp. 395-412, 2005. (Pubitemid 40929803)
19. J. Harding, M. Shahbaz, and A. Kusiak, "Data mining in manufacturing: A review," J. Manuf. Sci. Eng., vol. 128, no. 5, p. 969, 2006.
20. A. K. Choudhary, J. A. Harding, and M. K. Tiwari, "Data mining in manufacturing: A review based on the kind of knowledge," J. Intell. Manuf., vol. 20, no. 5, pp. 501-521, 2009.
21. N. Lohse, H. Hirani, and S. Ratchev, "Equipment ontology for modular reconfigurable assembly systems," Int. J. Flexible Manuf. Syst., vol. 17, no. 4, pp. 301-314, 2005. (Pubitemid 44568920)
22. M. Wiering and M. Van Otterlo, Reinforcement Learning: State-Of-The-Art. Berlin, Germany: Springer, 2012.
23. Y. Lei, X. Yangsheng, and C. S. Chen, "Human action learning via hidden Markov model," IEEE Trans. Syst., Man, Cybern. A, Syst, Humans, vol. 27, no. 1, pp. 34-44, Jan. 1997. (Pubitemid 127770688)
24. X. Li, J. Wang, and R. Sawhney, "Reinforcement learning for joint pricing, lead-time and scheduling decisions in make-to-order systems," Eur. J. Oper. Res., vol. 221, no. 1, pp. 99-109, 2012.
25. Y. C. Wang and J. M. Usher, "Application of reinforcement learning for agent-based production scheduling," Eng. Applicat. Artif. Intell., vol. 18, no. 10, pp. 73-82, 2005.
26. M. Shin, K. Ryu, and M. Jung, "Reinforcement learning approach to goal-regulation in a self-evolutionary manufacturing system," Expert Syst. Appl., vol. 39, no. 1, pp. 8736-8743, 2012.
27. K. Ueda, I. Hatono, N. Fujii, and J. Vaario, "Reinforcement learning approaches to biological manufacturing system," CIRP Ann. Manuf. Technol., vol. 49, no. 1, pp. 343-346, 2000.
28. S. Kalyanakrishnan and P. Stone, "Batch reinforcement learning in a complex domain," in Proc. AAMAS, 2007, pp. 662-669.
29. S. Adam, L. Busoniu, and R. Babuska, "Experience replay for real-time reinforcement learning control," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 42, no. 2, pp. 201-212, Mar. 2012.
30. S. Doltsinis, P. Ferreira, and N. Lohse, "Reinforcement learning for production ramp-up: A Q-batch learning approach," in Proc. ICMLA, Dec. 2012, pp. 610-615.
31. K. Konrad, M. Hoffmeister, M. Zapp, A. Verl, and J. Busse, "Enabling fast ramp-up of assembly lines through context-mapping of implicit operator knowledge and machine-derived data," in Proc. IPAS, vol. 371 AICT. 2012, pp. 163-174.
32. E. Project. (2012). Fast Ramp-up and Adaptive Manufacturing Environement (FRAME) [Online]. Available: http://www.frame-eu.org
33. M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, 2nd ed. New York, NY, USA: Wiley, 2005.
34. S. C. Doltsinis, S. Ratchev, and N. Lohse, "A framework for performance measurement during production ramp-up of assembly stations," Eur. J. Oper. Res., vol. 229, pp. 85-94, Aug. 2013.