Dynamic delay predictions for large-scale railway networks: Deep and shallow extreme learning machines tuned via thresholdout

IEEE Transactions on Systems, Man, and Cybernetics: Systems

Volumn 47, Issue 10, 2017, Pages 2754-2767

  Oneto, Luca ^a   Fumeo, Emanuele ^a   Clerico, Giorgio ^a   Canepa, Renzo ^b   Papa, Federico ^c   Dambra, Carlo ^c   Mazzino, Nadia ^c   Anguita, Davide ^a  

^a UNIVERSITY OF GENOA   (Italy)

^b Rete Ferroviaria Italiana SpA   (Italy)

^c ANSALDO ENERGIA   (Italy)

Author keywords

Apache Spark; big data; deep extreme learning machine (DELM); delay prediction; dynamic varying systems; in memory computing; intelligent transportation systems; model selection (MS); railway; shallow extreme learning machine (SELM); thresholdout

Indexed keywords

DATA HANDLING; FORECASTING; KNOWLEDGE ACQUISITION; LEARNING SYSTEMS; RAILROAD TRANSPORTATION; RAILROADS; REGRESSION ANALYSIS; TRANSPORTATION;

DIFFERENTIAL PRIVACIES; EFFICIENT IMPLEMENTATION; EXTREME LEARNING MACHINE; LARGE-SCALE DATA PROCESSING; MULTIVARIATE REGRESSION; NATIONAL WEATHER SERVICES; STATE-OF-THE-ART SYSTEM; TOOLS AND TECHNIQUES;

BIG DATA;

EID: 85019023963     PISSN: 21682216     EISSN: 21682232     Source Type: Journal    
DOI: 10.1109/TSMC.2017.2693209     Document Type: Article

References (110)

1. E. Fumeo, L. Oneto, and D. Anguita, "Condition based maintenance in railway transportation systems based on big data streaming analysis," in Proc. INNS Big Data Conf., San Francisco, CA, USA, 2015, pp. 437-446
2. H. Li, B. Qian, D. Parikh, and A. Hampapur, "Alarm prediction in large-scale sensor networks-A case study in railroad," in Proc. IEEE Int. Conf. Big Data, Silicon Valley, CA, USA, 2013, pp. 7-14
3. H. Li et al., "Improving rail network velocity: A machine learning approach to predictive maintenance," Transp. Res. C Emerg. Technol., vol. 45, pp. 17-26, Aug. 2014
4. H. Feng et al., "Automatic fastener classification and defect detection in vision-based railway inspection systems," IEEE Trans. Instrum. Meas., vol. 63, no. 4, pp. 877-888, Apr. 2014
5. C. Aytekin, Y. Rezaeitabar, S. Dogru, and I. Ulusoy, "Railway fastener inspection by real-time machine vision," IEEE Trans. Syst., Man, Cybern., Syst., vol. 45, no. 7, pp. 1101-1107, Jul. 2015
6. S. A. Branishtov, Y. A. Vershinin, D. A. Tumchenok, and A. M. Shirvanyan, "Graph methods for estimation of railway capacity," in Proc. IEEE 17th Int. Conf. Intell. Transp. Syst., Qingdao, China, 2014, pp. 525-530
7. Y. Bai, T. K. Ho, B. Mao, Y. Ding, and S. Chen, "Energy-efficient locomotive operation for Chinese mainline railways by fuzzy predictive control," IEEE Trans. Intell. Transp. Syst., vol. 15, no. 3, pp. 938-948, Jun. 2014
8. X. Zhang and D. Gong, "Application of big data technology in marketing decisions for railway freight," in Proc. ICLEM Syst. Plan. Supply Chain Manag. Safety, Shanghai, China, 2014, pp. 1136-1141
9. C. Morris, J. Easton, and C. Roberts, "Applications of linked data in the rail domain," in Proc. IEEE Int. Conf. Big Data, Washington, DC, USA, 2014, pp. 35-41
10. J. Tutcher, "Ontology-driven data integration for railway asset monitoring applications," in Proc. IEEE Int. Conf. Big Data, Washington, DC, USA, 2014, pp. 85-95
11. Q. Li, Z. Zhong, Z. Liang, and Y. Liang, "Rail inspection meets big data: Methods and trends," in Proc. Int. Conf. Netw. Based Inf. Syst., Taipei, Taiwan, 2015, pp. 302-308
12. M. Ma, P. Wang, C.-H. Chu, and L. Liu, "Efficient multipattern event processing over high-speed train data streams," IEEE Internet Things J., vol. 2, no. 4, pp. 295-309, Aug. 2015
13. F. Wang, T.-H. Xu, Y. Zhao, and Y.-R. Huang, "Prior LDA and SVM based fault diagnosis of vehicle on-board equipment for high speed railway," in Proc. IEEE Int. Conf. Intell. Transp. Syst., 2015, pp. 818-823
14. Y. Zhao, T.-H. Xu, and W. Hai-Feng, "Text mining based fault diagnosis of vehicle on-board equipment for high speed railway," in Proc. IEEE Int. Conf. Intell. Transp. Syst., Qingdao, China, 2014, pp. 900-905
15. K. Noori and K. Jenab, "Fuzzy reliability-based traction control model for intelligent transportation systems," IEEE Trans. Syst., Man, Cybern., Syst., vol. 43, no. 1, pp. 229-234, Jan. 2013
16. Z. Bin and X. Wensheng, "An improved algorithm for high speed train's maintenance data mining based on MapReduce," in Proc. Int. Conf. Cloud Comput. Big Data, Shanghai, China, 2015, pp. 59-66
17. B. Wang, F. Li, X. Hei, W. Ma, and L. Yu, "Research on storage and retrieval method of mass data for high-speed train," in Proc. 11th Int. Conf. Comput. Intell. Security, Shenzhen, China, 2015, pp. 474-477
18. J. Sadler et al., "GeoSRM-Online geospatial safety risk model for the GB rail network," IET Intell. Transp. Syst., vol. 10, no. 1, pp. 17-24, Jan. 2016
19. Y. Qingyang and Y. Xiaoyun, "Scheduling optimization model and algorithm design for two-way marshalling train," in Proc. Int. Conf. Intell. Transp. Big Data. Smart City, 2015, pp. 705-708
20. Y.-T. Zhu, F.-Z. Wang, X.-H. Shan, and X.-Y. Lv, "K-medoids clustering based on MapReduce and optimal search of medoids," in Proc. Int. Conf. Comput. Sci. Educ., Vancouver, BC, Canada, 2014, pp. 573-577
21. E. Davey, "Rail traffic management systems (TMS)," in Proc. IET Prof. Develop. Course Railway Signalling Control Syst., 2012, pp. 126-143
22. M. Müller-Hannemann and M. Schnee, "Efficient timetable information in the presence of delays," in Proc. Robust Online Large-Scale Optim., 2009, pp. 249-272
23. M. Dotoli, N. Epicoco, M. Falagario, C. Seatzu, and B. Turchiano, "A decision support system for optimizing operations at intermodal railroad terminals," IEEE Trans. Syst., Man, Cybern., Syst., vol. 47, no. 3, pp. 487-501, Mar. 2017
24. J.-F. Cordeau, P. Toth, and D. Vigo, "A survey of optimization models for train routing and scheduling," Transp. Sci., vol. 32, no. 4, pp. 380-404, 1998
25. T. Dollevoet, F. Corman, A. D'Ariano, and D. Huisman, "An iterative optimization framework for delay management and train scheduling," Flexible Serv. Manuf. J., vol. 26, no. 4, pp. 490-515, 2014
26. X. Li, B. Shou, and D. Ralescu, "Train rescheduling with stochastic recovery time: A new track-backup approach," IEEE Trans. Syst., Man, Cybern., Syst., vol. 44, no. 9, pp. 1216-1233, Sep. 2014
27. S. Milinkovic, M. Markovic, S. Veskovic, M. Ivic, and N. Pavlovic, "A fuzzy Petri net model to estimate train delays," Simulat. Model. Pract. Theory, vol. 33, pp. 144-157, Apr. 2013
28. A. Berger, A. Gebhardt, M. Müller-Hannemann, and M. Ostrowski, "Stochastic delay prediction in large train networks," in Proc. 11th Workshop on Algorithmic Approaches for Transportation Modelling, Optimization, and Systems, OASIcs-OpenAccess Series in Informatics, vol. 20. Dagstuhl, Germany: Schloss Dagstuhl Leibniz-Zentrum für Informatik, 2011, pp. 100-111
29. S. Pongnumkul, T. Pechprasarn, N. Kunaseth, and K. Chaipah, "Improving arrival time prediction of Thailand's passenger trains using historical travel times," in Proc. Int. Joint Conf. Comput. Sci. Softw. Eng., Chon Buri, Thailand, 2014, pp. 307-312
30. R. M. P. Goverde, "A delay propagation algorithm for large-scale railway traffic networks," Transp. Res. C Emerg. Technol., vol. 18, no. 3, pp. 269-287, 2010
31. I. A. Hansen, R. M. P. Goverde, and D. J. Van Der Meer, "Online train delay recognition and running time prediction," in Proc. IEEE Int. Conf. Intell. Transp. Syst., 2010, Funchal, Portugal, pp. 1783-1788
32. P. Kecman, Models for Predictive Railway Traffic Management, Ph.D. dissertation, Dept. Transp. Plan., Delft Univ. Technol., Delft, The Netherlands, 2014
33. P. Kecman and R. M. P. Goverde, "Online data-driven adaptive prediction of train event times," IEEE Trans. Intell. Transp. Syst., vol. 16, no. 1, pp. 465-474, Feb. 2015
34. G.-B. Huang, H. Zhou, X. Ding, and R. Zhang, "Extreme learning machine for regression and multiclass classification," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 42, no. 2, pp. 513-529, Apr. 2012
35. B. M. Bishop, Neural Networks for Pattern Recognition. Oxford, U.K.: Oxford Univ. Press, 1995
36. H. Zhang, Z. Liu, G.-B. Huang, and Z. Wang, "Novel weighting-delaybased stability criteria for recurrent neural networks with time-varying delay," IEEE Trans. Neural Netw., vol. 21, no. 1, pp. 91-106, Jan. 2010
37. R. Kamalapurkar, J. R. Klotz, P. Walters, and W. E. Dixon, "Model-based reinforcement learning in differential graphical games," IEEE Trans. Control Netw. Syst., to be published, doi: 10.1109/TCNS.2016.2617622
38. H. Zhang and Y. Wang, "Stability analysis of Markovian jumping stochastic Cohen-Grossberg neural networks with mixed time delays," IEEE Trans. Neural Netw., vol. 19, no. 2, pp. 366-370, Feb. 2008
39. H. Zhang, Z. Wang, and D. Liu, "A comprehensive review of stability analysis of continuous-time recurrent neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 7, pp. 1229-1262, Jul. 2014
40. H. Hong, W. Yu, G. Wen, and X. Yu, "Distributed robust fixed-time consensus for nonlinear and disturbed multiagent systems," IEEE Trans. Syst., Man, Cybern., Syst., to be published, doi: 10.1109/TSMC.2016.2623634
41. Q. Jiao, H. Modares, S. Xu, F. L. Lewis, and K. G. Vamvoudakis, "Multi-agent zero-sum differential graphical games for disturbance rejection in distributed control," Automatica, vol. 69, pp. 24-34, Jul. 2016
42. E. Cambria et al., "Extreme learning machines," IEEE Intell. Syst., vol. 28, no. 6, pp. 30-59, Nov./Dec. 2013
43. G. Huang, G.-B. Huang, S. Song, and K. You, "Trends in extreme learning machines: A review," Neural Netw., vol. 61, pp. 32-48, Jan. 2015
44. G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, "Extreme learning machine: Theory and applications," Neurocomputing, vol. 70, nos. 1-3, pp. 489-501, 2006
45. G.-B. Huang, "An insight into extreme learning machines: Random neurons, random features and kernels," Cogn. Comput., vol. 6, no. 3, pp. 376-390, 2014
46. G.-B. Huang, "What are extreme learning machines? Filling the gap between frank Rosenblatt's dream and John von Neumann's puzzle," Cogn. Comput., vol. 7, no. 3, pp. 263-278, 2015
47. S. Ridella, S. Rovetta, and R. Zunino, "Circular backpropagation networks for classification," IEEE Trans. Neural Netw., vol. 8, no. 1, pp. 84-97, Jan. 1997
48. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors," Cogn. Model., vol. 5, no. 3, p. 1, 1988
49. G.-B. Huang, M.-B. Li, L. Chen, and C.-K. Siew, "Incremental extreme learning machine with fully complex hidden nodes," Neurocomputing, vol. 71, nos. 4-6, pp. 576-583, 2008
50. G.-B. Huang, L. Chen, and C. K. Siew, "Universal approximation using incremental constructive feedforward networks with random hidden nodes," IEEE Trans. Neural Netw., vol. 17, no. 4, pp. 879-892, Jul. 2006
51. G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, "Extreme learning machine: A new learning scheme of feedforward neural networks," in Proc. IEEE Int. Joint Conf. Neural Netw., Budapest, Hungary, 2004, pp. 985-990
52. Y. Bengio, "Learning deep architectures for AI," Found. Trends Mach. Learn., vol. 2, no. 1, pp. 1-127, 2009
53. Y. Bengio, A. Courville, and P. Vincent, "Representation learning: A review and new perspectives," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1798-1828, Aug. 2013
54. P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, "Extracting and composing robust features with denoising autoencoders," in Proc. Int. Conf. Mach. Learn., Helsinki, Finland, 2008, pp. 1096-1103
55. G. E. Hinton, S. Osindero, and Y.-W. Teh, "A fast learning algorithm for deep belief nets," Neural Comput., vol. 18, no. 7, pp. 1527-1554, 2006
56. H. Zhou, G.-B. Huang, Z. Lin, H. Wang, and Y. C. Soh, "Stacked extreme learning machines," IEEE Trans. Cybern., vol. 45, no. 9, pp. 2013-2025, Sep. 2015
57. L. L. C. Kasun, H. Zhou, G.-B. Huang, and C. M. Vong, "Representational learning with ELMs for big data," IEEE Intell. Syst., vol. 28, no. 6, pp. 31-34, Nov./Dec. 2013
58. J. Tang, C. Deng, and G.-B. Huang, "Extreme learning machine for multilayer perceptron," IEEE Trans. Neural Netw. Learn. Syst., vol. 27, no. 4, pp. 809-821, Apr. 2016
59. M. D. Tissera and M. D. McDonnell, "Deep extreme learning machines: Supervised autoencoding architecture for classification," Neurocomputing, vol. 174, pp. 42-49, Jan. 2016
60. M. Zaharia et al., "Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing," in Proc. USENIX Conf. Netw. Syst. Design Implement., San Jose, CA, USA, 2012, p. 2
61. X. Meng et al., "MLlib: Machine learning in apache spark," J. Mach. Learn. Res., vol. 17, no. 1, pp. 1235-1241, 2016
62. J. L. Reyes-Ortiz, L. Oneto, and D. Anguita, "Big data analytics in the cloud: Spark on Hadoop vs MPI/OpenMP on Beowulf," Proc. Comput. Sci., vol. 53, pp. 121-130, 2015
63. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," Commun. ACM, vol. 51, no. 1, pp. 107-113, 2008
64. T. White, Hadoop: The Definitive Guide. Beijing, China: O'Reilly Media, 2012
65. Google. (2016). Google Compute Engine. Accessed on May 3, 2016. [Online]. Available: https://cloud.google.com/compute
66. L. Oneto et al., "Advanced analytics for train delay prediction systems by including exogenous weather data," in Proc. IEEE Int. Conf. Data Sci. Adv. Anal., Montreal, QC, Canada, 2016, pp. 458-467
67. P. L. Bartlett, S. Boucheron, and G. Lugosi, "Model selection and error estimation," Mach. Learn., vol. 48, nos. 1-3, pp. 85-113, 2002
68. D. Anguita, A. Ghio, L. Oneto, and S. Ridella, "In-sample model selection for support vector machines," in Proc. Int. Joint Conf. Neural Netw., San Jose, CA, USA, 2011, pp. 1154-1161
69. R. B. Rao, G. Fung, and R. Rosales, "On the dangers of crossvalidation. An experimental evaluation," in Proc. Int. Conf. Data Min., Atlanta, GA, USA, 2008, pp. 588-596
70. C. Dwork et al., "Preserving statistical validity in adaptive data analysis," in Proc. 47th Annu. ACM Symp. Theory Comput., Portland, OR, USA, Jun. 14-17, 2015, pp. 117-126
71. C. Dwork et al., "Generalization in adaptive data analysis and holdout reuse," in Proc. Neural Inf. Process. Syst., Montreal, QC, Canada, 2015, pp. 2341-2349
72. C. Dwork and A. Roth, "The algorithmic foundations of differential privacy," Found. Trends Theor. Comput. Sci., vol. 9, nos. 3-4, pp. 211-407, 2014
73. C. Dwork et al., "Preserving statistical validity in adaptive data analysis," in Proc. Annu. ACM Symp. Theory Comput., Portland, OR, USA, 2015, pp. 117-126
74. C. Dwork et al., "Generalization in adaptive data analysis and holdout reuse," arXiv Preprint arXiv:1506.02629 (v2), 2015
75. C. Dwork et al., "The reusable holdout: Preserving validity in adaptive data analysis," Science, vol. 349, no. 6248, pp. 636-638, 2015
76. Rete Ferroviaria Italiana. (2016). Gruppo Ferrovie Dello Stato Italiane. Accessed on May 3, 2016. [Online]. Available: http://www.rfi.it
77. Regione Liguria. (2016). Weather Data of Regione Liguria. Accessed on May 3, 2016. [Online]. Available: http://www2.arpalombardia.it/ siti/arpalombardia/meteo/richiesta-dati-misurati/Pagine/RichiestaDati Misurati.aspx
78. Regione Lombardia. (2016). Weather Data of Regione Lombardia. Accessed on 3 May 2016. [Online]. Available: http:// www.cartografiarl.regione.liguria.it/SiraQualMeteo/script/PubAccesso DatiMeteo.asp
79. F. Takens, Detecting Strange Attractors in Turbulence. Heidelberg, Germany: Springer, 1981
80. N. H. Packard, J. P. Crutchfield, J. D. Farmer, and R. S. Shaw, "Geometry from a time series," Phys. Rev. Lett., vol. 45, no. 9, pp. 712-716, 1980
81. V. N. Vapnik, Statistical Learning Theory. New York, NY, USA: Wiley, 1998
82. J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis. Cambridge, U.K.: Cambridge Univ. Press, 2004
83. E. E. Elattar, J. Goulermas, and Q. H. Wu, "Electric load forecasting based on locally weighted support vector regression," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 40, no. 4, pp. 438-447, Jul. 2010
84. L. Ghelardoni, A. Ghio, and D. Anguita, "Energy load forecasting using empirical mode decomposition and support vector regression," IEEE Trans. Smart Grid, vol. 4, no. 1, pp. 549-556, Mar. 2013
85. F. Bisio, P. Gastaldo, R. Zunino, and E. Cambria, "A learning scheme based on similarity functions for affective common-sense reasoning," in Proc. Int. Joint Conf. Neural Netw., Killarney, Ireland, 2015, pp. 1-6
86. L. Oneto, F. Bisio, E. Cambria, and D. Anguita, "Statistical learning theory and ELM for big social data analysis," IEEE Comput. Intell. Mag., vol. 11, no. 3, pp. 45-55, Aug. 2016
87. R. Caruana, S. Lawrence, and L. Giles, "Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping," in Proc. Neural Inf. Process. Syst., Denver, CO, USA, 2001, pp. 381-387
88. L. Prechelt, "Automatic early stopping using cross validation: Quantifying the criteria," Neural Netw., vol. 11, no. 4, pp. 761-767, 1998
89. D. Anguita, A. Ghio, L. Oneto, and S. Ridella, "In-sample and outof-sample model selection and error estimation for support vector machines," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 9, pp. 1390-1406, Sep. 2012
90. J. Langford. (2005). Clever Methods of Overfitting. [Online]. Available: http://hunch.net/?p=22
91. P. L. Bartlett and S. Mendelson, "Rademacher and Gaussian complexities: Risk bounds and structural results," J. Mach. Learn. Res., vol. 3, pp. 463-482, Nov. 2002
92. P. L. Bartlett, O. Bousquet, and S. Mendelson, "Local Rademacher complexities," Ann. Statist., vol. 33, no. 4, pp. 1497-1537, 2005
93. S. Floyd and M. Warmuth, "Sample compression, learnability, and the Vapnik-Chervonenkis dimension," Mach. learn., vol. 21, no. 3, pp. 269-304, 1995
94. J. Langford and D. McAllester, "Computable shell decomposition bounds," J. Mach. Learn. Res., vol. 5, pp. 529-547, 2004
95. O. Bousquet and A. Elisseeff, "Stability and generalization," J. Mach. Learn. Res., vol. 2, pp. 499-526, Mar. 2002
96. T. Poggio, R. Rifkin, S. Mukherjee, and P. Niyogi, "General conditions for predictivity in learning theory," Nature, vol. 428, no. 6981, pp. 419-422, 2004
97. L. Oneto, A. Ghio, S. Ridella, and D. Anguita, "Fully empirical and data-dependent stability-based bounds," IEEE Trans. Cybern., vol. 45, no. 9, pp. 1913-1926, Sep. 2015
98. G. Lever, F. Laviolette, and F. Shawe-Taylor, "Tighter PAC-Bayes bounds through distribution-dependent priors," Theor. Comput. Sci., vol. 473, pp. 4-28, Feb. 2013
99. I. O. Tolstikhin and Y. Seldin, "PAC-Bayes-empirical-Bernstein inequality," in Proc. Neural Inf. Process. Syst., 2013, pp. 109-117
100. P. Germain, A. Lacasse, M. Laviolette, M. Marchand, and J.-F. Roy, "Risk bounds for the majority vote: From a PAC-Bayesian analysis to a learning algorithm," J. Mach. Learn. Res., vol. 16, no. 1, pp. 787-860, 2015
101. L. Bégin, P. Germain, F. Laviolette, and J.-F. Roy, "PAC-Bayesian bounds based on the Rényi divergence," in Proc. Int. Conf. Artif. Intell. Statist., 2016, pp. 435-444
102. S. Nitzan and J. Paroush, "Optimal decision rules in uncertain dichotomous choice situations," Int. Econ. Rev., vol. 23, no. 2, pp. 289-297, 1982
103. O. Catoni, Pac-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning. Beachwood, OH, USA: Inst. Math. Statist., 2007
104. J. Bergstra and Y. Bengio, "Random search for hyper-parameter optimization," J. Mach. Learn. Res., vol. 13, no. 1, pp. 281-305, 2012
105. Y. Bengio, "Gradient-based optimization of hyperparameters," Neural Comput., vol. 12, no. 8, pp. 1889-1900, 2000
106. D. Maclaurin, D. Duvenaud, and R. P. Adams, "Gradient-based hyperparameter optimization through reversible learning," in Proc. Int. Conf. Mach. Learn., Lille, France, 2015, pp. 2113-2122
107. J. S. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, "Algorithms for hyper-parameter optimization," in Proc. Neural Inf. Process. Syst., Granada, Spain, 2011, pp. 2546-2554
108. N. Baba, "Convergence of a random optimization method for constrained optimization problems," J. Optim. Theory Appl., vol. 33, no. 4, pp. 451-461, 1981
109. A. Blum and M. Hardt, "The ladder: A reliable leaderboard for machine learning competitions," in Proc. Int. Conf. Mach. Learn., Lille, France, 2015, pp. 1006-1014
110. M. Hardt and J. Ullman, "Preventing false discovery in interactive data analysis is hard," in Proc. Found. Comput. Sci., Philadelphia, PA, USA, 2014, pp. 454-463