An Ingestion and Analytics Architecture for IoT Applied to Smart City Use Cases

IEEE Internet of Things Journal

Volumn 5, Issue 2, 2018, Pages 765-774

  Ta Shma, Paula ^a   Akbar, Adnan ^b   Gerson Golan, Guy ^a   Hadash, Guy ^a   Carrez, Francois ^b   Moessner, Klaus ^b  

^a IBM RESEARCH   (United States)

^b UNIVERSITY OF SURREY   (United Kingdom)

Author keywords

Big data; complex event processing (CEP); context aware; energy management; ingestion; Internet of Things (IoT); machine learning; smart cities; spark; transportation

Indexed keywords

BATCH DATA PROCESSING; COMPUTER ARCHITECTURE; ELECTRIC SPARKS; ENERGY MANAGEMENT; INFORMATION MANAGEMENT; INGESTION (ENGINES); INTERNET OF THINGS; LEARNING SYSTEMS; REAL TIME SYSTEMS; SCALABILITY; SMART CITY; TRANSPORTATION;

BATCH PRODUCTION; BIG DATA APPLICATIONS; COMPLEX EVENT PROCESSING; CONTEXT-AWARE; INTERNET OF THING (IOT); OPEN-SOURCE COMPONENTS; REAL TIME ANALYSIS; SCALABLE ARCHITECTURES;

BIG DATA;

EID: 85021912902     PISSN: None     EISSN: 23274662     Source Type: Journal    
DOI: 10.1109/JIOT.2017.2722378     Document Type: Article

References (36)

1. R. Sakhuja. (2016). 5 Reasons Why Microservices Have Become So Popular in the Last 2 Years. [Online]. Available: https://www.linkedin.com/pulse/5-reasons-why-microservices-have-become-sopopular-last-sakhuja
2. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," Commun. ACM, vol. 51, no. 1, pp. 107-113, Jan. 2008. [Online]. Available: http://doi.acm.org/10.1145/1327452.1327492
3. Apache Hadoop. Accessed on Dec. 12, 2016. [Online]. Available: http://hadoop.apache.org/
4. Amazon Elastic MapReduce. Accessed on Dec. 12, 2016. [Online]. Available: https://aws.amazon.com/emr/
5. Amazon EC2-Virtual Server Hosting. Accessed on Dec. 15, 2016. [Online]. Available: https://aws.amazon.com/ec2/
6. Amazon S3 AWS Cloud-Simple, Durable, Massively Scalable Object Storage. Accessed on Dec. 15, 2016. [Online]. Available: https://aws.amazon.com/s3/
7. OpenStack Sahara. Accessed on Dec. 10, 2016. [Online]. Available: https://wiki.openstack.org/wiki/Sahara
8. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, "Spark: Cluster computing with working sets," in Proc. HotCloud, vol. 10. 2010, p. 10.
9. M. Zaharia et al., "Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing," in Proc. 9th USENIX Conf. Netw. Syst. Design Implement. (NSDI), San Jose, CA, USA, 2012, p. 2. [Online]. Available: http://dl.acm.org/citation.cfm?id=2228298.2228301
10. M. Zaharia et al., "Discretized streams: Fault-tolerant streaming computation at scale," in Proc. 24th ACM Symp. Oper. Syst. Principles, Farmington, PA, USA, 2013, pp. 423-438.
11. G. Cugola and A. Margara, "Processing flows of information: From data stream to complex event processing," ACM Comput. Surveys, vol. 44, no. 3, pp. 1-62, Jun. 2012. [Online]. Available: http://doi.acm.org/10.1145/2187671.2187677
12. N. Marz and J. Warren, Big Data: Principles and Best Practices of Scalable Realtime Data Systems, 1st ed. Greenwich, CT, USA: Manning, 2015.
13. Simplifying the (Complex) Lambda Architecture. Accessed on Dec. 28, 2016. [Online]. Available: https://voltdb.com/blog/simplifyingcomplex-lambda-architecture
14. Questioning the Lambda Architecture. Accessed on Dec. 28, 2016. [Online]. Available: https://www.oreilly.com/ideas/questioning-the-lambda-architecture
15. V. Chandola, A. Banerjee, and V. Kumar, "Anomaly detection: A survey," ACM Comput. Surveys, vol. 41, no. 3, pp. 1-58, Jul. 2009. [Online]. Available: http://doi.acm.org/10.1145/1541880.1541882
16. A. Akbar, F. Carrez, K. Moessner, and A. Zoha, "Predicting complex events for pro-active IoT applications," in Proc. IEEE 2nd World Forum Internet Things (WF IoT), Milan, Italy, Dec. 2015, pp. 327-332.
17. Node-RED. (2016). Node-RED: A Visual Tool for Wiring the Internet of Things. Accessed on May 6, 2016. [Online]. Available: http://nodered.org//
18. J. Kreps, N. Narkhede, and J. Rao, "Kafka: A distributed messaging system for log processing," in Proc. NetDB, 2011, pp. 1-7.
19. Pinterest Secor. Accessed on Dec. 28, 2016. [Online]. Available: https://github.com/pinterest/secor
20. OpenStack: Open Source Software for Creating Private and Public Clouds. Accessed on Nov. 5, 2016. [Online]. Available: https://www.openstack.org/
21. CloudFiles: Scalable Cloud Object Storage. Accessed on Nov. 13, 2016. [Online]. Available: https://www.rackspace.com/cloud/files
22. OpenStack Swift Documentation. Accessed on Apr. 22, 2017. [Online]. Available: http://docs.openstack.org/developer/swift/
23. Apache Parquet Documentation. Accessed on Apr. 22, 2017. [Online]. Available: https://parquet.apache.org/documentation/latest/
24. J. Kestelyn. (2016). Benchmarking Apache Parquet: The Allstate Experience. Accessed on Apr. 22, 2017. [Online]. Available: https://blog.cloudera.com/blog/2016/04/benchmarking-apacheparquet-the-allstate-experience/
25. SoftLayer-API Operations for Search Services. Accessed on Apr. 22, 2017. [Online]. Available: http://sldn.softlayer.com/article/API-Operations-Search-Services
26. Elastic Search GitHub Repository. Accessed on May 15, 2017. [Online]. Available: https://github.com/elastic/elasticsearch
27. Welcome to Apache Lucene. Accessed on May 15, 2017. [Online]. Available: http://lucene.apache.org/
28. Learn More About Solr. Accessed on May 15, 2017. [Online]. Available: http://lucene.apache.org/solr/
29. X. Meng et al., "MLlib: Machine learning in apache spark," J. Mach. Learn. Res., vol. 17, no. 1, pp. 1235-1241, 2016.
30. A. Akbar, F. Carrez, K. Moessner, J. Sancho, and J. Rico, "Contextaware stream processing for distributed IoT applications," in Proc. IEEE 2nd World Forum Internet Things (WF IoT), Milan, Italy, Dec. 2015, pp. 663-668.
31. S. Petrovic, "A comparison between the silhouette index and the Davies-Bouldin index in labelling IDS clusters," in Proc. 11th Nordic Workshop Secure IT Syst., 2006, pp. 53-64.
32. A. Margara, G. Cugola, and G. Tamburrelli, "Learning from the past: Automated rule generation for complex event processing," in Proc. 8th ACM Int. Conf. Distrib. Event Based Syst. (DEBS), Mumbai, India, 2014, pp. 47-58. [Online]. Available: http://doi.acm.org/10.1145/2611286.2611289
33. A. Akbar, A. Khan, F. Carrez, and K. Moessner, "Predictive analytics for complex IoT data streams," IEEE Internet Things J., to be published, doi: 10.1109/JIOT.2017.2712672.
34. T. Hargreaves, M. Nye, and J. Burgess, "Making energy visible: A qualitative field study of how householders interact with feedback from smart energy monitors," Energy Policy, vol. 38, no. 10, pp. 6111-6119, 2010.
35. Real Time Traffic Analysis Sample Application. [Online]. Available: https://github.com/cfsworkload/data-analytics-transportation
36. P. Ta-Shma. (2017). End-to-End IoT Data Pipelines in IBM Bluemix: Introducing the Message Hub Object Storage Bridge. [Online]. Available: https://www.ibm.com/blogs/bluemix/2017/03/end-to-end-iot-data-pipelines-introducing-the-message-hub-objectstorage/