Hourglass: A library for incremental processing on Hadoop

Proceedings - 2013 IEEE International Conference on Big Data, Big Data 2013

Volumn , Issue , 2013, Pages 742-752

  Hayes, Matthew ^a   Shah, Sam ^a  

^a LINKEDIN CORPORATION   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BIG DATA; INTERFACE STATES; SEMANTICS; SOCIAL NETWORKING (ONLINE);

INCREMENTAL PROCESSING; LARGE DATASETS; LINKEDIN; ON-LINE SOCIAL NETWORKS; OPEN SOURCES; STATE MANAGEMENT; SUB-COMPUTATIONS;

DATA HANDLING;

EID: 84893318699     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/BigData.2013.6691647     Document Type: Conference Paper

References (26)

1. P. Bhatotia, A. Wieder, R. Rodrigues, U. A. Acar, and R. Pasquin. Incoop: MapReduce for incremental computations. In SoCC, pages 7:1-7:14, 2011.
2. P. Bhatotia, M. Dischinger, R. Rodrigues, and U. A. Acar. Slider: Incremental sliding-window computations for largescale data analysis. Technical Report 2012-004, MPI-SWS, 2012.
3. Y. Bu, B. Howe, M. Balazinska, and M. D. Ernst. The HaLoop approach to large-scale iterative data analysis. The VLDB Journal, 21(2):169-190, Apr. 2012.
4. T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sears. MapReduce online. In NSDI, 2010.
5. T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein. Introduction to Algorithms. The MIT Press, 3rd edition, 2009.
6. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, 2004.
7. P. Flajolet, ́ E. Fusy, O. Gandouet, and F. Meunier. Hyperloglog: the analysis of a near-optimal cardinality estimation algorithm. DMTCS Proceedings, (1), 2008.
8. P. K. Gunda, L. Ravindranath, C. A. Thekkath, Y. Yu, and L. Zhuang. Nectar: automatic management of data and computation in datacenters. In OSDI, pages 1-8, 2010.
9. S. Heule, M. Nunkesser, and A. Hall. Hyperloglog in practice: algorithmic engineering of a state of the art cardinality estimation algorithm. In Proceedings of the 16th International Conference on Extending Database Technology, pages 683-692. ACM, 2013.
10. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. In EuroSys, pages 59-72, 2007.
11. G. Lee, J. Lin, C. Liu, A. Lorek, and D. Ryaboy. The unified logging infrastructure for data analytics at Twitter. Proc. VLDB Endow., 5(12):1771-1780, Aug. 2012.
12. Y. Li and Y. Zhang. Generating ordered list of recommended items: a hybrid recommender system of microblog. arXiv preprint arXiv:1208.4147, 2012.
13. J. Lin. Monoidify! monoids as a design principle for efficient mapreduce algorithms. CoRR, abs/1304.7544, 2013.
14. Y. Liu, Z. Hu, and K. Matsuzaki. Towards systematic parallel programming over mapreduce. In Euro-Par 2011 Parallel Processing, pages 39-50. Springer, 2011.
15. Y. Liu, K. Emoto, and Z. Hu. A generate-test-aggregate parallel programming library. 2013.
16. D. Logothetis, C. Olston, B. Reed, K. C. Webb, and K. Yocum. Stateful bulk processing for incremental analytics. In SoCC, pages 51-62, 2010.
17. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: a not-so-foreign language for data processing. In SIGMOD, pages 1099-1110, 2008.
18. C. Olston, G. Chiou, L. Chitnis, F. Liu, Y. Han, M. Larsson, A. Neumann, V. B. N. Rao, V. Sankarasubramanian, S. Seth, C. Tian, T. ZiCornell, and X. Wang. Nova: continuous Pig/Hadoop workflows. In SIGMOD, pages 1081-1090, 2011.
19. D. Peng and F. Dabek. Large-scale incremental processing using distributed transactions and notifications. In OSDI, pages 1-15, 2010.
20. L. Popa, M. Budiu, Y. Yu, and M. Isard. DryadInc: reusing work in large-scale computations. In HotCloud, 2009.
21. D. Saile. Mapreduce with deltas. Master's thesis, Universitt Koblenz-Landau, Campus Koblenz, 2011.
22. K. Shvachko, H. Kuang, S. Radia, and R. Chansler. The Hadoop Distributed File System. In Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pages 1-10, 2010.
23. R. Sumbaly, J. Kreps, and S. Shah. The "Big Data" ecosystem at LinkedIn. In SIGMOD, 2013.
24. A. Thusoo, Z. Shao, S. Anthony, D. Borthakur, N. Jain, J. S. Sarma, R. Murthy, and H. Liu. Data warehousing and analytics infrastructure at Facebook. In SIGMOD, pages 1013-1020, 2010.
25. T. White. Hadoop: The Definitive Guide. O'Reilly Media, 2010.
26. Y. Yu, P. K. Gunda, and M. Isard. Distributed aggregation for data-parallel computing: interfaces and implementations. In SOSP, pages 247-260, 2009.