Architecting big data: Challenges, studies and forecasts

Jisuanji Xuebao/Chinese Journal of Computers

Volumn 34, Issue 10, 2011, Pages 1741-1752

  Wang, Shan ^a,b   Wang, Hui Ju ^a,b   Qin, Xiong Pai ^a,b   Zhou, Xuan ^a,b  

^a BEIJING UNIVERSITY OF POSTS AND TELECOMMUNICATIONS   (China)

^b RENMIN UNIVERSITY OF CHINA   (China)

Author keywords

Big data; Deep analytics; Large scale; MapReduce; Parallel database

Indexed keywords

BIG DATA; DEEP ANALYTICS; LARGE SCALE; MAP-REDUCE; PARALLEL DATABASE;

HARDWARE; SOFTWARE ENGINEERING;

DATA WAREHOUSES;

EID: 81055156446     PISSN: 02544164     EISSN: None     Source Type: Journal    
DOI: 10.3724/SP.J.1016.2011.01741     Document Type: Article

References (42)

1. WinterCorp: 2005 TopTen Program Summary. http://www.wintercorp.com/WhitePapers/WC_TopTenWP.pdf
2. TDWI Checklist Report: Big Data Analytics. http://tdwi.org/research/2010/08/Big-Data-Analytics.aspx
3. Chaudhuri S, Dayal U. An overview of data warehousing and OLAP technology. SIGMOD Rec, 1997, 26(1): 65-74
4. Madden S, DeWitt D J, Stonebraker M. Database parallelism choices greatly impact scalability. DatabaseColumn Blog. http://www.databasecolumn.com/2007/10/ database-parallelism-choices.html
5. Dean J, Ghemawat S. MapReduce: Simplified data processing on large clusters//Proceedings of the 6th Symposium on Operating System Design and Implementation (OSDI'04). San Francisco, California, USA, 2004: 137-150
6. DeWitt D J, Gerber R H, Graefe G, Heytens M L, Kumar K B, Muralikrishna M. GAMMA-A high performance dataflow database machine//Proceedings of the 12th International Conference on Very Large Data Bases (VLDB'86). Kyoto, Japan, 1986: 228-237
7. Fushimi S, Kitsuregawa M, Tanaka H. An overview of the system software of a parallel relational database machine// Proceedings of the 12th International Conference on Very Large Data Bases (VLDB'86). Kyoto, Japan, 1986: 209-219
8. Brewer E A. Towards robust distributed systems//Proceedings of the 19th Annual ACM Symposium on Principles of Distributed Computing (PODC'00). Portland, Oregon, USA, 2000: 7
9. http://www.dbms2.com/2008/08/26/known-applications-of-mapreduce/
10. http://hadoop.apache.org
11. Pavlo A, Paulson E, Rasin A, Abadi D J, DeWitt D J, Madden S, Stonebraker M. A comparison of approaches to large-scale data analysis//Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD'09). Providence, Rhode Island, USA, 2009: 165-178
12. Jiang D, Ooi B C, Shi L, Wu S. The performance of MapReduce: An in-depth study. PVLDB, 2010, 3(1): 472-483
13. Stonebraker M, Abadi D J, DeWitt D J, Madden S, Paulson E, Pavlo A, Rasin A. MapReduce and parallel DBMSs: Friends or foes? Communications of the ACM, 2010, 53(1): 64-71
14. Dean J, Ghemawat S. MapReduce: A flexible data processing tool. Communications of the ACM, 2010, 53(1): 72-77
15. http://www.asterdata.com/product/mapreduce.php
16. http://www.greenplum.com/technology/mapreduce/
17. http://hive.apache.org/
18. Olston C, Reed B, Srivastava U, Kumar R, Tomkins Andrew. Pig latin: A not-so-foreign language for data processing//Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD'08). Vancouver, BC, Canada, 2008: 1099-1110
19. Azza Abouzeid, Kamil Bajda-Pawlikowski, Daniel J Abadi, Alexander Rasin, Avi Silberschatz. HadoopDB: An architectural hybrid of MapReduce and DBMS technologies for analytical workloads//Proceedings of the 35th International Conference on Very Large Data Bases (VLDB'09). Lyon, France, 2009: 733-743
20. Hadapt Inc. http://www.hadapt.com
21. http://www.vertica.com/the-analytics-platform/ native-bi-etl-and-hadoop-mapreduce-integration/
22. Xu Y, Kostamaa P. Integrating hadoop and parallel DBMs//Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD'10). Indianapolis, Indiana, USA, 2010: 969-974
23. Upadhyaya P, Kwon Y C, Balazinska M. A latency and fault-tolerance optimizer for online parallel query plans//Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD'11). Athens, Greece, 2011: 241-252
24. Yang C, Yen C, Tan C, Madden S. Osprey: Implementing MapReduce-style fault tolerance in a shared-nothing distributed database//Proceedings of the 24th International Conference on Data Engineering (ICDE'10). Long Beach, California, USA, 2010: 657-668
25. He Yongqiang, Lee Rubao, Huai Yin, Shao Zheng, Jain Namit, Zhang Xiaodong, Xu Zhiwei. RCFile: A fast and space-efficient data placement structure in MapReduce-based warehouse systems//Proceedings of the 24th International Conference on Data Engineering (ICDE'11). Hannover, Germany, 2011: 1199-1208
26. Floratou A, Patel J M, Shekita E J, Tata Sandeep. Column-oriented storage techniques for MapReduce. PVLDB, 2011, 4(7): 419-429
27. Jens Dittrich, Jorge-Arnulfo Quiané-Ruiz, Alekh Jindal, Yagiz Kargin, Vinay Setty, Jörg Schad. Hadoop++: Making a yellow elephant run like a cheetah (without it even noticing). PVLDB, 2010, 3(1): 518-529
28. Condie T, Conway N, Alvaro P, Hellerstein J M, Elmeleegy K, Sears R. MapReduce online//Proceedings of the 7th USENIX Symposium on Networked Systems Design and Implementation (NSDI'10). San Jose, California, 2010: 313-328
29. Li Boduo, Mazur Edward, Diao Yanlei, McGregor Andrew, Shenoy Prashant J. A platform for scalable one-pass analytics using MapReduce//Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD'11). Athens, Greece, 2011: 985-996
30. Nykiel T, Potamias M, Mishra C, Kollios G, Koudas N. MRShare: Sharing across multiple queries in MapReduce. PVLDB, 2010, 3(1): 494-505
31. Blanas S, Patel Jignesh, Ercegovac V, Rao J, Shekita E J, Tian Y. A comparison of join algorithms for log processing in MaPreduce//Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD'10). Indianapolis, Indiana, USA, 2010: 975-986
32. Yang H-C, Dasdan A, Hsiao R-L, Parker D S. Map-reduce-merge: Simplified relational data processing on large clusters//Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD'07). Beijing, China, 2007: 1029-1040
33. Afrati F N, Ullman J D. Optimizing joins in a map-reduce environment//Proceedings of the 13th International Conference on Extending Database Technology. Lausanne, Switzerland, 2010: 99-110
34. Jiang D, Tung A K H, Chen G. Map-join-reduce: Towards scalable and efficient data analysis on large clusters. TKDE, 2010, 23(9): 1299-1311
35. Lin Y, Agrawal D, Chen C, Ooi B C, Wu S. Llama: Leveraging columnar storage for scalable join processing in the MapReduce framework//Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD'11). Athens, Greece, 2011: 961-972
36. Okcan A, Riedewald M. Processing theta-joins using MapReduce //Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD'11). Athens, Greece, 2011: 949-960
37. Wang Huiju, Wang Shan, Qin Xiongpai, Li Furong, Zhou Xuan, Qin Zuoyan, Zhu Qing. Efficient star query processing on Hadoop-A hierarchy encoding based approach (Technical report)
38. Bajda-Pawlikowski Kamil, Abadi Daniel J, Silberschatz Avi, Paulson Erik. Efficient processing of data warehousing queries in a split execution environment//Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD'11). Athens, Greece, 2011: 985-996. 2011: 1165-1176
39. Wang Huijui, Qin Xiongpai, Zhang Yansong, Wang Shan, Wang Zhanwei. LinearDB: A relational approach to make data warehouse scale like MapReduce//Proceedings of the Database Systems for Advanced Applications-16th International Conference (DASFAA'11). Hong Kong, China, 2011: 306-320
40. Karayannidis N, Tsois A, Sellis T K, Pieringer R, Markl V, Ramsak F, Fenk R, Elhardt K, Bayer R. Processing star queries on hierarchically-clustered fact tables//Proceedings of the 28th International Conference on Very Large Data Bases (VLDB'02). Hong Kong, China, 2002: 730-741
41. Qin Xiongpai, Wang Huiju, Du Xiaoyong, Wang Shan. Parallel aggregation queries over star schema: A hierarchical encoding scheme and efficient percentile computing as a case//Proceedings of the 9th IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA'11). Busan, Korea, 2011: 329-334
42. Das S, Sismanis Y, Beyer K S, Gemulla R, Haas P J, McPherson J. Ricardo: Integrating R and Hadoop//Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD'10). Athens, Greece, Indianapolis, Indiana, USA, 2010: 987-998