Large-scale incremental processing using distributed transactions and notifications

Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2010

Volumn , Issue , 2019, Pages 251-264

  Peng, Daniel ^a   Dabek, Frank ^a  

^a GOOGLE INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BATCH DATA PROCESSING; DATA HANDLING; DIGITAL STORAGE; INDEXING (MATERIALS WORKING); INDEXING (OF INFORMATION); SYSTEMS ANALYSIS;

AVERAGE AGES; BATCH PROCESSING SYSTEM; DISTRIBUTED TRANSACTION; INCREMENTAL PROCESSING; INDEXING SYSTEMS; LARGE DATASETS; MAP-REDUCE; WEB SEARCHES;

SEARCH ENGINES;

EID: 85076901170     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (35)

1. TPC benchmark E standard specification version 1.9.0. Tech. rep., Transaction Processing Performance Council, September 2009.
2. AGUILERA, M. K., KARAMANOLIS, C., MERCHANT, A., SHAH, M., AND VEITCH, A. Building distributed applications using Sinfonia. Tech. rep., Hewlett-Packard Labs, 2006.
3. AGUILERA, M. K., MERCHANT, A., SHAH, M., VEITCH, A., AND KARAMANOLIS, C. Sinfonia: a new paradigm for building scalable distributed systems. In SOSP'07 (2007), ACM, pp. 159-174.
4. BARU, C., FECTEAU, G., GOYAL, A., HSIAO, H.-I., JHINGRAN, A., PADMANABHAN, S., WILSON, W., AND I HSIAO, A. G. H. DB2 parallel edition, 1995.
5. BERENSON, H., BERNSTEIN, P., GRAY, J., MELTON, J., O'NEIL, E., AND O'NEIL, P. A critique of ANSI SQL isolation levels. In SIGMOD (New York, NY, USA, 1995), ACM, pp. 1-10.
6. BERNSTEIN, P. A., AND GOODMAN, N. Concurrency control in distributed database systems. ACM Computer Surveys 13, 2 (1981), 185-221.
7. BORAL, H., ALEXANDER, W., CLAY, L., COPELAND, G., DANFORTH, S., FRANKLIN, M., HART, B., SMITH, M., AND VALDURIEZ, P. Prototyping Bubba, a highly parallel database system. IEEE Transactions on Knowledge and Data Engineering 2, 1 (1990), 4-24.
8. BURROWS, M. The Chubby lock service for loosely-coupled distributed systems. In 7th OSDI (Nov. 2006).
9. CHANG, F., DEAN, J., GHEMAWAT, S., HSIEH, W. C., WALLACH, D. A., BURROWS, M., CHANDRA, T., FIKES, A., AND GRUBER, R. E. Bigtable: A distributed storage system for structured data. In 7th OSDI (Nov. 2006), pp. 205-218.
10. CONDIE, T., CONWAY, N., ALVARO, P., AND HELLERSTIEN, J. M. MapReduce online. In 7th NSDI (2010).
11. COOPER, B. F., RAMAKRISHNAN, R., SRIVASTAVA, U., SILBERSTEIN, A., BOHANNON, P., JACOBSEN, H.-A., PUZ, N., WEAVER, D., AND YERNENI, R. PNUTS: Yahoo!'s hosted data serving platform. In Proceedings of VLDB (2008).
12. DAS, S., AGRAWAL, D., AND ABBADI, A. E. ElasTraS: An elastic transactional data store in the cloud. In USENIX HotCloud (June 2009).
13. DEAN, J., AND GHEMAWAT, S. MapReduce: Simplified data processing on large clusters. In 6th OSDI (Dec. 2004), pp. 137-150.
14. DECANDIA, G., HASTORUN, D., JAMPANI, M., KAKULAPATI, G., LAKSHMAN, A., PILCHIN, A., SIVASUBRAMANIAN, S., VOSSHALL, P., AND VOGELS, W. Dynamo: Amazon's highly available key-value store. In SOSP'07 (2007), pp. 205-220.
15. DEWITT, D., GHANDEHARIZADEH, S., SCHNEIDER, D., BRICKER, A., HSIAO, H.-I., AND RASMUSSEN, R. The gamma database machine project. IEEE Transactions on Knowledge and Data Engineering 2 (1990), 44-62.
16. DEWITT, D., AND STONEBRAKER, M. MapReduce: A major step backwards. http://databasecolumn.vertica.com/database-innovation/mapreduce-a-major-step-backwards/..
17. DEWITT, D. J., AND GRAY, J. Parallel database systems: the future of database processing or a passing fad? SIGMOD Rec. 19, 4 (1990), 104-112.
18. EKANAYAKE, J., LI, H., ZHANG, B., GUNARATHNE, T., BAE, S.-H., QIU, J., AND FOX, G. Twister: A runtime for iterative MapReduce. In The First International Workshop on MapReduce and its Applications (2010).
19. GERSHENSON, C., AND PINEDA, L. A. Why does public transport not arrive on time? The pervasiveness of equal headway instability. PLoS ONE 4, 10 (10 2009).
20. GHEMAWAT, S., GOBIOFF, H., AND LEUNG, S.-T. The Google file system. vol. 37, pp. 29-43.
21. GUPTA, A., AND MUMICK, I. S. Maintenance of materialized views: Problems, techniques, and applications, 1995.
22. Hadoop. http://hadoop.apache.org/.
23. HBase. http://hbase.apache.org/.
24. ISARD, M., BUDIU, M., YU, Y., BIRRELL, A., AND FETTERLY, D. Dryad: Distributed data-parallel programs from sequential building blocks. In EuroSys'07 (New York, NY, USA, 2007), ACM, pp. 59-72.
25. IYER, S., AND CUTTS, M. Help test some next-generation infrstructure. http://googlewebmastercentral.blogspot.com/2009/08/help-test-some-next-generation.html, August 2009.
26. LOGOTHETIS, D., OLSTON, C., REED, B., WEBB, K. C., AND YOCUM, K. Stateful bulk processing for incremental analytics. In SoCC'10: Proceedings of the 1st ACM symposium on cloud computing (2010), pp. 51-62.
27. MongoDB. http://mongodb.org/.
28. PAGE, L., BRIN, S., MOTWANI, R., AND WINOGRAD, T. The PageRank citation ranking: Bringing order to the web. Tech. rep., Stanford Digital Library Technologies Project, 1998.
29. PATON, N. W., AND DÍAZ, O. Active database systems. ACM Computing Surveys 31, 1 (1999), 63-103.
30. PAVLO, A., PAULSON, E., RASIN, A., ABADI, D. J., DEWITT, D. J., MADDEN, S., AND STONEBRAKER, M. A comparison of approaches to large-scale data analysis. In SIGMOD'09 (June 2009), ACM.
31. POPA, L., BUDIU, M., YU, Y., AND ISARD, M. DryadInc: Reusing work in large-scale computations. In USENIX workshop on Hot Topics in Cloud Computing (2009).
32. STONEBRAKER, M. Operating system support for database management. Communications of the ACM 24, 7 (1981), 412-418.
33. NEC Express5800/A1080a-E TPC-E results. http://www.tpc.org/tpce/results/tpce result detail.asp?id=110033001, Mar. 2010.
34. WEI, Z., PIERRE, G., AND CHI, C.-H. CloudTPS: Scalable transactions for Web applications in the cloud. Tech. Rep. IR-CS-053, Vrije Universiteit, Amsterdam, The Netherlands, Feb. 2010. http://www.globule.org/publi/CSTWAC ircs53.html.
35. ZAHARIA, M., CHOWDHURY, M., FRANKLIN, M., SHENKER, S., AND STOICA, I. Spark: Cluster computing with working sets. In 2nd USENIX workshop on Hot Topics in Cloud Computing (2010).