All aboard the databus!: Linkedin's scalable consistent change data capture platform

Proceedings of the 3rd ACM Symposium on Cloud Computing, SoCC 2012

Volumn , Issue , 2012, Pages

  Das, Shirshanka ^a   Botev, Chavdar ^a   Surlaker, Kapil ^a   Ghosh, Bhaskar ^a   Varadarajan, Balaji ^a   Nagaraj, Sunil ^a   Zhang, David ^a   Gao, Lei ^a   Westerman, Jemiah ^a   Ganti, Phanindra ^a   Shkolnik, Boris ^a   Topiwala, Sajid ^a   Pachev, Alexander ^a   Somasundaram, Naveen ^a   Subramaniam, Subbu ^a  

^a LINKEDIN CORPORATION   (United States)

Author keywords

Algorithms; CDC; Change data capture; Design; Replication; Stream processing

Indexed keywords

BUSINESS LOGIC; CDC; COMPLEX PROCESSING; COMPLEX QUERIES; DATA ARCHITECTURES; DATA BUS; DATA CAPTURE SYSTEMS; DATA PROCESSING PIPELINES; DATA SYSTEMS; DERIVED DATA; INTEGRAL PART; INTERNET ARCHITECTURE; PRIMARY DATA; REPLICATION; STREAM PROCESSING; TRANSPORT LAYERS;

ALGORITHMS; CLOUD COMPUTING; DATA PROCESSING; DESIGN; SOCIAL NETWORKING (ONLINE);

DIGITAL STORAGE;

EID: 84870551992     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2391229.2391247     Document Type: Conference Paper

References (14)

1. ActiveMQ. http://activemq.apache.org/.
2. Avro. http://avro.apache.org.
3. Linkedin infrastructure. QCON'2007, http://bit.ly/FQNqWI.
4. MySQL Replication. http://dev.mysql.com/doc/refman/5.0/en/replication. html.
5. Oracle DataGuard. http://www.oracle.com/technetwork/database/features/ availability/dataguardoverview-083155.html.
6. Oracle GoldenGate. http://www.oracle.com/technetwork/middleware/ goldengate/overview/index.html.
7. Tungsten Replicator. http://www.continuent.com/solutions/tungsten- replicator.
8. J. Gray. Notes on data base operating systems. In Operating Systems, An Advanced Course, pages 393-481, London, UK, UK, 1978. Springer-Verlag.
9. J. Kreps, N. Narkhede, and J. Rao. Kafka: a distributed messaging system for log processing, 2011.
10. L. Lamport. The part-time parliament. ACM Trans. Comput. Syst., 16(2):133-169, 1998.
11. LinkedIn Data Infrastructure Team. Data infrastructure at LinkedIn. In ICDE, 2012.
12. R. Ramamurthy, D. J. DeWitt, and Q. Su. A case for fractured mirrors. In Proceedings of the 28th international conference on Very Large Data Bases, VLDB '02, pages 430-441. VLDB Endowment, 2002.
13. M. Stonebraker, S. Madden, D. J. Abadi, S. Harizopoulos, N. Hachem, and P. Helland. The end of an architectural era: (it's time for a complete rewrite). In Proceedings of the 33rd international conference on Very large data bases, VLDB '07, pages 1150-1160. VLDB Endowment, 2007.
14. L. Wong, N. S. Arora, L. Gao, T. Hoang, and J. Wu. Oracle streams: a high performance implementation for near real time asynchronous replication, 2009.