The unified logging infrastructure for data analytics at twitter

Proceedings of the VLDB Endowment

Volumn 5, Issue 12, 2012, Pages 1771-1780

  Lee, George ^a   Lin, Jimmy ^a   Liu, Chuang ^a   Lorek, Andrew ^a   Ryaboy, Dmitriy ^a  

^a TWITTER INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

QUERY PROCESSING; SOCIAL NETWORKING (ONLINE);

APPLICATION SPECIFIC; BASIC UNITS; DATA ANALYTICS; LARGE CLUSTERS; LARGE SCALE DATA; LOGGING INFRASTRUCTURE; PRODUCTION LOGGING; USER SESSIONS;

WELL LOGGING;

EID: 84873206169     PISSN: None     EISSN: 21508097     Source Type: Conference Proceeding    
DOI: 10.14778/2367502.2367516     Document Type: Article

References (30)

1. A. Abouzeid, K. Bajda-Pawlikowski, D. Abadi, A. Silberschatz, and A. Rasin. HadoopDB: An architectural hybrid of MapReduce and DBMS technologies for analytical workloads. VLDB, pp. 922-933, 2009.
2. R. Agrawal and R. Srikant. Mining sequential patterns. ICDE, pp. 3-14, 1995.
3. S. Altschul, W. Gish, W. Miller, E. Myers, and D. Lipman. Basic local alignment search tool. Journal of Molecular Biology, 215(3):403-410, 1990.
4. K. Bajda-Pawlikowski, D. Abadi, A. Silberschatz, and E. Paulson. Efficient processing of data warehousing queries in a split execution environment. SIGMOD, pp. 1165-1176, 2011.
5. D. Borthakur, J. Gray, J. Sarma, K. Muthukkaruppan, N. Spiegelberg, H. Kuang, K. Ranganathan, D. Molkov, A. Menon, S. Rash, R. Schmidt, and A. Aiyer. Apache Hadoop goes realtime at Facebook. SIGMOD, pp. 1071-1080, 2011.
6. K. Church and P. Hanks. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22-29, 1990.
7. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. OSDI, pp. 137-150, 2004.
8. J. Dittrich, J.-A. Quiané-Ruiz, A. Jindal, Y. Kargin, V. Setty, and J. Schad. Hadoop++: Making a yellow elephant run like a cheetah (without it even noticing). VLDB, pp. 515-529, 2010.
9. T. Dunning. Accurate methods for the statistics of surprise and coincidence. Computational Linguistics, 19(1):61-74, 1993.
10. A. Gates, O. Natkovich, S. Chopra, P. Kamath, S. Narayanamurthy, C. Olston, B. Reed, S. Srinivasan, and U. Srivastava. Building a high-level dataflow system on top of MapReduce: The Pig experience. VLDB, pp. 1414-1425, 2009.
11. Y. He, R. Lee, Y. Huai, Z. Shao, N. Jain, X. Zhang, and Z. Xu. RCFile: A fast and space-efficient data placement structure in MapReduce-based warehouse systems. ICDE, pp. 1199-1208, 2011.
12. P. Hunt, M. Konar, F. Junqueira, and B. Reed. ZooKeeper: Wait-free coordination for Internet-scale systems. USENIX, pp. 145-158, 2010.
13. A. Jindal, J.-A. Quiané-Ruiz, and J. Dittrich. Trojan data layouts: Right shoes for a running elephant. SoCC, pp. 21:1-21:14, 2011.
14. D. Jurafsky and J. Martin. Speech and Language Processing. Pearson, 2009.
15. D. Klein and C. Manning. A generative constituent-context model for improved grammar induction. ACL, pp. 128-135, 2002.
16. R. Kohavi, R. Henne, and D. Sommerfield. Practical guide to controlled experiments on the web: Listen to your customers not to the HiPPO. KDD, pp. 959-967, 2007.
17. J. Kreps, N. Narkhede, and J. Rao. Kafka: A distributed messaging system for log processing. NetDB, 2011.
18. J. Lin and A. Kolcz. Large-scale machine learning at twitter. SIGMOD, pp. 793-804, 2012.
19. J. Lin, D. Ryaboy, and K. Weil. Full-text indexing for optimizing selection operations in large-scale data analytics. MAPREDUCE Workshop, pp. 59-66, 2011.
20. J. Lin and W. Wilbur. Modeling actions of PubMed users with n-gram language models. Information Retrieval, 12(4):487-503, 2009.
21. Y. Lin, D. Agrawal, C. Chen, B. Ooi, and S. Wu. Llama: Leveraging columnar storage for scalable join processing in the MapReduce framework. SIGMOD, pp. 961-972, 2011.
22. T. Mah, H. Hoek, and Y. Li. Funnel report mining for the MSN network. KDD, pp. 450-455, 2001.
23. C. Manning and H. Schütze. Foundations of Statistical Natural Language Processing. MIT Press, 1999.
24. A. Montgomery, S. Li, K. Srinivasan, and J. Liechty. Modeling online browsing and path analysis using clickstream data. Marketing Science, 23(4):579-595, 2004.
25. A. Nandi, C. Yu, P. Bohannon, and R. Ramakrishnan. Distributed cube materialization on holistic measures. ICDE, pp. 183-194, 2011.
26. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: A not-so-foreign language for data processing. SIGMOD, pp. 1099-1110, 2008.
27. D. Pearce. A comparative evaluation of collocation extraction techniques. LREC, pp. 1530-1536, 2002.
28. A. Thusoo, J. Sarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Anthony, H. Liu, and R. Murthy. Hive-a petabyte scale data warehouse using Hadoop. ICDE, 2010.
29. A. Thusoo, Z. Shao, S. Anthony, D. Borthakur, N. Jain, J. Sarma, R. Murthy, and H. Liu. Data warehousing and analytics infrastructure at Facebook. SIGMOD, 2010.
30. K. Wongsuphasawat, J. Gómez, C. Plaisant, T. Wang, M. Taieb-Maimon, and B. Shneiderman. LifeFlow: Visualizing an overview of event sequences. CHI Extended Abstracts, pp. 507-510, 2011.