Earlybird: Real-time search at Twitter

Proceedings - International Conference on Data Engineering

Volumn , Issue , 2012, Pages 1360-1369

  Busch, Michael ^a   Gade, Krishna ^a   Larson, Brian ^a   Lok, Patrick ^a   Luckenbill, Samuel ^a   Lin, Jimmy ^a  

^a Twitter ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DESIGN SPACES; HIGH-THROUGHPUT; INDEX STRUCTURE; INVERTED INDICES; LOW-LATENCY; QUERY EVALUATION; REAL-TIME SEARCH SERVICE; REAL-TIME SIGNALS; RETRIEVAL ENGINES; WEB SEARCHES;

SEARCH ENGINES; STRUCTURES (BUILT OBJECTS);

SOCIAL NETWORKING (ONLINE);

EID: 84864197927     PISSN: 10844627     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICDE.2012.149     Document Type: Conference Paper

References (51)

1. T.-Y. Liu, "Learning to rank for information retrieval," Foundations and Trends in Information Retrieval, vol. 3, no. 3, pp. 225-331, 2009.
2. Y. Ganjisaffar, R. Caruana, and C. V. Lopes, "Bagging gradient-boosted trees for high precision, low variance ranking models," in Proceedings of the 34rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2011), Beijing, China, 2011, pp. 85-94.
3. H. Li, Learning to Rank for Information Retrieval and Natural Language Processing. Morgan & Claypool Publishers, 2011.
4. H. Cui, J.-R. Wen, and W.-Y. Ma, "Probabilistic query expansion using query logs," in Proceedings of the Eleventh International World Wide Web Conference (WWW 2002), Honolulu, Hawaii, 2002, pp. 325-332.
5. E. Agichtein, E. Brill, S. Dumais, and R. Ragno, "Learning user interaction models for predicting Web search result preferences," in Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006), Seattle, Washington, 2006, pp. 3-10.
6. T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay, "Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search," ACM Transactions on Information Systems, vol. 25, no. 2, pp. 1-27, 2007.
7. S. M. Beitzel, E. C. Jensen, D. D. Lewis, A. Chowdhury, and O. Frieder, "Automatic classification of Web queries using very large unlabeled query logs," ACM Transactions on Information Systems, vol. 25, no. 2, pp. 1-29, 2007.
8. C. Olston and M. Najork, "Web crawling," Foundations and Trends in Information Retrieval, vol. 4, no. 3, pp. 175-246, 2010.
9. L. A. Barroso, J. Dean, and U. Hölzle, "Web search for a planet: The Google cluster architecture," IEEE Micro, vol. 23, no. 2, pp. 22-28, 2003.
10. R. Baeza-Yates, C. Castillo, F. Junqueira, V. Plachouras, and F. Silvestri, "Challenges on distributed web retrieval," in Proceedings of the IEEE 23rd International Conference on Data Engineering (ICDE 2007), Istanbul, Turkey, 2007, pp. 6-20.
11. M. V. Vieira, B. M. Fonseca, R. Damazio, P. B. Golgher, D. de Castro Reis, and B. Ribeiro-Neto, "Efficient search ranking in social networks," in Proceedings of the Sixteenth International Conference on Information and Knowledge Management (CIKM 2007), Lisbon, Portugal, 2007, pp. 563-572.
12. D. Horowitz and S. D. Kamvar, "The anatomy of a large-scale social search engine," in Proceedings of the 19th International World Wide Web Conference (WWW 2010), Raleigh, North Carolina, 2010, pp. 431-440.
13. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," in Proceedings of the 6th USENIX Symposium on Operating System Design and Implementation (OSDI 2004), San Francisco, California, 2004, pp. 137-150.
14. D. Peng and F. Dabek, "Large-scale incremental processing using distributed transactions and notifications," in Proceedings of the 9th USENIX Symposium on Operating System Design and Implementation (OSDI 2010), Vancouver, British Columbia, Canada, 2010, pp. 251-264.
15. B. B. Cambazoglu, V. Plachouras, and R. Baeza-Yates, "Quantifying performance and quality gains in distributed web search engines," in Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2009), 2009, pp. 411-418.
16. B. B. Cambazoglu, E. Varol, E. Kayaaslan, C. Aykanat, and R. Baeza-Yates, "Query forwarding in geographically distributed search engines," in Proceedings of the 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2010), Geneva, Switzerland, 2010, pp. 90-97.
17. E. Kayaaslan, B. B. Cambazoglu, R. Blanco, F. P. Junqueira, and C. Aykanat, "Energy-price-driven query processing in multi-center web search engines," in Proceedings of the 34rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2011), Beijing, China, 2011, pp. 983-992.
18. J. Zobel and A. Moffat, "Inverted files for text search engines," ACM Computing Surveys, vol. 38, no. 6, pp. 1-56, 2006.
19. F. Leibert, J. Mannix, J. Lin, and B. Hamadani, "Automatic management of partitioned, replicated search services," in Proceedings of the 2nd ACM Symposium on Cloud Computing (SoCC '11), Cascais, Portugal, 2011.
20. R. Baeza-Yates, A. Gionis, F. Junqueira, V. Murdock, V. Plachouras, and F. Silvestri, "The impact of caching on search engines," in Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007), Amsterdam, The Netherlands, 2007, pp. 183-190.
21. G. Skobeltsyn, F. P. Junqueira, V. Plachouras, and R. Baeza-Yates, "ResIn: A combination of results caching and index pruning for high-performance web search engines," in Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2008), Singapore, 2008, pp. 131-138.
22. M. Najork, "The Scalable Hyperlink Store," in Proceedings of the 20th ACM Conference on Hypertext and Hypermedia (HT 2009), Torino, Italy, 2009, pp. 89-98.
23. J. Hamilton, "On designing and deploying Internet-scale services," in Proceedings of the 21st Large Installation System Administration Conference (LISA '07), Dallas, Texas, 2007, pp. 233-244.
24. L. Barroso and U. Hölzle, The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines. Morgan & Claypool, 2009.
25. A. Z. Broder, D. Carmel, M. Herscovici, A. Soffer, and J. Zien, "Efficient query evaluation using a two-level retrieval process," in Proceedings of the Twelfth International Conference on Information and Knowledge Management (CIKM 2003), New Orleans, Louisiana, 2003, pp. 426-434.
26. I. Matveeva, C. Burges, T. Burkard, A. Laucius, and L. Wong, "High accuracy retrieval with multiple nested ranker," in Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006), Seattle, Washington, 2006, pp. 437-444.
27. S. Tatikonda, B. B. Cambazoglu, and F. P. Junqueira, "Posting list intersection on multicore architectures," in Proceedings of the 34rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2011), Beijing, China, 2011, pp. 963-972.
28. S. E. Robertson, S. Walker, S. Jones, M. Hancock-Beaulieu, and M. Gatford, "Okapi at TREC-3," in Proceedings of the 3rd Text REtrieval Conference (TREC-3), Gaithersburg, Maryland, 1994, pp. 109-126.
29. L. Page, S. Brin, R. Motwani, and T. Winograd, "The PageRank citation ranking: Bringing order to the Web," Stanford University, Stanford Digital Library Working Paper SIDL-WP-1999-0120, 1999.
30. J. M. Kleinberg, "Authoritative sources in a hyperlinked environment," Journal of the ACM, vol. 46, no. 5, pp. 604-632, 1999.
31. J. Lin, D. Metzler, T. Elsayed, and L. Wang, "Of Ivory and Smurfs: Loxodontan MapReduce experiments for web search," in Proceedings of the Eighteenth Text REtrieval Conference (TREC 2009), Gaithersburg, Maryland, 2009.
32. C. J. C. Burges, "From RankNet to LambdaRank to LambdaMART: An overview," Microsoft Research, Tech. Rep. MSR-TR-2010-82, 2010.
33. B. B. Cambazoglu, H. Zaragoza, O. Chapelle, J. Chen, C. Liao, Z. Zheng, and J. Degenhardt, "Early exit optimizations for additive machine learned ranking systems," in Proceedings of the Third ACM International Conference on Web Search and Data Mining (WSDM 2010), New York, New York, 2010, pp. 411-420.
34. L. Wang, J. Lin, and D. Metzler, "A cascade ranking model for efficient ranked retrieval," in Proceedings of the 34th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2011), Beijing, China, 2011, pp. 105-114.
35. V. N. Anh, O. de Kretser, and A. Moffat, "Vector-space ranking with effective early termination," in Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2001), New Orleans, Louisiana, 2001, pp. 35-42.
36. V. N. Anh and A. Moffat, "Simplified similarity scoring using term ranks," in Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2005), Salvador, Brazil, 2005, pp. 226-233.
37. H. Turtle and J. Flood, "Query evaluation: Strategies and optimizations," Information Processing and Management, vol. 31, no. 6, pp. 831-850, 1995.
38. A. Moffat and J. Zobel, "Self-indexing inverted files for fast text retrieval," ACM Transactions on Information Systems, vol. 14, no. 4, pp. 349-379, 1996.
39. T. Strohman, H. Turtle, and W. B. Croft, "Optimization strategies for complex queries," in Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2005), Salvador, Brazil, 2005, pp. 219-225.
40. T. Strohman and W. B. Croft, "Efficient document retrieval in main memory," in Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007), Amsterdam, The Netherlands, 2007, pp. 175-182.
41. F. Diaz, "Integration of news content into web results," in Proceedings of the Second ACM International Conference on Web Search and Data Mining (WSDM 2009), Barcelona, Spain, 2009, pp. 182-191.
42. K. M. Risvik, Y. Aasheim, and M. Lidal, "Multi-tier architecture for web search engines," in Proceedings of the 1st Latin American Web Congress, Santiago, Chile, 2003, pp. 132-143.
43. R. A. Baeza-Yates, V. Murdock, and C. Hauff, "Efficiency trade-offs in two-tier web search systems," in Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2009), 2009, pp. 163-170.
44. H. Yan, S. Ding, and T. Suel, "Inverted index compression and query processing with optimized document ordering," in Proceedings of the 18th International World Wide Web Conference (WWW 2009), Madrid, Spain, 2009, pp. 401-410.
45. J. Zhang, X. Long, and T. Suel, "Performance of compressed inverted list caching in search engines," in Proceedings of the 17th International World Wide Web Conference (WWW 2008), Beijing, China, 2008, pp. 387-396.
46. V. N. Anh and A. Moffat, "Inverted index compression using word-aligned binary codes," Information Retrieval, vol. 8, no. 1, pp. 151-166, 2005.
47. B. Goetz, Java Concurrency in Practice. Addison-Wesley, 2006.
48. S. Mizzaro, "How many relevances in information retrieval?" Interacting With Computers, vol. 10, no. 3, pp. 305-322, 1998.
49. K. Gimpel, N. Schneider, B. O'Connor, D. Das, D. Mills, J. Eisenstein, M. Heilman, D. Yogatama, J. Flanigan, and N. A. Smith, "Part-of-speech tagging for Twitter: Annotation, features, and experiments," in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT 2011), Portland, Oregon, 2011, pp. 42-47.
50. A. Ritter, S. Clark, Mausam, and O. Etzioni, "Named entity recognition in Tweets: An experimental study," in Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP 2010), Edinburgh, Scotland, 2011, pp. 1524-1534.
51. S. Gouws, D. Metzler, C. Cai, and E. Hovy, "Contextual bearing on linguistic variation in social media," in Proceedings of the ACL-HLT 2011 Workshop on Language in Social Media, Portland, Oregon, 2011.