Scalability of the Nutch search engine

Proceedings of the International Conference on Supercomputing

Volumn , Issue , 2007, Pages 3-12

  Moreira, José E ^a   Michael, Maged M ^a   Da Silva, Dilma ^a   Shiloach, Doron ^a   Dube, Parijat ^a   Zhang, Li ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

Lucene; Nutch; Query; Scale out; Search

Indexed keywords

OPEN SYSTEMS; PARALLEL ALGORITHMS; QUERY LANGUAGES;

COMMERCIAL SCALE OUT PROJECT; QUERY RATE; SCALE-OUT ARCHITECTURES;

SEARCH ENGINES;

EID: 34548041191     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1274971.1274975     Document Type: Conference Paper

References (18)

1. M. Cafarella and D. Cutting. Building Nutch: open source search. ACM Queue. Vol. 2, no. 2, pp. 54-61, 2004.
2. M. Cafarella and O. Etzioni. A search engine for natural language applications. WWW '05: Proceedings of the 14th International World Wide Web Conference. Chiba, Japan. 2005. pp. 442-452.
3. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. Proceedings of the Sixth Symposium on Operating System Design and Implementation (OSDI'04). San Francisco, CA, December, 2004.
4. D. M. Desai, T. M. Bradicich, D. Champion, W. G. Holland, and B. M. Kreuz. BladeCenter system overview. IBM Journal of Research and Development. Vol. 49, no. 6. 2005.
5. D. Gruhl, L. Chavet, D. Gibson, J. Meyer, P. Pattanayak, A. Tomkins, and J. Zien. How to build a WebFountain: An architecture for very large-scale text analytics. IBM Systems Journal. Vol. 13, no 1, 2006 pp. 64-77.
6. E. Hatcher and O. Gospodnetic. Lucene in Action. Manning Publications. 2004.
7. J. E. Hughes, P. S. Patel, I. R. Zapata, T. D. Pahel, Jr., J. P. Wong, D. M. Desai, and B. D. Herrman. BladeCenter midplane and media interface card. IBM Journal of Research and Development. Vol. 49, no. 6. 2005.
8. J. E. Hughes, M. L. Scollard, R. Land, J. Parsonese, C. C. West, V. A. Stankevich, C. L. Purrington, D. Q. Hoang, G. R. Shippy, M. L. Loeb, M. W. Williams, B. A. Smith, and D. M. Desai. BladeCenter processor blades, I/O expansion adapters, and units. IBM Journal of Research and Development. Vol. 49, no. 6. 2005.
9. Z. Liu, C. H. Xia, P. Momcilovic, and L. Zhang. AMBIENCE: Automatic Model Building using InfErENCE. In Congress MSR03, Metz, France, Oct. 2003.
10. H. M. Mathis, J. D. McCalpin, and J. Thomas. IBM p5 575 ultra-dense, modular cluster node for high performance computing. IBM Systems and Technology Group. October 2005.
11. H. M. Mathis, A. E. Mericas, J. D. McCalpin, R. J. Eickemeyer, and S. R. Kunkel. Characterization of simultaneous multithreading (SMT) efficiency in POWERS. IBM Journal of Research and Development. Vol. 49, no. 4/5. 2005.
12. M. Michael, J. E. Moreira, D. Shiloach, and R. Wisniewski. Scale-up x Scale-out: A Case Study using Nutch/Lucene. Third International Workshop on System Management Techniques, Processes, and Services (SMTPS). Held in conjunction with the 2007 International Parallel and Distributed. Processing Symposium. (IPDPS 2007). Long Beach, CA, March 30th, 2007.
13. F. Schmuck and R. Haskin. GPFS: A Shared-Disk File System for Large Computing Clusters. First Conference on File and Storage Technologies (FAST), pp. 231-244. January 2002.
14. B. Sinharoy, R. N. Kalla, J. M. Tendler, R. J. Eickemeyer, and J. B. Joyner. POWER5 system microarchitecture. IBM Journal of Research and Development. Vol. 49, no. 4/5. 2005.
15. University of Mannheim, University of Tennessee, and NERSC/LBNL. TOP500 Supercomputer sites. http://www.top500.org/.
16. E. Varki. Mean value technique for closed fork-join networks. In ACM Sigmetrics, pages 103-112, 1999.
17. rd Edition. Wiley, 2005.
18. L. Zhang, C. Xia, M. S. Squillante, and W. N. Mills III. Web workload service requirement analysis: A queueing network approach. In MASCOTS, 2002.