Architecture of a grid-enabled Web search engine

Information Processing and Management

Volumn 43, Issue 3, 2007, Pages 609-623

  Cambazoglu, B Barla ^a   Karaca, Evren ^a   Kucukyilmaz, Tayfun ^a   Turk, Ata ^a   Aykanat, Cevdet ^a  

^a BILKENT UNIVERSITY   (Turkey)

Author keywords

Grid computing; Search engine; Text classification; Web crawling

Indexed keywords

SOCIAL ASPECTS; WEBSITES; WORLD WIDE WEB;

GRID COMPUTING; TEXT CLASSIFICATION; WEB CRAWLING;

SEARCH ENGINES;

EID: 33846441233     PISSN: 03064573     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ipm.2006.10.011     Document Type: Article

References (46)

1. Altingovde I.S., and Ulusoy O. Exploiting interclass rules for focused crawling. IEEE Intelligent Systems 19 6 (2004) 66-73
2. Arasu A., Cho J., Garcia-Molina H., and Raghavan S. Searching the Web. ACM Transactions on Internet Technologies 1 1 (2001) 2-43
3. Baeza-Yates, R., Castillo, C., Marin, M., & Rodriguez, A. (2005). Crawling a country: better strategies than breadth-first for Web page ordering. In Special interest tracks and posters of the 14th international conference on World Wide Web. Chiba, Japan.
4. Baeza-Yates R., and Ribeiro-Neto B. Modern information retrieval (1999), Addison-Wesley, New York
5. Bender, M., Michel, S., Triantafillou, P., Weikum, G., & Zimmer, C. (2005). Improving collection selection with overlap awareness in P2P search engines. In Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval (pp. 67-74). Salvador, Brazil.
6. Boldi, P., Codenotti, B., Santini, M., & Vigna, S. (2002). Ubicrawler: a scalable fully distributed Web crawler. In Proceedings of AusWeb02, the eighth Australian World Wide Web conference.
7. Cambazoglu, B. B., & Aykanat, C. (2005). Harbinger machine learning toolkit manual. Technical Report, BU-CE-0502, Bilkent University, Department of Computer Engineering. Ankara, Turkey.
8. Cambazoglu B.B., and Aykanat C. Performance of query processing implementations in ranking-based text retrieval systems using inverted indices. Information Processing & Management 42 4 (2006) 875-898
9. Cambazoglu B.B., Turk A., and Aykanat C. Data-parallel Web crawling models. Lecture Notes in Computer Science 3280 (2004) 801-809
10. Can F., Altingovde I.S., and Demir E. Efficiency and effectiveness of query processing in cluster-based retrieval. Information Systems 29 8 (2004) 697-717
11. Chakrabarti S., van den Berg M., and Dom B. Focused crawling: a new approach to topic-specific Web resource discovery. Computer Networks 31 11-16 (1999) 1623-1640
12. Cho, J., & Garcia-Molina, H. (2000). The evolution of the Web and implications for an incremental crawler. In Proceedings of the 26th international conference on very large data bases (pp. 200-209). Cairo, Egypt.
13. Cho, J., & Garcia-Molina, H. (2002). Parallel Crawlers. In Proceedings of the seventh World-Wide Web conference (pp. 124-135).
14. Cho, J., Garcia-Molina, H., & Page, L. (1998). Efficient crawling through URL ordering. In Proceedings of the 7th international World Wide Web conference (pp. 161-172). Brisbane, Australia.
15. Clarke C.L.A., Cormack G.V., and Tudhope E.A. Relevance ranking for one to three term queries. Information Processing and Management 36 2 (2000) 291-311
16. Diligenti, M., Coetzee, F., Lawrence, S., Giles, C. L., & Gori, M. (2000). Focused crawling using context graphs. In Proceedings of the 26th international conference on very large data bases (pp. 527-534). Cairo, Egypt.
17. Foster I., and Kesselman C. The grid 2: Blueprint for a new computing infrastructure (2003), Morgan Kaufman, San Francisco
18. Han, E., Karypis, G., & Kumar, V. (2002). Text categorization using weight adjusted k-nearest neighbor classification. In Proceedings of the 5th Pacific-Asia conference on knowledge discovery and data mining (pp. 53-65).
19. Heydon A., and Najork M. Mercator: a scalable, extensible Web crawler. World Wide Web 2 4 (1999) 219-229
20. Kan, M.-Y. (2004). Web page categorization without the Web page. In Proceedings of the 13th international World Wide Web conference (pp. 262-263).
21. Khoussainov R., Zuo X., and Kushmerick N. Grid-enabled Weka: a toolkit for machine learning on the grid. ERCIM News 59 (2004)
22. Lam W., Ruiz M.E., and Srinivasan P. Automatic text categorization and its applications to text retrieval. IEEE Transactions on Knowledge and Data Engineering 11 6 (1999) 865-879
23. Lee D.L., Chuang H., and Seamons K. Document ranking and the vector-space model. IEEE Software 14 2 (1997) 67-75
24. Lewis, D. D. (1992). Feature selection and feature extraction for text categorization. In Proceedings of speech and natural language workshop (pp. 212-217).
25. Lewis, D. D., & Ringuette, M. (1994). A comparison of two learning algorithms for text categorization. In Proceedings of the third annual symposium on document analysis and information retrieval (pp. 81-93).
26. Long, X., & Suel, T. (2003). Optimized query execution in large search engines. In Proceedings of the 29th international conference on very large databases. Berlin, Germany.
27. McCallum, A., & Nigam, K. (1998). A comparison of event models for naive bayes text classification. In AAAI-98 Workshop on learning for text categorization.
28. Melnik S., Raghavan S., Yang B., and Garcia-Molina H. Building a distributed full-text index for the Web. ACM Transactions on Information Systems 19 3 (2001) 217-241
29. Moffat A., Zobel J., and Sacks-Davis R. Memory efficient ranking. Information Processing and Management 30 6 (1994) 733-744
30. Najork, M., & Wiener, J. L. (2001). Breadth-first crawling yields high-quality pages. In Proceedings of the 10th international conference on World Wide Web (pp. 114-118). Hong Kong, Hong Kong.
31. Ng, H. T., Goh, W. B., & Low, K. L. (1997). Feature selection, perceptron learning, and a usability case study for text categorization. In Proceedings of the 20th international conference on research and development in information retrieval (pp. 67-73).
32. Page, L., & Brin, S. (1998). The anatomy of a large-scale hypertextual Web search engine. In Proceedings of the seventh World-Wide Web conference (pp. 107-117).
33. Ribeiro-Neto, B. A., & Barbosa, R. A. (1998). Query performance for tightly coupled distributed digital libraries. In Proceedings of the third ACM conference on digital libraries (pp. 182-190).
34. Scholze, F., Haya, G., Vigen, J., & Prazak, P. (2004). Project GRACE: a grid based search tool for the global digital library. In 7th international conference on electronic theses and dissertations. Lexington, KY.
35. Sebastiani F. Machine learning in automated text categorization. ACM Computing Surveys 34 1 (2002) 1-47
36. Shkapenyuk, V., & Suel, T. (2002). Design and implementation of a high-performance distributed Web crawler. In International conference on data engineering (pp. 357-368).
37. Sun, A., Lim, E. P., & Ng, W. K. (2002). Web classification using support vector machine. In Proceedings of the 4th international workshop on Web information and data management (pp. 96-99).
38. Teng, S., Lu, Q., Eichstaedt, M., Ford, D., & Lehman, T. (1999). Collaborative Web crawling: information gathering/processing over Internet. In 32nd Hawaii international conference on system sciences.
39. Tomasic, A., Garcia-Molina, H., & Shoens, K. (1994). Incremental updates of inverted lists for text document retrieval. In Proceedings of the 1994 ACM SIGMOD international conference on management of data (pp. 289-300). Minneapolis, Minnesota.
40. Turtle H., and Flood J. Query evaluation: strategies and optimizations. Information Processing and Management 31 6 (1995) 831-850
41. Wilkinson, R., Zobel, J., & Sacks-Davis, R. (1995). Similarity measures for short queries. In Fourth text retrieval conference (TREC-4) (pp. 277-285). Gaithersburg, Maryland.
42. Witten I.H., and Frank E. Data mining: Practical machine learning tools and techniques. 2nd ed. (2005), Morgan Kaufman, San Francisco
43. Wong W.Y.P., and Lee D.K. Implementations of partial document ranking using inverted files. Information Processing and Management 29 5 (1993) 647-669
44. Yang Y. An evaluation of statistical approaches to text categorization. Journal of Information Retrieval 1 1/2 (1999) 67-88
45. Zeinalipour-Yazti, D., & Dikaiakos, M. D. (2002). Design and implementation of a distributed crawler and filtering processor. In Proceedings of the next generation information technologies and systems (pp. 58-74).
46. Zobel, J., Moffat, A., & Sacks-Davis, R. (1992). An efficient indexing technique for full-text database systems. In Proceedings of the 18th international conference on very large databases (pp. 352-362). Vancouver, Canada.