The implementation of a web crawler URL filter algorithm based on caching

2nd International Workshop on Computer Science and Engineering, WCSE 2009

Volumn 2, Issue , 2009, Pages 453-456

  Wang, Hui Chang ^a   Ruan, Shu Hua ^a   Tang, Qi Jie ^a  

^a SICHUAN UNIVERSITY   (China)

Author keywords

Caching; URL filter; Web crawler

Indexed keywords

CENTRAL COMPONENT; COLLECTION SYSTEMS; FILTER ALGORITHM; FILTER MODULE; INDEX PAGES; WEB CRAWLERS; WEB INFORMATION; WEB PAGE;

ALGORITHMS; COMPUTER SCIENCE; EXPERIMENTS; SEARCH ENGINES; WEBSITES;

INTERNET;

EID: 77950230236     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/WCSE.2009.851     Document Type: Conference Paper

References (4)

1. Yuan Wan, and Hengqing Tong, "URL Assignment Algorithm of Crawler in Distributed System Based on Hash," IEEE International Conference on Networking, Sensing and Control, 2008.
2. WU Lihui, WANG Bin, and YU Zhihua, "Design and Realization of a General Web Crawler," Computer Engineering, February 2005, pp.123-124.
3. Christopher Martinez, Wei-Ming Lin, and Parimal Patel, "Optimal XOR Hashing for A Linearly Distributed Address Lookup in Computer Networks," Symposium On Architecture For Networking And Communications Systems, 2005, pp. 203-210.
4. Xiao-Guang Liu, and Jun Lee, "K-Divided Bloom Filter Algorithm and Its Analysis," Future generation communication and networking (fgcn 2007), 2007, 1(6-8), pp. 220-224.