A large-scale study of robots.txt

16th International World Wide Web Conference, WWW2007

Volumn , Issue , 2007, Pages 1123-1124

  Sun, Yang ^a   Zhuang, Ziming ^a   Giles, C Lee ^a  

^a PENNSYLVANIA STATE UNIVERSITY   (United States)

Author keywords

Crawler; Robots exclusion protocol; Robots.txt; Search engine

Indexed keywords

ADMINISTRATIVE DATA PROCESSING; LARGE SCALE SYSTEMS; LOGIC PROGRAMMING; SEARCH ENGINES; WEB SERVICES;

CRAWLERS; ROBOTS EXCLUSION PROTOCOLS;

INTELLIGENT ROBOTS;

EID: 35348856355     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1242572.1242726     Document Type: Conference Paper

References (4)

1. M. Drott. Indexing aids at corporate websites: The use of robots.txt and meta tags. Information Processing and Management, 38(2):209-219, 2002.
2. B. Kelly and I. Peacock. Webwatching uk web communities: Final report for the webwatch project. British Library Research and Innovation Report, 1999.
3. M. Koster. A method for web robots control. In the Internet Draft, The Internet Engineering Task Force (IETF), 1996.
4. G. Pant, P. Srinivasan, and F. Menczer. Crawling the Web, chapter Web Dynamics. Springer-Verlag, 2004.