Determining bias to search engines from robots.txt

Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence, WI 2007

Volumn , Issue , 2007, Pages 149-155

  Sun, Yang ^a   Zhuang, Ziming ^a   Councill, Isaac G ^a   Giles, C Lee ^a  

^a PENNSYLVANIA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPETITION; COMPUTER SOFTWARE; ENGINES; INDUSTRIAL ECONOMICS; INDUSTRIAL MANAGEMENT; INFORMATION RETRIEVAL; INFORMATION SERVICES; INTERNET; INTERNET PROTOCOLS; LAWS AND LEGISLATION; PORTALS; ROBOTICS; ROBOTS; SEARCH ENGINES; STANDARDS; TELECOMMUNICATION NETWORKS;

BUSINESS DOMAINS; DE FACTO STANDARDS; INFORMATION ACCESS; INFORMATION PORTALS; INTERNATIONAL CONFERENCES; MARKET SHARES; RICH GET RICHER; SERVER SIDE; STATISTICAL ANALYSIS; STRONG CORRELATIONS; WEB INTELLIGENCE;

WORLD WIDE WEB;

EID: 48349091114     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/WI.2007.45     Document Type: Conference Paper

References (16)

1. Fortune magazine, http://money.cnn.com/magazines/fortune/ fortune500, 2005.
2. M. L. Boonk, D. R. A. d. Groot, F. M. T. Brazier, and A. Oskamp. Agent exclusion on websites. In Proceedings of The 4th Workshop on the Law and Electronic Agents, 2005.
3. S. Chakrabarti, M. Van den Berg, , and B. Dom. Focused crawling: A new approach to topic-specific web resource discovery. In Proc. of the 8th WWW Conference, pages 545-562, 1999.
4. J. Cho, H. Garcia-Molina, and L. Page. Efficient crawling through url ordering. In Proceedings of the 7th International World Wide Web Conference, 1998.
5. T. Y. Chun. World wide web robots: An overview. Online Information Review, 23(3): 135-142, 1999.
6. DMOZ. The open directory project, http://dmoz.org, 2005.
7. M. Drott. Indexing aids at corporate websites: The use of robots.txt and meta tags. Information Processing and Management, 38(2):209-219, 2002.
8. D. Eichmann. Ethical web agents. Computer Networks and ISDN Systems, 28(1-2): 127-136, 1995.
9. T. Joachims. Optimizing search engines using clickthrough data. In KDD '02: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133-142, New York, NY, USA, 2002. ACM Press.
10. B. Kelly and I. Peacock. Webwatching uk web communities: Final report for the webwatch project. British Library Research and Innovation Report, 1999.
11. M. Kendall. Rank Correlation Methods. Hafner, 1955.
12. M. Koster. A method for web robots control. In the Internet Draft, The Internet Engineering Task Force (IETF), 1996.
13. R. L. Ott and M. T. Longnecker. An Introduction to Statistical Methods and Data Analysis. Duxbury Press; 5th edition, 2000.
14. G. Pant, P. Srinivasan, and F Menczer. Crawling the Web, chapter Web Dynamics. Springer-Verlag, 2004.
15. R. R. Sokal and F J. Rohlf. Biometry. Freeman New York, 2001.
16. Y Sun, Z. Zhuang, and C. L. Giles. A large-scale study ofrobots.txt. In WWW '07: Proceedings of the 16th international conference on World Wide Web, pages 1123-1124, New York, NY, USA, 2007. ACM Press.