Boilerplate detection using shallow text features

WSDM 2010 - Proceedings of the 3rd ACM International Conference on Web Search and Data Mining

Volumn , Issue , 2010, Pages 441-450

  Kohlschütter, Christian ^a   Fankhauser, Peter ^a   Nejdl, Wolfgang ^a  

^a L3S RESEARCH CENTER   (Germany)

Author keywords

Boilerplate removal; Full text extraction; Template detection; Text cleaning; Web document modeling

Indexed keywords

CREATION PROCESS; DETECTION ACCURACY; RETRIEVAL PERFORMANCE; TEMPLATE DETECTION; TEXT ELEMENTS; TEXT EXTRACTION; TEXT FEATURE; WEB DOCUMENT; WEB DOCUMENT MODELING; WEB PAGE;

INFORMATION RETRIEVAL; STOCHASTIC MODELS;

WORLD WIDE WEB;

EID: 77950904942     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1718487.1718542     Document Type: Conference Paper

References (26)

1. G. Altmann. Quantitative Linguistics - an international handbook, chapter Diversification processes. de Gruyter, 2005.
2. S. Baluja. Browsing on small screens: recasting web-page segmentation into an efficient machine learning framework. In WWW '06: Proceedings of the 15th international conference on World Wide Web, pages 33-42, New York, NY, USA, 2006. ACM.
3. Z. Bar-Yossef and S. Rajagopalan. Template detection via data mining and its applications. In WWW, pages 580-591, 2002.
4. M. Baroni, F. Chantree, A. Kilgarriff, and S. Sharoff. Cleaneval: a competition for cleaning web pages. In N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odjik, S. Piperidis, and D. Tapias, editors, Proceedings of the Sixth International Language Resources and Evaluation (LREC'08), 2008.
5. D. Cai, S. Yu, J.-R. Wen, and W.-Y. Ma. Extracting content structure for web pages based on visual representation. In X. Zhou, Y. Zhang, and M. E. Orlowska, editors, APWeb, volume 2642 of LNCS, pages 406-417. Springer, 2003.
6. D. Chakrabarti, R. Kumar, and K. Punera. Page-level template detection via isotonic smoothing. In WWW '07: Proc. of the 16th int. conf. on World Wide Web, pages 61-70, New York, NY, USA, 2007. ACM.
7. D. Chakrabarti, R. Kumar, and K. Punera. A graph-theoretic approach to webpage segmentation. In WWW '08: Proceeding of the 17th international conference on World Wide Web, pages 377-386, New York, NY, USA, 2008. ACM.
8. Y. Chen, W.-Y. Ma, and H.-J. Zhang. Detecting web page structure for adaptive viewing on small form factor devices. In WWW '03: Proceedings of the 12th international conference on World Wide Web, pages 225-233, New York, NY, USA, 2003. ACM.
9. S. Debnath, P. Mitra, N. Pal, and C. L. Giles. Automatic identification of informative sections of web pages. IEEE Trans. on Knowledge and Data Engineering, 17(9):1233-1246, 2005.
10. S. Evert. A lightweight and efficient tool for cleaning web pages. In N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odjik, S. Piperidis, and D. Tapias, editors, Proceedings of the Sixth International Language Resources and Evaluation (LREC'08), Marrakech, Morocco, may 2008. European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2008/.
11. D. Fernandes, E. S. de Moura, B. Ribeiro-Neto, A. S. da Silva, and M. A. Gonçalves. Computing block importance for searching on web sites. In CIKM '07, pages 165-174, 2007.
12. A. Ferraresi, E. Zanchetta, M. Baroni, and S. Bernardini. Introducing and evaluating ukwac, a very large web-derived corpus of english. In Proceedings of the WAC4 Workshop at LREC 2008.
13. A. Finn, N. Kushmerick, and B. Smyth. Fact or fiction: Content classification for digital libraries. Joint DELOS-NSF Workshop on Personalisation and Recommender Systems in Digital Libraries (Dublin), 2001.
14. D. Gibson, K. Punera, and A. Tomkins. The volume and evolution of web page templates. In WWW'05, pages 830-839, New York, NY, USA, 2005. ACM.
15. J. Gibson, B. Wellner, and S. Lubar. Adaptive web-page content identification. In WIDM '07: Proceedings of the 9th annual ACM international workshop on Web information and data management, pages 105-112, New York, NY, USA, 2007. ACM.
16. K. Hofmann and W. Weerkamp. Web corpus cleaning using content and structure. In Building and Exploring Web Corpora, pages 145-154. UCL Presses Universitaires de Louvain, September 2007.
17. H.-Y. Kao, J.-M. Ho, and M.-S. Chen. Wisdom: Web intrapage informative structure mining based on document object model. Knowledge and Data Engineering, IEEE Transactions on, 17(5):614-627, May 2005.
18. C. Kohlschütter. A densitometric analysis of web template content. In WWW '09: Proc. of the 18th intl. conf. on World Wide Web, New York, NY, USA, 2009. ACM.
19. C. Kohlschütter and W. Nejdl. A Densitometric Approach to Web Page Segmentation. In ACM 17th Conf. on Information and Knowledge Management (CIKM 2008), 2008.
20. I. Ounis, C. Macdonald, M. de Rijke, G. Mishne, and I. Soboroff. Overview of the trec 2006 blog track. In E. M. Voorhees and L. P. Buckland, editors, TREC, volume Special Publication 500-272. National Institute of Standards and Technology (NIST), 2006.
21. J. Pasternack and D. Roth. Extracting article text from the web with maximum subsequence segmentation. In WWW '09: Proceedings of the 18th international conference on World wide web, pages 971-980, New York, NY, USA, 2009. ACM.
22. C. E. Shannon. A mathematical theory of communication. Bell system techn. journal, 27, 1948.
23. M. Spousta, M. Marek, and P. Pecina. Victor: the web-page cleaning tool. In WaC4, 2008.
24. K. Vieira, A. S. da Silva, N. Pinto, E. S. de Moura, a. M. B. C. Jo and J. Freire. A fast and robust method for web page template detection and removal. In CIKM '06: Proc. 15th ACM int. conf. on Information and knowledge management, pages 258-267, 2006.
25. R. Vulanovic and R. Köhler. Quantitative Linguistics - An international Handbook, chapter Syntactic units and structures, pages 274-291. de Gruyter, 2005.
26. L. Yi, B. Liu, and X. Li. Eliminating noisy information in web pages for data mining. In KDD '03: Proc. of the 9th ACM SIGKDD int. conf. on Knowledge discovery and data mining, pages 296-305, 2003.