A survey of Web page cleaning research

Jisuanji Yanjiu yu Fazhan/Computer Research and Development

Volumn 47, Issue 12, 2010, Pages 2025-2036

  Mao, Xianling ^a   He, Jing ^a   Yan, Hongfei ^a  

^a PEKING UNIVERSITY   (China)

Author keywords

Data mining; Information retrieval; Web mining; Web page cleaning; WWW

Indexed keywords

CLEANING METHODS; DATA SETS; DE-NOISE; EXISTING PROBLEMS; EXPERIMENTAL METHODS; FUTURE DIRECTIONS; IN-DEPTH STUDY; MODEL-BASED; MULTI-MODEL; RAPID DEVELOPMENT; RESEARCH AND APPLICATION; WEB APPLICATION; WEB DATA; WEB MINING; WEB PAGE; WWW;

CLEANING; DATA MINING; INFORMATION RETRIEVAL; RESEARCH; SEARCH ENGINES;

WORLD WIDE WEB;

EID: 78650954485     PISSN: 10001239     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (48)

1. Fetterly D, Manasse M, Najork M, et al. A large-scale study of the evolution of Web pages[J]. Software: Practice and Experience, 2004, 34(2): 213-237
2. Gibson D, Punera K, Tomkins A. The volume and evolution of Web page templates[C]//Proc of the 14th Int Conf on World Wide Web. New York: ACM, 2005: 830-839
3. Vieira K, Silva A S D, Pinto N, et al. A fast and robust method for Web page template detection and removal[C]//Proc of the 15th ACM Int Conf on Information and Knowledge Management. New York: ACM, 2006: 258-267
4. Chen L, Ye S, Li X. Template detection for large scale search engines[C]//Proc of the 2006 ACM Symp on Applied Computing. New York: ACM, 2006: 1094-1098
5. Bar-Yossef Z, Rajagopalan S. Template detection via data mining and its applications[C]//Proc of the 11th Int Conf on World Wide Web. New York: ACM, 2002: 580-591
6. Manku G S, Jain A, Sarma A D. Detecting near-duplicates for Web crawling[C]//Proc of the 16th Int Conf on World Wide Web. New York: ACM, 2007: 141-150
7. Manasse F M, Najork M. Detecting phrase-level duplication on the World Wide Web[C]//Proc of the 28th Annual Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2005: 170-177
8. Carvalho A L D C, Chirita P A, Moura E S D, et al. Site level noise removal for search engines[C]//Proc of the 15th Int Conf on World Wide Web. New York: ACM, 2006: 73-82
9. Coughlan J, Yuille A, English C, et al. Efficient deformable template detection and localization without user initialization[J]. Computer Vision Image Understanding, 2000, 22(78): 303-319
10. Wang Jiying, Lochovsky F H. Data-rich section extraction from HTML pages[C]//Proc of the 3rd Int Conf on Web Information Systems Engineering(Workshops). Los Alamitos, CA: IEEE Computer Society, 2002: 313-322
11. Yi L, Liu B, Li X. Eliminating noisy information in Web pages for data mining[C]//Proc of the 9th ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining. New York: ACM, 2003: 296-305
12. Yi L, Liu B. Web Page Cleaning for Web Mining through Feature Weighting[C]//Proc of the 18th Int Joint Conf on Artificial Intelligence (IJCAI-03). San Francisco: Morgan Kaufmann, 2003: 43-50
13. Lin S H, Ho J M. Discovering informative content blocks from Web documents[C]//Proc of the 8th ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining. New York: ACM, 2002: 588-593
14. Ma L, Goharian N, Chowdhury A, et al. Extracting unstructured data from template generated Web documents[C]//Proc of the 12th Int Conf on Information and Knowledge Management. New York: ACM, 2003: 512-518
15. Chakrabarti D, Kumar R, Punera K. Page-level template detection via isotonic smoothing[C]//Proc of the 16th Int Conf on World Wide Web. New York: ACM, 2007: 61-70
16. Kushmerick N. Learning to remove Internet advertisements[C]//Proc of the 3rd Annual Conf on Autonomous Agents. New York: ACM, 1999: 175-181
17. Gupta S, Kaiser G, Neistadt D, et al. DOM-based content extraction of HTML documents[C]//Proc of the 12th Int Conf on World Wide Web. New York: ACM, 2003: 207-214
18. Zhang Zhigang, Chen Jin, Li Xiaoming. An approach to reducing noise in HTML pages[J]. Journal of the China Society for Scientific and Technical Information, 2004, 23(4): 387-393 (in Chinese)
19. Wang Jiandong, Wang Jimin, Tian Feijia. An algorithm for noise reduction in Web pages based on a group of content-related rules[J]. New Technology of Library and Information Service, 2008, 22(3): 51-54 (in Chinese)
20. Shi Daming, Lin Hongfei, Yang Zhihao, et al. An approach to eliminate noise based on framework of Web pages and rules[J]. Computer Engineering, 2007, 33(19): 276-278 (in Chinese)
21. Song Ruihua, Ma Shaoping, Chen Gang, et al. An HTML parser to improve Chinese search engines[J]. Journal of Chinese Information Processing, 2003, 17(4): 19-26 (in Chinese)
22. Kovacevic M, Dilligenti M, Gori M, et al. Recognition of common areas in a Web page using a visualization approach[C]//Proc of the 10th Int Conf on Artificial Intelligence: Methodology, Systems, and Applications. Berlin: Springer, 2002: 37-45
23. Cai D, Yu S, Wen J R, et al. Extracting content structure for Web pages based on visual representation[C]//Web Technologies and Applications: 5th Asia-Pacific Web Conf. Berlin: Springer, 2003: 406-417
24. Song R, Liu H, Wen J R, et al. Learning block importance models for Web pages[C]//Proc of the 13th Int Conf on World Wide Web. New York: ACM, 2004: 211-220
25. Cai D, Yu S, Wen J R, et al. VIPS: A vision based page segmentation algorithm, MSR-TR-2003-79[R/OL]. Seattle, USA: Microsoft, (2003-11) [2009-02-01]. http://research.microsoft.com/apps/pubs/default.aspx?id=70027
26. Yu S, Cai D, Wen J R, et al. Improving pseudo-relevance feedback in Web information retrieval using Web page segmentation[C]//Proc of the 12th World Wide Web Conf. New York: ACM, 2003
27. Liu W, Meng X, et al. Vision-based Web data records extraction[C]//Proc of the 9th Int Workshop on the Web and Databases (WebDB 2006). New York: ACM, 2006: 61-70
28. Jing Tao, Zuo Wanli. An algorithm for the elimination of the noise in Web pages based on visual layout information[J]. Journal of South China University of Technology: Natural Science Edition, 2004, 32(Suppl 1): 84-88 (in Chinese)
29. Yuan Mingxuan, Zhang Xuanping, Jiang Yu, et al. Noise elimination method in Web pages based on the similarity of same layer ages[J]. Computer Engineering, 2006, 32(23): 61-63 (in Chinese)
30. Feng Huamin, Liu Biao, Liu Yanmin, et al. Framework of Web page analysis and content extraction with coordinate trees[J]. Journal of Tsinghua University: Science and Technology, 2005, 45(9): 1767-1771 (in Chinese)
31. Chibane I, Doan B L. A Web page topic segmentation algorithm based on visual criteria and content layout[C]//Proc of the 30th Annual Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2007: 817-818
32. Chakrabarti D, Kumar R, Punera K. A graph-theoretic approach to webpage segmentation[C]//Proc of the 17th Int Conf on World Wide Web. New York: ACM, 2008: 377-386
33. Debnath S, Mitra P, Pal N, et al. Automatic identification of informative sections of Web pages[J]. IEEE Trans on Knowledge and Data Engineering, 2005, 2(17): 1233-1246
34. Kao H Y, Ho J M, Chen M S. WISDOM: Web intrapage informative structure mining based on document object model[J]. IEEE Trans on Knowledge and Data Engineering, 2005, 2(17): 614-627
35. Wan Le, Zuo Wanli, Gao Jin. Web pages noise removal based on focused topics[J]. Computer Engineering and Design, 2008, 29(8): 2072-2076 (in Chinese)
36. Davison B D. Recognizing nepotistic links on the Web[C]//Proc of the AAAI-2000 Workshop on Artificial Intelligence for Web Search. Menlo Park: AAAI Press, 2000: 23-28
37. Pardalos P M, Xue G. Algorithms for a class of isotonic regression problems[J]. Algorithmica, 1999, 2(23): 211-222
38. Restrepo A, Bovik A C. Locally monotonic regression[J]. IEEE Trans on Signal Processing, 1994, 41(9): 2796-2780
39. Best M J, Chakravarti N. Active set algorithms for isotonic regression: A unifying framework[J]. Mathematical Programming, 1990, 47(1): 425-439
40. Best M J, Tan R Y. An O(n̂3 log n) strong polynomial algorithm for an isotonic regression knapsack problem[J]. Optimization Theory and Applications, 1993, 79(3): 463-478
41. Bing L, Wang Y, Zhang Y, et al. Primary content extraction with mountain model[C]//Proc of the IEEE CIT2008. Piscataway, NJ: IEEE, 2008: 479-484
42. Gupta S, Becker H, Kaiser G, et al. Verifying genre-based clustering approach to content extraction[C]//Proc of the 15th Int Conf on World Wide Web. New York: ACM, 2006: 875-876
43. Wang Y, Fang B, Cheng X, et al. Incremental Web page template detection[C]//Proc of the 17th Int Conf on World Wide Web (Poster). New York: ACM, 2008: 1247-1248
44. Ou Jianwen, Dong Shoubin, Cai Bin. Topic information extraction from template Web pages[J]. Journal of Tsinghua University: Science and Technology, 2005, 45(9): 1743-1747 (in Chinese)
45. Peking University. CWT: Chinese Web Test Collection[EB/OL]. (2004-06) [2009-11-25]. http://www.cwirf.org/SharedRes/DataSet/cwt.html (in Chinese)
46. Liu Chenxi, Wu Yangyang. A block-analysis-based approach to eliminate noise in Web pages[J]. Journal of Guangxi Normal University (Natural Science Edition), 2007, 25(2): 61-63 (in Chinese)
47. Sougou Labs. The data resources of Sogou labs[EB/OL]. (2006-11-02) [2007-04-05]. http://www.sogou.com/labs/resources.html (in Chinese)
48. Zhang Heng, Qu Jinghui, Zhang Liang. Web page text information extraction and result estimation[J]. Microcomputer Applications, 2007, 28(9): 921-924 (in Chinese)