Information extraction for search engines using fast heuristic techniques

Data and Knowledge Engineering

Volumn 69, Issue 2, 2010, Pages 169-196

  Hong, Jer Lang ^a   Siew, Eu Gene ^a   Egerton, Simon ^a  

^a MONASH UNIVERSITY MALAYSIA   (Malaysia)

Author keywords

Automatic wrapper; Information extraction; Search engine

Indexed keywords

AUTOMATIC WRAPPER; DATA ALIGNMENTS; DATA EXTRACTION; DATA ITEMS; DATA MERGING; DATA RECORDS; DATA REGIONS; FILTERING RULES; FREQUENCY MEASURES; HEURISTIC TECHNIQUES; INFORMATION EXTRACTION; META SEARCH ENGINES; NOVEL TECHNIQUES; NUMBER AND SIZE; PARTITIONING METHODS; STATE OF THE ART; TREE MATCHING ALGORITHM; VISUAL CUES; WEB PAGE;

ALGORITHMS; ALIGNMENT; HEURISTIC METHODS; INFORMATION RETRIEVAL; INTERNET; MARKUP LANGUAGES; SEARCH ENGINES;

WEBSITES;

EID: 72649103406     PISSN: 0169023X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.datak.2009.10.002     Document Type: Article

References (58)

1. Laender A.H.F., Ribeiro-Neto B.A., da Silva A.S., and Teixeira J.S. A brief survey of web data extraction tools. SIGMOD Rec. 31 (2002) 84-93
2. Laender A.H.F., Ribeiro-Neto B.A., and da Silva A.S. DEByE - Date extraction by example. Data Knowl. Eng. 40 (2002) 121-154
3. Andrew Hogue, David Karger, Thresher: automating the unwrapping of semantic content from the World Wide Web, in: Proceedings of the 14th International Conference on World Wide Web Chiba, ACM, Japan, 2005.
4. Apostolico A., and Guerra C. The longest common subsequence problem revisited. Algorithmica 2 (1987)
5. Sahuguet A., and Azavant F. Building intelligent web applications using lightweight wrappers. Data Knowl. Eng. 36 (2001) 283-316
6. Arasu A., and Garcia-Molina H. Extracting structured data from web pages. Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data (2003), ACM, San Diego, California
7. Liu B., Grossman R., and Zhai Y. Mining data records in web pages. Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2003), ACM, Washington, D.C.
8. Bing Liu, Yanhong Zhai, NET - a system for extracting web data from flat and nested data records, in: Web Information Systems Engineering - WISE 2005, 2005, pp. 487-495.
9. Brad Adelberg, NoDoSE{minus 45 degree rule}-a tool for semi-automatically extracting structured and semistructured data from text documents, in: Proceedings of the 1998 ACM SIGMOD International Conference on Management of Data Seattle, ACM, Washington, United States, 1998.
10. Chang C.-H., Kayed M., Girgis M.R., and Shaalan K. A survey of web information extraction systems. IEEE T. Knowl. Data Eng. 18 (2006) 1411-1428
11. Chang C.-H., and Lui S.-C. IEPAD: information extraction based on pattern discovery. Proceedings of the 10th International Conference on World Wide Web Hong Kong (2001), ACM, Hong Kong
12. Chang C.-H., and Kuo S.-C. Olera: semisupervised Web-data extraction with visual support. IEEE Intell. Syst. 19 (2004) 56-64
13. Hsu C.-N., and Dung M.-T. Generating finite-state transducers for semi-structured data extraction from the Web. Inf. Syst. 23 (1998) 521-538
14. Tao C., and Embley D.W. Automatic hidden-web table interpretation, conceptualization, and semantic annotation. Data Knowl. Eng. 68 (2009) 683-703
15. Gusfield D. Algorithms on strings, trees, and sequences: computer science and computational biology (1997), Cambridge University Press
16. David Sankoff and Joseph Kruskal, Time Warps, String Edits, and Macromolecules: The Theory and Practice of Sequence Comparison, Center for the Study of Language and Inf, 1999.
17. D.W. Embley, D.M. Campbell, Y.S. Jiang, S.W. Liddle, D.W. Lonsdale, Conceptual-model-based data extraction from multiple-record Web pages, Data Knowl. Eng. (1999).
18. Dayne Freitag, Information extraction from HTML: Application of a general learning approach, in: Proceedings of the Fifteenth National/Tenth Conference on Artificial Intelligence/ Innovative Applications of Artificial Intelligence, 1998.
19. Tanaka E., and Tanaka K. The tree-to-tree editing problem. Int. J. Pattern Recognit. Artif. Intell. (1988) 221-240
20. Gabriel Valiente, An efficient bottom-up distance between trees, in: Proc. Eighth International Symposium String Processing and Information Retrieval, 2001, pp. 212-219.
21. Gautam Das, Rudolf Fleischer, Leszek Gasieniec, Dimitris Gunopulos, Juha Karkkainen, Episode matching, in: Proceedings of the Eighth Annual Symposium on Combinatorial Pattern Matching, 1997.
22. Gengxin Miao, Junichi Tatemura, Wang-Pin Hsiung, Arsany Sawires, Louise E. Moser, Extracting data records from the web using tag path clustering, in: Proceedings of the 18th International Conference on World Wide Web, Spain, Madrid, 2009.
23. Navarro G. A guided tour to approximate string matching. ACM Comput. Surv. 33 (2001) 31-88
24. Gustavo O. Arocena, Alberto O. Mendelzon, WebOQL: restructuring documents, databases and webs, in: Proceedings of the 14th International Conference on Data Engineering, 1998, pp. 24-33.
25. Hongkun Zhao, Weiyi Meng, Clement Yu, Automatic extraction of dynamic record sections from search engine result pages, in: Proceedings of the 32nd International Conference on Very Large Data Bases, VLDB Endowment, Seoul, Korea, 2006.
26. Zhao H., Meng W., and Yu C. Mining templates from search result records of search engines. Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2007), ACM, San Jose, California, USA
27. Zhao H., Meng W., Wu Z., Raghavan V., and Yu C. Fully automatic wrapper generation for search engines. Proceedings of the 14th International Conference on World Wide Web Chiba (2005), ACM, Japan
28. Kao H.-Y., Lin S.-H., Ho J.-M., and Chen M.-S. Mining web informative structures and contents based on entropy analysis. IEEE T. Knowl. Data Eng. 16 (2004) 41-55
29. Muslea I., Minton S., and Knoblock C. A hierarchical approach to wrapper induction. Proceedings of the Third Annual Conference on Autonomous Agents Seattle (1999), ACM, Washington, United States
30. Jiying Wang, Frederick H. Lochovsky, Data extraction and label assignment for web databases, in: Proceedings of the 12th International Conference on World Wide Web Budapest, ACM, Hungary, 2003.
31. Joachim Hammer, Jason McHugh, Hector Garcia-Molina, Semistructured data: the TSIMMIS experience, in: Proc. First East-European Symposium Advances in Databases and Information Systems (ADBIS), 1997, pp. 1-8.
32. Wang J., Chen C., Wang C., Pei J., Bu J., Guan Z., and Zhang W.V. Can we learn a template-independent wrapper for news article extraction from a single training site?. Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Paris (2009), ACM, France
33. Simon K., and Lausen G. ViPER: augmenting automatic information extraction with visual perceptions. Proceedings of the 14th ACM International Conference on Information and Knowledge Management Bremen (2005), ACM, Germany
34. Liu K.-L., Meng W., Qiu J., Yu C., Raghavan V., Wu Z., Lu Y., He H., and Zhao H. AllInOneNews: development and evaluation of a large-scale news metasearch engine. Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data Beijing (2007), ACM, China
35. Tai K.-C. The tree-to-tree correction problem. J. ACM 26 (1979) 422-433
36. Ling Liu, Calton Pu, Wei Han, XWRAP: an XML-enabled wrapper construction system for web information sources, in: Proceedings of the 16th International Conference on Data Engineering, 2000, pp. 611-621.
37. Longzhuang Li, Yonghuai Liu, Abel Obregon, Matt Weatherston, Visual segmentation-based data record extraction from web documents, in: IEEE International Conference on Information Reuse and Integration, IRI 2007, 2007, pp. 502-507.
38. lvarez M., Pan A., Raposo J., Bellas F., and Cacheda F. Extracting lists of data records from semi-structured web pages. Data Knowl. Eng. 64 (2008) 491-509
39. Mary E. Califf, Raymond J. Mooney, Relational Learning of Pattern-Match Rules for Information Extraction, University of Texas at Austin, 1998.
40. Song M., Song I.-Y., Hu X., and Allen R.B. Integration of association rules and ontologies for semantic query expansion. Data Knowl. Eng. 63 (2007) 63-75
41. Nicholas Kushmerick, Wrapper induction for information extraction, University of Washington, 1997, p. 246.
42. Baeza-Yates R.A. Algorithms for string searching. SIGIR Forum 23 (1989) 34-58
43. Needleman S.B., and Wünsch C.D. A general method applicable to the search for similarities in the amino acid sequences of two proteins. J. Mol. Biol. (1970)
44. Jiang T., Wang L., and Zhang K. Alignment of trees - an alternative to tree edit. Proceedings of the Fifth Annual Symposium on Combinatorial Pattern Matching (1994), Springer
45. Crescenzi V., and Mecca G. Grammars have exceptions. Inf. Syst. 23 (1998) 539-565
46. Crescenzi V., Mecca G., and Merialdo P. RoadRunner: towards automatic data extraction from large web sites. Proceedings of the 27th International Conference on Very Large Data Bases (2001), Morgan Kaufmann Publishers Inc.
47. Levenshtein V.I. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady 10 (1966) 707
48. Wei Liu, Xiaofeng Meng, Weiyi Meng, Vision-based web data records extraction, ACM Ninth International Workshop on the Web and Databases (WebDB 2006), 2006.
49. Wei Liu, Xiaofeng Meng, Weiyi Meng, ViDE: a vision-based approach for deep web data extraction, IEEE T. Knowl. Data Eng. (2009).
50. Weifeng Su, Jiying Wang, Frederick H. Lochovsky, ODE: ontology-assisted data extraction, ACM T. Database Syst. (2009).
51. Yang W. Identifying syntactic differences between two programs. Softw. Pract. Exper. 21 (1991) 739-755
52. Zhai Y., and Liu B. Web data extraction based on partial tree alignment. Proceedings of the 14th International Conference on World Wide Web Chiba (2005), ACM, Japan
53. Zhai Y., and Liu B. Structured data extraction from the web based on partial tree alignment. IEEE T. Knowl. Data Eng. 18 (2006) 1614-1628
54. Zhai Y., and Liu B. Extracting web data using instance-based learning. World Wide Web 10 (2007) 113-132
55. Yipu Wu, Jing Chen, Qing Li, Extracting loosely structured data records through mining strict patterns, in: ICDE 2008, IEEE 24th International Conference on Data Engineering, 2008, pp. 1322-1324.
56. Yuan Kui Shen, Automatic record extraction for the World Wide Web, in: Department of Electrical Engineering and Computer Science, Master of Science in Computer Science and Engineering, MIT, 2006.
57. Shen Y.K., and Karger D.R. U-REST: an unsupervised record extraction system. Proceedings of the 16th International Conference on World Wide Web Banff (2007), ACM, Alberta, Canada
58. http://en.wikipedia.org/wiki/Web_template.