OXPath: A language for scalable data extraction, automation, and crawling on the deep web

VLDB Journal

Volumn 22, Issue 1, 2013, Pages 47-72

  Furche, Tim ^a   Gottlob, Georg ^a   Grasso, Giovanni ^a   Schallhart, Christian ^a   Sellers, Andrew ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

AJAX; Automation; Crawling; Data extraction; DOM; Web applications; Web extraction; XPath

Indexed keywords

AUTOMATION; DATA MINING; EXTRACTION;

AJAX; CRAWLING; DATA EXTRACTION; WEB APPLICATION; WEB EXTRACTION; XPATH;

WEB CRAWLER;

EID: 84872659606     PISSN: 10668888     EISSN: 0949877X     Source Type: Journal    
DOI: 10.1007/s00778-012-0286-6     Document Type: Article

References (48)

1. http://www.iopus.com/iMacros
2. http://www.newprosoft.com/web-content-extractor.htm
3. http://www.visualwebripper.com
4. http://www.web-harvest.sourceforge.net
5. http://www.w3.org/TR/CSS2/selector.html
6. Alba, A., Bhagwan, V., Grandison, T.: Accessing the deep web: when good ideas go bad. In: OOPSLA (2008)
7. Anton, T.:XPath-wrapper induction by generalizing tree traversal patterns. In: LWA (2005)
8. Anupam, V., Freire, J., Kumar, B., Lieuwen, D.: Automating web navigation with the webvcr. In: WWW (2000)
9. Arocena, G.O., Mendelzon, A.O.: Weboql: Restructuring documents, databases, and webs. In: ICDE (1998)
10. Badica, C., Badica, A., Popescu, E., Abraham, A.: L-wrappers: concepts, properties and construction: A declarative approach to data extraction from web sources. Soft Comput. 11(8), 753-772 (2007)
11. Banko, M., Cafarella, M.J., Soderland, S., Broadhead, M., Etzioni, O.: Open information extraction from the Web. In: IJCAI (2007)
12. Baumgartner, R., Flesca, S., Gottlob, G.: Visual web information extraction with Lixto. In: VLDB (2001)
13. Benedikt, M., Koch, C.: Xpath leashed. CSUR 41, 3:1-3:54 (2009)
14. Bergman, M.K.:The deep web: Surfacing hidden value. J. Electron. Publ. 7(1), 1-17 (2001)
15. Bigham, J.P., Cavender, A.C., Kaminsky, R.S., Prince, C.M., Obison T.S.: Transcendence: enabling a personal viewof the deep web. In: IUI (2008)
16. Boldi, P., Codenotti, B., Santini, M., Vigna, S.: Ubicrawler: a scalable fully distributed web crawler. Softw. Practice Experience 34, 711-726 (2004)
17. Bolin, M., Webber, M., Rha, P., Wilson, T., Miller, R.C.:. Automation and customization of rendered web pages. In: UIST (2005)
18. Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine. Comput. Netw. ISDN Syst. 30(1-7), 107-117 (1998)
19. Cafarella, M.J., Halevy, A.Y., Wang, D.Z., Wy, E., Zhang, Y.: WebTables: exploring the power of tables on the web. PVLDB 1(1), 538-549 (2008)
20. Centeno, V.L., Kloos, C.D., Fernández, L.S.: García, N.F.: Intelligent automated navigation through the deep web. In: Advances in Web Intelligence (2004)
21. Chang, C.-H., Kayed, M., Girgis, M.R., Shaalan, K.F.: A survey of web information extraction systems. TKDE 18(10), 1411-1428 (2006)
22. Crescenzi, V., Mecca, G., Merialdo, P.: Roadrunner: automatic data extraction from data-intensive web sites. In: SIGMOD (2002)
23. Cafarella, M.J., Downey, D., Popescu, A.-M., Shaked, T., Soderland, S., Weld, D.S., Yates, A.: Unsupervised named-entity extrac-tion from the Web: an experimental study. Artif. Intell. 165(1), 91-134 (2005)
24. Furche, T., Gottlob, G., Grasso, G., Gunes, O., Guo, X., Kravchenko, A., Orsi, G., Schallhart, C., Sellers, A., Wang, C.: DIADEM: Domain-centric, intelligent, automated data extraction methodology. In: WWW (2012)
25. Furche, T., Gottlob, G., Grasso, G., Schallhart, C., Sellers, A.: Oxpath: A language for scalable, memory-efficient data extraction from web applications. PVLDB 4(11), 1016-1027 (2011)
26. Gottlob, G., Koch, C., Pichler, R.: Efficient algorithms for processing XPath queries. In: TODS (2005)
27. Gruhl, D., Chavet, L., Gibson, D., Meyer, J., Pattanayak, P., Tomkins, A., Zien, J.: How to build a webfountain: an architecture for very large-scale text analytics. IBM Syst. J. 43, 64-77 (2004)
28. He, B., Patel, M., Zhang, Z., Chang, K.C.-C.: Accessing the deep web. Commun. ACM 50(5), 94-101 (2007)
29. Heydon, A., Najork, M.: Mercator: a scalable, extensible web crawler. World Wide Web 2(4), 219-229 (1999)
30. Kranzdorf, J., Sellers, A., Grasso, G., Schallhart, C., Furche, T: Spotting the tracks on the oxpath. In: WWW (2012)
31. Leshed, G., Haber, E.M., Matthews, T., Lau, T.: Coscripter: automating & sharing how-to knowledge in the enterprise. In: CHI (2008)
32. Lin, J., Wong, J., Nichols, J., Cypher, A., Lau, T.A.: End-user programming of mashups with vegemite. In: IUI (2009)
33. Liu, L., Pu, C., Han, W.: Xwrap: an xml-enabled wrapper construction system for web information sources. In: ICDE (2000)
34. Liu, M., Ling, T.W.: A rule-based query language for html. In: DASFAA (2001)
35. Marx, M.: Conditional XPath. ACM Trans. Database Syst. 30(4), 929-959 (2005)
36. Marx, M., de Rijke, M.: Semantic characterizations of navigational XPath. ACM SIGMOD Rec. 34(2), 41-46 (2005)
37. Mendelzon, A.O., Mihaila, G.A., Milo, T.: Querying the world wide web. Int. J. Digit. Libr. 1(1), 54-67 (1997)
38. Mir, S., Staab, S., Rojas, I.: Web-prospector-an automatic, sitewide wrapper induction approach for scientific deep-web databases. In: BTW (2009)
39. Montoto, P., Pan, A., Raposo, J., Bellas, F., López, J: Automating navigation sequences in ajax websites. In: ICWE (2009)
40. Myllymaki, J.: Effective web data extraction with standard xml technologies. Comput. Netw. 39(5), 635-644 (2002)
41. Olteanu, D., Meuss, H., Furche, T., Bry, F.: XPath: looking Forward. In: EDBT-XML-Based Data Management, LNCS 2490 (2002)
42. Raposo, J., Pan, A., Álvarez, M., Hidalgo, J., Viña., A.: The wargo system: semi-automatic wrapper generation in presence of complex data access modes. In: DEXA (2002)
43. Safonov, A.: Web macros by example: users managing the www of applications. In: CHI, pp. 71-72. ACM (1999)
44. Sahuguet, A., Azavant, F.: Building light-weight wrappers for legacy web data-sources usingw4f. In: VLDB, pp. 738-741 (1999)
45. Sawa, N., Morishima, A., Sugimoto, S., Kitagawa, H.: Wraplet: Wrapping your web contents with a lightweight language. In: SITIS, pp. 387-394 (2007)
46. Shen, W., Doan, A., Naughton, J.F., Ramakrishnan, R: Declarative information extraction using datalogwith embedded extraction predicates. In: VLDB (2007)
47. Su, J.-Y., Sun, D.-J., Wu, I.-C., Chen, L.-P.: On design of browseroriented data extraction system and plug-ins. J. Mar. Sci. Technol. 18(2), 189-200 (2010)
48. Wang, Y., Hornung, T.: Deep web navigation by example. Scalable Comput. Practice Experience 9, 281-292 (2008)