Search engine technology and digital libraries: Libraries need to discover the academic internet

D-Lib Magazine

Volumn 10, Issue 6, 2004, Pages

  Lossau, Norbert ^a  

^a BIELEFELD UNIVERSITY   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 4644281713     PISSN: 10829873     EISSN: None     Source Type: Journal    
DOI: 10.1045/june2004-lossau     Document Type: Conference Paper

References (33)

1. Very recently Stephen Arnold has addressed the threat that libraries could become marginalized in his paper "Information boundaries and libraries", February 2004 (http://www.arnoldit.com/articles/ elib_Feb2004_final.doc).
2. Michael K. Bergman (on behalf of the BrightPlanet Corporation) "The deep web: Surfacing hidden value". In: The Journal of Electronic Publishing. Michigan University Press. July 2001 (http://www.press. umich.edu/jep/07-01/bergman.html).
3. Bergman uses the term "surface" web.
4. See 〈http://www.scirus.com〉.
5. The majority of the deep web content comes from Medline and Elsevier's Science Direct database. Web sites that have been crawled include domains like ".edu": 58.5 million pages, ".ac.uk": 6.8 million pages (http://www.scirus.com/srsapp/aboutus/#range).
6. Partners can in principle include library service centres and also publishers or other content integrating commercial services.
7. Web services seem to be a promising technology that allow the modular integration of remote services into a local environment.
8. Using navigation tools like thesauri, cross-concordances or classifications.
9. See 〈http://www.vascoda.de〉.
10. See 〈http://www.arl.org/access/scholarsportal/〉.
11. See 〈http://www.rdn.ac.uk/〉.
12. See 〈http://www.renardus.org〉.
13. See 〈http://scout.wisc.edu/〉.
14. Three initiatives were present at the DLF Spring Forum 2004 by the University of Michigan Libraries (http://www.diglib.org/forums/Spring2004/ hagedorn0404.htm), the University of Illinois, Urbana-Champaign (http://www.diglib.org/forums/Spring2004/spring04bios.htm#habing) and OCLC (http://www.diglib.org/ forums/Spring2004/young0404.htm).
15. The Fast software, for example, also includes a file traverser and some generic database connectors to load data from heterogeneous content sources.
16. 625 undergraduates and graduates returned the questionnaire within a week.
17. See 〈http://www.ub.uni-bielefeld.de/portal/english/Metasuche〉. The development of the Digital Library North Rhine-Westphalia was coordinated by the former director of Bielefeld University Library, Dr. Karl Wilhelm Neubauer and funded through a major grant of the State Ministry for Science and Research. The project began in 1998 and officially ended in 2001 with the regular service coming into operation at the regional Service Centre for Academic Libraries in Cologne (HBZ). The portal technology (IPS system) has been developed by IHS according to the specifications made by the project's development group.
18. Bonita Wilson. "Using the Internet for Searching," D-Lib Magazine, March 2004, Volume 10 Number 3, 〈doi:10.1045/march2004- editorial〉.
19. Stephen Arnold provided a figure of "$24 million per year to index one billion content sources". In: "The future of search", Proceedings of the 26th Online Information conference 2002. p 51.
20. Fast Search & Transfer is based in Oslo, Norway with branches in some European countries, the U.S. and Japan (http://www.fastsearch.com). After selling their internet assets (like AllTheWeb) to Overture in April 2003 Fast has been focussing on innovative developments of their search technology through a major company research lab in Norway and external research collaborations with the Universities of Munich (Germany), Cornell, Penn. State (U.S.) and Trondheim/Tromsoe (Norway).
21. Customers come from multinational enterprises, e-Government, e-Health etc.
22. February 2002 to August 2002, Convera, Google, Mnogo, Lucene (2003).
23. See 〈http://www.nutch.org〉.
24. See 〈http://jakarta.apache.org/lucene/docs/index.html〉.
25. Bielefeld UL is collaborating with the external research laboratory of Fast in Munich, Germany.
26. Although the index data itself can be, for performance reasons, distributed on large clusters of computers.
27. There is no local client software that needs to be licensed and installed.
28. E.g., for the integration of search templates within local web sites.
29. In general low-cost hardware such as Linux-PCs with appropriate mass-storage attached. Further details can be obtained from the technical project group at Bielefeld UL.
30. Obviously a principle that applies to many other areas in the library world.
31. Elsevier is using the Fast search software for some of their "integrated search" products (www.scirus. com and www.scopus.com), Thomson's Web of Knowledge is based on WebFeat Prism technology, 〈http://isiwebofknowledge.com/technology.html〉.
32. The article "Moving from theory to praxis" is scheduled to appear in an issue of D-Lib-Magazine later this year. A PowerPoint presentation that describes some of the details can be viewed on the website of the Digital Library Federation 〈http://www.diglib.org/forums/Spring2004/summann0404. htm〉.
33. See 〈http://base.ub.uni-bielefeld.de/〉. Core technology for the search index is Fast Data Search; additional data processing scripts and user-interfaces have been developed at Bielefeld UL with other programming languages and open source technology.