Information retrieval in digital libraries: Bringing search to the net

Science

Volumn 275, Issue 5298, 1997, Pages 327-334

  Schatz, Bruce R ^a,b  

^a University Library   (United States)

^b UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; COMPUTER NETWORK; DATA BASE; INFORMATION RETRIEVAL; INFORMATION SYSTEM; INTERNET; LIBRARY; MEDICAL RESEARCH; PRIORITY JOURNAL; SCIENCE;

ABSTRACTING AND INDEXING; COMPUTATIONAL BIOLOGY; COMPUTER COMMUNICATION NETWORKS; DATABASES, BIBLIOGRAPHIC; DATABASES, FACTUAL; HYPERMEDIA; INFORMATION STORAGE AND RETRIEVAL; MEDLARS; SOFTWARE; USER-COMPUTER INTERFACE;

EID: 0031026901     PISSN: 00368075     EISSN: None     Source Type: Journal    
DOI: 10.1126/science.275.5298.327     Document Type: Article

References (49)

1. The term "digital library" has recently displaced the more traditional "electronic library" [E. Fox et al., Commun. ACM 38, 23 (April 1995)]. The advent of computer networks built upon optical fibers made the term "electronic" seem inappropriate because the fibers carry light, not electricity; however, the term "digital" sometimes has the unfortunate connotation of "digitization." There are many aspects of digital libraries that are important to complete systems, such as intellectual property and permanent archiving issues. The discussion here concentrates on the search-and-display issues most relevant to bringing search to the Net.
2. H. G. Wells, World Brain (Methuen, London, 1938).
3. J. L. Borges, "The Library of Babel," reprinted in Labyrinths: Selected Stories and Other Writings (New Directions, New York, 1964), pp. 51-58.
4. V. Bush, Atl. Mon. 176, 101 (July 1945) [reprinted in (48)].
5. M. E. Lesk, The Seven Ages of Information Retrieval, in As We May Think: A 50th Anniversary Celebration of Bush's Vision, MIT, October 1995. Available at http://www-eecs.mit.edu/AY95-96/events/bush/ index.html
6. T. H. Nelson, "As We Will Think," in On-line 72 Conference Proceedings (1972), vol. 1, pp. 439-454 [reprinted in (48)].
7. J. C. R. Licklider, Libraries of the Future (MIT Press, Cambridge, MA, 1965).
8. J. C. R. Licklider and W. E. Clark, Proc. Am. Fed. Inf. Processing Soc. 21, 113 (1962).
9. Y. Wilks, Ed., special issue on Natural Language Processing, Commun. ACM 39 (January 1996).
10. Licklider went on to found the Information Processing Techniques Office (IPTO) at ARPA, where he funded a number of revolutionary systems projects concerned with information retrieval across remote networks. These include the first demonstrations of network information retrieval. For example, Bourne in 1963 demonstrated a prototype online retrieval system at Stanford Research Institute (SRI) operating on a local terminal from a remote computer at SDC (C. Bourne, personal communication based on SRI internal reports). The most notable ARPA IPTO project was NLS (oNLine System) in the 1960s at SRI. D. Engelbart was the visionary behind NLS, which was the first and still is almost the only complete system for navigating information spaces (24). All documents were divided into paragraphs, which could be arbitrarily linked and followed to other documents across the network. This system inspired many pioneers of later generations. The most immediate outcome of Licklider's book was project INTREX at MIT [C. Overhage and R. Harman, Eds., INTREX: Report of a Planning Conference on Information Technology Experiments (MIT Press, Cambridge, MA, 1965)], which discussed what would today be called community systems with federated repositories. Such systems in the 1960s never developed past the demonstration phase, because of the limitations of hardware and software technology. The realization of personal workstations and local-area networks was pioneered at the Xerox Palo Alto Research Center during the 1970s, including many of the ideas from NLS. The first operational prototypes of community systems were built in the 1980s, after the advent of commercial workstations in research laboratories.
11. G. Salton, Ed., The SMART Retrieval System: Experiments in Automatic Document Processing (Prentice-Hall, Englewood Cliffs, NJ, 1971).
12. M. E. Williams, Science 228, 445 (1985).
13. F. W. Lancaster and E. G. Fayen, Information Retrieval On-Line (Melville, Los Angeles, 1973).
14. R. K. Summit, in Proceedings of the 22nd National Conference of the Association on Computing Machinery (Thompson, 1967), pp. 51-56.
15. C. T. Meadow, Database (October 1988), p. 23.
16. D. B. McCarn and J. Leiter, Science 181, 318 (1973).
17. G. Salton and M. McGill, Introduction to Modern Information Retrieval (McGraw-Hill, New York, 1983).
18. R. S. Marcus and J. F. Reintjes, IEEE Trans. Syst. Man Cybern. 12, 116 (March-April 1982).
19. M. E. Williams, J. Am. Soc. Inf. Sci. 37, 204 (1986).
20. B. R. Schatz, in Proceedings of IEEE Globecom +87 (IEEE, New York, November 1987), pp. 1181-1186.
21. For example, as a member of the Internet Research Task Force, I was one of the few members of the generation after the pioneers invited to speak at the 20th Anniversary Symposium for the ARPANET at the University of California at Los Angeles in 1989. My talk, "Telesophy: Towards World-Wide Information Spaces," although full of technical details and projections, seemed grand and futuristic at that point (August 1989).
22. B. R. Schatz, "Telesophy," Bellcore TM-ARH-002487 (August 1984).
23. _, in Proceedings of the 5th IEEE International Conference on Data Engineering (IEEE, New York, 1989), pp. 188-197.
24. D. C. Engelbart and W. K. English, in Proceedings of the Fall Joint Computer Conference (AFIPS Press, New York, 1968), vol. 33, part 1, pp. 395-410.
25. C. F. Herot, Assoc. Comput. Mach. Trans. Database Syst. 5, 493 (1980).
26. An inspiration for knowledge regions was T. Nelson, who designed a grand system called Xanadu to handle all the world's knowledge as a single hyperliterature across multiple collections. His unimplemented treatise, Literary Machines (1981), contained many suggestions for building new documents by annotating and linking parts of old.
27. A. Goldberg and D. Robson, Smalltalk-80: The Language and Its Implementation (Addison-Wesley, Reading, MA, 1983).
28. B. Kahle et al., Electron. Networking 2, 59 (spring 1992).
29. B. Kahle, personal communication. Kahle developed the WAIS software at Thinking Machines with funding from Apple Computer and later started WAIS Inc., which was purchased by America Online.
30. B. R. Schatz and J. B. Hardin, Science 265, 895 (1994).
31. The two predominant Web browsers are derived from Mosaic: Netscape Navigator was built by the original developers after they left NCSA, and Microsoft's Internet Explorer has at its core a licensed version of Enhanced Mosaic which is produced by Spyglass as the official commercial distributor of NCSA Mosaic. Historically, Telesophy played a role in Mosaic as well, because I have been the scientific advisor for information systems at NCSA since 1989, and Mosaic was one of several attempts at NCSA to reproduce the functionality of Telesophy for the general scientific community.
32. Lycos is a spin-off company from digital library projects at Carnegie-Mellon University. See http:// www.lycos.com/
33. Alta Vista was a project, now a service, from Digital Equipment Corporation's Research Laboratories. See http://altavista.digital.com/
34. B. Schatz and H. Chen, IEEE Comput. 29, 22 (May 1996).
35. The May 1996 special issue of IEEE Computer contains overview articles from all six DLI projects. See http://www.computer.org/pubs/computer/dli/
36. B. Schatz et al., Computer 29, 28 (May 1996).
37. E. van Herwinjnen, Practical SGML (Kluwer, Boston, 1994).
38. F. W. Lancaster, Vocabulary Control for Information Retrieval (Information Resources Press, Arlington, VA, 1986).
39. C. Lynch and H. Molina-Garcia, Eds., "Interoperability, Scaling, and the Digital Libraries Research Agenda," 22 August 1995. Available at http://www. hppc.gov/reports/report-nco/reports/iita-dlw/ main.html. The Information Infrastructure Technology and Applications (IITA) group is the highest level technical committee of the Federal NII Program.
40. S. Nadis, Science 272, 1419 (1996).
41. H. Chen et al., IEEE Trans. Pattern Anal. Mach. Intell. 18, 771 (1996).
42. H. Chen, J. Martinez, T. Ng, B. Schatz, J. Am. Soc. Inf. Sci. 48, 17 (1997).
43. P. B. Kantor, Annu. Rev. Inf. Sci. Technol. 29, 53 (1994).
44. R. T. Niehoff, J. Am. Soc. Inf. Sci. 27, 3 (1976).
45. The vocabulary switching computation used bibliographic abstracts from Compendex (engineering and science) and Inspec (electrical engineering and computer science). Compendex has 40 broad subject classes (for example, computer science) and 600 class codes total. Inspec is narrower and deeper than Compendex, and the computation included about 150 classes at its highest level, the same as the lowest level of Compendex. Because Inspec has roughly 2500 classes all together, the collection spanned in total about (600/150)2500 = 10,000 community repositories across all of science and engineering. This size is similar to that calculated by Licklider, who stated 100 fields and 1000 subfields, because communities are the next deeper level (for example, Smalltalk is a community within the subfield of programming languages, within the field of computer science). A typical community repository in this computation or in the previous molecular biology computations has 5000 documents, at 20 kilobytes per document for full text. The size of a subfield literature is thus 10 times this, 1 gigabyte, just as computed by Licklider. The vocabulary switching computation thus spanned a representative set of all scientific literature (it used abstracts, not documents, and a sample of communities, so it did not compute the complete literature in toto).
46. B. R. Schatz, "Information Analysis in the Net: The Interspace of the Twenty-First Century", a CIC Forum White Paper for America in the Age of Information: A Forum, Committee on Information and Communications (CIC) of the National Science and Technology Council, July 1995. Available at http://www. hpcc.gov/cic/forum/CIC_Cover.html. The CIC is one of nine committees reporting directly to the Science Adviser to the President of the United States.
47. B. R. Schatz, "Building the Interspace," http://csl. ncsa.uiuc.edu/interspace.html
48. J. M. Nyce and P. Kahn, From Memex to Hypertext: Vannevar Bush and the Mind's Machine (Academic Press, San Diego, CA, 1991).
49. I thank the members of the DLI project at the University of Illinois in general and the Interspace project in particular, especially H. Chen, K. Powell, and C. Herring. C. Bourne, who was a pioneer in the early days of online information retrieval, carefully reviewed the historical details and suggested many corrections. L. Smith and P. Cochrane also kindly helped with the periods that predated my direct experiences. K. Powell helped with preparation of the figures. Support was provided through NSF-ARPA-NASA DLI grant IRI-94-11318COOP and my NSF Young Investigator award IRI-9257252 in science information systems.