External memory algorithms

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 1461 LNCS, Issue , 1998, Pages 1-25

  Vitter, Jeffrey Scott ^a  

^a Duke University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

MEMORY ARCHITECTURE; PROGRAM PROCESSORS; PULSE WIDTH MODULATION;

ALGORITHM'S PERFORMANCE; DESIGN AND ANALYSIS; EXTERNAL MEMORY ALGORITHMS; HIERARCHICAL MEMORY; OUT-OF-CORE ALGORITHMS; PARALLEL DISK MODELS; PERFORMANCE BOTTLENECKS; PROGRAMMING ENVIRONMENT;

CACHE MEMORY;

EID: 84896772765     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/3-540-68530-8_1     Document Type: Conference Paper

References (114)

1. J. Abello, A. Buchsbaum, and J. Westbrook. A functional approach to external memory graph algorithms. In J. Abello and J. S. Vitter, editors, External Memory Algorithms and Visualization, DIMACS series. American Mathematical Society, 1998.
2. M. Adler. New coding techniques for improved bandwidth utilization. In 37th IEEE Symp. on Foundations of Computer Science, 173-182, Burlington, VT, October 1996.
3. P. K. Agarwal, L. Arge, J. Erickson, P. G. Franciosa, and J. S. Vitter. Efficient searching with linear constraints. In Proc. 17th ACM Symp. on Princ. of Database Systems, 1998.
4. P. K. Agarwal, L. Arge, T. M. Murali, K. Varadarajan, and J. S. Vitter. I/O-efficient algorithms for contour line extraction and planar graph blocking. In Proc. ACM-SIAM Symp. on Discrete Algorithms, 1998.
5. P. K. Agarwal and J. Erickson. Geometric range searching and its relatives. In B. Chazelle, E. Goodman, and R. Pollack, editors, Discrete and Computational Geometry: Ten Years Later, 63-72. American Mathematical Society Press, to appear.
6. A. Aggarwal and C. G. Plaxton. Optimal parallel sorting in multi-level storage. Proc. Fifth Annual ACM-SIAM Symp. on Discrete Algorithms, 659-668, 1994.
7. A. Aggarwal and J. S. Vitter. The input/output complexity of sorting and related problems. Communications of the ACM, 31(9), 1116-1127, 1988.
8. M. Ajtai, M. Fredman, and Komlos. Hash functions for priority queues. Information and Control, 63(3), 217-225, 1984.
9. L. Arge. The buffer tree: A new technique for optimal I/O-algorithms. In Proc. Workshop on Algorithms and Data Structures, LNCS 955, 334-345, 1995. A complete version appears as BRICS technical report RS-96-28, University of Aarhus.
10. L. Arge. The I/O-complexity of ordered binary-decision diagram manipulation. In Proc. Int. Symp. on Algorithms and Computation, LNCS 1004, 82-91, 1995.
11. L. Arge. External-memory algorithms with applications in geographic information systems. In M. van Kreveld, J. Nievergelt, T. Roos, and P. Widmayer, editors, Algorithmic Foundations of GIS. Springer-Verlag, LNCS 1340, 1997.
12. L. Arge, P. Ferragina, R. Grossi, and J. Vitter. On sorting strings in external memory. In Proc. ACM Symposium on Theory of Computation, 540-548, 1997.
13. L. Arge, K. H. Hinrichs, J. Vahrenhold, and J. S. Vitter. Efficient bulk operations on dynamic R-trees, 1998. Manuscript.
14. L. Arge, M. Knudsen, and K. Larsen. A general lower bound on the I/O-complexity of comparison-based algorithms. In Proc. 3rd Workshop on Algorithms and Data Structures, Volume 709, 83-94. Lecture Notes in Computer Science, Springer-Verlag, 1993.
15. L. Arge and P. Miltersen. On showing lower bounds for external-memory computational geometry problems. In J. Abello and J. S. Vitter, editors, External Memory Algorithms and Visualization, DIMACS series. American Mathematical Society, 1998.
16. L. Arge, O. Procopiuc, S. Ramaswamy, T. Suel, and J. S. Vitter. Scalable sweeping-based spatial join. In Proc. 24th Intl. Conf. on Very Large Databases, New York, August 1998.
17. L. Arge, O. Procopiuc, S. Ramaswamy, T. Suel, and J. S. Vitter. Theory and practice of I/O-efficient algorithms for multidimensional batched searching problems. In Proc. ACM-SIAM Symp. on Discrete Algorithms, 1998.
18. L. Arge, D. E. Vengroff, and J. S. Vitter. External-memory algorithms for processing line segments in geographic information systems. Algorithmica, to appear. Special issue on cartography and geographic information systems.
19. L. Arge and J. S. Vitter. Optimal interval management in external memory. In Proc. 37th IEEE Symp. on Found. of Computer Sci., 560-569, Burlington, VT, October 1996.
20. R. Barve, P. B. Gibbons, B. Hillyer, Y. Matias, E. Shriver, and J. S. Vitter. Modeling and optimizing I/O throughput of multiple disks on a bus: the long version. Technical report, Bell Labs, 1997.
21. R. Barve, E. F. Grove, and J. S. Vitter. Simple randomized mergesort on parallel disks. Parallel Computing, 23(4), 601-631, 1997.
22. R. Barve and J. S. Vitter. External memory algorithms with dynamically changing memory, 1998. Manuscript.
23. R. Bayer and E. McCreight. Organization of large ordered indexes. Acta Inform., 1, 173-189, 1972.
24. B. Becker, S. Gschwind, T. Ohler, B. Seeger, and P. Widmayer. An asymptotically optimal multiversion B-tree. The VLDB Journal, 5(4), 264-275, Dec. 1996.
25. *-tree: An efficient and robust access method for points and rectangles. In Proc. SIGMOD International Conf. on Management of Data, 322-331, 1990.
26. S. Berchtold, C. Böhm, and H.-P. Kriegel. Improving the query performance of high-dimensional index structures by bulk load operations. In International Conf. on Extending Database Technology, 1998.
27. G. S. Brodal and J. Katajainen. Worst-case efficient external-memory priority queues. Technical Report DIKU Report 97/25, University of Copenhagen, October 1997.
28. P. Callahan, M. T. Goodrich, and K. Ramaiyer. Topology B-trees and their applications. In Proc. Workshop on Algorithms and Data Structures, LNCS 955, 381-392, 1995.
29. P. M. Chen, E. K. Lee, G. A. Gibson, R. H. Katz, and D. A. Patterson. RAID: highperformance, reliable secondary storage. ACM Comp. Surveys, 26(2), 145-185, June 1994.
30. Y.-J. Chiang, M. T. Goodrich, E. F. Grove, R. Tamassia, D. E. Vengroff, and J. S. Vitter. External-memory graph algorithms. In Proc. ACM-SIAM Symp. on Discrete Algorithms, 139-149, January 1995.
31. Y.-J. Chiang. Experiments on the practical I/O efficiency of geometric algorithms: Distribution sweep vs. plane sweep. In Proc. 1995 Work. Algs. and Data Structures, 1995.
32. Y.-J. Chiang and C. T. Silva. I/O optimal isosurface extraction. In Proc. IEEE Visualization Conf., 1997.
33. D. R. Clark and J. I. Munro. Efficient suffix trees on secondary storage. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms, 383-391, Atlanta, June 1996.
34. K. L. Clarkson and P. W. Shor. Applications of random sampling in computational geometry, II. Discrete and Computational Geometry, 4, 387-421, 1989.
35. P. Corbett, D. Feitelson, S. Fineberg, Y. Hsu, B. Nitzberg, J.-P. Prost, M. Snir, B. Traversat, and P. Wong. Overview of the MPI-IO parallel I/O interface. In R. Jain, J. Werth, and J. C. Browne, editors, Input/Output in Parallel and Distributed Computer Systems, Volume 362 of The Kluwer International Series in Engineering and Computer Science, chapter 5, 127-146. Kluwer Academic Publishers, 1996.
36. T. H. Cormen and D. M. Nicol. Performing out-of-core FFTs on parallel disk systems. Parallel Computing, 1998. To appear; available as Dartmouth Report PCS-TR96-294.
37. T. H. Cormen, T. Sundquist, and L. F. Wisniewski. Asymptotically tight bounds for performing BMMC permutations on parallel disk systems. SIAM J. Computing, to appear.
38. A. Crauser, P. Ferragina, K. Mehlhorn, U. Meyer, and E. Ramos. Randomized external-memory algorithms for geometric problems. In Proc. 14th ACM Symp. on Computational Geometry, June 1998.
39. R. Cypher and G. Plaxton. Deterministic sorting in nearly logarithmic time on the hypercube and related computers. J. Computer and System Sci., 47(3), 501-548, 1993.
40. F. Dehne, W. Dittrich, and D. Hutchinson. Efficient external memory algorithms by simulating coarse-grained parallel algorithms. In Proc. 9th ACM Symp. on Parallel Algorithms and Architectures, 106-115, June 1997.
41. H. B. Demuth. Electronic Data Sorting. PhD thesis, Stanford University, 1956.
42. A shortened version appears in IEEE Transactions on Computing, C-34(4): 296-310, April 1985, special issue on sorting, E. E. Lindstrom, C. K. Wong, and J. S. Vitter, editors.
43. D. J. DeWitt, J. F. Naughton, and D. A. Schneider. Parallel sorting on a shared-nothing architecture using probabilistic splitting. In Proc. First International Conf. on Parallel and Distributed Information Systems, 280-291, December 1991.
44. J. R. Driscoll, N. Sarnak, D. D. Sleator, and R. E. Tarjan. Making data structures persistent. Journal of Computer and System Sciences, 38, 86-124, 1989.
45. M. Farach and P. Ferragina. Optimal suffix tree construction in external memory, November 1997. Manuscript.
46. W. Feller. An Introduction to Probability Theory and its Applications, Volume 1. John Wiley & Sons, New York, third edition, 1968.
47. P. Ferragina and R. Grossi. A fully-dynamic data structure for external substring search. In Proc. 27th Annual ACM Symp. on Theory of Computing, 693-702, Las Vegas, 1995.
48. P. Ferragina and R. Grossi. Fast string searching in secondary storage: Theoretical developments and experimental results. In Proc. ACM-SIAM Symp. on Discrete Algorithms, 373-382, Atlanta, June 1996.
49. R. W. Floyd. Permuting information in idealized two-level storage. In R. Miller and J. Thatcher, editors, Complexity of Computer Computations, 105-109. Plenum, 1972.
50. V. Gaede and O. Günther. Multidimensional access methods. Computing Surveys, 1998.
51. M. Gardner. Magic Show, chapter 7. Knopf, New York, 1977.
52. G. A. Gibson, J. S. Vitter, and J. Wilkes. Report of the working group on storage I/O issues in large-scale computing. ACM Computing Surveys, 28(4), December 1996. Also available as http://www.cs.duke.edu/~jsv/SDCR96-IO/report. ps.
53. M. T. Goodrich, J.-J. Tsay, D. E. Vengroff, and J. S. Vitter. External-memory computational geometry. In IEEE Foundations of Computer Science, 714-723, 1993.
54. D. Greene. An implementation and performance analysis of spatial data access methods. In Proc. IEEE International Conf. on Data Engineering, 606-615, 1989.
55. R. Grossi and G. F. Italiano. Efficient splitting and merging algorithms for order decomposable problems. In 24th International Colloquium on Automata, Languages and Programming, Volume 1256 of LNCS, 605-615, Bologna, Italy, July 1997.
56. R. Grossi and G. F. Italiano. Efficient cross-trees for external memory. In J. Abello and J. S. Vitter, editors, External Memory Algorithms and Visualization, DIMACS series. American Mathematical Society, 1998.
57. S. K. S. Gupta, Z. Li, and J. H. Reif. Generating efficient programs for two-level memories from tensor-products. In Proc. Seventh IASTED/ISMM International Conf. on Parallel and Distributed Computing and Systems, 510-513, Washington, D.C., October 1995.
58. A. Guttman. R-trees: A dynamic index structure for spatial searching. In Proc. ACM SIGMOD Conf. on Management of Data, 47-57, 1985.
59. J. M. Hellerstein, E. Koutsoupias, and C. H. Papadimitriou. On the analysis of indexing schemes. In Proc. 16th ACM Symp. on Principles of Database Systems, 249-256, Tucson, Arizona, May 1997.
60. L. Hellerstein, G. Gibson, R. M. Karp, R. H. Katz, and D. A. Patterson. Coding techniques for handling failures in large disk arrays. Algorithmica, 12(2-3), 182-208, 1994.
61. J. W. Hong and H. T. Kung. I/O complexity: The red-blue pebble game. Proc. 13th Annual ACM Symp. on Theory of Computation, 326-333, May 1981.
62. D. Hutchinson, A. Maheshwari, J.-R. Sack, and R. Velicescu. Early experiences in implementing the buffer tree. Workshop on Algorithm Engineering, 1997. Electronic proceedings available at http://www.dsi.unive.it/~wae97/ proceedings/.
63. I. Kamel and C. Faloutsos. On packing R-trees. In Proc. 2nd International Conf. on Information and Knowledge Management (CIKM), 490-499, 1993.
64. I. Kamel and C. Faloutsos. Hilbert R-tree: An improved R-tree using fractals. In Proc. 20th International Conf. on Very Large Databases, 500-509, 1994.
65. I. Kamel, M. Khalil, and V. Kouramajian. Bulk insertion in dynamic R-trees. In Proc. 4th International Symp. on Spatial Data Handling, 3B, 31-42, 1996.
66. P. C. Kanellakis, G. M. Kuper, and P. Z. Revesz. Constraint query languages. Proc. 9th ACM Conf. on Princ. of Database Systems, 299-313, 1990.
67. P. C. Kanellakis, S. Ramaswamy, D. E. Vengroff, and J. S. Vitter. Indexing for data models with constraints and classes. In Proc. ACM Symp. on Principles of Database Systems, 233-243, 1993. To appear in a special issue of Journ. Comput. Sys. Science.
68. K. V. R. Kanth and A. K. Sing. Optimal dynamic range searching in non-replicating index structures. Technical Report CS97-13, UCSB, July 1997.
69. D. G. Kirkpatrick and R. Seidel. The ultimate planar convex hull algorithm? SIAM Journal on Computing, 15, 287-299, 1986.
70. D. E. Knuth. Sorting and Searching, Volume 3 of The Art of Computer Programming. Addison-Wesley, Reading MA, second edition, 1998.
71. E. Koutsoupias and D. S. Taylor. Tight bounds for 2-dimensional indexing schemes. In Proc. 17th ACM Conf. on Princ. of Database Systems, Seattle, WA, June 1998.
72. V. Kumar and E. Schwabe. Improved algorithms and data structures for solving graph problems in external memory. In Proc. 8th IEEE Symp. on Parallel and Distributed Processing, 169-176, October 1996.
73. R. Laurini and D. Thompson. Fundamentals of Spatial Information Systems. Academic Press, 1992.
74. T. Leighton. Tight bounds on the complexity of parallel sorting. IEEE Transactions on Computers, C-34(4), 344-354, April 1985. Special issue on sorting, E. E. Lindstrom and C. K. Wong and J. S. Vitter, editors.
75. C. E. Leiserson, S. Rao, and S. Toledo. Efficient out-of-core algorithms for linear relaxation using blocking covers. In IEEE Foundations of Comp. Sci., 704-713, 1993.
76. Z. Li, P. H. Mills, and J. H. Reif. Models and resource metrics for parallel and distributed computation. Parallel Algorithms and Applications, 8, 35-59, 1996.
77. J. W. H. Liu. On the storage requirement in the out-of-core multifrontal method for sparse factorization. ACM Transactions on Mathematical Software, 12(3), 249-264, Sept. 1986.
78. D. B. Lomet and B. Salzberg. Concurrency and recovery for index trees. The VLDB Journal, 6(3), 224-240, 1997.
79. J. Nievergelt and P. Widmayer. Spatial data structures: Concepts and design choices. In M. van Kreveld, J. Nievergelt, T. Roos, and P. Widmayer, editors, Algorithmic Foundations of GIS. Springer-Verlag, LNCS 1340, 1997.
80. M. H. Nodine, M. T. Goodrich, and J. S. Vitter. Blocking for external graph searching. Algorithmica, 16(2), 181-214, August 1996.
81. M. H. Nodine, D. P. Lopresti, and J. S. Vitter. I/O overhead and parallel vlsi architectures for lattice computations. IEEE Transactions on Computers, 40(7), 843-852, July 1991.
82. M. H. Nodine and J. S. Vitter. Deterministic distribution sort in shared and distributed memory multiprocessors. In Proc. 5th Annual ACM Symp. on Parallel Algorithms and Architectures, 120-129, June-July 1993.
83. M. H. Nodine and J. S. Vitter. Greed Sort: An optimal sorting algorithm for multiple disks. J. ACM, 42(4), 919-933, July 1995.
84. H. Pang, M. Carey, and M. Livny. Memory-adaptive external sorts. Proc. 19th Conf. on Very Large Data Bases, 1993.
85. J. M. Patel and D. J. DeWitt. Partition based spatial-mergejoin. In Proc. ACM SIGMOD International Conf. on Management of Data, Volume 25, 2 of ACM SIGMOD Record, 259-270, June 1996.
86. S. Ramaswamy and S. Subramanian. Path caching: a technique for optimal external searching. Proc. 13th ACM Conf. on Princ. of Database Systems, 1994.
87. C. Ruemmler and J. Wilkes. An introduction to disk drive modeling. IEEE Computer, 17-28, Mar. 1994.
88. J. Salmon and M. Warren. Parallel out-of-core methods for N-body simulation. In Proc. Eighth SIAM Conf. on Parallel Processing for Scientific Computing, 1997.
89. H. Samet. Applications of Spatial Data Structures: Computer Graphics, Image Processing, and GIS. Addison-Wesley, 1989.
90. H. Samet. The Design and Analysis of Spatial Data Structures. Addison-Wesley, 1989.
91. J. E. Savage and J. S. Vitter. Parallelism in space-time tradeoffs. In F. P. Preparata, editor, Advances in Computing Research, Volume 4, 117-146. JAI Press, 1987.
92. E. Shriver, A. Merchant, and J. Wilkes. An analytic behavior model for disk drives with readahead caches and request reordering. In Joint International Conf. on Measurement and Modeling of Computer Systems, June 1998.
93. E. A. M. Shriver and M. H. Nodine. An introduction to parallel I/O models and algorithms. In R. Jain, J. Werth, and J. C. Browne, editors, Input/Output in Parallel and Distributed Computer Systems, chapter 2, 31-68. Kluwer Academic Publishers, 1996.
94. J. F. Sibeyn. From parallel to external list ranking. Technical Report MPI-I-97-1-021, Max-Planck-Institut, Sept. 1997.
95. J. F. Sibeyn and M. Kaufmann. BSP-like external-memory computation. In Proc. 3rd Italian Conf. on Algorithms and Complexity, 229-240, 1997.
96. R. Tamassia and J. S. Vitter. Optimal cooperative search in fractional cascaded data structures. Algorithmica, 15(2), 154-171, February 1996.
97. S. Toledo. Out of core algorithms in numerical linear algebra, a survey. In J. Abello and J. S. Vitter, editors, External Memory Algorithms and Visualization, DIMACS series. American Mathematical Society, 1998.
98. S. Toledo and F. G. Gustavson. The design and implementation of SOLAR, a portable library for scalable out-of-core linear algebra computations. In Proc. Fourth Workshop on Input/Output in Parallel and Distributed Systems, 28-40, Philadelphia, May 1996.
99. J. D. Ullman and M. Yannakakis. The input/output complexity of transitive closure. Annals of Mathematics and Artificial Intellegence, 3, 331-360, 1991.
100. J. van den Bercken, B. Seeger, and P. Widmayer. A generic approach to bulk loading multidimensional index structures. In Proceedings 23rd VLDB Conf., 406-415, 1997.
101. M. van Kreveld, J. Nievergelt, T. Roos, and P. W. (Eds.). Algorithmic Foundations of GIS. LNCS 1340. Springer-Verlag, 1997.
102. P. J. Varman and R. M. Verma. An efficient multiversion access structure. IEEE Transactions on Knowledge and Data Engineering, 9(3), 391-409 May/June 1997.
103. D. E. Vengroff. TPIE User Manual and Reference. Duke University, 1995. The manual and software distribution are available on the web at http://www.cs.duke.edu/TPIE/.
104. D. E. Vengroff and J. S. Vitter. Efficient 3-d range searching in external memory. In Proc. 28th Annual ACM Symp. on Theory of Computing, Philadelphia, PA, May 1996.
105. D. E. Vengroff and J. S. Vitter. I/O-efficient scientific computation using TPIE. In Proc. Goddard Conf. on Mass Storage Systems and Technologies, Volume II of NASA Conf. Publication 3340, 553-570, College Park, MD, September 1996.
106. J. S. Vitter. Efficient memory access in large-scale computation. In Proc. 1991 Symp. on Theor. Aspects of Comp. Sci., LNCS. Springer-Verlag, 1991.
107. J. S. Vitter. External memory algorithms. In J. Abello and J. S. Vitter, editors, External Memory Algorithms and Visualization, DIMACS series. American Mathematical Society, 1998.
108. J. S. Vitter and E. A. M. Shriver. Algorithms for parallel memory I: Two-level memories. Algorithmica, 12(2-3), 110-147, 1994.
109. J. S. Vitter and E. A. M. Shriver. Algorithms for parallel memory II: Hierarchical multilevel memories. Algorithmica, 12(2-3), 148-169, 1994.
110. M. Wang, J. S. Vitter, and B. R. Iyer. Scalable mining for classification rules in relational databases. In Proc. International Database Engineering & Application Symp., Cardiff, Wales, July 1998.
111. D. Womble, D. Greenberg, S. Wheat, and R. Riesen. Beyond core: Making parallel computer I/O practical. In Proc. 1993 DAGS/PC Symp., 56-63, Hanover, NH, June 1993. Dartmouth Institute for Advanced Graduate Studies.
112. C. Wu and T. Feng. The universality of the shuffle-exchange network. IEEE Transactions on Computers, C-30, 324-332, May 1981.
113. S. B. Zdonik and D. Maier, editors. Readings in Object-Oriented Database Systems. Morgan Kauffman, 1990.
114. W. Zhang and P.-A. Larson. Dynamic memory adjustment for external mergesort. Proc. 23rd Intl. Conf. on Very Large Data Bases, 1997.