Statistical models for empirical search-based performance tuning

International Journal of High Performance Computing Applications

Volumn 18, Issue 1, 2004, Pages 65-94

  Vuduc, Richard ^a   Demmel, James W ^a   Bilmes, Jeff A ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b University of Washington   (United States)

Author keywords

Algorithm selection; Automatic performance tuning; Early stopping; Feedback directed optimization; Matrix multiplication; Performance distribution; Performance optimization; Software engineering; Support vector method

Indexed keywords

ALGORITHMS; CODES (SYMBOLS); COMPUTER ARCHITECTURE; COMPUTER HARDWARE; DIGITAL LIBRARIES; FAST FOURIER TRANSFORMS; LINEAR ALGEBRA; OPTIMIZATION; PROBLEM SOLVING; PROGRAM COMPILERS; STATISTICAL METHODS; VECTORS;

ALGORITHM SELECTION; EMPIRICAL SEARCH-BASED PERFORMANCE TUNING; FEEDBACK-DIRECTED OPTIMIZATION; SUPPORT VECTOR METHOD;

COMPUTATION THEORY;

EID: 1542710758     PISSN: 10943420     EISSN: None     Source Type: Journal    
DOI: 10.1177/1094342004041293     Document Type: Article

References (97)

1. Andersen, B.S., Gustavson, F., Karaivanov, A., Wasniewski, J., and Yalamov, P. Y. September 1999. LAWRA - Linear algebra with recursive algorithms. In Proceedings of the Conference on Parallel Processing and Applied Mathematics, Kazimierz Dolny, Poland.
2. Arge, L., Chase, J., Vitter, J. S., and Wickremesinghe, R. 2001. Efficient sorting using registers and caches. ACM Journal on Experimental Algorithmics 6:1-18.
3. Arnold, M., Fink, S., Grove, D., Hind, M., and Sweeney, P. F. December 2000. Adaptive optimization in the Jalapeño JVM: The controller's analytical model. In MICRO-33: Third ACM Workshop on Feedback-Directed Dynamic Optimization, Monterey, CA.
4. Ball, T. and Larus, J. R. December 1996. Efficient path profiling. In Proceedings of MICRO 96, Paris, France, pp. 46-57.
5. Barnes, R. November 1999. Feedback-directed data cache optimizations for the x86. In Proceedings of the 32nd Annual International Symposium on Microarchitecture, Second Workshop on Feedback-Directed Optimization, Haifa, Israel.
6. Baumgartner, G., Bernholdt, D. E., Cociorva, D., Harrison, R., Hirata, S., Lam, C.-C., Nooijen, M., Pitzer, R., Ramanujam, J., and Saddayappan, P. November 2002. A high-level approach to synthesis of high-performance codes for quantum chemistry. In Proceedings of the IEEE/ACM Conference on Supercomputing, Baltimore, MD.
7. Beckmann, O. and Kelley, P. H. J. August 1997. Run-time interprocedural data placement optimization for lazy parallel libraries. In EuroPar, Lecture Notes in Computer Science, Springer-Verlag, Berlin.
8. Bickel, P. J. and Doksum, K. A. 1977. Mathematical Statistics: Basic Ideas and Selected Topics, Holden-Day, San Francisco, CA.
9. Bik, A. J. C. and Wijshoff, H. A. G. 1995. Advanced compiler optimizations for sparse computations. Journal of Parallel and Distributed Computing 31(1):14-24.
10. Bilmes, J., Asanović, K., Chin, C., and Demmel, J. July 1997. Optimizing matrix multiply using PHiPAC: a Portable, High-Performance, ANSI C coding methodology. In Proceedings of the International Conference on Supercomputing, Vienna, Austria.
11. Bilmes, J., Asanović, K., Demmel, J., Lam, D., and Chin, C. October 1998. The PHiPAC v1.0 matrix-multiply distribution, Technical Report UCB/CSD-98-1020, University of California, Berkeley, CA.
12. Birnbaum, Z. W. 1952. Numerical tabulation of the distribution of Kolmogorov's statistic for finite sample size. Journal of the American Statistical Association 47:425-441.
13. Blackford, S. et al. 2001. Document for the Basic Linear Algebra Subprograms (BLAS) standard: BLAS Technical Forum, http://www.netlib.org/blas/blast-forum.
14. Brewer, E. July 1995. High-level optimization via automated statistical modeling. In Symposium on Parallel Architectures and Algorithms, Santa Barbara, CA.
15. Bruening, D., Garnett, T., and Amarsinghe, S. March 2003. An infrastructure for adaptive dynamic optimization. In Proceedings of the 1st International Symposium on Code Generation and Optimization, San Francisco, CA.
16. Carr, S. and Kennedy, K. 1992. Compiler blockability of numerical algorithms. In Proceedings of Supercomputing, Minneapolis, MN, pp. 114-124.
17. Chang, P. P., Mahlke, S. A., and Hwu, W. W. 1991. Using profile information to assist classic code optimizations. Software - Practice and Experience 21(12):1301-1321.
18. Chatterjee, S., Parker, E., Hanlon, P. J., and Lebeck, A. R. June 2001. Exact analysis of the cache behavior of nested loops. In Proceedings of the ACM SIGPLAN 2001 Conference on Programming Language Design and Implementation, Snowbird, UT, pp. 286-297.
19. Chen, Z., Dongarra, J., Luszczek, P., and Roche, K. January 2003. Self-adapting software for numerical linear algebra and LAPACK for clusters, Technical Report UT-CS-03-499, University of Tennessee.
20. Chow, K. and Wu, Y. November 1999. Feedback-directed selection and characterization of compiler optimizations. In Second Workshop on Feedback-Directed Optimization, Haifa, Israel.
21. Chow, Y. S., Robbins, H., and Siegmund, D. 1971. Great Expectations: The Theory of Optimal Stopping, Houghton-Mifflin, Boston, MA.
22. Cooper, K. D., Subramanian, D., and Torczon, L. 2002a. Adaptive optimizing compilers for the 21st century. Journal of Supercomputing 23(1):7-22.
23. Cooper, K. D., Harvey, T. J., Subramanian, D., and Torczon, L. 2002b. Compilation order matters, Technical Report, Rice University, Houston, TX.
24. Darcy, J. D. 2002. Finding a fast quicksort implementation for Java, http://www.sonic.net/~jddarcy/Research/cs339-quick-sort.pdf.
25. Diniz, P. and Rinard, M. June 1997. Dynamic feedback: An effective technique for adaptive computing. In Proceedings of Programming Language Design and Implementation, Las Vegas, NV.
26. Dolan, E. D. and Moré, J. J. 2002. Benchmarking optimization software with performance profiles. Mathematical Programming 91:201-213.
27. Dongarra, J., Croz, J. D., Duff, I., and Hammarling, S. 1990. A set of level 3 basic linear algebra subprograms. ACM Transactions on Mathematical Software 16(1):1-17.
28. Dongarra, J. and Eijkhout, V. June 2003. Self-adapting numerical software and automatic tuning of heuristics. In Proceedings of the International Conference on Computational Science, Melbourne, Australia.
29. Fraguela, B. B., Doallo, R., and Zapta, E. L. October 1999. Automatic analytic modeling for the estimation of cache misses. In Proceedings of the International Conference on Parallel Architectures and Compilation Techniques, Newport Beach, CA, pp. 221-231.
30. Frens, J. D. and Wise, D. S. July 1997. Auto-blocking matrix-multiplication or tracking BLAS3 performance from source code. In Proceedings of the 6th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, Las Vegas, NV, pp. 206-216.
31. Frey, B. 1998. Graphical Models for Machine Learning and Digital Communications, MIT Press, Boston, MA.
32. Frigo, M. and Johnson, S. May 1998. FFTW: An adaptive software architecture for the FFT. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, Seattle, WA.
33. Frigo, M., Leiserson, C. E., Prokop, H., and Ramachandran, S. October 1999. Cache-oblivious algorithms. In Proceedings of the 40th Annual Symposium on Foundations of Computer Science, New York, NY.
34. Gatlin, K. S. and Carter, L. November 1999. Architecture-cognizant divide and conquer algorithms. In Proceedings of Supercomputing, Portland, OR.
35. Geist, A., Gropp, W., Huss-Lederman, S., Lumsdaine, A., Lusk, E., Saphir, W., Skjellum, T., and Snir, M. 1996. MPI-2: extending the message-passing interface. In Proceedings of the 2nd European Conference on Parallel Processing (Euro-Par'96), Lyon, France, Lecture Notes in Computer Science Vol. 1123-1124, Springer-Verlag, Berlin, pp. 128-135. http://www.mpi-forum.org.
36. Ghosh, S., Martonosi, M., and Malik, S. 1999. Cache miss equations: a compiler framework for analyzing and tuning memory behavior. ACM Transactions on Programming Languages and Systems 21(4):703-746.
37. Goto, K. and Van de Geijn, R. November 2002. On reducing TLB misses in matrix multiplication, Technical Report TR-2002-55, University of Texas at Austin.
38. Graham, S. L., Kessler, P. B., and McKusick, M. K. 1982. gprof: a call graph execution profiler. SIGPLAN Notices 17(6):120-126.
39. Granlund, T. and Krenner, R. 1992. Eliminating branches using a superoptimizer and the GNU C compiler. SIGPLAN Notices 27(7):341-352.
40. Gunnels, J. A., Gustavson, F. G., Henry, G. M., and Van de Geijn, R. A. 2001. FLAME: Formal Linear Algebra Methods Environment. ACM Transactions on Mathematical Software 27(4):422-455.
41. Hong, J. W. and Kung, H. T. May 1981. I/O complexity: the red-blue pebble game. In Proceedings of the 13th Annual ACM Symposium on Theory of Computing, Milwaukee, WI, pp. 326-333.
42. Houstis, E. N., Catlin, A. C., Rice, J. R., Verykios, V. S., Ramakrishnan, N., and Houstis, C. E. 2000. PYTHIA-II: a knowledge/database system for managing performance data and recommending scientific software. ACM Transactions on Mathematical Software 26(2):277-253.
43. Huss-Lederman, S., Jacobson, E. M., Johnson, J. R., Tsao, A., and Turnbull, T. November 1996. Implementation of Strassen's algorithm for matrix multiplication. In Proceedings of Supercomputing, Pittsburgh, PA.
44. Im, E.-J. and Yelick, K. March 1999. Optimizing sparse matrix vector multiplication on SMPs. In Proceedings of the 9th SIAM Conference on Parallel Processing for Scientific Computing, San Antonio, TX.
45. Jordan, M. I. 1995. Why the logistic function?, Technical Report 9503, MIT, Cambridge, MA.
46. Joshi, R., Nelson, G., and Randall, K. August 2001. Denali: A goal-directed superoptimizer, Technical Report 171, Compaq SRC.
47. Kagstrom, B., Ling, P., and Loan, C. V. 1998. GEMM-based level 3 BLAS: high-performance model implementations and performance evaluation benchmark. ACM Transactions on Mathematical Software 24(3):268-302.
48. Kistler, T. and Franz, M. 2003. Continuous program optimization: a case study. ACM Transactions on Programming Languages and Systems 25(4):500-548.
49. Kisuki, T., Knijnenburg, P. M., O'Boyle, M. F., and Wijshoff, H. 2000. Iterative compilation in program optimization. In Proceedings of the 8th International Workshop on Compilers for Parallel Computers, Aussois, France, pp. 35-44.
50. Knuth, D. 1971. An empirical study of FORTRAN programs. Software - Practice and Experience 1(2):105-133.
51. Ko, A. N. and Izaguirre, J. A. 2003. MDSimAid: automatic optimization of fast electrostatics algorithms for molecular simulations. In Proceedings of the International Conference on Computational Science, Melbourne, Australia, LNCS Vol. 2659, Springer-Verlag, Berlin.
52. Lagoudakis, M. G. and Littman, M. L. June 2000. Algorithm selection using reinforcement learning. In Proceedings of the 17th International Conference on Machine Learning, Stanford, CA, pp. 511-518.
53. Lam, M. S., Rothberg, E. E., and Wolf, M. E. April 1991. The cache performance and optimizations of blocked algorithms. In Proceedings of the 4th International Conference on Architectural Support for Programming Languages and Operating Systems, Santa Clara, CA.
54. Leone, M. and Dybvig, R. K. September 1997. Dynamo: a staged compiler architecture for dynamic program optimization, Technical Report TR-490, Department of Computer Science, Indiana University.
55. Liniker, P., Beckmann, O., and Kelly, P. H. J. August 2002. Delayed evaluation, self-optimising software components as a programming model. In Euro-Par, Paderborn, Germany.
56. Lucks, M. and Gladwell, I. 1992. Automated selection of mathematical software. ACM Transactions on Mathematical Software 18(1):11-34.
57. McCalpin, J. D. and Smotherman, M. March 1995. Automatic benchmark generation for cache optimization of matrix algorithms. In Proceedings of the 33rd Annual Southeast Conference, Clemson, SC, USA, R. Geist and S. Junkins, editors, ACM, New York, pp. 195-204.
58. McKinley, K. S., Carr, S., and Tseng, C.-W. 1996. Improving data locality with loop transformations. ACM Transactions on Programming Languages and Systems 18(4):424-453.
59. Massalin, H. 1987. Superoptimizer-a look at the smallest program. In Proceedings of the 2nd International Conference on Architectural Support for Programming Languages and Operating Systems, Palo Alto, CA, pp. 122-126.
60. Mirkovic, D., Mahasoom, R., and Johnsson, L. May 2000. An adaptive software library for fast Fourier transforms. In Proceedings of the International Conference on Supercomputing, Sante Fe, NM, pp. 215-224.
61. Mitchell, N., Hogstedt, K., Carter, L., and Ferrante, J. 1998. Quantifying the multi-level nature of tiling interactions. International Journal of Parallel Programming 26(6):641-670.
62. Mitchell, N., Carter, L., and Ferrante, J. May 2001. A modal model of memory. In Proceedings of the International Conference on Computational Science, San Francisco, CA, LNCS Vol. 2073, Springer-Verlag, Berlin, pp. 81-96.
63. Nisbet, A. June 1998. GAPS: Iterative feedback directed parallelization using genetic algorithms. In Proceedings of the Workshop on Profile and Feedback Directed Compilation, Paris, France.
64. Noether, G. E. 1963. Note on the Kolmogorov statistic in the discrete case. Metrika 7:115-116.
65. Olsen, J. H. and Skov, S. C. 2002. Cache-oblivious algorithms in practice, Master's thesis, University of Copenhagen, Copenhagen, Denmark.
66. Parello, D., Temam, O., and Verdun, J.-M. November 2002. On increasing architecture awareness in program optimizations to bridge the gap between peak and sustained processor performance-matrix multiply revisited. In Proceedings of the IEEE/ACM Conference on Supercomputing, Baltimore, MD.
67. Petrank, E. and Rawitz, D. January 2002. The hardness of cache conscious data placement. In Proceedings of the 29th ACM SIGPLAN-SIGACT Symposium on the Principles of Programming Languages, Portland, OR, ACM, New York, pp. 101-112.
68. Pike, G. and Hilfinger, P. November 2002. Better tiling and array contraction for compiling scientific programs. In Proceedings of the IEEE/ACM Conference on Supercomputing, Baltimore, MD.
69. Platt, J. January 1999. Fast training of support vector machines using sequential minimal optimization. In Advances in Kernel Methods - Support Vector Learning, B. Schölkopf, C. Burges, and A. Smola, editors, MIT Press, Cambridge, MA, pp. 185-208.
70. Pugh, W. and Shpeisman, T. August 1998. Generation of efficient code for sparse matrix computations. In Proceedings of the 11th Workshop on Languages and Compilers for Parallel Computing, Chapel Hill, NC, LNCS, Springer-Verlag, Berlin.
71. Püschel, M., Singer, B., Veloso, M., and Moura, J. M. F. May 2001. Fast automatic generation of DSP algorithms. In Proceedings of the International Conference on Computational Science, San Francisco, CA, LNCS Vol. 2073, Springer-Verlag, Berlin, pp. 97-106.
72. Ramakrishnan, N. and Valdés-Pérez, R. E. 2000. Note on generalization in experimental algorithmics. ACM Transactions on Mathematical Software 26(4)568-580.
73. Rice, J. R. 1976. The algorithm selection problem. Advances in Computers 15:65-118.
74. Santiago, N. G., Rover, D. T. and Rodriguez, D. August 2002. A statistical approach for the analysis of the relation between low-level performance information, the code, and the environment. In Proceedings of the ICPP 4th Workshop on High Performance Scientific and Engineering Computing with Applications, Vancouver, BC, Canada, pp. 282-289.
75. Savage, J. E. 1995. Extending the Hong-Kung model to memory hierarchies. In Computing and Combinatorics, D.-Z. Du and M. Li, editors, LNCS Vol. 959, Springer-Verlag, Berlin, pp. 270-281.
76. Schwartz, D. A., Judd, R. R., Harrod, W. J., and Manley, D. P. March 2000. VSIPL 1.0 API. http://www.vsipl.org.
77. Siek, J. G. and Lumsdaine, A. 1998. A rational approach to portable high performance: the Basic Linear Algebra Instruction Set (BLAIS) and the Fixed Algorithm Size Template (FAST) library. In Proceedings of ECOOP, Brussels, Belgium.
78. Smith, M. D. January 2000. Overcoming the challenges to feedback-directed optimization. In Proceedings of the ACM SIGPLAN Workshop on Dynamic and Adaptive Compilation and Optimization (Dynamo), Boston, MA.
79. Smola, A. J. and Schölkopf, B. 1998. A tutorial on support vector regression, Technical Report NC2-TR-1998-030. European Community ESPRIT Working Group in Neural and Computational Learning Theory. http://www.neurocolt.com.
80. Stephenson, M., Amarasinghe, S., Martin, M., and O'Reilly, U. M. June 2003. Meta optimization: improving compiler heuristics with machine learning. In Proceedings of the ACM Conference on Programming Language Design and Implementation, San Diego, CA.
81. Stodghill, P. August 1997. A relational approach to the automatic generation of sequential sparse matrix codes, Ph.D. Thesis, Cornell University.
82. Thottethodi, M., Chatterjee, S., and Lebeck, A. R. November 1998. Tuning Strassen's matrix multiplication for memory efficiency. In Proceedings of Supercomputing '98, Orlando, FL.
83. Toledo, S. 1997. Locality of reference in LU decomposition with partial pivoting, SIAM Journal on Matrix Analysis and Applications 18(4):1065-1081.
84. Triantafyllis, S., Vachharajani, M., Vachharajani, N., and August, D. I. March 2003. Compiler optimization-space exploration. In Proceedings of the International Symposium on Code Generation and Optimization, San Francisco, CA, pp. 204-215.
85. Tǎpus, C., Chung, I.-H., and Hollingsworth, J. K. November 2002. Active Harmony: towards automated performance tuning. In Proceedings of the IEEE/ACM Conference on Supercomputing, Baltimore, MD.
86. Vadhiyar, S. S., Fagg, G. E., and Dongarra, J. November 2000. Automatically tuned collective operations. In Proceedings of Supercomputing 2000, Dallas, TX.
87. Van Der Mark, P., Rohou, E., Bodin, F., Chamski, Z., and Eisenbeis, C. September 1999. Using iterative compilation for managing software pipeline-unrolling trade-offs. In Proceedings of the 4th International Workshop on Compilers for Embedded Systems, St. Goar, Germany.
88. Vapnik, V. N. 1998. Statistical Learning Theory, Wiley, New York.
89. Veldhuizen, T. 1998. Arrays in Blitz++. In Proceedings of ISCOPE, LNCS Vol. 1505, Springer-Verlag, Berlin.
90. Veldhuizen, T. L. and Gannon, D. 1998. Active libraries: rethinking the roles of compilers and libraries. In Proceedings of the SIAM Workshop on Object Oriented Methods for Interoperable Scientific and Engineering Computing, Philadelphia, PA.
91. Voss, M. J. and Eigenmann, R. August 2000. ADAPT: automated de-coupled adaptive program transformation. In Proceedings of the International Conference on Parallel Processing, Toronto, Canada.
92. Vuduc, R., Demmel, J. W., Yelick, K. A., Kamil, S., Nishtala, R., and Lee, B. November 2002. Performance optimizations and bounds for sparse matrix-vector multiply. In Proceedings of Supercomputing, Baltimore, MD.
93. Whaley, R. C., Petitet, A., and Dongarra, J. 2001. Automated empirical optimizations of software and the ATLAS project. Parallel Computing 27(1):3-25.
94. Wise, D. S., Frens, J. D., Gu, Y., and Alexander, G. A. 2001. Language support for Morton-order matrices. In Proceedings of the 8th ACM SIGPLAN Symposium on Principles and Practices of Parallel Programming, Snowbird, UT, ACM, New York, pp. 24-33.
95. Wolf, M. E. and Lam, M. S. June 1991. A data locality optimizing algorithm. In Proceedings of the ACM SIGPLAN '91 Conference on Programming Language Design and Implementation, Toronto, Ontario, Canada.
96. Yi, Q., Adve, V., and Kennedy, K. June 2000. Transforming loops to recursion for multi-level memory hierarchies. In Proceedings of the ACM SIGPLAN 2000 Conference on Programming Language Design and Implementation, Vancouver, BC, Canada, pp. 169-181.
97. Yotov, K., Li, X., Ren, G., Cibulskis, M., DeJong, G., Garzaran, M., Padua, D., Pingali, K., Stodghill, P., and Wu, P. June 2003. A comparison of empirical and model-driven optimization. In Proceedings of the ACM Conference on Programming Language Design and Implementation, San Diego, CA.