Statistical Analysis of Computational Tests of Algorithms and Heuristics

INFORMS Journal on Computing

Volumn 12, Issue 1, 2000, Pages 24-44

  Coffin, Marie ^a   Saltzman, Matthew J ^a  

^a CLEMSON UNIVERSITY   (United States)

Author keywords

Algorithms; Heuristics; Statistical analysis

Indexed keywords

EID: 0141557747     PISSN: 10919856     EISSN: None     Source Type: Journal    
DOI: 10.1287/ijoc.12.1.24.11899     Document Type: Article

References (68)

1. R.K. AHUJA, T.L. MAGNANTI, and J. B. ORLIN, 1993. Network Flows, Prentice Hall, Englewood Cliffs, New Jersey.
2. R.K. AHUJA and J.B. ORLIN, 1996. Use of Representative Operation Counts in Computational Testing of Algorithms, INFORMS Journal on Computing 8:3, 318-330.
3. M.G. AKRITAS, S.A. MURPHY, and M.P. LAVALLE, 1995. The Thiel-Sen Estimator with Doubly Censored Data and Applications to Astronomy, Journal of the American Statistical Society 90:429, 170-177.
4. M.M. AMINI and R.S. BARR, 1993. Network Reoptimization Algorithms: A Statistically Designed Comparison, INFORMS Journal on Computing 5:4, 395-409.
5. M.M. AMINI and M. RACER, 1994. A Rigorous Computational Comparison of Alternative Solution Methods for the Generalized Assignment Problem, Management Science 40:7, 868-890.
6. R.J. ANDERSON, 1996. The Role of Experiment in the Theory of Algorithms, in Fifth DIMACS Challenge Workshop: Experimental Methodology Day, www.cs.amherst.edu/̃dsj/methday.html.
7. E. BALAS and M.J. SALTZMAN, 1991. An Algorithm for the Three-Index Assignment Problem, Operations Research 39:1, 150-161.
8. R.S. BARR, B. L. GOLDEN, J.P. KELLY, M.G.C. RESENDE, and W.R. STEWART, JR., 1995. Designing and Reporting on Computational Experiments with Heuristic Methods, Journal of Heuristics 1, 1-32.
9. R.E. BIXBY, E.A. BOYD, and R.R. INDOVINA, 1992. MIPLIB: A Test of Real-World Mixed-Integer Programming Problems, SIAM News 25, 16.
10. G.E.P. BOX and D.R. COX, 1964. An Analysis of Transformations, Journal of the Royal Statistical Society 26, 211-243.
11. G.E.P. Box, W.G. HUNTER, and J.S. HUNTER, 1978. Statistics for Experimenters, Wiley, New York, 205-207.
12. M. COFFIN and M.J. SALTZMAN, 1999. Robust Regression with Doubly Censored Data, Technical Report 671, Department of Mathematical Sciences, Clemson University, Clemson, SC.
13. H.P. CROWDER, R.S. DEMBO, and J.M. MULVEY, 1978. Reporting Computational Results in Mathematical Programming, Mathematical Programming 15, 316-329.
14. H.P. CROWDER, R.S. DEMBO, and J.M. MULVEY, 1979. On Reporting Computational Experiments with Mathematical Software, ACM Transactions on Mathematical Software 5:2, 193-203.
15. DATA ANALYSIS PRODUCTS DIVISION, MATHSOFT, 1998. S-Plus 5 for UNIX Guide to Statistics, Seattle, WA.
16. R.S. DEMBO and J.M. MULVEY, 1976. On the Analysis and Comparison of Mathematical Programming Algorithms and Software, in Computers and Mathematical Programming, W.W. White (ed.), National Bureau of Standards, Washington, DC, 106-116.
17. A.T. ERNST and M. KRISHNAMOORTHY, 1996. Efficient Algorithms for the Uncapacitated Single Allocation p-hub Median Problem, Location Science 4, 139-154.
18. A.T. ERNST and M KRISHNAMOORTHY, 1998. An Exact Solution Approach Based on Shortest-Paths for p-hub Median Problems, INFORMS Journal on Computing 10:2, 149-162.
19. R. FOURER and S. MEHROTRA, 1991. Performance of an Augmented System Approach for Solving Least-Squares Problems in an Interior-Point Method for Linear Programming, COAL Newsletter 19, 26-31.
20. D.M. GAY, 1985. Electronic Mail Distribution of Linear Programming Test Problems, COAL Newsletter 13, 10-12.
21. 1 Approximation Codes, Communications in Statistics, Simulation, and Computations B6:4, 399-413.
22. B.L. GOLDEN, A.A. ASSAD, E.A. WASIL, and E. BAKER, 1986. Experimentation in Optimization, European Journal of Operational Research 27, 1-16.
23. B.L. GOLDEN and W.R. STEWART, 1985. Empirical Analysis of Heuristics, in The Traveling Salesman Problem, E.L. Lawler, J.K. Lenstra, A.H.G. Rinnooy Kan, and D.B. Shmoys (eds) Wiley, New York, 207-249.
24. H.J. GREENBERG, 1990. Computational Testing: Why, How and How Much, ORSA Journal on Computing 2:1, 94-97.
25. Z. GU, G.L. NEMHAUSER, and M.W.P. SAVELSBERGH, 1998. Lifted Cover Inequalities for 0-1 Integer Programs: Computation, INFORMS Journal on Computing 10:4, 427-437.
26. P. HANSEN, B. JAUMARD, and V. MATHON, 1993. Constrained Nonlinear 0-1 Programming, INFORMS Journal on Computing 5:2, 97-119.
27. T. P. HETTMANSPERGER, 1984. Statistical Inference Based on Ranks, Wiley, New York.
28. C.R. HICKS, 1982. Fundamental Concepts in the Design of Experiments, 3rd ed., Holt, Reinhart & Winston, New York.
29. D.C. HOAGLIN, V.C. KLEMA, and S.C. PETERS, 1982 Exploratory Data Analysis in a Study of the Performance of Nonlinear Optimization Routines, ACM Transactions on Mathematical Software 8:2, 145-162.
30. C.A.R. HOARE, 1962. Quicksort, Computer Journal 5:1, 10-15.
31. W. HOCK and K. SCHITTKOWSKI, 1981. Test Examples for Nonlinear Programming Codes, number 187, in Lecture Notes in Economics and Mathematical Systems, Springer-Verlag, New York.
32. A. HOFFMAN, M. MANNOS, D. SOLOKOWSKY, and N. WEIGMANN, 1953. Computational Experience in Solving Linear Programs, SIAM Journal 1, 1-33.
33. K.L. HOFFMAN and R.H.F. JACKSON, 1982. In Pursuit of a Methodology for Testing Mathematical Programming Software, in Evaluating Mathematical Programming Techniques, J.M. Mulvey (ed.), number 199, in Lecture Notes in Economics and Mathematical Systems, Springer-Verlag, New York, 177-199.
34. M. HOLLANDER and D.A. WOLFE, 1999. Nonparametric Statistical Methods 2nd ed., Wiley, New York, 56-59.
35. J.N. HOOKER, 1995. Testing Heuristics: We Have It All Wrong, Journal of Heuristics 1, 33-42.
36. R.H.F. JACKSON, 1985. On the State of the Art of Computational Testing of Mathematical Programming Algorithms, COAL Newsletter 12, 8-13.
37. R.H.F. JACKSON, P.T. BOGGS, S.G. NASH, and S. POWELL, 1989. Report of the Ad Hoc Committee to Revise the Guidelines for Reporting Computational Experiments in Mathematical Programming, COAL Newsletter 18, 3-14.
38. R.H.F. JACKSON, P.T. BOGGS, S.G. NASH, and S. POWELL, 1991. Guidelines for Reporting Results of Computational Experiments: Report of the Ad Hoc Committee, Mathematical Programming 49, 413-425.
39. R.H.F. JACKSON and J.M. MULVEY, 1978. A Critical Review of Comparisons of Mathematical Programming Algorithms and Software (1953-1977), Journal of Research of the National Bureau of Standards 83:6, 563-584.
40. D.S. JOHNSON, 1996. A Theoretician's Guide to the Experimental Analysis of Algorithms, in Fifth DIMACS Challenge Workshop: Experimental Methodology Day, www.cs.amherst.edu/̃dsj/methday.html.
41. N. KARMARKAR, 1984. Special presentation, TIMS/ORSA National Meeting, Dallas.
42. C.H. LAYMAN and R.P. O'NEILL, 1976. A Study of the Effect of LP Parameters on Algorithm Performance, in Computers and Mathematical Programming, W.W. White (ed.), National Bureau of Standards, Washington, DC, 251-260.
43. P. L'ECUYER, 1996. Simulation of Algorithms for Performance Analysis, INFORMS Journal on Computing 8:1, 16-20.
44. F.A. LOOTSMA, 1982. Performance Evaluation of Nonlinear Optimization Methods via Multi-criteria Decision Analysis and via Linear Model Analysis, in Nonlinear Optimization 1981, M.J.D. Powell (ed.), Academic, New York, 419-453.
45. I.J. LUSTIG, R.E. MARSTEN, and D.F. SHANNO, 1991. Computational Experience with a Primal-Dual Interior Point Method for Linear Programming, Linear Algebra and its Applications 152, 191-222.
46. C. MCGEOCH, 1992. Analyzing Algorithms by Simulation: Variance Reduction Techniques and Simulation Speedups, Computing Surveys 24:2, 195-212.
47. C. MCGEOCH, 1996. Challenges in Algorithm Simulation, INFORMS Journal on Computing 8:1, 27-28.
48. C. MCGEOCH, 1996. Toward an Experimental Method for Algorithm Simulation, INFORMS Journal on Computing 8:1, 1-15.
49. A. MIELE and S. GONZALEZ, 1978. On the Comparative Evaluation of Algorithms for Mathematical Programming Problems, in Nonlinear Programming, Vol. 3, O.L. Mangasarian, R.R. Meyer, and S.M. Robinson (eds.), Academic, New York, 337-359.
50. D.C. MONTGOMERY, 1991. Design and Analysis of Experiments, Wiley, New York.
51. D.S. MOORE and G.P. MCCABE, 1999. Introduction to the Practice of Statistics, W.H. Freeman, New York.
52. L. NAZARETH and F. SCHLICK, 1976. The Evaluation of Unconstrained Optimization Routines, in Computers and Mathematical Programming, W.W. White (ed.), National Bureau of Standards, Washington, DC, 117-131.
53. P.J. NELSON, 1998. Application of the Analysis of Means, Proceedings of the SAS Users Group International Conference. 13, 225-230.
54. W. NELSON, 1982. Applied Life Data Analysis, Wiley, New York.
55. J. NETER, M.H. KUTNER, C.J. NACHTSHIEM, and J.W. WASSERMAN, 1998. Applied Linear Statistical Models, 4th ed., Irwin, Homewood, IL.
56. J.B. ORLIN, 1996. On Experimental Methods for Algorithm Simulation, INFORMS Journal on Computing 8:1, 21-23.
57. H.D. RATLIFF and W. PIERSKALLA, 1981. Reporting Computational Experience, Operations Research, 29:2, xi-xiv.
58. Y. ROCHAT and E.D. TAILLARD, 1995. Probabilistic Diversification and Intensification in Local Search for Vehicle Routing, Journal of Heuristics 1:1, 147-165.
59. P.J. ROUSSEEUW and A.M. LEROY, 1997. Robust Regression and Outlier Detection, Wiley, New York.
60. T. SAATY, 1980. The Analytic Hierarchy Process, McGraw-Hill, New York.
61. SAS INSTITUTE, 1989. SAS/STAT User's Guide, Version 6, 4th ed., SAS Institute, Cary, NC.
62. R. SEDGEWICK, 1977. The Analysis of Quicksort Programs, Acta Informatica 7, 327-355.
63. E. SHAYAN, 1986. A Methodology for Algorithm Comparison in Mathematical Programming, COAL Newsletter 15, 3-11.
64. D.R. SHIER, 1996. On Algorithm Analysis, INFORMS Journal on Computing 8:1, 24-26.
65. J.W. TUKEY, 1977. Exploratory Data Analysis, Addison-Wesley, Reading, MA.
66. R.J. VANDERBEI, 1996. Linear Programming: Foundations and Extensions, Kluwer, Boston, MA.
67. W.W. WHITE, Ed., 1976. Computers and Mathematical Programming, National Bureau of Standards, Washington, DC.
68. E. ZEMEL, 1981. Measuring the Quality of Approximate Solutions to Zero-One Programming Problems, Mathematics of Operations Research 6:3, 319-332.