Assessing the reliability of statistical software: Part I

American Statistician

Volumn 52, Issue 4, 1998, Pages 358-366

  McCullough, B D ^a  

^a FEDERAL COMMUNICATIONS COMMISSION   (United States)

Author keywords

Accuracy; Benchmarks; Random number generator; Software testing; StRD

Indexed keywords

EID: 22444455170     PISSN: 00031305     EISSN: 15372731     Source Type: Journal    
DOI: 10.1080/00031305.1998.10480597     Document Type: Article

References (58)

1. Bankhofer, U., and Hilbert, A. (1997), “Statistical Software Packages for Windows: A Market Survey,” Statistical Papers, 38, 393-407.
2. Brown, B. W. (1998), DCDFLIB v1.1 (Double Precision Cumulative Distribution Function Library), available at ftp://odin.mdacc.tmc.edu/pub/source
3. Brown, B. W., and Levy, L. B. (1994), “Certification of Algorithm 708: Significant Digit Computation of the Incomplete Beta,” ACM Transactions on Mathematical Software, 20, 393-397.
4. Dahlquist, G., and Björck, A. (1974), Numerical Methods, New York: Prentice-Hall.
5. Dennis, J. E. (1984), “A User’s Guide to Nonlinear Optimization Algorithms,” in Proceedings of the IEEE, 72, 1765-1776.
6. Donaldson, J. R., and Schnabel, R. B. (1987), “Computational Experience With Confidence Regions and Confidence Intervals for Nonlinear Least Squares,” Technometrics, 29, 67-82.
7. Eddy, W. F., Howe, S. E., Ryan, B. F., Teitel, R. F., and Young, F. (1991), The Future of Statistical Software: Proceedings of a Forum, Washington, DC: National Academy Press.
8. Elliott, A. C., Reisch, J. S., and Campbell, N. P. (1989), “Benchmark Data Sets for Evaluating Microcomputer Statistical Programs,” Collegiate Microcomputer, 11, 289-299.
9. Francis, I. (1981), Statistical Software: A Comparative Review, New York: North-Holland.
10. Francis, I., Heiberger, R. M., and Velleman, P. F. (1975), “Criteria and Considerations in the Evaluation of Statistical Software,” The American Statistician, 29, 52-56.
11. Gentle, J. (1998a), Numerical Linear Algebra with Applications in Statistics, New York: Springer.
12. Gentle, J. (1998b), Random Number Generation and Monte Carlo Methods, New York: Springer.
13. Goldberg, D. (1991), “What Every Computer Scientist Should Know about Floating-Point Arithmetic,” ACM Computing Surveys, 23, 5-48.
14. Hammarling, S. (1985), “The Singular Value Decomposition in Multivariate Statistics,” ACM Special Interest Group on Numerical Mathematics, 20, 2-25.
15. Higham, N. J. (1996), Accuracy and Stability of Numerical Algorithms, Philadelphia: SIAM.
16. IBM (1968), System/360 Scientific Subroutine Package, Version III, Programmer’s Manual, White Plains, NY: author.
17. Kennedy, W. J., and Gentle, J. E. (1980), Statistical Computing, New York: Marcel-Dekker.
18. Knüsel, L. (1986), “Computation of the Chisquare and Poisson Distribution,” SIAM Journal on Scientific and Statistical Computing, 7, 1022-1036.
19. Knüsel, L. (1989), “Computergestützte Berechnung Statistischer Verteilungen,” Oldenburg, München-Wien (an English version of the program can be obtained) at www.stat.uni-muenchen.de/~knuesel/elv
20. Knüsel, L. (1995), “On the Accuracy of Statistical Distributions in Gauss,” Computational Statistics and Data Analyis, 20, 699-702.
21. Knüsel, L. (1996), ‘Telegrams,” Computational Statistics and Data Analysis, 21, 116.
22. Knüsel, L. (1998), “On the Accuracy of Statistical Distributions in Microsoft Excel 97,” Computational Statistics and Data Analysis, 26, 375-377.
23. Knüsel, L., and Bablok, B. (1996), “Computation of the Noncentral Gamma Distribution,” SIAM Journal on Scientific Computing, 17, 1224-1231.
24. Knuth, D. E. (1997), The Art of Computer Programming (vol. 2, 3rd ed.), Reading, MA: Addison-Wesley.
25. Lachenbruch, P. A. (1983), “Statistical Programs for Microcomputers,” Byte, 8, 560-570.
26. L’Ecuyer, P. (1994), “Uniform Random Number Generation,” Annals of Operations Research, 53, 77-120.
27. L’Ecuyer, P. (1999), “Random Number Generation,” in Handbook on Simulation, ed. J. Banks, New York: Wiley, pp. 93-138.
28. Lesage, J. P., and Simon, S. D. (1985), “Numerical Accuracy of Statistical Algorithms for Microcomputers,” Computational Statistics and Data Analysis, 3, 47-57.
29. Letson, D., and McCullough, B. D. (1998), “Better Bootstrap Confidence Intervals: The Double Bootstrap with No Pivot,” American Journal of Agricultural Economics, 80, 552-559.
30. Ling, R. F. (1974), “Comparison of Several Algorithms for Computing Sample Means and Variances,” Journal of the American Statistical Association, 69, 859-866.
31. Longley, J. W. (1967), “An Appraisal of Computer Programs for the Electronic Computer from the Point of View of the User,” Journal of the American Statistical Association, 62, 819-841.
32. Marsaglia, G. (1968), “Random Numbers Fall Mainly in the Planes,” in Proceedings of the National Academy of Sciences of the United States of America, 60, pp. 25-28.
33. Marsaglia, G. (1993), “Monkey Tests for Random Number Generators,” Computers and Mathematics with Applications, 26, 1-10.
34. Marsaglia, G. (1996), “DIEHARD: A Battery of Tests of Randomness,” http://stat.f su.edu/“geo
35. MacKinnon, J. G. (1996), “Numerical Distribution Functions for Unit Root and Cointegration Tests,” Journal of Applied Econometrics, 11,601-618.
36. McCullough, B. D. (1997), “Benchmarking Numerical Accuracy: A Review of RATS v4.2,” Journal of Applied Econometrics, 12, 181-190.
37. McCullough, B. D. (in press), “Wilkinson’s Tests and Econometric Software,” Journal of Economic and Social Measurement.
38. Park, S. K., and Miller, K. W. (1988), “Random Number Generators: Good Ones are Hard to Find,” Communications of the ACM, 31, 1192-1201.
39. Press, W., Teukolsky, S., Vetterling, W., and Flannery, B. (1992), Numerical Recipes in FORTRAN (2nd ed.), New York: Cambridge University Press.
40. Ripley, B. D. (1987), Stochastic Simulation, New York: Wiley.
41. Ripley, B. D. (1990), “Thoughts on Pseudorandom Number Generators,” Journal of Computational and Applied Mathematics, 31, 153-163.
42. Rogers, J., Filliben, J., Gill, L Guthrie, W., Lagergren, E and Vangel, M. (1998), “StRD: Statistical Reference Datasets for Assessing the Numerical Accuracy of Statistical Software,” NIST TN# 1396, National Institute of Standards and Technology.
43. Sawitzki, G. (1985), “Another Random Number Generator Which Should Be Avoided,” Statistical Software Newsletter, 11, 81-82.
44. Sawitzki, G. (1994a), ‘Testing Numerical Reliability of Data Analysis Systems,” Computational Statistics and Data Analysis, 18, 269-286.
45. Sawitzki, G. (1994b), “Report on the Reliability of Data Analysis Systems,” Computational Statistics and Data Analysis (SSN), 18, 289-301.
46. Simon, S. D., and Lesage, J. P. (1988), “Benchmarking Numerical Accuracy of Statistical Algorithms,” Computational Statistics and Data Analysis, 1, 197-209.
47. Simon, S. D., and Lesage, J. P. (1989), “Assessing the Accuracy of ANOVA Calculations in Statistical Software,” Computational Statistics and Data Analysis, 8, 325-332.
48. Stephens, M. A. (1986), “Tests for the Uniform Distribution,” in Goodness-of-Fit Techniques, eds. R. D’Agostino and M. Stephens, New York: Marcel-Dekker, pp. 331-366.
49. Stewart, G. W. (1996), Afte motes on Numerical Analysis, Philadelphia, PA: SIAM Press.
50. Tezuka, S. (1995), Uniform Random Numbers: Theory and Practice, Boston: Kluwer.
51. Thisted, R. A. (1988), Elements of Statistical Computing, New York: Chapman and Hall.
52. Veall, M. R. (1991), “Shazam 6.2: A Review,” Journal of Applied Econometrics, 6, 317-320.
53. Vinod, H. D. (1989), “A Review of Soritec 6.2,” American Statistician, 43, 266-269.
54. Wampler R. H. (1970), “A Report on the Accuracy of Some Widely-Used Least Squares Computer Programs,” Journal of the American Statistical Association, 65, 549-565.
55. Wampler R. H. (1980), ‘Test Procedures and Test Problems for Least Squares Algorithms,” Journal of Econometrics, 12, 3-22.
56. Wilkinson, L. (1985), Statistics Quiz, Evanston, IL: SYSTAT, Inc. (available at http://www.tspintl.com/benchmarks)
57. Wilkinson, L. (1994), “Practical Guidelines for Testing Statistical Software,” Computational Statistics, eds. P. DirschedI and Riidiger Ostermann, Berlin: Physica-Verlag, pp. 111-124.
58. Wilkinson, L., and Dallai, G. E. (1977), “Accuracy of Sample Moments Calculations Among Widely Used Statistical Programs,” The American Statistician, 31, 128-131.