Software Reliability-Status and Perspectives

IEEE Transactions on Software Engineering

Volumn SE-8, Issue 4, 1982, Pages 354-371

  Ramamoorthy, C V ^a   Bastani, Farokh B ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF HOUSTON   (United States)

Author keywords

Correctness probability; error seeding; error size; error counting models; evaluation of test cases; nonerror counting models; software fault; software reliability models; testing and debugging phase; testing process; validation phase

Indexed keywords

COMPUTER SOFTWARE;

EID: 0020154271     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.1982.235728     Document Type: Article

References (114)

1. J. Abe, K. Sakamura, and H. Aiso, “An analysis of software project failure,” in Proc. 4th Int. Conf. Software Eng., Munich, Germany, Sept. 1979, pp. 378-385.
2. F. Akiyama, “An example of software system debugging,” in Proc. IFIP1971. Amsterdam, The Netherlands: North-Holland, 1972, pp. 353-359.
3. P. G. Anderson, “Redundancy techniques for software quality,” in Proc. Annu. Rel and Maintainability Symp., 1978, pp. 86-93.
4. J. E. Angus, R. E. Schafer, and A. Sukert, “Software reliability model validation,” in Proc. Annu. Rel. and Maintainability Symp., San Francisco, CA, Jan. 1980, pp. 191-199.
5. R. G. Babb, II and L. L. Trip, “An engineering framework for software standards,” in Proc. Annu. Rel. and Maintainability Symp., San Francisco, CA, Jan. 1980, pp. 214-219.
6. R. E. Barlow and F. Proschan, Statistical Theory of Reliability and Life Testing. New York: Holt, Rinehart and Winston, 1975.
7. F. B. Bastani, “An input domain based theory of software reliability and its application,” Ph.D. dissertation, Dep. Elec. Eng. Comput. Sci., Univ. of California, Berkeley, 1980.
8. L. A. Belady and M. M. Lehmann, “An introduction to growth dynamics,” in Statistical Computer Performance Evaluation, W. Freiberger, Ed. New York: Academic, 1972, pp. 503-511.
9. “A model of large program development,” IBM Syst. J., vol. 15, no. 3, pp. 225-252, 1976.
10. “The characteristic of large systems,” IBM Tech. Rep. RC6785 (#28969), Sept. 13, 1977.
11. W. R. Bezanson, “Reliable software through requirements definition using data abstraction,” in Microelectronics and Reliability, vol. 17. New York: Pergamon, 1978, pp. 85-92.
12. P. J. Bickel and K. A. Doksum, Mathematical Statistics: Basic Ideas and Selected Topics. San Francisco: Holden-Day, 1977.
13. B. W. Boehm, J. R. Brown, and M. Lipow, “Quantitative evaluation of software quality,” in Proc. 2nd Int. Conf. Software Eng., San Francisco, CA, Oct. 1976, pp. 592-605.
14. S. Bologna and W. Ehrenberger, “Applicability of statistical models for reactor safety software verification,” Rep., 1978.
15. J. R. Brown and M. Lipow, “Testing for software reliability,” in Proc. 1975 Int. Conf Rel. Software, Los Angeles, CA, Apr. 1975, pp. 518-527.
16. W. L. Bunce, “Hardware and software: An analytical approach,” in Proc. Annu. Rel. and Maintainability Symp., San Francisco, CA, Jan. 1980, pp. 209-213.
17. N. Chapin, “A measure of software complexity,” in Proc. 1979 Nat. Comput. Conf, AFIPS, New York, June 1979, pp. 995-1002.
18. R. C. Cheung, “A user-oriented software reliability model,” in Proc. COMPSAC1978, Chicago, IL., Nov. 1978, pp. 565-570.
19. D. R. Cox and P.A.W. Lewis, The Statistical Analysis of Series of Events. London: Methuen, 1966.
20. B. Curtis, S. B. Sheppard, and P. Milliman, “Third time charm: Stronger prediction of programmer performance by software complexity metrics,” in Proc. 4th Int. Conf. Software Eng., Munich, Germany, Sept. 1979, pp. 356-360.
21. G. Dahll and J. Lahti, “Investigation of methods for production and verification of computer programmes with high requirements for reliability,” OECD Halden Reactor Project, Preliminary Rep., 1978.
22. R. A. DeMillo, R. J. Lipton, and F. G. Say ward, “Hints on test data selection: Help for the practicing programmer,” Computer, pp. 34-41, Apr. 1978.
23. R. A. Demillo, R. J. Lipton, and A. J, Perlis, “Social processes and proofs of theorems and programs,” Commun. Ass. Comput. Mach., vol. 22, pp. 271-280, May 1979.
24. J. C. Dickson et al., “Quantitative analysis of software reliability,” in Proc. Annu. Rel. and Maintainability Symp., San Francisco, CA, 1972, pp. 148-157.
25. J. T. Duane, “Learning curve approach to reliability monitoring,” IEEE Trans. Aerosp., vol. 2, pp. 563-566, 1964.
26. L. Duvall et al., “Data needs for software reliability modelling,” in Proc. Annu. Rel. and Maintainability Symp., San Francisco, CA, Jan. 1980, pp. 200-208.
27. A. Endres, “An analysis of errors and their causes in system programs,” in Proc. Int. Conf on Rel. Software, Los Angeles, CA, 1975, pp. 327-336.
28. M. E. Fagan, “Design and code inspections to reduce errors in program development,” IBM Syst. J., vol. 15, no. 3, pp. 182-211, 1976.
29. A. R. Feuer and E. B. Fowlkes, “Relating computer program maintainability to software measures,” in Proc. 1979 Nat. Comput. Conf, AFIPS, New York, June 1979, pp. 1003-1012.
30. “Some results from an empirical study of computer software,” in Proc. 4th Int. Conf. Software Eng., Munich, Germany, Sept. 1979, pp. 351-355.
31. J. M. Finkelstein, “Starting and limiting values for reliability growth,” IEEE Trans. Rel, vol. R-28, pp. 111-114, June 1979.
32. K. F. Fischer and M. G. Walker, “Improved software reliability through requirements verification,” IEEE Trans. Rel., vol. R-28, pp. 233-240, Aug. 1979.
33. E. H. Forman and N. D. Singpurwalla, “An empirical stopping rule for debugging and testing computer software,” J. Amer. Stat. Ass., vol. 72, pp. 750-757, Dec, 1977.
34. “Optimal time intervals for testing hypotheses on computer software errors,” IEEE Trans. Rel, vol. R-28, pp. 250-253, Aug. 1979.
35. J. D. Gannon and J. J. Horning, “The impact of language design on the production of reliable software,” in Proc. Int. Conf Rel. Software, Los Angeles, CA, 1975, pp. 10-22.
36. E. Girard and J.-C. Rault, “A programming technique for software reliability,” in Rec. 1973 IEEE Symp. Comput. Software Rel., New York, May 1973, pp. 44-50.
37. A. L. Goel and K. Okumoto, “Time-dependent error-detection model for software reliability and other performance measures,” IEEE Trans. Rel, vol. R-28, pp. 206-211, Aug. 1979.
38. J. B. Goodenough and S. L. Gerhart, “Toward a theory of test data selection,” IEEE Trans. Software Eng., vol. SE-1, pp. 156-173, June 1975.
39. S. J. Greenspan and C. L. McGowan, “Structuring software development for reliability,” in Microelectronics and Reliability, vol. 17. New York: Pergamon, 1978, pp. 75-84.
40. M. H. Halstead, Elements of Software Science. New York: Elsevier, 1977.
41. P. A. Hamilton and J. D. Musa, “Measuring reliability of computation center software,” in Proc. 3rd Int. Conf. on Software Eng., Atlanta, GA, 1978, pp. 29-36.
42. H. Hecht, “Fault-tolerant software for real-time applications,” Comput. Surveys, vol. 8, pp. 391-407, Dec. 1976.
43. “Fault-tolerant software,” IEEE Trans. Rel, vol. R-28, pp. 227-232, Aug. 1979.
44. P. G. Hoel, S. C. Port, and C. J. Stone, Introduction to Probability Theory. Boston: Houghton Mifflin, 1971.
45. P. P. Howley, Jr., “Software quality assurance for reliable software,” in Proc. Annu. Rel. and Maintainability Symp., 1978, pp. 73-78.
46. D. R. Jeffrey and M. J. Lawrence, “An inter-organizational comparison of programmer productivity,” in Proc. 4th Int. Conf Software Eng., Munich, Germany, Sept. 1979, pp. 369-377.
47. Z. Jelinski and P. Moranda, “Software reliability research,” in Statistical Computer Performance Evaluation, W. Freiberger, Ed. New York: Academic, 1972, pp. 465-484.
48. “Applications of a probability-based model to code reading,” in Rec. 1973 IEEE Symp. Comput. Software Rel, New York, May 1973, pp. 78-81.
49. R. H. Keegan and R. C. Howard, “Approximation method for estimating meaningful parameters for a software-controlled electromechanical system,” in Proc. Annu. Rel and Maintainability Symp., 1976, pp. 434-439.
50. M. B. Kline, “Software and hardware R & M: What are the differences?,” in Proc. Annu. Rel, and Maintainability Symp., San Francisco, CA, Jan. 1980, pp. 179-185.
51. H. Kopetz, “Software redundancy in real-time systems,” in Proc. IFIP Congress 1974. Amsterdam, The Netherlands: North-Holland, 1974, pp. 182-186.
52. M. M. Lehman and F. N. Parr, “Program evolution and its impact on software engineering,” in Proc. 2nd Int. Conf Software Eng., San Francisco, CA, Oct, 1976, pp. 350-357.
53. B. Littlewood and J. L. Verrall, “A Bayesian reliability growth model for computer software,” J. Roy. Stat. Soc, vol. 22, no. 3, pp. 332-346, 1973; also in Rec. 1973 IEEE Symp. Comput. Software Rel, New York, May 1973, pp. 70-77.
54. B. Littlewood, “MTBF is meaningless in software reliability,” IEEE Trans. Rel, vol. R-24, p. 82, Apr. 1975.
55. “A reliability model for Markov structured software,” in Proc. 1975 Int. Conf. Rel. Software, Los Angeles, CA, pp. 204-207.
56. “A Bayesian differential debugging model for software reliability,” Dep. Math., City Univ., London, England, June 1979; also in Proc. COMPSAC 1980, Chicago, IL, 1980, pp. 511-519.
57. “How to measure software reliability and how not to,” IEEE Trans. Rel., vol. R-28, pp. 103-110, June 1979; also in Proc. 3rd Int. Conf. on Software Eng., Atlanta, GA, May 1978, pp. 37-45.
58. “Software reliability model for modular program structure,” IEEE Trans. Rel, vol. R-28, pp. 241-246, Aug. 1979.
59. private communication.
60. B. Littlewood and J. L. Verrall, “On the likelihood function of a debugging model for computer software reliability,” Dep. Math., City Univ., London, England, 1980.
61. B. Littlewood, “Theories of software reliability: How good are they and how can they be improved?,” IEEE Trans. Software Eng., vol. SE-6, pp. 489-500, Sept. 1980.
62. A. B. Long et al., “A methodology for the development and validation of critical software for nuclear power plants,” in Proc. 1st Int. Conf on Software Appl., Nov. 1977.
63. W. H. MacWilliams, “Reliability of large real-time control software systems,” in Rec. 1973 IEEE Symp. Comput. Software Rel., New York, May 1973, pp. 1-6.
64. T. J. McCabe, “A complexity measure,” IEEE Trans. Software Eng., vol. SE-2, pp. 308-320, Dec. 1976.
65. H. D. Mills, “On the development of large reliable software,” in Rec. IEEE Symp. Comput. Software Rel., New York, May 1973, pp. 155-159.
66. I. Miyamoto, “Software reliability in on-line real-time environment,” in Proc. Int. Conf Rel. Software, Los Angeles, CA, 1975, pp. 194-203.
67. “Toward an effective software reliability evaluation,” in Proc. 3rd Int. Conf on Software Eng., Atlanta, GA, 1978, pp.46-55.
68. S. N. Mohanty, “Models and measurements for quality assessment of software,” Comput. Surveys, vol. 11, pp. 251-275, Sept. 1979.
69. P. B. Moranda, “Prediction of software reliability during debugging,” in Proc. 1975 Annu. Rel. and Maintainability Symp., Washington, DC, Jan. 1975, pp. 327-332.
70. “The (sad) status of; (unapproached) limits to; and (manifold) alternatives for software measurement techniques,” in Proc. COMPCON 1978, San Francisco, CA, Mar. 1978, pp. 353-354.
71. “Software reliability revisited,” Computer, vol. 11, pp. 92-94, April. 1978.
72. “Limits to program testing with random number inputs,” in Proc. COMPSAC 1978, Chicago, IL, Nov. 1978, pp. 521 526.
73. “Event-altered rate models for general reliability analysis,” IEEE Trans. Rel., vol. R-28, pp. 376-381, Dec. 1979.
74. F. Mosteller and J. W. Tukey, Data Analysis and Regression: A Second Course in Statistics. Reading, MA: Addison-Wesley, 1977.
75. J. D. Musa, “A theory of software reliability and its applications,” IEEE Trans. Software Eng., vol. SE-1, pp. 312-327, Sept. 1975.
76. “An exploratory experiment with foreign debugging of programs,” in Proc. Symp. Comput. Software Eng., New York, Apr. 1976, pp. 499-511.
77. “Software reliability measures applied to system engineering,” in Proc. 1979 Nat. Comput. Conf, AFIPS, New York, June 1979, pp. 941-946.
78. “Validity of execution-time theory of software reliability,” IEEE Trans. Rel., vol. R-28, pp. 181-191, Aug. 1979.
79. E. Nelson, “Estimating software reliability from test data,” in Microelectronics and Reliability, vol. 17. New York: Pergamon, 1978, pp. 67-74.
80. D. L. Parnas, “Information distribution aspects of design methodology,” in Proc. IFIP 1971. Amsterdam, The Netherlands: North-Holland, 1972, pp. 339-344.
81. “The influence of software structure on reliability,” in Proc. Int. Conf. Rel. Software, Los Angeles, CA, 1975, pp. 358-362.
82. C. V. Ramamoorthy and F. B. Bastani, “An input domain based approach to the quantitative estimation of software reliability,” in Proc. Taipei Seminar on Software Eng., Taipei, Taiwan, 1979.
83. C. V. Ramamoorthy et al., “A systematic approach to the development and validation of critical software for nuclear power plants,” in Proc. Taipei Seminar on Software Eng., Taipei, Taiwan, 1979.
84. C. V. Ramamoorthy and F. B. Bastani, “Practical considerations for the development of process control software,” presented at INTERKAMA 1980, Dusseldorf, West Germany, Oct. 1980.
85. “Modelling of the software reliability growth process,” in Proc. COMPSAC 1980, Chicago, IL, 1980, pp. 161-169.
86. J.-C. Rault, “An approach towards reliable software,” in Proc. 4th Int. Conf on Software Eng., Munich, Germany, Sept. 1979, pp. 220-230.
87. D. J. Reifer, “A new assurance technology for computer software,” in Proc. Annu. Rel. and Maintainability Symp., 1916, pp. 446-451.
88. “Software failure modes and effect analysis,” IEEE Trans. Rel., vol. R-28, pp. 247-249, Aug. 1979.
89. H. Remus and S. Zilles, “Prediction and management of program quality,” in Proc. 4th Int. Conf Software Eng., Munich, Germany, Sept. 1979, pp. 341-350.
90. R. J. Rubey, “Quantitative aspects of software validation,” in Proc. Int. Conf. on Rel. Software, Los Angeles, CA, 1975, pp. 246-251.
91. G. J. Schick and R. W. Wolverton, “Achieving reliability in large scale software systems,” in Proc. 1974 Rel. and Maintainability Symp., Los Angeles, CA, Jan. 1974, pp. 302-319.
92. “An analysis of competing software reliability models,” IEEE Trans. Software Eng., vol. SE-4, pp. 104-120, Mar. 1978.
93. N. F. Schneidewirid, “An approach to software reliability prediction and quality control,” in Proc. 1972 Fall Joint Comput. Conf, AFIPS, vol. 41, pp. 837-847.
94. “Analysis of error processes in computer software,” in Proc. Int. Conf. on Rel. Software, Los Angeles, CA, 1975, pp. 337-346.
95. M. L. Shooman, “Probability models for software reliability prediction,” in Statistical Computer Performance Evaluation, W. Freiberger, Ed. New York:. Academic, 1972, pp. 485-502.
96. “Operational testing and software reliability estimation during program development,” in Rec. 1973 IEEE Symp. Comput. Software Rel, New York, May 1973, pp. 51-57.
97. M. L. Shooman and S. J. Amster, “Software reliability: An overview,” in Reliability and Fault Tree Analysis, Barlow, Fussel, and Singpurwalla, Eds. Philadelphia, SIAM, 1975, pp. 655-685.
98. M. L. Shooman, “Software reliability: Measurement and models,” in Proc. 1975 Annu. Rel and Maintainability Symp., Washington, DC, Jan. 1975, pp. 485-491.
99. M. L. Shooman and M. I. Bolsky, “Types, distribution, and test and correction times for programming errors,” in Proc. Int. Conf. Rel. Software, Los Angeles, CA, 1975, pp. 347-357.
100. M. L. Shooman, “Structured models for software reliability prediction,” in Proc. 2nd Int. Conf Software Eng., San Francisco, CA, 1976, pp. 268-280.
101. A. N. Sukert, “A four-project empirical study of software error prediction models,” in Proc. COMPSAC 1978, Chicago, IL, Nov. 1978, pp. 577-582.
102. “Empirical validation of three error prediction models,” IEEE Trans. Rel, vol. R-28, pp. 199-205, Aug. 1979.
103. A. N. Sukert and A. L. Goel, “A guidebook for software reliability assessment,” in Proc. Annu. Rel and Maintainability Symp., San Francisco, CA, Jan. 1980, pp. 186-190.
104. E. B. Swanson, “The dimensions of maintenance,” in Proc. 2nd Int. Conf Software Eng., San Francisco, CA, Oct. 1976, pp. 492-497.
105. “On the user-requisite variety of computer application software,” IEEE Trans. Rel, vol. R-28, pp. 221-226, Aug. 1979.
106. J. Tal, “Development and evaluation of software reliability estimators,” Dep. Elec. Eng., Univ. of Utah, Salt Lake City, UTECSR 77-013, 1976.
107. W. E. Thomson and P. O. Chelson, “On the specification and testing of software reliabiity,” in Proc. Annu. Rel. and Maintainability Symp., San Francisco, CA, Jan. 1980, pp. 379-383.
108. A. K. Trivedi and M. L. Shooman, “A many-state Markov model for the estimation and prediction of computer software performance parameters,” in Proc. 1975 Int. Conf. Rel. Software, Los Angeles, CA, pp. 208-220.
109. TRW Defense and Space Systems Group, “Software reliability study,” Redondo Beach, CA, Rep. 76-2260.1-9-5, 1976.
110. G. F. Walters and J. A. McCall, “The development of metrics for software R & M,” in Proc. Annu. Rel. and Maintainability Symp., 1978, pp. 78-85.
111. “Software quality metrics for life-cycle cost-reduction,” IEEE Trans. Rel, vol. R-28, pp. 212-220, Aug. 1979.
112. E. J. Weyuker and T. J. Ostrand, “Theories of program testing and the application of revealing subdomains,” in Dig. Workshop on Software Testing and Test Documentation, FL, Dec. 1978, pp. 1-18.
113. B. B. White, “Program standards help software maintainability,” in Proc. Annu. Rel and Maintainability Symp., 1978, pp. 94-98.
114. M. V. Zelkowitz, “Perspectives on software engineering,” Comput. Surveys, vol. 10, pp. 197-216, June 1978.