Validation, Verification, and Testing of Computer Software

ACM Computing Surveys (CSUR)

Volumn 14, Issue 2, 1982, Pages 159-192

  Adrion, W Richards ^a   Branstad, Martha A ^b   Cherniavsky, John C ^a  

^a NATIONAL SCIENCE FOUNDATION   (United States)

^b NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER SOFTWARE;

EID: 0020141868     PISSN: 03600300     EISSN: 15577341     Source Type: Journal    
DOI: 10.1145/356876.356879     Document Type: Article

References (100)

1. Acree, A. “On Mutation”, Ph.D dissertation, Information and Computer Science Dep, Georgia Institute of Technology, Atlanta, June, 1980
2. Adrion, W R. “Issues in software validation, verification, and testing”, ORSA/TIMS Bull. (1980 TIMS-ORSA Conf.) 10 (Sept. 1980), 80.
3. Alford, M.W. “A requirement engineering methodology for real-time processing requirements”, IEEE Trans Softw Eng SE-2, 1 (1977), 60–69
4. Allen, F. E. “Interprocedural data flow analysis”, in Proc IFIP Congress 1974, North-Holland, Amsterdam, 1974, pp. 398–402.
5. Allen, F. E., and Cocke, J. “A program data flow procedure”, Commun. ACM 19, 3 (March 1976), 137–147.
6. Ambler, A. L., Good, D. I., Browne, J. C., Burger, W. F., Cohen, R. M., Hoch, C. G., and Wells, R. E. “Gypsy: A language for specification and implementation of verifiable programs”, in Proc. Conf. Language Design for Reliable Software, D. B. Wortman (Ed.), ACM, New York, pp. 1–10.
7. Andrews, D. M., and Benson, J. P. “An automated program testing methodology, and its implementation”, in Proc. 5th Int. Conf. Software Engineering (San Diego, Calif., March 912), IEEE Computer Society Press, Silver Spring, Maryland, 1981, pp. 254–261.
8. ANSI X3 9-1978, “Fortran”, American National Standards Institute, New York, 1978.
9. Apt, K. R., “Ten years of Hoare's logic: A survey—Part I”, Trans. Program Lang. Syst. 3, 4 (Oct. 1981), 431–483.
10. Baker, F. T. “Chief programmer team management of production programming”, IBM Syst. J 11, 1 (1972), 56–73.
11. Barstow, D. R., and Shrobe, H. E. (Eds). Special Issue on Programming Environments, IEEE Trans. Softw. Eng. SE-7, 5 (Sept. 1981).
12. Barstow, D.R., Shrobe, H., and San-Dewall, E., Eds. Interactive programming environments, McGraw-Hill, New York, 1981
13. Bell, T. E., Bixler, D. C., and Dyer, M. E. “An extendable approach to computer-aided software requirements engineering”, IEEE Trans. Softw Eng. SE-3, 1 (1977), 49–60.
14. Boehm, B. W., “Seven basic principles of software engineering”, in Software engineering techniques, Infotech State of the Art Report, Infotech, London, 1977.
15. Boehm, B. W., Brown, J. R., Kaspar, H., Lipow, M., MacLeod, G. J., and Merrit, M. J. Characteristics of software quality, North-Holland, New York, 1978.
16. Boyer, R. S., Elspas, B., and Levitt, K. N. “SELECT—A formal system for testing and debugging programs by symbolic execution”, in Proc. 1975 Int. Conf. Reliable Software (Los Angeles, April), 1975, pp 234–245.
17. Branstad, M. A., Cherniavsky, J. C., and Adrion, W. R. “Validation, verification, and testing for the individual programmer”, Computer 13, 12 (Dec 1980), 24–30
18. Branstad, M. A., Adrion, W. R., and Cherniavsky, J. C. “A view of software development support systems”, in Proc. Nat Electronics Conf., vol. 35, National Engineering Consortium, Oakbrook, 111., Oct. 1981, pp. 257–262.
19. Branstad, M. A. and Adrion, W. R., Eds. “NBS programming environment workshop”, Softw. Eng Notes 6, 4 (Aug 1981), 1–51.
20. Brown, J.R., et al. “Automated software quality assurance”, in W DeMi79 Hetzel (Ed.), Program test methods, Prentice-Hall, Englewood Cliffs, N.J, 1973, Chap 15.
21. Buckley, F. “A standard for software quality assurance plans”, Computer 12, 8 (Aug 1979), 43–50
22. Budd, T., DeMillo, R. A., Lipton, R. J., and Sayward, F. G. “The design of a prototype mutation system for program testing”, in Proc AFIPS Nat Computer Conf., vol 47, AFIPS Press, Arlington, Va., 1978, pp 623–627.
23. Budd, T.A., and Lipton, R. J. “Mutation analysis of decision table programs”, in Proc. Conf. Information Science and Systems, Johns Hopkins Univ, Baltimore, Md., pp. 346–349, 1978.
24. Caine, S. H., and Gordon, E. K. “PDL Baltimore, A tool for software design”, in Proc National Computer Conf., vol. 44, AFIPS Press, Arlington, Va., 1975, pp. 271–276.
25. Carpenter, L. C., and Tripp, L. L. “Software design validation tool”, EDP81 in Proc 1975 Int Conf Reliable Software (Apr 1975)
26. Chapin, N. “A measure of software complexity”, in Proc. AFIPS National Computer Conf., vol 48, AFIPS Press, FIPS76 Arlington, Va., 1979, pp 995–1002.
27. Cherniavsky, J. C. “On finding test data sets for loop free programs”, Inform. Process. Lett 8, 2 (1979).
28. Cherniavsky, J.C., Adrion, W.R., and Branstad, M.A. “The role of testing tools and techniques m the procurement of quality software and systems”, in Proc. 13th Annu. Alisomar Conf Circuits, Systems, and Computers, IEEE Computer Society, Long Beach, Calif., 1979, pp. 309–313
29. Clarke, A. “A system to generate test data and symbolically execute programs”, IEEE Trans Softw Eng. SE-2, 3 (Sept. 1977), 215–222.
30. Constable, R. L., and O'Donnell, M.J. A programming logic, Winthrop, Cambridge, Mass., 1978.
31. Davis, M. D., and Weyuker, E.J. “Pseudo-oracles for montestable programs”, Tech. Rep., Courant Institute of Mathematical Sciences, New York, 1981.
32. DeMillo, R. A., Lipton, R. J., and Sayward, F.G. “Hints on test data selection: Help for the practicing programmer”, Computer 11, 4 (1978), 34–43.
33. Demillo, R. A., Lipton, R. J., and Perlis, A. J. “Social processes and the proofs of theorems and programs”, Commun. ACM 225 (May 1979), 271–280.
34. Dijkstra, E.W. “Notes on structured programming”, in O. J. Dahl, E. J. Dijkstra, and C. A. R. Hoare (Eds.), Structured programming, Academic Press, London, 1972.
35. Dijkstra, E. W. “On a political pamphlet from the Middle Ages (regarding the POPL paper of R. A. DeMillo, R. J. Lipton, and A. J. Perlis)”, Softw Eng Notes 3, 2 (Apr. 1978), 14–15.
36. Duran, J. W., and Wiorkowski, J. J. “Capture-recapture sampling for estimating software error content”, IEEE Trans Softw. Eng SE-7 (Jan 1981), 147–148.
37. Duran, J. W., and Ntafos, S. “A report on random testing”, in Proc. 5th Int. Conf. Software Engineering, IEEE Computer Society Press, Silver Spring, Md, 1981, pp. 179–183.
38. Fagan, M. E. “Design and code inspections to reduce errors in program development”, IBM Syst. J 15, 3 (1976), 182–211.
39. Floyd, R. W. “Assigning meaning to programs”, in Proc. Symposia Applied Mathematics, vol. 19, American Mathematics Society, Providence, R.I., 1967, pp. 19–32.
40. Fosdick, L. D., and Osterweil, L.J. “Data flow analysis in software reliability”, Comput Surv (ACM) 8, 3 (Sept. 1976), 305–330.
41. General Accounting Office “Federal agencies maintenance of computer programs Expensive and underman-aged”, GAO, Washington, D. C., 1981.
42. Gerhart, S.L. “Program verification in the 1980s: Problems, perspectives, and opportunities”, Rep. ISI/RR-78-71, Information Sciences Institute, Marina del Rey, Calif., Aug. 1978.
43. Gerhart, S. L., Musser, D. R., Thompson, D. H., Baker, D. A., Bates, R.L., Erickson, R. W., London, R. L., Taylor, D. G., and Wile, D.S. “An overview of AFFIRM, A specification and verification system”, in Proc. IFIP Congress 1980, North-Holland, Amsterdam, pp. 343–347.
44. Glass, R. L. “Persistent software errors”, IEEE Trans Softw Eng. SE-7, 2(March 1981), 162–168.
45. Goodenough, J. B., and Gerhart, S. L. “Toward a theory of test data selection”, IEEE Trans Softw. Eng SE-1, 2 (March 1975).
46. Halstead, M. H. Elements of software science, Elsevier North-Holland, New York, 1977
47. Hamilton, N., and Zeldin, S. “Higher order software—A methodology for defining software”, IEEE Trans Softw Eng SE-2, 1 (1976), 932.
48. Hantler, S L, and King, J. C. “An introduction to proving the correctness of programs”, Comput. Surv. (ACM) 8, 3(Sept. 1976), 331–353
49. Hecht, M., and Ullman, J. “Flow-graph reducibility”, SIAM J Appl. Math 1 (1972), 188–202.
50. Howden, W. E. “Reliability of the path analysis testing strategy”, IEEE Trans. Softw. Eng. SE-2, 3 (1976).
51. Howden, W. E. “Symbolic testing and the DISSECT symbolic evaluation system”, IEEE Trans Softw Eng SE-3, 4 (1977), 266–278
52. Howden, W.E. “A survey of dynamic analysis methods”, in E. Miller (Eds), Tutorial. Software testing and validation techniques, IEEE Computer Soc, New York, 1978.
53. Howden, W. E. “Functional program testing”, IEEE Trans. Soft. Eng SE-6, 2 (1980), 162–169
54. Howden, W. E. “Applicability of software validation techniques to scientific programs”, Trans Program Lang. Syst 2, 3 (June 1980), 307–320.
55. Howden, W. E. “Completeness criteria for testing elementary program functions”, in Proc 5th Int Conf on Software Engineering (San Diego, March 9-12), IEEE Computer Society Press, Silver Spring, Md., 1981, pp 235–243.
56. Howden, W.E. “Errors in data processing programs and the refinement of current program test methodologies”, Final Rep, NBS Contract NB79BCA0069, National Bureau of Standards, Washington, D C., July 1981.
57. Jackson, M. A. Principles of program design, Academic Press, New York, 1975.
58. Jones, C. “Program quality and programmer productivity”, IBM Tech Rep., International Business Machines Corp., San Jose, Calif, 1976.
59. Kernighan, B. W. “RATFOR—A preprocessor for a rational FORTRAN”, Bell Labs. Internal Memorandum, Bell Laboratories, Murray Hill, N.J., 1974.
60. King, J. C. “Symbolic execution and program testing”, Commun. ACM 19, 7 (July 1976), 385–394.
61. Koppang, R. G. “Process design system—An integrated set of software development tools”, in Proc. 2nd Int Software Engineering Conf (San Fran-ciisco, Oct. 13-15), IEEE, New York, 1976, pp. 86–90.
62. Lamb, S. S., Leck, V. G., Peters, L. J., and Smith, G. L. “SAMM—A modeling tool for requirements and design specification”, in Proc COMPSAC 78, IEEE Computer Society, New York, 1978, pp 48–53.
63. Lipton, R. J., and Sayward, F. G. “The status of research on program mutation”, in Proc Workshop on Software Testing and Test Documentation, IEEE Computer Society, New York, 1978, pp. 355–367.
64. Luckham, D., German, S., Von Henke, F., Karp, R., Milne, P., Oppen, D., Polak, W., and Schenlis, W. “Stanford Pascal Verifier user's manual”, AI Memo. CS-79-731, Computer Science Dep., Stanford University, Stanford, Calif, 1979.
65. Lyon, G., and Stillman, R.B. “A FORTRAN analyzer”, NBS Tech Note 849, National Bureau of Standards, Washington, D.C., 1974.
66. Maitland, R. “NODAL”, in NBS software tools database, R. Houghton and K, Oakley (Eds.), NBSIR, National Bureau of Standards, Washington, D. C., 1980.
67. Manna, Z. Mathematical theory of computation, McGraw-Hill, New York, 1974
68. McCabe, T. J. “A complexity measure”, IEEE Trans. Softw Eng SE-2, 4 (1976), 308–320
69. McCall, J., Richards, P., and Walters, G. Factors in software quality, vols. 1-3, NTIS Rep File Nos. AD-A049-014, 0X5, 055, 1977.
70. Metzner, J. R., and Barnes, B. Prat77 H Decision table languages and systems, Academic Press, New York, 1977.
71. Mills, H. D. “Top down programming in large systems”, in Debugging techniques in large systems, R. Rustin (Ed), Prentice-Hall, Englewood Cliffs, Rama74 N J., 1970, pp 41–55.
72. Mills, H. D. “On statistical validation of computer programs”, IBM Rep. FSC72-6015, Federal Systems Division, IBM, Gaithersburg, Md, 1972.
73. Miller, E. F., Jr. “RXVP—An automated verification system for FORTRAN”, in Proc. Workshop 4, Computer Science and Statistics 8th Ann. Symp on the Interface (Los Angeles, Calif, Feb), 1975.
74. Miller, E. R., Jr. “Program testing Robi79 Art meets theory”, Computer 10, 7 (1977), 42–51
75. Military Standard. “Technical reviews and audits for systems, equipment, and computer programs”, MIL-STD-1521A (USAF), U.S. Department of the Air Force, Washington, D.C, Roub76 1976.
76. Myers, G. J. Software reliability Principles and practices, Wiley, New York, 1976
77. Myers, G. J. The art of software testing, Wiley, New York, 1979.
78. Neumann, P. G., Robinson, L., Levitt, K., Boyer, R. S., and Saxema, A. R. “A provably secure operating system”, SRI Project 2581, SRI International, Menlo Park, Calif., 1975
79. Osterweil, L. J., and Fosdick, L. D. “DAVEA validation, error detection, and documentation system for FORTRAN programs”, Softw. Pract. SRS79 Exper 6 (1976), 473–486
80. Osterweil, L. J. “A strategy for effective integration of verification and testing techniques”, Tech Rep.CU-CS181-80, Computer Science Dep., Univ. of Colorado, Boulder, 1980.
81. Ostrand, T. J., and Weyucker, E. J. “Current directions in the theory of testing”, in Proc IEEE Computer Software and Applictions Conf. (COMP-SAC80), IEEE Press, Silver Spring, Taus77 Md, 1980, pp. 386–389.
82. Paige, M. R. “On partitioning program graphs”, IEEE Trans. Softw Eng. SE-3, 6 (1977), 386–393
83. Panzl, D. J. “Automatic revision of formal test procedures”, in Proc 3rd Int Conf Software Engineering (Atlanta, May 10-12), ACM, New York, 1978, pp. 320–326
84. Parnas, D. L. “The use of precise wass81a specifications in the development of software”, in Information processing 77, B. Gilchrist (Ed.), North-Holland, Amsterdam, 1977, pp. 861–867.
85. Pratt, V R. “Semantic considerations in Floyd-Hoare logic”, Proc. 17th Annu. IEEE Symp. on the Foundations of Computer Science, IEEE Computer Society Press, Long Beach, Calif, 1976, pp. 109–112.
86. Ramamoorthy, C. V., and Ho, S. F. FORTRAN automated code evaluation system, ERL—M466, Electronics Research Lab., Univ. of California, Berkeley, 1974.
87. Richardson, D. J., and Clarke, L. A. “A partition analysis method to increase program reliability”, in Proc 5th Int Conference SoftwareEngmeering (San Diego, March 9-12), IEEE Computer Society Press, Silver Spring, Md, 1981, pp. 244–253.
88. Robinson, L. The HDM handbook, vol. I-III, SRI Project 4828, SRI International, Menlo Park, Calif., 1979.
89. Ross, D. T., and Schoman, K. E., Jr. “Structured analysis for requirements definition”, IEEE Trans Softw. Eng. SE-3, 1 (1977), 6–15.
90. Roubine, O., and Robinson, L. Special Reference Manual, Stanford Research Institute Tech. Rep CSG-45, Menlo Park, Calif., 1976.
91. Schick, G. J., and Wolverton, R. W. “An analysis of competing software reliability models”, IEEE Trans. Softw Eng. SE-4 (March, 1978), 104–120.
92. Sneed, H., and Kirchoff, K. “PrufstandA testbed for systematic software components”, in Proc INFOTECH State of the Art Conf. Software Testing, Infotech, London, 1978.
93. Stucki, L. G. “New directions in automated tools for improving software quality”, in R. Yeh (Ed), Current trends in programming methodology, vol II Program validation, Prentice-Hall, Englewood Cliffs, N J., 1977, pp. 80–111.
94. Tausworthe, R. C. Standardized development of computer software, Prentice-Hall, Englewood Cliffs, NJ, 1977.
95. Teichroew, D., and Hershey, E. A., “PSL/PSA: A computer-aided technique for structured documentation and analysis of information processing systems”, IEEE Trans. Softw. Eng SE-3, 1 (Jan. 1977), 41–48.
96. Weinberg, G. M. The psychology of computer programming, Van Nos-trand-Reinhold, Princeton, N J., 1971 weyucker, E. J., and ostrand, T. J. “Theories of program testing and the application of revealing subdo-mains”, IEEE Trans. Softw. Eng. SE-6 (May, 1980), 236–246.
97. Weyucker, E. J., and Ostrand, T. J. “Theories of program testing and the application of revealing subdo-mains”, IEEE Trans. Softw. Eng. SE-6 (May, 1980), 236–246.
98. White, L. J., and Cohen, E. I. “A domain strategy for computer program testing”, Digest for the Workshop on Software Testing and Test Documentation (Ft. Lauderdale, Fla), pp 335–354.
99. Yourdon, E., and Constantine, L. L. Structured design, Prentice-Hall, Englewood Cliffs, N.J., 1979.
100. Zelkowitz, M. V. “Perspectives on software engineering”, Comput. Surv. (ACM) 10, 2 (June 1978), 197–216