The oracle problem in software testing: A survey

IEEE Transactions on Software Engineering

Volumn 41, Issue 5, 2015, Pages 507-525

  Barr, Earl T ^a   Harman, Mark ^a   McMinn, Phil ^b   Shahbaz, Muzammil ^b   Yoo, Shin ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Automatic testing; Test oracle; Testing formalism

Indexed keywords

AUTOMATIC TESTING; AUTOMATION; SPECIFICATIONS; SURVEYS;

DOMAIN-SPECIFIC INFORMATION; METAMORPHIC TESTING; ORACLE PROBLEM; POTENTIAL FAULTS; REDUCING COSTS; TEST AUTOMATION; TEST ORACLES;

SOFTWARE TESTING;

EID: 84929331583     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2014.2372785     Document Type: Article

References (211)

1. S. Afshan, P. McMinn, and M. Stevenson, "Evolving readable string test inputs using a natural language model to reduce human oracle cost," in Proc. Int. Conf. Softw. Testing, Verification Validation, Mar. 2013, pp. 352-361.
2. M. Afshari, E. T. Barr, and Z. Su, "Liberating the programmer with prorogued programming," in Proc ACM Int. Symp. New Ideas, New Paradigms, Reflections Programm. Softw., 2012, pp. 11-26.
3. W. Afzal, R. Torkar, and R. Feldt, "A systematic review of search-based testing for non-functional system properties," Inf. Softw. Technol., vol. 51, no. 6, pp. 957-976, 2009.
4. B. K. Aichernig, "Automated black-box testing with abstract VDM oracles," in Proc. 18th Int. Conf. Comput. Comput. Safety, Rel. Security, 1999, pp. 250-259.
5. S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege, "A systematic review of the application and empirical investigation of search-based test-case generation," IEEE Trans. Softw. Eng., vol. 36, no. 6, pp. 742-762, Nov. 2010.
6. N. Alshahwan and M. Harman, "Automated session data repair for web application regression testing," in Proc. Int. Conf. Softw. Testing, Verification, Validation, 2008, pp. 298-307.
7. D. Angluin, "Learning regular sets from queries and counterexamples," Inf. Comput., vol. 75, no. 2, pp. 87-106, 1987.
8. W. Araujo, L. C. Briand, and Y. Labiche, "Enabling the runtime assertion checking of concurrent contracts for the Java modeling language," in Proc. 33rd Int. Conf. Softw. Eng., 2011, pp. 786-795.
9. W. Araujo, L. C. Briand, and Y. Labiche, "On the effectiveness of contracts as test oracles in the detection and diagnosis of race conditions and deadlocks in concurrent object-oriented software," in Proc. Int. Symp. Empirical Softw. Eng. Meas., 2011, pp. 10-19.
10. S. Artzi, M. D. Ernst, A. Kiez-un, C. Pacheco, and J. H. Perkins, "Finding the needles in the haystack: Generating legal test inputs for object-oriented programs," in Proc. 1st Workshop Model-Based Testing Object-Oriented Syst., Oct. 23, 2006.
11. E. Astesiano, M. Bidoit, H. Kirchner, B. Krieg-Bruckner, P. D. Mosses, D. Sannella, and A. Tarlecki, "CASL: The common algebraic specification language," Theor. Comput. Sci., vol. 286, no. 2, pp. 153-196, 2002.
12. T. M. Austin, S. E. Breach, and G. S. Sohi, "Efficient detection of all pointer and array access errors," in Proc. ACM SIGPLAN1994 Conf. Programm. Lang. Des. Implementation, 1994, pp. 290-301.
13. A. Avizienis, "The N-version approach to fault-tolerant software," IEEE Trans. Softw. Eng., vol. 11, no. 12, pp. 1491-1501, Dec. 1985.
14. A. Avizienis and L. Chen, "On the implementation of N-version programming for software fault-tolerance during execution," in Proc. 1st Int. Comput. Softw. Appl. Conf., 1977, pp. 149-155.
15. F. Baader, and T. Nipkow, Term Rewriting and All That. New York, NY, USA: Cambridge Univ. Press, 1998.
16. A. F. Babich, "Proving total correctness of parallel programs," IEEE Trans. Softw. Eng., vol. 5, no. 6, pp. 558-574, Nov. 1979.
17. L. Baresi and M. Young. (2001, Aug.). Test oracles. Univ. of Oregon, Dept. Comput. Inform. Sci., Eugene, OR, USA. Tech. Rep. CIS-TR-01-02. [Online]. Available: http://www.cs.uoregon.edu/~michal/pubs/oracles.html
18. S. Bekrar, C. Bekrar, R. Groz, and L. Mounier, "Finding software vulnerabilities by smart fuzzing," in IEEE 4th Int. Conf. Softw. Testing, Verification Validation, 2011, pp. 427-430.
19. G. Bernot, "Testing against formal specifications: A theoretical view," in Proc. Int. Joint Conf. Theory Prac. Softw. Develop. Adv. Distrib. Comput. Colloquium Combining Paradigms Softw. Develop., 1991, pp. 99-119.
20. G Bernot, M. C. Gaudel, and B. Marre, "Software testing based on formal specifications: A theory and a tool," Softw. Eng. J., vol. 6, no. 6, pp. 387-405, Nov. 1991.
21. A. Bertolino, P. Inverardi, P. Pelliccione, and M. Tivoli, "Automatic synthesis of behavior protocols for composable web-services," in Proc. 7th Joint Meeting Eur. Softw. Eng. Conf. ACM SIGSOFT Symp. Found. Softw. Eng., 2009, pp. 141-150.
22. D. Beyer, T. Henzinger, R. Jhala, and R. Majumdar, "Checking memory safety with Blast," in Proc. 8th Int. Conf. Held as Part Joint Eur. Conf. Theory Pract. Softw. Conf. Fundam. Approaches Softw. Eng., 2005, pp. 2-18.
23. R. Binder, Testing Object-Oriented Systems: Models, Patterns, and Tools. Reading, MA, USA: Addison-Wesley, 2000.
24. A. Blikle, "Proving programs by sets of computations," in Proc. 3rd Symp. Math. Found. Comput. Sci., 1975, pp. 333-358.
25. C. Blum and A. Roli, "Metaheuristics in combinatorial optimization: Overview and conceptual comparison," ACM Comput. Surveys, vol. 35, no. 3, pp. 268-308, 2003.
26. G V. Bochmann and A. Petrenko, "Protocol testing: Review of methods and relevance for software testing," in Proc. ACM SIGSOFT Int. Symp. Softw. Testing Anal, 1994, pp. 109-124.
27. G V. Bochmann, "Finite state description of communication protocols," Comput. Netw., vol. 2, no. 4, pp. 361-372, 1978.
28. E. Borger, A. Cavarra, and E. Riccobene, "Modeling the dynamics of UML state machines," in Abstract State Machines-Theory and Applications. New York, NY, USA: Springer, 2000, pp. 167-186.
29. E. Borger, "High level system design and analysis using abstract state machines," in Proc. Int. Workshop Current Trends Appl. Formal Method: Appl. Formal Methods, 1999, pp. 1-43.
30. K. Bottger, R. Schwitter, D. Molla, and D. Richards, "Towards reconciling use cases via controlled language and graphical models," in Proc. Appl. Prolog 14th Int. Conf. Web Knowl. Manag. Decis. Support, 2003, pp. 115-128.
31. F. Bouquet, C. Grandpierre, B. Legeard, F. Peureux, N. Vacelet, and M. Utting, "A subset of precise UML for model-based testing," in Proc. 3rd Int. Workshop Adv. Model-Based Testing, 2007, pp. 95-104.
32. M. Bozkurt and M. Harman, "Automatically generating realistic test input from web services," in Proc. IEEE 6th Int. Symp. Serv. Oriented Syst. Eng., 2011, pp. 13-24.
33. L. C. Briand, Y. Labiche, and H. Sun, "Investigating the use of analysis contracts to improve the testability of object-oriented code," Softw. Pract. Exp., vol. 33, no. 7, pp. 637-672, Jun. 2003.
34. C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Eng-ler, "EXE: Automatically generating inputs of death," ACM Trans. Inf. Syst. Secur., vol. 12, no. 2, pp. 10:1-10:38, Dec. 2008.
35. J. Callahan, F. Schneider, and S. Easterbrook, "Automated software testing using model-checking," in Proc. SPIN Workshop, volume 353, 1996.
36. F. T. Chan, T. Y. Chen, S. C. Cheung, M. F. Lau, and S. M. Yiu, "Application of metamorphic testing in numerical analysis," in Proc. IASTED Int. Conf. Softw. Eng., 1998, pp. 191-197.
37. W. K. Chan, S. C. Cheung, and Karl R. P. H. Leung, "A metamor-phic testing approach for online testing of service-oriented software applications," in Software Applications: Concepts, Methodologies, Tools, and Applications. Hershey, PA, USA: IGI Global, 2009, ch. 7, pp. 2894-2914.
38. W. K. Chan, S. C. Cheung, and K. R. P. H. Leung, "Towards a metamorphic testing methodology for service-oriented software applications," in Proc. 5th Int. Conf. Quality Softw., Sep. 2005, pp. 470-476.
39. F. Chen, N. Tillmann, and W. Schulte, "Discovering specifications," Microsoft Research, Tech. Rep. MSR-TR-2005-146, Oct. 2005.
40. H. Y. Chen, T. H. Tse, F. T. Chan, and T. Y. Chen, "In black and white: An integrated approach to class-level testing of object-oriented programs," ACM Trans. Softw. Eng. Methodol., vol. 7, pp. 250-295, Jul. 1998.
41. H. Y. Chen, T. H. Tse, and T. Y. Chen, "TACCLE: A methodology for object-oriented software testing at the class and cluster levels," ACM Trans. Softw. Eng. Methodol., vol. 10, no. 1, pp. 56-109, Jan. 2001.
42. T. Y. Chen, F.-C. Kuo, T. H. Tse, and Z. Q. Zhou, "Metamorphic testing and beyond," in Proc. Int. Workshop Softw. Technol. Eng. Prac., Sep. 2004, pp. 94-100.
43. T. Chen, D. Huang, H. Huang, T.-H. Tse, Z. Yang, and Z. Zhou, "Metamorphic testing and its applications," in Proc. 8th Int. Symp. Future Softw. Technol., 2004, pp. 310-319.
44. Y. Cheon, "Abstraction in assertion-based test oracles," in Proc. 7th Int. Conf. Quality Softw., 2007, pp. 410-414.
45. Y. Cheon and G. T. Leavens, "A simple and practical approach to unit testing: The JML and JUnit way," in Proc. 16th Eur. Conf. Object-Oriented Programm., 2002, pp. 231-255.
46. L. A. Clarke, "A system to generate test data and symbolically execute programs," IEEE Trans. Softw. Eng., vol. 2, no. 3, pp. 215-222, Sep. 1976.
47. P. C. Clements, "Managing variability for software product lines: Working with variability mechanisms," in Proc. 10th Int. Conf. Softw Product Lines, 2006, pp. 207-208.
48. M. Clermont and D. Parnas, "Using information about functions in selecting test cases," in Proc. 1st Int. Workshop Adv. Model-Based Testing, 2005, pp. 1-7.
49. D. Coppit and J. M. Haddox-Schatz, "On the use of specification-based assertions as test oracles," in Proc. 29th Annu. IEEE/NASA Softw. Eng. Workshop, 2005, pp. 305-314.
50. M. Davies and E. Weyuker, "Pseudo-oracles for non-testable programs," in Proc. ACM '81 Conf., 1981, pp. 254-257.
51. L. K. Dillon, "Automated support for testing and debugging of real-time programs using oracles," SIGSOFT Softw. Eng. Notes, vol. 25, no. 1, pp. 45-46, Jan. 2000.
52. R.-K. Doong and P. G. Frankl, "The ASTOOT approach to testing object-oriented programs," ACM Trans. Softw. Eng. Methodol., vol. 3, pp. 101-130, Apr. 1994.
53. E. Elkind, B. Genest, D. Peled, and H. Qu, "Grey-box checking," in Proc. 26th IFIP WG 6.1 Int. Conf. Formal Techn. Netw. Distrib. Syst., 2006, pp. 420-435.
54. J. Ellsberger, D. Hogrefe, and A. Sarma, SDL: Formal Object-Oriented Language for Communicating Systems. Englewood Cliffs, NJ, USA: Prentice-Hall, 1997.
55. M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao, "The Daikon system for dynamic detection of likely invariants," Sci. Comput. Programm., vol. 69, no. 1, pp. 35-45, 2007.
56. M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin, "Dynamically discovering likely program invariants to support program evolution," IEEE Trans. Softw. Eng., vol. 27, no. 2, pp. 99-123, Feb. 2001.
57. R. A. Eyre-Todd, "The detection of dangling references in C++ programs," ACM Lett. Programm. Lang. Syst., vol. 2, no. 1-4, pp. 127-134, 1993.
58. R. Feldt, "Generating diverse software versions with genetic programming: An experimental study," in Softw., IEE Proc., vol. 145, pp. 228-236, Dec. 1998.
59. X. Feng, D. L. Parnas, T. H. Tse, and T. O'Callaghan, "A comparison of tabular expression-based testing strategies," IEEE Trans. Softw. Eng., vol. 37, no. 5, pp. 616-634, Sep./Oct. 2011.
60. X. Feng, D. L. Parnas, and T. H. Tse, "Tabular expression-based testing strategies: A comparison," in Proc. Testing: Acad. Indus. Conf. Prac. Res. Techn.-MUTATION, 2007, p. 134.
61. J. Ferrer, F. Chicano, and E. Alba, "Evolutionary algorithms for the multi-objective test data generation problem," Softw.: Prac. Exp., vol. 42, no. 11, pp. 1331-1362, 2011.
62. J. S. Fitzgerald and P. G. Larsen, "Modelling Systems-Practical Tools and Techniques in Software Development, 2nd ed. Cambridge, U.K.: Cambridge Univ. Press, 2009.
63. C. Flanagan and K. R. M. Leino, "Houdini, an annotation assistant for ESC/Java," Lecture Notes in Comput. Sci., vol. 2021, pp. 500-517, 2001.
64. R. W. Floyd, "Assigning meanings to programs," in Mathematical Aspects of Computer Science, volume 19 of Symposia in Applied Mathematics, J. T. Schwartz Ed., American Providence, RI, USA: Mathematical Society, 1967, pp. 19-32.
65. G. Fraser and A. Arcuri, "Whole test suite generation," IEEE Trans. Softw. Eng., vol. 39, no. 2, pp. 276-291, Feb. 2013.
66. G. Fraser and A. Zeller, "Exploiting common object usage in test case generation," in Proc. 4th IEEE Int. Conf. Softw. Testing, Verification Validation, 2011, pp. 80-89.
67. K. Frounchi, L. C. Briand, L. Grady, Y. Labiche, and R. Subra-manyan, "Automating image segmentation verification and validation by learning test oracles," Inf. Softw. Technol., vol. 53, no. 12, pp. 1337-1348, 2011.
68. P. Gall and A. Arnould, "Formal specifications and test: Correctness and oracle," in Recent Trends in Data Type Specification, M. Haveraaen, O. Owe, and O.-J. Dahl, Ed. Berlin, Germany, Springer, 1996, pp. 342-358.
69. J. Gannon, P. McMullin, and R. Hamlet, "Data abstraction, implementation, specification, and testing," ACM Trans. Programm. Lang. Syst, vol. 3, no. 3, pp. 211-223, 1981.
70. A. Gargantini and E. Riccobene, "ASM-based testing: Coverage criteria and automatic test sequence," J. Universal Comput. Sci., vol. 7, no. 11, pp. 1050-1067, Nov. 2001.
71. S. J. Garland, J. V. Guttag, and J. J. Horning, "An overview of Larch," in Functional Programming, Concurrency, Simulation and Automated Reasoning, Springer Berlin Heidelberg, 1993, pp. 329-348.
72. M.-C. Gaudel, "Testing from formal specifications, a generic approach," in Proc. 6th Ade-Eur. Int. Conf. Leuven Rel. Softw. Tech-nol., 2001, pp. 35-48.
73. M.-C. Gaudel and P. R. James, "Testing algebraic data types and processes: A unifying theory," Formal Asp. Comput., vol. 10, no. 5-6, pp. 436-451, 1998.
74. G. Gay, S. Rayadurgam, and M. Heimdah, "Improving the accuracy of oracle verdicts through automated model steering," in Proc. 29th ACM/IEEE Int. Conf. Automated Softw. Eng., 2014, pp. 527-538.
75. P. Godefroid, N. Klarlund, and K. Sen, "DART: Directed automated random testing," in Proc. ACM SIGPLAN Conf. Programm. Lang. Des. Implementation, 2005, pp. 213-223.
76. A. Groce, M. A. Alipour, C. Zhang, Y. Chen, and J. Regehr, "Cause reduction for quick testing," in Proc. IEEE Int. Conf. Softw. Testing, Verification, Validation, 2014, pp. 243-252.
77. A. Groce, D. Peled, and M. Yannakakis, "AMC: An adaptive model checker," in Proc. Int. Conf. Comput. Aided Verification, 2002, pp. 521-525.
78. R. Groz, K. Li, A. Petrenko, and M. Shahbaz, "Modular system verification by inference, testing and reachability analysis," in Proc. TestCom/FATES, 2008, pp. 216-233.
79. Z. Gu, E. T. Barr, D. J. Hamilton, and Z. Su, "Has the bug really been fixed?" in Proc. Int. Conf. Softw. Eng., 2010, pp. 55-64.
80. R. Guderlei and J. Mayer, "Statistical metamorphic testing testing programs with random output by means of statistical hypothesis tests and metamorphic testing," in 7th Int. Conf. Quality Softw., Oct. 2007, pp. 404-409.
81. D. Harel, "Statecharts: A visual formalism for complex systems," Sci. Comput. Programm., vol. 8, no. 3, pp. 231-274, 1987.
82. M. Harman, S. G. Kim, K. Lakhotia, P. McMinn, and S. Yoo, "Optimizing for the number of tests generated in search based test data generation with an application to the oracle cost problem," in Proc. Int. Workshop Search-Based Softw. Testing, Apr. 2010, pp. 182-191.
83. M. Harman, A. Mansouri, and Y. Zhang, "Search based software engineering: A comprehensive analysis and review of trends techniques and applications," Dept. of Comput. Sci., King's College London, London, U.K., Tech. Rep. TR-09-03, Apr. 2009.
84. M. Harman, P. McMinn, J. T. de Souza, and S. Yoo, "Search based software engineering: Techniques, taxonomy, tutorial," in Empirical Software Engeneering and Verification, B. Meyer and M. Nordio, Eds. New York, NY, USA: Springer, 2012 pp. 1-59.
85. M. J. Harrold, R. Gupta, and M. L. Soffa, "A methodology for controlling the size of a test suite," ACM Trans. Softw. Eng. Meth-odol., vol. 2, no. 3, pp. 270-285, Jul. 1993.
86. M. J. Harrold, G. Rothermel, K. Sayre, R. Wu, and L. Yi, "An empirical investigation of the relationship between spectra differences and regression faults," Softw. Testing, Verification Reliability, vol. 10, no. 3, pp. 171-194, 2000.
87. D. L. Heine and M. S. Lam, "A practical flow-sensitive and context-sensitive C and C++ memory leak detector," in Proc. ACM SIGPLAN Conf. Programm. Lang. Des. Implementation, 2003, pp. 168-181.
88. J. Henkel and A. Diwan, "Discovering algebraic specifications from Java classes," Lecture Notes in Comput. Sci., vol. 2743, pp. 431-456, 2003.
89. F. C. Hennie, Finite-State Models for Logical Machines. Hoboken, NJ, USA: Wiley, 1968.
90. M. H. Heule and S. Verwer, "Software model synthesis using satisfiability solvers," Empirical Softw. Eng., vol. 18, pp. 825-856, Aug. 2013.
91. R. M. Hierons, "Oracles for distributed testing," IEEE Trans. Softw. Eng., vol. 38, no. 3, pp. 629-641, May/Jun. 2012.
92. R. M. Hierons, "Verdict functions in testing with a fault domain or test hypotheses," ACM Trans. Softw. Eng. Methodol., vol. 18, no. 4, pp. 14:1-14:19, Jul. 2009.
93. R. M. Hierons, K. Bogdanov, J. P. Bowen, R. Cleaveland, J. Derrick, J. Dick, M. Gheorghe, M. Harman, K. Kapoor, P. Krause, G. Lüttgen, A. J. H. Simons, S. Vilkomir, M. R. Woodward, and H. Zedan, "Using formal specifications to support testing," ACM Comput. Surv., vol. 41, pp. 9:1-9:76, Feb. 2009.
94. C. A. R. Hoare, "An axiomatic basis of computer programming," Commun. ACM, vol. 12, pp. 576-580, 1969.
95. M. Holcombe, "X-machines as a basis for dynamic system specification," Softw. Eng. J., vol. 3, no. 2, pp. 69-76, 1988.
96. M. Holcombe and F. Ipate, "Correct systems: Building a business process solution," Softw. Testing VerificationRel., vol. 9, no. 1, pp. 76-77, 1999.
97. G. J. Holzmann, "The model checker SPIN," IEEE Trans. Softw. Eng., vol. 23, no. 5, pp. 279-295, May 1997.
98. W. E. Howden, "A functional approach to program testing and analysis," IEEE Trans. Softw. Eng., vol. 12, no. 10, pp. 997-1005, Oct. 1986.
99. W. E. Howden, "Theoretical and empirical studies of program testing," IEEE Trans. Softw. Eng., vol. 4, no. 4, pp. 293-298, Jul. 1978.
100. M. Hughes and D. Stotts, "Daistish: Systematic algebraic testing for OO programs in the presence of side-effects," SIGSOFT Softw. Eng. Notes, vol. 21, no. 3, pp. 53-61, May 1996.
101. D. Jackson, "Software Abstractions: Logic, Language, and Analysis. Cambridge, MA, USA: MIT Press, 2006.
102. C. Jard and G. V. Bochmann, "An approach to testing specifications," J. Syst. Softw., vol. 3, no. 4, pp. 315-323, 1983.
103. D. Jeffrey and R. Gupta, "Test case prioritization using relevant slices," in Proc. 30th Annu. Int. Comput. Softw. Appl. Conf., 2006, pp. 411-420, 2006.
104. Y. Jin and D. L. Parnas, "Defining the meaning of tabular mathematical expressions," Sci. Comput. Program., vol. 75, no. 11, pp. 980-1000, Nov. 2010.
105. M. J. Kearns and U. V. Vazirani, An Introduction to Computational Learning Theory. Cambridge, MA, USA: MIT Press, 1994.
106. R. M. Keller, "Formal verification of parallel programs," Commun. ACM, vol. 19, no. 7, pp. 371-384, 1976.
107. J. C. King, "Symbolic execution and program testing," Commun. ACM, vol. 19, no. 7, pp. 385-394, Jul. 1976.
108. K. Lakhotia, P. McMinn, and M. Harman, "An empirical investigation into branch coverage for C programs using CUTE and AUSTIN," J. Syst. Softw., vol. 83, no. 12, pp. 2379-2391, 2010.
109. A. van Lamsweerde, "Formal specification: A roadmap," in Proc. Conf. Future Softw. Eng., 2000, pp. 147-159.
110. K. Lano and H. Haughton, "Specification in B: An Introduction Using the B Toolkit. London, U.K.: Imperial College Press, 1996.
111. D. Lee and M. Yannakakis, "Principles and methods of testing finite state machines-a survey," Proc. IEEE, vol. 84, no. 8, pp. 1090-1123, Aug. 1996.
112. A. Leitner, M. Oriol, A Zeller, I. Ciupa, and B. Meyer, "Efficient unit test case minimization," in Proc. Autom. Softw. Eng., 2007, pp. 417-420.
113. K. Li, R. Groz, and M. Shahbaz, "Integration testing of components guided by incremental state machine learning," in Proc. Testing: Acad. Ind. Conf. Prac. Res. Tech., 2006, pp. 59-70.
114. D. Lorenzoli, L. Mariani, and M. Pezze, "Automatic generation of software behavioral models," in Proc. 30th Int. Conf. Softw. Eng., 2008, pp. 501-510.
115. P. Loyola, M. Staats, I.-Y. Ko, and G. Rothermel, "Dodona: Automated oracle data set selection," in Proc. Int. Symp. Softw. Testing Anal., 2014, pp. 193-203.
116. D. Luckham and F. W. Henke, "An overview of ANNA-a specification language for ADA," IEEE Soft., vol. 2, no. 2, pp. 9-22, 1984.
117. N. A. Lynch and M. R. Tuttle, "An introduction to input/output automata," CWI Quarterly, vol. 2, pp. 219-246, 1989.
118. P. D. L. Machado, "On oracles for interpreting test results against algebraic specifications," in Proc. Int. Conf. Algebraic Methodol. Softw. Technol., 1999, pp. 502-518.
119. P. Maker.(1998). GNU Nana: Improved support for assertion checking and logging in GNU C/C++ [Online]. Available: http://gnu.cs.pu.edu.tw/software/nana/
120. H. Malik, H. Hemmati, and A. E. Hassan, "Automatic detection of performance deviations in the load testing of large scale systems," in Proc. Int. Conf. Softw. Eng.-Softw. Eng. Prac. Track, 2013, pp. 1012-1021.
121. L. Mariani, F. Pastore, and M. Pezze, "Dynamic analysis for diagnosing integration faults," IEEE Trans. Softw. Eng., vol. 37, no. 4, pp. 486-508, Jul./Aug. 2011.
122. B. Marre, "Loft: A tool for assisting selection of test data sets from algebraic specifications," in Proc. 6th Int. Joint Conf. CAAP/FASE Theory Pract. Softw. Develop., 1995, pp. 799-800.
123. A. P. Mathur, "Performance, effectiveness, and reliability issues in software testing," in Proc. 15th Annu. Int. Comput. Softw. Appl. Conf., 1991, pp. 604-605.
124. J. Mayer and R. Guderlei, "Test oracles using statistical methods," in Proc. 1st Int. Workshop Softw. Quality, Lecture Notes Inform. P-58, Kllen Druck+Verlag GmbH, 2004, pp. 179-189.
125. P. McMinn, M. Stevenson, and M. Harman, "Reducing qualitative human oracle costs associated with automatically generated test data," in Proc. 1st Int. Workshop Softw. Test Output Validation, 2010, pp. 1-4.
126. P. McMinn, "Search-based software test data generation: A survey," Softw. Test. Verif. Rel., vol. 14, no. 2, pp. 105-156, Jun. 2004.
127. P. McMinn, "Search-based failure discovery using testability transformations to generate pseudo-oracles," in Proc. Genetic Evol. Comput. Conf., 2009, pp. 1689-1696.
128. P. McMinn, "Search-based software testing: Past, present and future," in Proc. Int. Workshop Search-Based Softw. Testing, 2011, pp. 153-163.
129. P. McMinn, M. Shahbaz, and M. Stevenson, "Search-based test input generation for string data types using the results of web queries," in Proc. IEEE 5th Int. Conf. Softw. Testing, Verification Validation, 2012, pp. 141-150.
130. P. McMinn, M. Stevenson, and M. Harman, "Reducing qualitative human oracle costs associated with automatically generated test data," in Proc. Int. Workshop Softw. Test Output Validation, Jul. 2010, pp. 1-4.
131. A. M. Memon, "Automatically repairing event sequence-based GUI test suites for regression testing," ACM Trans. Softw. Eng. Methodol., vol. 18, no. 2, pp. 1-36, 2008.
132. A. M. Memon, M. E. Pollack, and M. L. Soffa, "Automated test oracles for GUIs," in Proc. 8th ACM SIGSOFT Int. Symp. Found. Softw. Eng., 2000, pp. 30-39.
133. A. M. Memon and Q. Xie, "Using transient/persistent errors to develop automated test oracles for event-driven software," in Proc. 19th IEEE Int. Conf. Automated Softw. Eng., 2004, pp. 186-195.
134. M. Merten, F. Howar, B. Steffen, P. Pellicione, and M. Tivoli, "Automated inference of models for black box systems based on interface descriptions," in Proc. 5th Int. Conf. Leveraging Appl. Formal Methods, Verification Validation: Technol. Mastering Change-Volume Part I, 2012, pp. 79-96.
135. B. Meyer, "Eiffel: A language and environment for software engineering," J. Syst. Softw., vol. 8, no. 3, pp. 199-246, Jun. 1988.
136. B. P. Miller, L. Fredriksen, and B. So, "An empirical study of the reliability of UNIX utilities," Commun. ACM, vol. 33, no. 12, pp. 32-44, 1990.
137. S. Mouchawrab, L. C. Briand, Y. Labiche, and M. Di Penta, "Assessing, comparing, and combining state machine-based testing and structural testing: A series of experiments," IEEE Trans. Softw. Eng., vol. 37, no. 2, pp. 161-187, Mar./Apr. 2011.
138. C. Murphy, K. Shen, and G. Kaiser, "Automatic system testing of programs without test oracles," in Proc. 18th Int. Symp. Softw. Testing Anal., 2009, pp. 189-200.
139. C. Murphy, K. Shen, and G. Kaiser, "Using JML runtime assertion checking to automate metamorphic testing in applications without test oracles," in Proc. Int. Conf. Softw. Testing Verification and Validation, 2009, pp. 436-445.
140. A. J. Offutt, J. Pan, and J. M. Voas, "Procedures for reducing the size of coverage-based test sets," in Int. Conf. Testing Comput. Softw., 1995, pp. 111-123.
141. C. Pacheco and M. Ernst, "Eclat: Automatic generation and classification of test inputs," in Proc. 19th Eur. Conf. Object-Oriented Programming, pp. 734-734, 2005.
142. D. L. Parnas, J. Madey, and M. Iglewski, "Precise documentation of well-structured programs," IEEE Trans. Softw. Eng., vol. 20, no. 12, pp. 948-976, Dec. 1994.
143. D. L. Parnas, "Document based rational software development," J. Knowl. Based Syst., vol. 22, pp. 132-141, Apr. 2009.
144. D. L. Parnas, "Precise documentation: The key to better software," in The Future of Software Engineering, S. Nanz, Ed. Berlin, Germany, Springer, 2011, pp. 125-148.
145. D. L. Parnas and J. Madey, "Functional documents for computer systems," Sci. Comput. Program., vol. 25, no. 1, pp. 41-61, Oct. 1995.
146. F. Pastore, L. Mariani, and G Fraser, "Crowdoracles: Can the crowd solve the oracle problem?" in Proc. Int. Conf. Softw. Testing, Verification Validation, 2013, pp. 342-351.
147. D. Peled, M. Y. Vardi, and M. Yannakakis, "Black box checking," J. Automata, Lang. Combinatorics, vol 7, no. 2, pp. 225-246, 2002.
148. D. K. Peters and D. L. Parnas, "Using test oracles generated from program documentation," IEEE Trans. Softw. Eng., vol. 24, no. 3, pp. 161-173, Mar. 1998.
149. D. K. Peters and D. L. Parnas, "Requirements-based monitors for real-time systems," IEEE Trans. Softw. Eng., vol. 28, no. 2, pp. 146-158, Feb. 2002.
150. M. Pezze, and C. Zhang, "Automated test oracles: A survey," in A. Hurson and A. Memon, Eds., Advances in Computers, vol. 95, pp. 1-48. Elsevier Ltd., 2014.
151. N. Polikarpova, I. Ciupa, and B. Meyer, "A comparative study of programmer-written and automatically inferred contracts," in Proc. 18th Int. Symp. Softw. Testing Anal., 2009, pp. 93-104.
152. S. J. Prowell and J. H. Poore, "Foundations of sequence-based software specification," IEEE Trans, Softw. Eng., vol. 29, no. 5, pp. 417-429, May 2003.
153. S. Ratcliff, D. R. White, and J. A. Clark, "Searching for invariants using genetic programming and mutation testing," in Proc. 13th Annu. Conf. Genetic Evol. Comput., 2011, pp. 1907-1914.
154. F. Ricca and P. Tonella, "Detecting anomaly and failure in web applications," IEEE Multimedia, vol. 13, no. 2, pp. 44-51, Apr.-Jun. 2006.
155. D. S. Rosenblum, "A practical approach to programming with assertions," IEEE Trans., Softw. Eng., vol. 21, no. 1, pp. 19-31, Jan. 1995.
156. G Rothermel, M. J. Harrold, J. von Ronne, and C. Hong, "Empirical studies of test-suite reduction," Softw. Testing, Verification Rel., vol. 12, pp. 219-249, 2002.
157. N. Walkinshaw S. Ali, K. Bogdanov. (2007, Oct.). A comparative study of methods for dynamic reverse-engineering of state models. The Univ. of Sheffield, Dept. of Comput. Sci., Sheffield, U.K., Tech. Rep. CS-07-16, Oct.[Online]. Available: http://www.dcs. shef.ac.uk/intranet/research/resmes/CS0716.pdf
158. D. Schuler and A. Zeller, "Assessing oracle quality with checked coverage," in Proc. 4th IEEE Int. Conf. Softw. Testing, Verification Validation, 2011, pp. 90-99.
159. R. Schwitter, "English as a formal specification language," in Proc. 13th Int. Workshop Database Expert Syst. Appl., Sep. 2002, pp. 228-232.
160. S. Segura, R. M. Hierons, D. Benavides, and A. Ruiz-Cortes, "Automated metamorphic testing on the analyses of feature models," Inf. Softw. Technol., vol. 53, no. 3, pp. 245-258, 2011.
161. K. Sen, D. Marinov, and G. Agha, "CUTE: A concolic unit testing engine for C," in Proc. 10th Eur. Softw. Eng. Conf. Held Jointly 13th ACM SIGSOFT Int. Symp. Found. Softw. Eng., 2005, pp. 263-272.
162. S. Shahamiri, W. Wan-Kadir, S. Ibrahim, and S. Hashim, "Artificial neural networks as multi-networks automated test oracle," Autom. Softw. Eng., vol. 19, no. 3, pp. 303-334, 2012.
163. S. R. Shahamiri, W. M. N. Wan-Kadir, S. Ibrahim, and S. Z. M. Hashim, "An automated framework for software test oracle," Inf. Softw. Technol., vol. 53, no. 7, pp. 774-788, 2011.
164. S. R. Shahamiri, W. M. N. Wan-Kadir, and S. Z. M. Hashim, "A comparative study on automated software test oracle methods," in Proc. 4th Int. Conf. Softw. Eng. Adv., Porto, 2009, pp. 140-145.
165. M. Shahbaz, "Reverse Engineering and Testing of Black-Box Software Components. LAP Lambert Academic Publishing, 2012.
166. M. Shahbaz, P. McMinn, and M. Stevenson, "Automatic generation of valid and invalid test data for string validation routines," Sci. Comput. Programm., vol. 97, no. 4, pp. 405-425, 2015.
167. K. Shrestha and M. J. Rutherford, "An empirical evaluation of assertions as oracles," in Proc. 4th IEEE Int. Conf. Softw. Testing, Verification Validation, 2011, pp. 110-119.
168. G. Shu, Y. Hsu, and D. Lee, "Detecting communication protocol security flaws by formal fuzz testing and machine learning," in Proc. 28th IFIP WG 6.1 Int. Conf. Formal Techn. Networked Distrib. Syst., 2008, pp. 299-304.
169. J. Sifakis, "Deadlocks and livelocks in transition systems," in Proc. 9th Symp. Math. Found. Comput. Sci., 1980, pp. 587-600.
170. A. J. H. Simons, "JWalk: A tool for lazy systematic testing of Java classes by design introspection and user interaction," Autom. Softw. Eng., vol. 14, no. 4, pp. 369-418, Dec. 2007.
171. R. Singh, D. Giannakopoulou, and C. S. Pasareanu, "Learning component interfaces with may and must abstractions," in Proc. 22nd Int. Conf. Comput. Aided Verification, 2010, pp. 527-542.
172. J. M. Spivey, Z Notation-a Reference Manual, 2nd ed. Englewood Cliffs, NJ, USA: Prentice-Hall, 1992.
173. M. Staats, G. Gay, and M. P. E. Heimdahl, "Automated oracle creation support, or: How I learned to stop worrying about fault propagation and love mutation testing," in Proc. 34th Int. Conf. Softw. Eng., 2012, pp. 870-880.
174. M. Staats, M. W. Whalen, and M. P. E. Heimdahl, "Programs, tests, and oracles: The foundations of testing revisited," in Proc. 33rd Int. Conf. Softw. Eng., 2011, pp. 391-400.
175. M. Staats, S. Hong, M. Kim, and G. Rothermel, "Understanding user understanding: Determining correctness of generated program invariants," in Proc. Int. Symp. Softw. Testing Anal., 2012, pp. 188-198.
176. M. Staats, P. Loyola, and G. Rothermel, "Oracle-centric test case prioritization," in Proc. 23rd Int. Symp. Softw. Rel. Eng., 2012, pp. 311-320.
177. A. Takanen, J. DeMott, and C. Miller, Fuzzing for Software Security Testing and Quality Assurance, 1st ed. Norwood, MA, USA: Artech House, Inc., 2008.
178. R. Taylor, M. Hall, K. Bogdanov, and J. Derrick, "Using behaviour inference to optimise regression test sets," in Proc. 24th IFIP WG 601 Int. Conf. Testing Softw. Syst., 2012, pp. 184-199.
179. A. Tiwari. (2002). Formal semantics and analysis methods for Simulink Stateflow models. SRI International, Menlo Park, CA, USA: Tech. Rep. [Online]. Available: http://www.csl.sri.com/users/tiwari/html/stateflow.html
180. J. Tretmans, "Test generation with inputs, outputs and repetitive quiescence," Softw.-Concepts Tools, vol. 17, no. 3, pp. 103-120, 1996.
181. A. M. Turing, "Checking a large routine," in Report of a Conference on High Speed Automatic Calculating Machines. Cambridge, England, University Mathematical Laboratory, Jun. 1949, pp. 67-69.
182. M. Utting and B. Legeard, Practical Model-Based Testing: A Tools Approach. San Francisco, CA, USA: Morgan Kaufmann, 2007.
183. M. Utting, A. Pretschner, and B. Legeard, "A taxonomy of model-based testing approaches," Softw. Test. Verif. Reliab., vol. 22, no. 5, pp. 297-312, Aug. 2012.
184. G. V. Bochmann, C. He, and D. Ouimet, "Protocol testing using automatic trace analysis," in Proc. Can. Conf. Elect. Comput. Eng., 1989, pp. 814-820.
185. A. Van Lamsweerde and M. Sintzoff, "Formal derivation of strongly correct concurrent programs," Acta Informatica, vol. 12, no. 1, pp. 1-31, 1979.
186. J. M. Voas, "PIE: A dynamic failure-based technique," IEEE Trans. Softw. Eng., vol. 18, no. 8, pp. 717-727, Aug. 1992.
187. N. Walkinshaw, K. Bogdanov, J. Derrick, and J. Paris, "Increasing functional coverage by inductive testing: A case study," in Proc. 22nd IFIP WG 6.1 Int. Conf. Testing Softw. Syst., 2010, pp. 126-141.
188. N. Walkinshaw and K. Bogdanov, "Inferring finite-state models with temporal constraints," in Proc. 23rd IEEE/ACM Int. Conf. Autom. Softw. Eng., 2008, pp. 248-257.
189. N. Walkinshaw, J. Derrick, and Q. Guo, "Iterative refinement of reverse-engineered models by model-based testing," in Proc. 2nd World Congress Formal Methods, 2009, pp. 305-320.
190. Y. Wang and D. Parnas, "Trace rewriting systems," in Conditional Term Rewriting Systems, vol. 656, M. Rusinowitch and J.-L. Remy Eds. Berlin, Germany, Springer, 1993, pp. 343-356.
191. Y. Wang and D. L. Parnas, "Simulating the behaviour of software modules by trace rewriting," in Proc. 15th Int. Conf. Softw. Eng., 1993, pp. 14-23.
192. Y. Wei, C. A. Furia, N. Kazmin, and B. Meyer, "Inferring better contracts," in Proc. 33rd Int. Conf. Softw. Eng., 2011, pp. 191-200.
193. Y. Wei, H. Roth, C. A. Furia, Y. Pei, A. Horton, M. Steindorfer, M. Nordio, and B. Meyer, "Stateful testing: Finding more errors in code and contracts," in Proc. 26th IEEE/ACM Int. conf. Autom. Softw. Eng., 2011, pp. 440-443.
194. E. J. Weyuker, "Assessing test data adequacy through program inference," ACM Trans. Programm. Lang. Syst., vol. 5, no. 4, pp. 641-655, 1983.
195. E. J. Weyuker and F. I. Vokolos, "Experience with performance testing of software systems: Issues, an approach, and case study," IEEE Trans. Softw. Eng., vol. 26, no. 12, pp. 1147-1156, Dec. 2000.
196. E. J. Weyuker, "On testing non-testable programs," Comput. J., vol. 25, no. 4, pp. 465-470, Nov. 1982.
197. J. M. Wing, "A specifier's introduction to formal methods," IEEE Comput., vol. 23, no. 9, pp. 8-24, Sep. 1990.
198. Q. Xie and A. M. Memon, "Designing and comparing automated test oracles for GUI-based software applications," ACM Trans. Softw. Eng. Methodol., vol. 16, no. 1, p. 4, 2007
199. T. Xie, "Augmenting automatically generated unit-test suites with regression oracle checking," in Proc. 20th Eur. Conf. Object-Oriented Programm., Jul. 2006, pp. 380-403.
200. T. Xie and D. Notkin, "Checking inside the black box: Regression testing by comparing value spectra," IEEE Trans. Softw. Eng., vol. 31, no. 10, pp. 869-883, Oct. 2005.
201. Y. Xie and A. Aiken, "Context-and path-sensitive memory leak detection," ACM SIGSOFT Softw. Eng. Notes, vol. 30, no. 5, pp. 115-125, 2005.
202. Z. Xu, Y. Kim, M. Kim, G Rothermel, and M. B. Cohen, "Directed test suite augmentation: Techniques and tradeoffs," in Proc. 18th ACM SIGSOFT Int. Symp. Found. Softw. Eng., 2010, pp. 257-266.
203. S. Yoo, "Metamorphic testing of stochastic optimisation," in Proc. 3rd Int. Conf. Softw. Testing, Verification, Validation Workshops, 2010, pp. 192-201.
204. S. Yoo and M. Harman, "Regression testing minimisation, selection and prioritisation: A survey," Softw. Testing, Verification, Rel., vol. 22, no. 2, pp. 67-120, Mar. 2012.
205. B. Yu, L. Kong, Y. Zhang, and H. Zhu, "Testing Java components based on algebraic specifications," in Proc. Int. Conf. Softw. Testing, Verification, Validation, 2008, pp. 190-199.
206. A. Zeller and R Hildebrandt, "Simplifying and isolating failure-inducing input," IEEE Trans. Softw. Eng., vol. 28, no. 2, pp. 183-200, Feb. 2002.
207. S. Zhang, D. Saff, Y. Bu, and M. D. Ernst, "Combined static and dynamic automated test generation," in Proc. Int. Symp. Softw. Testing Anal., 2011, vol. 11, pp. 353-363.
208. W. Zheng, H. Ma, M. R. Lyu, T. Xie, and I. King, "Mining test oracles of web search engines," in Proc. 26th IEEE/ACM Int. Conf. Autom. Softw. Eng., 2011, pp. 408-411.
209. Z. Q. Zhou, S. Zhang, M. Hagenbuchner, T. H. Tse, F.-C. Kuo, and T. Y. Chen, "Automated functional testing of online search services," Softw. Testing, Verification Rel., vol. 22, no. 4, pp. 221-243, 2012.
210. H. Zhu, "A note on test oracles and semantics of algebraic specifications," in Proc. 3rd Int. Conf. Q. Softw., 2003, pp. 91-98.
211. B. Zorn and P. Hilfinger, "A memory allocation profiler for C and Lisp programs," in Proc. Summer USENIX Conf., 1988, pp. 223-237.