Automated test oracles: A survey

Advances in Computers

Volumn 95, Issue , 2014, Pages 1-48

  Pezzè, Mauro ^a,b   Zhang, Cheng ^a,b  

^a UNIVERSITY OF LUGANO   (Switzerland)

^b UNIVERSITY OF MILANO BICOCCA   (Italy)

Author keywords

Automated Oracles; Code based oracles; Software Testing; Specification based Oracles; Testing Oracles

Indexed keywords

EID: 84906748710     PISSN: 00652458     EISSN: None     Source Type: Book Series    
DOI: 10.1016/B978-0-12-800160-8.00001-2     Document Type: Chapter

References (90)

1. Weyuker E.J. On testing non-testable programs. Comput. J. 1982, 25(4):465-470.
2. Luciano Baresi, Michal Young, Test oracles, Technical report CIS-TR-01-02, University of Oregon, Department of Computer and Information Science, Eugene, OR, USA, August 2001. http://ix.cs.uoregon.edu/~michal/pubs/oracles.html.
3. Pezzè M., Young M. Software Testing and Analysis: Process, Principles, and Techniques 2008, John Wiley & Sons Inc., New York.
4. Csallner C., Smaragdakis Y. JCrasher: an automatic robustness tester for Java. Softw. Pract. Exp. 2004, 34(11):1025-1050.
5. Csallner C., Smaragdakis Y. Check 'n' crash: combining static checking and testing 2005, 422-431. ACM, New York, NY.
6. Flanagan C., Rustan K., Leino M., Lillibridge M., Nelson G., Saxe J.B. Extended static checking for Java 2002, 234-245. ACM, New York, NY.
7. Csallner C., Smaragdakis Y., Xie T. Dsd-crasher: a hybrid analysis tool for bug finding. ACM Trans. Softw. Eng. Methodol. 2008, 17(2):8:1-8:37.
8. Ernst M.D. Dynamically discovering likely program invariants 2000, Ph.D, University of Washington, Department of Computer Science and Engineering, Seattle, Washington.
9. Pacheco C., Lahiri S.K., Ernst M.D., Ball T. Feedback-directed random test generation 2007, 75-84.
10. Ricca F., Tonella P. Detecting anomaly and failure in web applications. IEEE MultiMedia 2006, 13(2):44-51.
11. Anand S., Burke E., Chen T.Y., Clark J., Cohen M.B., Grieskamp W., Harman M., Harrold M.J., McMinn P., Bertolino A., et al. An orchestrated survey on automated software test case generation. J. Syst. Software 2013, 86(8):1978-2001.
12. Bertolino A. Software testing research: achievements, challenges, dreams. 2007 Future of Software Engineering, FOSE '07 1978-2007, 85-103. IEEE Computer Society, Washington, DC.
13. M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, A comprehensive survey of trends in oracles for software testing, Technical report, Technical Report Research Memoranda CS-13-01, Department of Computer Science, University of Sheffield, 2013.
14. Luckham D.C., Von Henke F.W. An overview of Anna, a specification language for Ada. IEEE Softw. 1985, 2(2):9-22.
15. Sankar S., Hayes R. ADL-an interface definition language for specifying and testing software 1994, 13-21. ACM, New York, NY.
16. Pacheco C., Ernst M.D. Eclat: automatic generation and classification of test inputs 2005, 504-527.
17. Daniel B., Jagannath V., Dig D., Marinov D. ReAssert: suggesting repairs for broken unit tests 2009, 433-444. IEEE Computer Society, Washington, DC.
18. Fraser G., Arcuri A. EvoSuite: automatic test suite generation for object-oriented software 2011, 416-419.
19. Fraser G., Zeller A. Mutation-driven generation of unit tests and oracles. IEEE Trans. Softw. Eng. 2012, 38(2):278-292.
20. Staats M., Gay G., Heimdahl M.P.E. Automated oracle creation support, or: how I learned to stop worrying about fault propagation and love mutation testing 2012, 870-880.
21. Pastore F., Mariani L., Fraser G. "Crowdoracles: can the crowd solve the oracle problem?" 2013,Proceedings - IEEE 6th International Conference on Software Testing, Verification and Validation, ICST 2013 ;Article number 6569746; 342-351.
22. Staats M., Hong S., Kim M., Rothermel G. Understanding user understanding: determining correctness of generated program invariants 2012, 188-198.
23. Spivey J.M. The Z Notation: A Reference Manual 1992, Prentice Hall International (UK) Ltd., Hertfordshire, UK.
24. Abrial J.-R., Lee M.K.O., Neilson D., Scharbach P.N., Sørensen I. The B-method 1991, 398-405. Springer-Verlag, London, UK.
25. Jones C.B. Systematic Software Development Using VDM 1986, Prentice Hall International (UK) Ltd., Hertfordshire, UK.
26. Meyer B. Object-Oriented Software Construction 1988, Prentice-Hall, Inc., Upper Saddle River, NJ. first ed.
27. Burdy L., Cheon Y., Cok D.R., Ernst M.D., Kiniry J.R., Leavens G.T., Rustan K., Leino M., Poll E. An overview of JML tools and applications. Int. J. Softw. Tools Technol. Transfer 2005, 7(3):212-232.
28. Peters D.K., Parnas D.L. Using test oracles generated from program documentation. IEEE Trans. Softw. Eng. 1998, 24(3):161-173.
29. Mikk E. Compilation of Z specifications into C for automatic test result evaluation 1995, 167-180. Springer-Verlag, London, UK.
30. Schmidt U., Hörcher H.-M. Programming with VDM domains 1990, 122-134. Springer, Berlin, Heidelberg.
31. McDonald J., Murray L., Strooper P. Translating object-z specifications to object-oriented test oracles 1997, 414. IEEE Computer Society, Washington, DC.
32. Richardson D.J., Aha S.L., O'Malley T.O. Specification-based test oracles for reactive systems 1992, 105-118.
33. Meyer B. Applying "design by contract". Computer 1992, 25(10):40-51.
34. Kramer R. iContract-the Java(tm) design by contract(tm) tool 1998, 295. IEEE Computer Society, Washington, DC.
35. A. Bhorkar, A run-time assertion checker for Java using JML, Technical report 00-08, Department of Computer Science, Iowa State University, 2000.
36. Boyapati C., Khurshid S., Marinov D. Korat: automated testing based on Java predicates 2002, 123-133. ACM, New York, NY.
37. Araujo W., Briand L.C., Labiche Y. Enabling the runtime assertion checking of concurrent contracts for the java modeling language 2011, 786-795. ACM, New York, NY.
38. Peters D., Parnas D.L. Generating a test oracle from program documentation: work in progress 1994, 58-65. ACM, New York, NY.
39. Bouquet F., Grandpierre C., Legeard B., Peureux F., Vacelet N., Utting M. A subset of precise UML for model-based testing 2007, 95-104. ACM, New York, NY.
40. Andrews J.H. Testing using log file analysis: tools, methods, and issues 1998, 157. IEEE Computer Society, Washington, DC.
41. Pradel M., Gross T.R. Leveraging test generation and specification mining for automated bug detection without false positives 2012, 288-298. IEEE Press, Piscataway, NJ.
42. Dillon L.K., Yu Q. Oracles for checking temporal properties of concurrent systems 1994, 140-153. ACM, New York, NY.
43. Fujiwara S., von Bochmann G., Khendek F., Amalou M., Ghedamsi A. Test selection based on finite state models. IEEE Trans. Softw. Eng. 1991, 17(6):591-603.
44. Dillon L.K., Kutty G., Moser L.E., Melliar-Smith P.M., Ramakrishna Y.S. Graphical specifications for concurrent software systems 1992, 214-224. ACM, New York, NY.
45. Dillon L.K., Ramakrishna Y.S. Generating oracles from your favorite temporal logic specifications 1996, 106-117. ACM, New York, NY.
46. Gargantini A., Heitmeyer C. Using model checking to generate tests from requirements specifications 1999, 146-162. Springer-Verlag, London, UK.
47. Holzmann G.J. The model checker spin. IEEE Trans. Softw. Eng. 1997, 23(5):279-295.
48. J.R. Callahan, S.M. Easterbrook, T.L. Montgomery, Generating test oracles via model checking, NASA/WVU Software Research Lab, Fairmont, WV, Technical report# NASA-IVV-98-015, 1998.
49. Memon A.M., Pollack M.E., Lou Soffa M. Automated test oracles for GUIs 2000, 30-39. ACM, New York, NY.
50. Gannon J., McMullin P., Hamlet R. Data abstraction, implementation, specification, and testing. ACM Trans. Program. Lang. Syst. 1981, 3(3):211-223.
51. Doong R.-K., Frankl P.G. The Astoot approach to testing object-oriented programs. ACM Trans. Softw. Eng. Methodol. 1994, 3(2):101-130.
52. Chen H.Y., Tse T.H., Chan F.T., Chen T.Y. In black and white: an integrated approach to class-level testing of object-oriented programs. ACM Trans. Softw. Eng. Methodol. 1998, 7(3):250-295.
53. Chen H.Y., Tse T.H., Chen T.Y. TACCLE: a methodology for object-oriented software testing at the class and cluster levels. ACM Trans. Softw. Eng. Methodol. 2001, 10(1):56-109.
54. Helm R., Holland I.M., Gangopadhyay D. Contracts: specifying behavioral compositions in object-oriented systems 1990, 169-180. ACM, New York, NY.
55. Yu B., Kong L., Zhang Y., Zhu H. Testing java components based on algebraic specifications 2008, 190-199. IEEE Computer Society, Washington, DC.
56. Antoy S., Hamlet D. Automatically checking an implementation against its formal specification. IEEE Trans. Softw. Eng. 2000, 26(1):55-69.
57. Memon A., Xie Q. Using transient/persistent errors to develop automated test oracles for event-driven software 2004, 186-195. IEEE Computer Society, Washington, DC.
58. Sprenkle S., Pollock L., Esquivel H., Hazelwood B., Ecott S. Automated oracle comparators for testing web applications 2007, 117-126. IEEE Computer Society, Washington, DC.
59. Xie T. Augmenting automatically generated unit-test suites with regression oracle checking 2006, 380-403.
60. Xie Q., Memon A.M. Designing and comparing automated test oracles for GUI-based software applications. ACM Trans. Softw. Eng. Methodol. 2007, 16(1):41-76.
61. Fraser G., Zeller A. Mutation-driven generation of unit tests and oracles 2010, 147-158. ACM, New York, NY.
62. Mitchell T.M. Machine Learning 1997, McGraw Hill, New York.
63. Jin H., Wang Y., Chen N.-W., Gou Z.-J., Wang S. Artificial neural network for automatic test oracles generation 2008, 727-730. IEEE Computer Society, Washington, DC.
64. Shahamiri S.R., Wan Kadir W.M.N., Ibrahim S., Mohd Hashim S.Z. An automated framework for software test oracle. Inform. Softw. Technol. 2011, 53(7):774-788.
65. Vanmali M., Last M., Kandel A. Using a neural network in the software testing process. Int. J. Intell. Syst. 2002, 17(1):45-62.
66. Zheng W., Ma H., Lyu M.R., Xie T., King I. Mining test oracles of web search engines 2011, 408-411. IEEE Computer Society, Washington, DC.
67. Frounchi K., Briand L.C., Grady L., Labiche Y., Subramanyan R. Automating image segmentation verification and validation by learning test oracles. Inform. Softw. Technol. 2011, 53(12):1337-1348.
68. Shahamiri S.R., Wan-Kadir W.M., Ibrahim S., Hashim S.Z. Artificial neural networks as multi-networks automated test oracle. Autom. Softw. Eng. 2012, 19(3):303-334.
69. T.Y. Chen, S.C. Cheung, S.M. Yiu, Metamorphic testing: a new approach for generating next test cases, Department of Computer Science, Hong Kong University of Science and Technology, Technical report HKUST-CS98-01, 1998.
70. Zhou Z.Q., Zhang S.J., Hagenbuchner M., Tse T.H., Kuo F.-C., Chen T.Y. Automated functional testing of online search services. Softw. Test. Verif. Reliab. 2012, 22(4):221-243.
71. Schroeder P.J., Faherty P., Korel B. Generating expected results for automated black-box testing 2002, 139. IEEE Computer Society, Washington, DC.
72. Murphy C., Shen K., Kaiser G. Using JML runtime assertion checking to automate metamorphic testing in applications without test oracles 2009, 436-445. IEEE Computer Society, Washington, DC.
73. Carzaniga A., Goffi A., Gorla A., Mattavelli A., Pezzè M. Cross-checking oracles from intrinsic software redundancy 2014, 931-942. ACM, New York, NY, USA.
74. Hughes M., Stotts D. Daistish: systematic algebraic testing for OO programs in the presence of side-effects 1996, 53-61. ACM, New York, NY.
75. Taneja K., Xie T. Diffgen: automated regression unit-test generation 2008, 407-410.
76. Mariani, L., Pastore, F."Supporting plug-in mashes to ease tool integration" IN Proceedings - International Conference on Software Engineering 2011, Pages 1-4
77. Aichernig B.K. Automated black-box testing with abstract VDM oracles 1999, 250-259. Springer-Verlag, London, UK.
78. Durrieu G., Waeselynck H., Wiels V. Leto-a lustre-based test oracle for airbus critical systems. Formal Methods for Industrial Critical Systems 2009, 7-22. Springer-Verlag, Berlin, Heidelberg. D. Cofer, A. Fantechi (Eds.).
79. Yu T., Sung A., Srisa-an W., Rothermel G. Using property-based oracles when testing embedded system applications 2011, 100-109. IEEE Computer Society, Washington, DC.
80. Padgham L., Thangarajah J., Zhang Z., Miller T. Model-based test oracle generation for automated unit testing of agent systems. IEEE Trans. Softw. Eng. 2013, 39(9):1230-1244.
81. Zhou Z.Q., Huang D.H., Tse T.H., Yang Z., Huang H., Chen T.Y. Metamorphic testing and its applications 2004, Software Engineers Association, Xian, China.
82. Chan W.K., Cheung S.C., Leung K.R.P.H. Towards a metamorphic testing methodology for service-oriented software applications 2005, 470-476. IEEE Computer Society, Washington, DC.
83. Hamlet R.G. Testing programs with the aid of a compiler. IEEE Trans. Softw. Eng. 1977, 3(4):279-290.
84. Panzl D.J. Automatic software test drivers. Computer 1978, 11(4):44-50.
85. Bai X., Hou K., Lu H., Zhang Y., Hu L., Ye H. Semantic-based test oracles 2011, 640-649. IEEE Computer Society, Washington, DC.
86. Kim-Park D.S., de la Riva C., Tuya J. An automated test oracle for xml processing programs 2010, 5-12. ACM, New York, NY.
87. Mayer J., Guderlei R. Test oracles using statistical methods 2004, 179-189. Springer, Berlin, Heidelberg.
88. Fuchs N.E. Specifications are (preferably) executable. Softw. Eng. J. 1992, 7(5):323-334.
89. Glässer U., Gurevich Y., Veanes M. High-level executable specification of the universal plug and play architecture 2002, 283. IEEE Computer Society, Washington, DC.
90. Wang X., Wang J., Qi Z.-C. Automatic generation of run-time test oracles for distributed real-time systems. Lecture Notes in Computer Science 2004, vol. 3235:199-212. Springer, Berlin, Heidelberg.