Designing and comparing automated test oracles for GUI-based software applications

ACM Transactions on Software Engineering and Methodology

Volumn 16, Issue 1, 2007, Pages

  Xie, Qing ^a,b   Memon, Atif M ^a,b  

^a UNIVERSITY OF MARYLAND   (United States)

^b University of Maryland ^*  (United States)

Author keywords

Graphical user interfaces; GUI state; GUI testing; Test oracles; User interfaces; Widgets

Indexed keywords

TEST ORACLES; WIDGETS;

AUTOMATA THEORY; COMPUTER SOFTWARE; DATABASE SYSTEMS; SOFTWARE ENGINEERING; SOFTWARE TESTING;

GRAPHICAL USER INTERFACES;

EID: 33847680195     PISSN: 1049331X     EISSN: 15577392     Source Type: Journal    
DOI: 10.1145/1189748.1189752     Document Type: Article

References (54)

1. ANTOY, S. AND HAMLET, D. 2000. Automatically checking an implementation against its formal specification. IEEE Trans. Softw. Eng. 26, 1, 55-69.
2. BARESI, L. AND YOUNG, M. 2001. Test oracles. Tech. Rep. CIS-TR-01-02, University of Oregon, Department of Computer and Information Science, Eugene, Oregon. August. http://www.cs.uoregon.edu/michal/ pubs/oracles.html.
3. BARRINGER, H., GOLDBERG, A., HAVELUND, K., AND SEN, K. 2004. Rule-Based runtime verification. In Proceedings of the 5th International Conference on Verification, Model Checking and Abstract Interpretation (VMCAI) (Venice). Springer Verlag. 44-57.
4. BEHNOT, G., GAUDEL, M. C., AND MAERE, B. 1991. Software testing based on formal specifications: A theory and a tool. Softw. Eng. J. 6, B, 387-405.
5. BLACKBURN, M., BUSSER, R., AND NAUMAN, A. 2002. Interface-Driven model-based test generation of Java test drivers. In Proceedings of the 16th International Software / Internet Quality Week Conference. Software Research, San Francisco, CA.
6. BURDY, L., CHEON, Y., COK, D., ERNST, M., KINIRY, J., LEAVENS, G. T., LEINO, K. R. M., AND POLL, E. 2003. An overview of JML tools and applications. In Proceedings of the 8th International Workshop on Formal Methods for Industrial Critical Systems (FMICS). Elsevier Science. 73-89.
7. CHEON, Y. AND LEAVENS, G. T. 2002. A simple and practical approach to unit testing: The JML and JUnit way. In ECOOP: Proceedings of the 16th European Conference on Object-Oriented Programming (London). Springer Verlag. 231-255.
8. DILLON, L. K. AND RAMAKHISHNA, Y S. 1996. Generating oracles from your favorite temporal logic specifications. In Proceedings of the 4th ACM SIGSOFT Symposium on the Foundations of Software Engineering. ACM Software Engineering Notes, vol. 21. ACM Press, New York. 106-117.
9. DILLON, L. K. AND YU, Q. 1994. Oracles for checking temporal properties of concurrent systems. In Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering (New Orleans, LA). ACM Press, New York. 140-153.
10. DOONG, R.-K. AND FRANKL, P. G. 1994. The ASTOOT approach to testing object-oriented programs. ACM Trans. Softw. Eng. Methodol. 3, 2, 101-130.
11. DU BOUSQUET, L., OUABDESSELAM, F., RICHIER, J.-L., AND ZUANON, N. 1999. Lutess: A specification-driven testing environment for synchronous software. In Proceedings of the 21st International Conference on Software Engineering (Los Angeles). ACM Press, New York. 267-276.
12. ELBAUM, S., KARHE, S., AND ROTHERMEL, G. 2003. Improving web application testing with user session data. In ICSE: Proceedings of the 25th International Conference on Software Engineering. IEEE Computer Society, Washington, DC. 49-59.
13. FINSTERWALDER, M. 2001. Automating acceptance tests for GUI applications in an extreme programming environment. In Proceedings of the 2nd International Conference on eXtreme Programming and Flexible Processes in Software Engineering (Villasimius, Sardinia, Italy). Addison-Wesley. 114-117.
14. GALL, P. L. AND ARNOULD, A. 1995. Formal specifications and test: Correctness and oracle. In COMPASS / ADT. Springer Verlag, Oslo, Norway. 342-358.
15. GANNON, J., MCMULLIN, P., AND HAMLET, R. 1981. Data abstraction, implementation, specification, and testing. ACM Trans. Program. Lang. Syst. 3, 3, 211-223.
16. GRAVES, T. L., HARROLD, M. J., KIM, J.-M., PORTER, A., AND ROTHERMEL, G. 2001. An empirical study of regression test selection techniques. ACM Trans. Softw. Eng. Methodol. 10, 2, 184-208.
17. GREEN, B. S. 2000. Software test automation. SIGSOFT Softw. Eng. Notes 25, 3, 66.
18. HAVELUND, K. AND ROU, G. 2004. An overview of the runtime verification tool Java PathExplorer. Form. Methods Syst. Des. 24, 2, 189-215.
19. HICINBOTHOM, J. H. AND ZACHARY, W. W. 1993. A tool for automatically generating transcripts of human-computer interaction. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting. Special Sessions: Demonstrations, vol. 2. Human Factors and Ergonomics Society, Seattle. 1042.
20. JAGADEESAN, L. J., PORTER, A., PUCHOL, C., RAMMING, J. C., AND VOTTA, L. G. 1997. Specification-Based testing of reactive software: Tools and experiments: Experience report. In ICSE: Proceedings of the 19th International Conference on Software Engineering. ACM Press, New York. 525-535.
21. JUNIT, JAVA UNIT TESTING FRAMEWORK. 2003. JUnit, Java unit testing framework, http://junit.sourceforge.net.
22. KEPPLE, L. R. 1994. The black art of GUI testing. Dr. Dobb's J. Softw. Tools 19, 2 (Feb.), 40.
23. KIM, M., VISWANATHAN, M., BEN-A BDALLAH, H., KANNAN, S., LEE, I., AND SOKOLSKY, O. 1998. Mac: A framework for run-time correctness assurance of real-time systems. Tech. Rep. MS-CIS-98-37, University of Pennsylvania, Department of Computer and Information Science.
24. LUCKHAM, D. AND HENKE, F. W. 1985. An overview of Anna - A specification language for Ada. IEEE Softw. 2, 2, 9-22.
25. MARICK, B. 1998. When should a test be automated? In Proceedings of the 11th International Software / Internet Quality Week. Software Research, San Francisco, CA.
26. MARICK, B. 2002. Bypassing the GUI. Softw. Test. Quality Eng. Mag. 4, 5, 41-47.
27. MEMON, A., BANERJEE, I., AND NAGARAJAN, A. 2003a. GUI ripping: Reverse engineering of graphical user interfaces for testing. In WCRE: Proceedings of the 10th Working Conference on Reverse Engineering (Victoria, BC, Canada). IEEE Computer Society. 260-269.
28. MEMON, A., NAGARAJAN, A., AND XIE, Q. 2005. Automating regression testing for evolving GUI software. J. Softw, Maintenance and Evolution: Res. Pract, 17, 1, 27-64.
29. MEMON, A. M. 2003. Advances in GUI testing. In Advances in Computers, Vol. 57, Marvin V. Zelkowitz, ed. Academic Press, New York.
30. MEMON, A. M., BANERJEE, I., AND NAGARAJAN, A. 2003b. What test oracle should I use for effective GUI testing? In Proceedings of the IEEE International Conference on Automated Software Engineering (Montreal, Canada). IEEE Computer Society. 164-173.
31. MEMON, A. M., POLLACK, M. E., AND SOFFA, M. L. 2000. Automated test oracles for GUIs. In Proceedings of the ACM SIGSOFT 8th International Symposium on the Foundations of Software Engineering (FSE-8) (Shelter Island, San Diego, CA). ACM Press, New York. 30-39.
32. MEMON, A. M., POLLACK, M. E., AND SOFFA, M. L. 2001. Hierarchical GUI test case generation using automated planning. IEEE Trans. Softw. Eng. 27, 2 (Feb.), 144-155.
33. MEMON, A. M. AND SOFFA, M. L. 2003. Regression testing of GUIs. In ESEC / FSE-11: Proceedings of the 9th European Software Engineering Conference held jointly with the 11th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM Press, New York. 118-127.
34. MEMON, A. M., SOFFA, M. L., AND POLLACK, M. E. 2001. Coverage criteria for GUI testing. In Proceedings of the 8th European Software Engineering Conference (ESEC) and 9th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE-9) (Vienna, Austria). ACM Press, New York. 256-267.
35. MEMON, A. M. AND XIE, Q. 2005. Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software. IEEE Trans. Softw. Eng. 31, 10 (Oct.), 884-896.
36. MYEHS, B. A. 1995. User interface software tools. ACM Trans. Comput. Hum. Interact. 2, 1, 64-103.
37. OSTRAND, T., ANODIDE, A., FOSTER, H., AND GORADIA, T. 1998. A visual test development environment for GUI systems. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM Press, New York. 82-92.
38. PETERS, D. AND PARNAS, D. L. 1994. Generating a test oracle from program documentation: Work in progress. In. ISSTA: Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM Press, New York. 58-65.
39. PETERS, D. K. AND PARNAS, D. L. 1998. Using test oracles generated from program documentation. IEEE Trans. Softw. Eng. 24, 3, 161-173.
40. RICHARDSON, D. J. 1994. Taos: Testing with analysis and oracle support. In ISSTA: Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM Press, New York. 138-153.
41. RICHARDSON, D. J., LEIF-AHA, S., AND O'MALLEY, T. O. 1992. Specification-Based test oracles for reactive systems. In Proceedings of the 14th International Conference on Software Engineering (Melbourne, Australia). ACM Press, New York. 105-118.
42. ROBINSON, H. 1999. Using pre-oracled data in model-based testing, http://www.geocities.com/harry_r-obinson.testing/pre-oracled.htm.
43. ROSENBLUM, D. S. 1991. Specifying concurrent systems with TSL. IEEE Softw. 8, 3, 52-61.
44. ROSENBLUM, D. S. 1992. Towards a method of programming with assertions. In ICSE: Proceedings of the 14th International Conference on Software Engineering. ACM Press, New York. 92-104.
45. ROTHERMEL, G., ELBAUM, S., MALISHEVSKY, A. G., KALLAKURI, P., AND QIU, X. 2004. On test suite composition and cost-effective regression testing. ACM Trans. Softw. Eng. Methodol. 13, 3, 277-331.
46. SIEPMANN, E. AND NEWTON, A. R. 1994. TOBAC: Test case browser for object-oriented software. In Proceedings of the International Symposium on Software Testing and Analysis. ACM Press, New York. 154-168.
47. SOFTWARE RESEARCH, INC. 2003. Capture-Replay tool 2003. http://soft.com.
48. SU, J. AND RITTER, P. R. 1991. Experience in testing the Motif interface. IEEE Softw. 8, 2 (Mar.), 26-33.
49. VIEIRA, M. E., DIAS, M. S., AND RICHARDSON, D. J. 2000. Object-Oriented specification-based testing using UML statechart diagrams. In Proceedings of the Workshop on Automated Program Analysis, Testing and Verification (Limerick, Ireland). ACM Press, New York.
50. VOGEL, P. A. 1993. An integrated general purpose automated test environment. In ISSTA: Proceedings of the 1993 ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM Press, New York. 61-69.
51. WEYUKER, E. J. 1982. On testing non-testable programs. Comput. J. 25, 4, 465-470.
52. WHITE, L., ALMEZEN, H., AND ALZEIDI, N. 2001. User-Based testing of GUI sequences and their interactions. In Proceedings of the 12th International Symposium Software Reliability Engineering (Hong Kong). IEEE Press. 54-63.
53. WOHLIN, C., RUNESON, P., HOST, M., OHLSSON, M. C., REGNELL, B., AND WESSLEN, A. 2000. Experimentation in Software Engineering: An Introduction. Kluwer Academic, Norwell, MA.
54. ZHU, H., HALL, P., AND MAY, J. 1997. Software unit test coverage and adequacy. ACM Comput. Surv. 29, 4 (Dec.), 366-427.