Integrating automated test generation into the WYSIWYT spreadsheet testing methodology

ACM Transactions on Software Engineering and Methodology

Volumn 15, Issue 2, 2006, Pages 150-194

  Fisher II, Marc ^a   Rothermel, Gregg ^a   Brown, Darren ^b   Cao, Mingming ^b   Cook, Curtis ^b   Burnett, Margaret ^b  

^a UNIVERSITY OF NEBRASKA LINCOLN   (United States)

^b OREGON STATE UNIVERSITY   (United States)

Author keywords

End user programming; End user software engineering; Test case generation; Testing

Indexed keywords

END-USER PROGRAMMING; END-USER SOFTWARE ENGINEERING; TEST CASE GENERATION;

COMPUTATION THEORY; COMPUTER PROGRAMMING LANGUAGES; FEEDBACK; SPREADSHEETS;

AUTOMATIC TESTING;

EID: 33745389959     PISSN: 1049331X     EISSN: 1049331X     Source Type: Journal    
DOI: 10.1145/1131421.1131423     Document Type: Article

References (61)

1. AVRITZER, A. AND WEYUKER, E. 1995. Automated generation ofload test suites and the assessment of the resulting software. IEEE Trans. Softw. Eng. 21, 9 (Sept.), 705-716.
2. BARESEL, A., BINKLEY, D., HARMAN, M., AND KOREL, B. 2004. Evolutionary testing in the presence of loop-assigned flags: A testability transformation approach. In Proceedings of the International Symposium, on Software Testing and Analysis (Boston, Mass). ACM Press, New York, 108-118.
3. BELKIN, N. 2000. Helping people find what they don't know. Commun. ACM 41, 8 (Aug.), 58-61.
4. BIRD, D. AND MUNOZ, C. 1983. Automatic generation of random self-checking test cases. IBM Syst. J. 22, 3, 229-245.
5. BOYAPATI, C., KHURSHID, S., AND MARINOV, D. 2002. Korat: Automated testing based on Java predicates. In Proceedings of the ACM International Symposium on Software Testing and Analysis (Rome). ACM Press, New York, 123-133.
6. BROWN, D., BURNETT, M., ROTHERMEL, G., FUJITA, H., AND NEGORO, F. 2003. Generalizing WYSIWYT visual testing to screen transition languages. In. Proceedings of the IEEE Symposium, on Human-Centric Computing Languages and Environments (Auckland, New Zealand). IEEE Computer Society Press, Los Alamitos, Calif.
7. BROWN, P. AND GOULD, J. 1987. Experimental study of people creating spreadsheets. ACM Trans. Office Inf. Syst. 5, 3 (July), 258-272.
8. BURNETT, M., ATWOOD, J., DJANG, R., GOTTFRIED, H., REICHWEIN, J., AND YANG, S. 2001. Forms/3: A first-order visual language to explore the boundaries of the spreadsheet paradigm. J. Functional Program. 11, 2, 155-206.
9. BURNETT, M., COOK, C., PENDSE, O., ROTHERMEL, G., SUMMET, J., AND WALLACE, C. 2003. End-user software engineering with assertions in the spreadsheet paradigm. In Proceedings of the International Conference on Software Engineering (Portland, Oreg.). IEEE Computer Society Press, Los Alamitos, Calif. 93-103.
10. BURNETT, M., COOK, C., AND ROTHERMEL, G. 2004. End-user software engineering. Commun. ACM 47, 9 (Sept.), 53-58.
11. BURNETT, M., HOSSLI, R., PULLIAM, T., VANVOORST, B., AND YANG, X. 1994. Toward visual programming languages for steering in scientific visualization: A taxonomy. IEEE Comput. Sci, Eng. 1, 4, 44-62.
12. BURNETT, M., SHERETOV, A., REN, B., AND ROTHERMEL, G. 2002. Testing homogeneous spreadsheet grids with the "What You See Is What You Test" methodology. IEEE Trans. Softw. Eng. 28, 6 (June), 576-594.
13. BURNETT, M., SHERETOV, A., AND ROTHERMEL, G. 1999. Scaling up a "What You See Is What You Test" methodology to spreadsheet grids. In Proceedings of the 1999 IEEE Symposium on Visual Languages (Tokyo). IEEE Computer Society Press, Los Alamitos, Calif. 30-37.
14. CHANG, J. AND RICHARDSON, D. 1999. Structural specification-based testing: Automated support and experimental evaluation. In Proceedings of the ACM Symposium on Foundations of Software Engineering (Toulouse, France). ACM Press, New York, 285-302.
15. CHI, E., BARRY, P., RIEDL, J., AND KONSTAN, J. 1997. A spreadsheet approach to information visualization. In Proceedings of the IEEE Symposium on Information Visualization (Phoenix, Ariz). IEEE Computer Society Press, Los Alamitos, Calif.
16. CLARKE, L. 1976. A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Eng. 2, 3 (Sept.), 215-222.
17. CORRITORE, C., KRACHER, B., AND WIEDENBECK, S. 2001. Trust in the online environment. In Proceedings of the HCI International (New Orleans, La). Lawrence Erlbaum, Mahwah, N.J., 1548-1552.
18. CULLEN, D. 2003. Excel snafu costs firm $24m. The Register.
19. DEMILLO, R. AND OFFUTT, A. 1991. Constraint-based automatic test data generation. IEEE Trans. Softw. Eng. 17, 9 (Sept.), 900-910.
20. DUESTERWALD, E., GUPTA, R., AND SOFFA, M. L. 1992. Rigorous data flow testing through output influences. In Proceedings of the 2nd Irvine Software Symposium (Irvine, Calif). University of California, Irvine.
21. ERNST, M., COCKRELL, J., GRISWOLD, W., AND NOTKIN, D. 2001. Dynamically discovering likely program invariants to support program evolution. IEEE Trans. Softw. Eng. 27, 2 (Feb.), 1-25.
22. FERGUSON, R. AND KOREL, B. 1996. The chaining approach for software test data generation. ACM Trans. Softw. Eng. Methodol. 5, l (Jan.), 63-86.
23. FISHER, M., II, CAO, M., ROTHERMEL, G., COOK, C., AND BURNETT, M. 2002. Automated test case generation for spreadsheets. In Proceedings of the 24th International Conference on Software Engineering (Orlando, Fla.). ACM Press, New York, 241-251.
24. FISHER, M., II, JIN, D., ROTHERMEL, G., AND BURNETT, M. 2002. Test reuse in the spreadsheet paradigm. In Proceedings of the International Symposium on Software Reliability Engineering (Annapolis, Md.). IEEE Computer Society, Los Alamitos, Calif.
25. FISHER, M., II AND ROTHERMEL, G. 2005. The EUSES Spreadsheet Corpus: A shared resource for supporting experimentation with spreadsheet dependability mechanisms. In Proceedings of the Workshop on End-User Software Engineering (St. Louis, Miss.). ACM Press, New York.
26. FRANKL, P. AND WEYUKER, E. 1988. An applicable family of data flow criteria. IEEE Trans. Softw. Eng. 14, 10 (Oct.), 1483-1498.
27. GOTLIEB, A., BOTELLA, B., AND REUHER, M. 1998. Automatic test data generation using constraint solving techniques. In Proceedings of the ACM International Symposium on Software Testing and Analysis (Clearwater Beach, Fla.). ACM Press, New York, 53-62.
28. GOTLIEB, A., BOTELLA, B., AND RUEHER, M. 2000. A CLP framework for computing structural test data. In Proceedings of the 1st International Conference on Computational Logic (London). Springer Verlag, New York, 399-412.
29. GUPTA, N., MATHUR, A., AND SOFFA, M. 1998. Automated test data generation using an iterative relaxation method. In Proceedings of the International Symposium on Foundations of Software Engineering (Lake Buena Vista, Fla.). ACM Press, New York, 231-244.
30. HARTMAN, A. AND NAGIN, K. 2003. Model driven testing - Agedis architecture interfaces and tools. In Proceedings of the 1st European Conference on Model Driven Software Engineering (Nuremberg), imbus AG, Möhrendorf, Germany, 1-11.
31. KOREL, B. 1990a. A dynamic approach of automated test data generation. In Proceedings of the International Conference on Software Maintenance (San Diego, Calif.). IEEE Computer Society Press, Los Alamitos, Calif., 311-317.
32. KOREL, B. 1990b. Automated software test data generation. IEEE Trans. Softw. Eng. 16, 8 (Aug.), 870-897.
33. KRISHNA, V., COOK, C., KELLER, D., CANTRELL, J., WALLACE, C., BURNETT, M., AND ROTHERMEL, G. 2001. Incorporating incremental validation and impact analysis into spreadsheet maintenance: An empirical study. In Proceedings of the International Conference on Software Maintenance (Florence). IEEE Computer Science Press, Los Alamitos, Calif.
34. LASKI, J. AND KOREL, B. 1993. A data flow oriented program testing strategy. IEEE Trans. Softw. Eng. 9, 3 (May), 347-354.
35. LEOPOLD, J. AND AMBLER, A. 1997. Keyboardless visual programming using voice, handwriting, and gesture. In Proceedings of the 1997 IEEE Symposium of Visual Languages (Capri, Italy). IEEE Computer Society Press, Los Alamitos, Calif., 28-35.
36. MARINOV, D. AND KHUHSHID, S. 2001. TestEra: A novel framework for automated testing of Java programs. In Proceedings of the International Conference on Automated Software Engineering (San Diego, Calif.). IEEE Computer Society Press, Los Alamitos, Calif.
37. MARRE, M. AND BERTOLINO, A. 1996a. Reducing and estimating the cost of test coverage criteria. In Proceedings of the International Conference on Software Engineering (Berlin). IEEE Computer Society Press, Los Alamitos, Calif.
38. MARRE, M. AND BERTOLINO, A. 1996b. Unconstrained duas and their use in achieving all-uses coverage. In Proceedings of the ACM International Symposium on Software Testing and Analysis (San Diego, Calif.). ACM Press, New York.
39. MICHAEL, C., MCGRAW, G., AND SHATZ, M. 2001. Generating software test data by evolution. IEEE Trans. Softw. Eng. 27, 12 (Dec.), 1085-1110.
40. MYERS, B. 1991. Graphical techniques in a spreadsheet for specifying user interfaces. In Proceedings of the ACM CHI '91 (New Orleans, La). ACM Press, New York, 243-249.
41. OEFUTT, A. 1991. An integrated automatic test data generation system. J. Syst. Integration 1, 3 (Nov.), 391-409.
42. OFUTT, J. AND ABDURAZIK, A. 1999. Generating tests from uml specifications. In Proceedings of the International Conference on the Unified Modeling Language (Fort Collins, Colo.). Springer Verlag, New York.
43. PANDE, H., LANDI, W., AND RYDER, B. 1994. Interprocedurai def-use associations in C programs. IEEE Trans. Softw. Eng. 20, 5 (May), 385-403.
44. PANKO, R. 1995. Finding spreadsheet errors: Most spreadsheet errors have design flaws that may lead to long-term miscalculation. Information Week, 100.
45. PANKO, R. 1998. What we know about spreadsheet errors. J. End User Comput. 10, 2 (Spring), 15-21.
46. RAMAMOORTHY, C., Ho, S., AND CHEN, W. 1976. On the automated generation of program test data. IEEE Trans. Softw. Eng. 2, 4 (Dec.), 293-300.
47. RAPPS, S. AND WEYUKER E. 1985. Selecting software test data using data flow information. IEEE Trans. Softw. Eng. 11, 4 (Apr.), 367-375.
48. RAZ, O., KOOPMAN, P., AND SHAW, M. 2002. Semantic anomaly detection on online data sources. In Proceedings of the International Conference on Software Engineering (Orlando, Fla). ACM Press, New York, 302-312.
49. ROTHERMEL, G., BURNETT, M., LI, L., DUPUIS AND SHERETOV, A. 2001. A methodology for testing spreadsheets. ACM Trans. Softw. Eng. Methodol. 10, 1 (Jan.), 110-147.
50. ROTHERMEL, G., LI, L., AND BURNETT, M. 1997. Testing strategies for form-based visual programs. In Proceedings of the 8th International Symposium on Software Reliability Engineering (Albuquerque, N. Mex.). IEEE Computer Society Press, Los Alamitos, Calif., 96-107.
51. ROTHERMEL, G., LI, L., DUPUIS, C., AND BURNETT, M. 1998. What You See Is What You Test: A methodology for testing form-based visual programs. In Proceedings of the 20th International Conference on Software Engineering (Kyoto). IEEE Computer Society Press, Los Alamitos, Calif., 198-207.
52. ROTHERMEL, K., COOK, C., BURNETT, M., SCHONFELD, J., GREEN, T., AND ROTHERMEL, G. 2000. WYSI-WYT testing in the spreadsheet paradigm: An empirical evaluation. In Proceedings of the 22nd International Conference on Software Engineering (Limerick, Ireland). ACM Press, New York.
53. SCOTT, A. 2003. Shurgard stock dives after auditor quits over company's accounting. The Seattle Times.
54. SMEDLEY, T., Cox, P., AND BYRNE, S. 1996. Expanding the utility of spreadsheets through the integration of visual programming and user interface objects. In Proceedings of the Conference on Advanced Visual Interfaces '96 (Gubbio, Italy). ACM Press, New York.
55. SMITH, R. 2004. University of Toledo loses $2.4m in projected revenue. The Toledo Blade.
56. SY, N. AND DEVILLE, Y. 2003. Consistency techniques for interprocedural test data generation. In Proceedings of the ESEC/FSE'03 (Helsinki). ACM Press, New York, 108-117.
57. VIEHSTAEDT, G. AND AMBLER, A. 1992. Visual representation and manipulation of matrices. J. Visual Lang. Comput. 3, 3 (Sept.), 273-298.
58. VISSER, W., PASAREANU, C., AND KHURSHID, S. 2004. Test input generation with Java Pathfinder. In Proceedings of the ACM International Symposium on Software Testing and Analysis (Boston, Mass). ACM Press, New York, 97-107.
59. WEYUKER, E. 1993. More experience with dataflow testing. IEEE Trans. Softw. Eng. 19, 9 (Sept.), 912-919.
60. WILCOX, E., ATWOOD, J., BURNETT, M., CADIZ, J., AND COOK, C. 1997. Does continuous visual feedback aid debugging in direct-manipulation programming systems? In Proceedings of the ACM CHI'97 (Atlanta, Ga). ACM Press, New York, 22-27.
61. WILSON, A., BURNETT, M., BECKWITH, L., GRANATIR, O., CASBURN, L., COOK, C., DURHAM, M., AND ROTHERMEL, G. 2003. Harnessing curiosity to increase correctness in end-user programming. In Proceedings of the ACM Conference on Human Factors in Computing Systems (Ft. Lauderdale, Fla). ACM Press, New York, 305-312.