A Methodology for Testing Spreadsheets

ACM Transactions on Software Engineering and Methodology

Volumn 10, Issue 1, 2001, Pages 110-147

  Rothermel, Gregg ^a   Burnett, Margaret ^a   Li, Lixin ^a   Dupuis, Christopher ^a   Sheretov, Andrei ^a  

^a Oregon State University   (United States)

Author keywords

Algorithms; D.1.7 Programming Techniques : Visual Programming; D.2.5 Software Engineering : Testing and Debugging; D.2.6 Software Engineering : Programming Environments; H.4.1 Information Systems Applications : Office Automation; Languages

Indexed keywords

EID: 0000377320     PISSN: 1049331X     EISSN: None     Source Type: Journal    
DOI: 10.1145/366378.366385     Document Type: Article

References (55)

1. AHO, A., SETHI, R., AND ULLMAN, J. 1986. Compilers: Principles, Techniques, and Tools. Addison-Wesley, Reading, MA.
2. AMBLER, A. L., BURNETT, M. M., AND ZIMMERMAN, B. A. 1992. Operational versus definitional: A perspective on programming paradigm. IEEE Computer 25, 9 (Sept.), 28-43.
3. AZEM, A., BELLI, F., JACK, O., AND JEDRZEJOWICZ, P. 1993. Testing and reliability of logic programs. In Proceedings of the Fourth International Symposium on Software Reliability Engineering. 318-327.
4. BELLI, F. AND JACK, O. 1995. A test coverage notion for logic programming. In Proceedings of the 6th IEEE International Symposium on Software Reliability Engineering (ISSRE '95, Toulouse, France). IEEE Computer Society, Washington, DC, 133-142.
5. BROWN, P. AND GOULD, J. 1987. Experimental study of people creating spreadsheets. ACM Trans. Off. Inf. Syst. 5, 3 (July), 258-272.
6. BURNETT, M. M. AND GOTTFRIED, H. J. 1998. Graphical definitions: Expanding spreadsheet languages through direct manipulation and gestures. ACM Trans. Comput. Hum. Interact. 5, 1 (Mar.), 1-33.
7. BURNETT, M., ATWOOD, J., AND WELCH, Z. 1998. Implementing level 4 liveness in declarative visual programming languages. In Proceedings of the IEEE Symposium on Visual Languages (Sept.).
8. BURNETT, M., HOSSLI, R., PULLIAM, T., VANVOORST, B., AND YANG, X. 1994. Toward visual programming languages for steering in scientific visualization: A taxonomy. IEEE Comput. Sci. Eng. 1, 4, 44-62.
9. BURNETT, M., SHERETOV, A., AND ROTHERMEL, G. 1999. Scaling up a "What you see is what you test" methodology to spreadsheet grids. In Proceedings of the IEEE Symposium on Visual Languages (Sept.). 30-37.
10. CHI, E. H.-H., KONSTAN, J., BARRY, P., AND RIEDL, J. 1997. A spreadsheet approach to information visualization. In Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology (UIST '97, Banff, Alberta, Canada, Oct. 14-17), G. Robertson and C. Schmandt, Chairs. ACM Press, New York, NY, 79-80.
11. CHRIST, R. 1975. Review and analysis of color coding research for visual displays. Hum. Factors 17, 6, 542-570.
12. CLARKE, L. A. 1976. A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Eng. SE-2, 3, 215-222.
13. CLARKE, L. A., PODGURSKI, A., RICHARDSON, D. J., AND ZEIL, S. J. 1989. A formal evaluation of data flow path selection criteria. IEEE Trans. Softw. Eng. 15, 11 (Nov.), 1318-1332.
14. DUESTERWALD, E., GUPTA, R., AND SOFFA, M. L. 1992. Rigorous data flow testing through output influences. In Proceedings of the 2nd Symposium on Irvine Software (Mar.).
15. DUPUIS, C. AND BURNETT, M. 1997. An animated turing machine simulator in Forms/3. i97-60-08 (July).
16. FERRANTE, J., OTTENSTEIN, K. J., AND WARREN, J. D. 1987. The program dependence graph and its use in optimization. ACM Trans. Program. Lang. Syst. 9, 3 (July), 319-349.
17. FRANKL, P. AND WEISS, S. 1993. An experimental comparison of the effectiveness of branch testing and data flow testing. IEEE Trans. Softw. Eng. 19, 8 (Aug.), 774-787.
18. FRANKL, P. G. AND WEYUKER, E. J. 1988. An applicable family of data flow testing criteria. IEEE Trans. Softw. Eng. 14, 10 (Oct.), 1483-1498.
19. GREN, T. R. G. AND PETRE, M. 1996. Usability analysis of visual programming environments: A "cognitive dimensions" framework. J. Visual Lang. Comput. 7, 2, 131-174.
20. GUPTA, R., HARROLD, M. J., AND SOFFA, M. L. 1996. Program slicing-based regression testing techniques. J. Softw. Test. Verific. Reliab. 6, 2 (June), 83-112.
21. HARROLD, M. J. AND SOFFA, M. L. 1988. An incremental approach to unit testing during maintenance. In Proceedings of the Conference on Software Maintenance (Oct.). 362-367.
22. HUDSON, S. E. 1991. Incremental attribute evaluation: A flexible algorithm for lazy update. ACM Trans. Program. Lang. Syst. 13, 3 (July), 315-341.
23. HUTCHINS, M., FOSTER, H., GORADIA, T., AND OSTRAND, T. 1994. Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proceedings of the 16th International Conference on Software Engineering (ICSE '94, Sorrento, Italy, May 16-21), B. Fadini, L. Osterweil, and A. van Lamsweerde, Chairs. IEEE Computer Society Press, Los Alamitos, CA, 191-200.
24. JOHNSON, R. 1992. Elementary Statistics. 6th. Duxbury Press, Boston, MA.
25. KUHN, W. AND FRANK, A. U. 1997. The use of functional programming in the specification and testing process. In Proceedings of the International Conference and Workshop on Interoperating Geographic Information Systems (Dec.).
26. LANDI, W. 1992. Undecidability of static analysis. ACM Lett. Program. Lang. Syst. 1, 4 (Dec.), 323-337.
27. LANDI, W. AND RYDER, B. G. 1991. Pointer-induced aliasing: A problem taxonomy. In Proceedings of the 18th Annual ACM Symposium on Principles of Programming Languages (POPL '91, Orlando, FL, Jan. 21-23), D. Wise, Chair. ACM Press, New York, NY, 93-103.
28. LANDI, W. AND RYDER, B. 1992. A safe approximate algorithm for interprocedural pointer aliasing. In Proceedings of the 5th ACM SIGPLAN Conference on Programming Language Design and Implementation (SIGPLAN '92, San Francisco, CA, June 17-19), R. L. Wexelblat, Ed. ACM Press, New York, NY, 235-248.
29. LASKI, J. AND KOREL, B. 1983. A data flow oriented program testing strategy. IEEE Trans. Softw. Eng. 9, 3 (May), 347-354.
30. LEOPOLD, J. AND AMBLER, A. 1997. Keyboardless visual programming using voice, handwriting, and gesture. In Proceedings of the IEEE Symposium on Visual Languages (VL97, Capri, Italy, Sept.). IEEE Computer Society Press, Los Alamitos, CA, 28-35.
31. LUO, G., BOCHMANN, G., SARIKAYA, B., AND BOYER, M. 1992. Control-flow based testing of Prolog programs. In Proceedings of the 3rd International Symposium on Software Reliability Engineering. 104-113.
32. MARLOWE, T. AND RYDER, B. 1990. An efficient hybrid algorithm for incremental data flow analysis. Prin. Prog. Lang. (Jan.), 184-196.
33. MURCH, G. M. 1984. Physiological principles for the effective use of color. IEEE Comput. Graph. Appl. 4, 11 (Nov.), 49-54.
34. MYERS, B. A. 1991. Graphical techniques in a spreadsheet for specifying user interfaces. In Proceedings of the Conference on Human Factors in Computing Systems: Reaching through Technology (CHI '91, New Orleans, LA, Apr. 27-May 2), S. P. Robertson, G. M. Olson, and J. S. Olson, Eds. ACM Press, New York, NY, 243-249.
35. NTAFOS, S. C. 1984. On required element testing. IEEE Trans. Softw. Eng. 10, 6 (Nov.), 132-139.
36. OFFUTT, A. J., PAN, J., TEWARY, K, AND ZHANG, T. 1996. An experimental evaluation of data flow and mutation testing. Softw. Pract. Exper. 26, 2, 165-176.
37. OUABDESSELAM, F. AND PARISSIS, I. 1995. Testing techniques for data-flow synchronous programs. In Proceedings of the Second International Workshop on Automated and Algorithmic Debugging (AADEBUG'95, May).
38. PANKO, R. AND HALVERSON, R. 1996. Spreadsheets on trial: A survey of research on spreadsheet risks. In Proceedings of the 29th Annual Hawaii International Conference on System Sciences (ICSS '96, Maui, Hawaii, Jan.). IEEE Computer Society Press, Los Alamitos, CA.
39. PERRY, D. E. AND KAISER, G. E. 1990. Adequate testing and object-oriented programming. J. Object Oriented Program. 2, 5 (Jan./Feb.), 13-19.
40. POLLOCK, L. L. AND SOFFA, M. L. 1989. An incremental version of iteractive data flow analysis. IEEE Trans. Softw. Eng. 15, 12 (Dec.), 1537-1549.
41. RAPPS, S. AND WEYUKER, E. J. 1985. Selecting software test data using data flow information. IEEE Trans. Softw. Eng. SE-11, 4 (Apr.), 367-375.
42. REICHWEIN, J., ROTHERMEL, G., AND BURNETT, M. 1999. Slicing spreadsheets: An integrated methodology for spreadsheet testing and debugging. In Proceedings of the 2nd Conference on Domain Specific Languages (Oct.). 25-38.
43. ROTHERMEL, G. AND HARROLD, M. J. 1997. A safe, efficient regression test selection technique. ACM Trans. Softw. Eng. Methodol. 6, 2, 173-210.
44. ROTHERMEL, G. AND HARROLD, M. J. 1994. Selecting tests and identifying test coverage requirements for modified software. In Proceedings of the 1994 International Symposium on Software Testing and Analysis (ISSTA '94, Seattle, WA, Aug. 17-19), T. Ostrand, Ed. ACM Press, New York, NY, 169-184.
45. ROTHERMEL, G., LI, L., AND BURNETT, M. 1997. Testing strategies for form-based visual programs. In Proceedings of the 8th International Symposium on Software Reliability Engineering (ISSRE 97, Nov.). 96-107.
46. ROTHERMEL, G., LI, L., DUPUIS, C., AND BURNETT, M. 1998. What you see is what you test: A methodology for testing form-based visual programs. In Proceedings of the 20th International Conference on Software Engineering (ICSE '98, Kyoto, Japan, Apr.). IEEE Press, Piscataway, NJ, 198-207.
47. SHNEIDERMAN, B. 1998. Designing the User Interface: Strategies for Effective Human-Computer Interaction. 3rd. Addison-Wesley, Reading, MA.
48. SMEDLEY, T., COX, P., AND BYRNE, S. 1996. Expanding the utility of spreadsheets through the integration of visual programming and user interface objects. In Proceedings of the Conference on Advanced Visual Interfaces (AVI'96, Gubbio, Italy, May), T. Catarci, M. F. Costabile, S. Levialdi, and G. Santucci, Eds. ACM Press, New York, NY.
49. VIEHSTAEDT, G. AND AMBLER, A. 1992. Visual representation and manipulation of matrices. J. Visual Lang. Comput. 3, 3 (Sept.), 273-298.
50. WEYUKER, E. J. 1982. On testing non-testable programs. Computer J. 15, 4, 465-470.
51. WEYUKER, E J. 1986. Axiomatizing software test data adequacy. IEEE Trans. Softw. Eng. SE-12, 12 (Dec.), 1128-1138.
52. WEYUKER, E. J. 1993. More experience with dataflow testing. IEEE Trans. Softw. Eng. 19, 9 (Sept.), 912-919.
53. WILCOX, E. M., ATWOOD, J. W., BURNETT, M. M., CADIZ, J. J., AND COOK, C. R. 1997. Does continuous visual feedback aid debugging in direct-manipulation programming systems? In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '97, Atlanta, GA, Mar. 22-27), S. Pemberton, Ed. ACM Press, New York, NY, 258-265.
54. WONG, W. E., HORGAN, J. R., LONDON, S., AND MATHUR, A. P. 1995. Effect of test set minimization on fault detection effectiveness. In Proceedings of the 17th International Conference on Software Engineering (ICSE-17, Seattle, WA, Apr. 23-30), D. Perry, Chair. ACM Press, New York, NY, 41-50.
55. YANG, S., BURNETT, M., DEKOVEN, E., AND ZLOOF, M. 1997. Representation design bench-marks: A design-time aid for VPL navigable static representations. J. Visual Lang. Comput. 8, 5/6 (Oct/Dec), 563-599.