Automatic programming assessment and test data generation: A review on its approaches

Proceedings 2010 International Symposium on Information Technology - System Development and Application and Knowledge Society, ITSim'10

Volumn 3, Issue , 2010, Pages 1186-1192

  Romli, Rohaida ^a   Sulaiman, Shahida ^b   Zamli, Kamal Zuhairi ^c  

^a UNIVERSITI UTARA MALAYSIA   (Malaysia)

^b SCHOOL OF PHYSICS   (Malaysia)

^c SCHOOL OF CHEMICAL ENGINEERING   (Malaysia)

Author keywords

Automatic programming assessment; Program testing; Test data generation

Indexed keywords

AUTOMATED METHODS; DYNAMIC TESTING; INTEGRAL PART; MAIN GAP; PROGRAM TESTING; PROGRAMMING COURSE; PROGRAMMING EXERCISE; PROGRAMMING SOLUTIONS; QUALITY PROGRAMS; RESEARCH AREAS; RESEARCH TOPICS; TEST DATA GENERATION;

COMPUTER SOFTWARE SELECTION AND EVALUATION; DATA COMMUNICATION SYSTEMS; INFORMATION TECHNOLOGY; SOFTWARE TESTING; STUDENTS; TEACHING; TESTING;

AUTOMATIC PROGRAMMING;

EID: 78049378289     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ITSIM.2010.5561488     Document Type: Conference Paper

References (100)

1. th SIGCSE Technical Symposium on Computer Science Education, San Jose, CA, 1997, pp. 335-339.
2. A. M. Zin, and E. Foxley, "Automatic Program Assessment System", 1994. Available: http://www.cs.nott.ac.uk/~ceilidh/papers/ASQA. html. Retrieved on: 10 Dec 2008.
3. nd Edition, Pearson-Addison Wesley, USA, 1997.
4. D. Gelperin, and B. Hetzel, "The Growth of Software Testing", Communication of ACM, Vol. 31, Issue 6, pp. 687-695, 1998.
5. K. M. Ala-Mutka, "A survey of Automated Assessment Approaches for Programming Assignments", Computer Science Education, Vol. 15, Issue 2, pp. 83-102, 2005.
6. C. Douce, "Automatic Test-based Assessment of Programming: A Review", Journal on Educational Resources in Computing, Vol. 5, Issue 3, 2006.
7. K.A. Rahman and M. D. Nordin, "A Review on the Static Analysis Approach in the Automated Programming Assessment Systems, Proceedings of National Conference on Programming 07, Kuala Lumpur, 2007.
8. M. Joy, N. Griffiths, and R. Boyatt, "The BOSS Online Submission and Assessment System", ACM Journal on Educational Resources in Computing, Vol. 5, No. 3, Article 2, 2005.
9. Z. Shukur, "The Automatic Assessment of Z Specification", PhD Thesis. University of Nottingham, 1999.
10. J. Hollingsworth, "Automatic Graders for Programming Classes", Communication of the ACM, Vol. 3, Issue 10, pp. 528-529, 1960.
11. G. E. Forsythe and N. Wirth, "Automatic Grading Programs", Communications of the ACM, Vol. 8, Issue 5, pp. 275-278, 1965.
12. M. J. Rees, "Automayic Assessment Aids for Pascal Programs", ACM SIGPLAN Notices, Vol. 17, Issue 10, pp.33-42, 1982.
13. P. B. Van-Verth, "A System for Automatically Grading Program Quality", Technical Report. SUNNY (Buffalo), 1985.
14. S. L. Hung, L. F. Kmok, and R. Chan R., "Automatic Programming Assessment", Computers and Education. Vol. 20, Issue 2, pp. 183-190, 1993.
15. M. Joy and M. Luck, "A User-Friendly On-Line Submission System", Proceeding of the 4th Annual Conference on Teaching of Computing, Dublin City University, 1996, pp. 92-95.
16. D. Jackson, "A Software System for Grading Student Computer Programs", Computers and Education, Vol. 27, Issue 3-4, pp. 171-180, 1996.
17. E. Foxley, C. Higgins and C. Gibbon, "The Ceilidh System: A General Overview 1996", 1996. Available: http://ww.cs.nott.ac.uk/~cmp/moreinfo/ html/Overview96.htm, Retrieved on: 10 Feb 2009.
18. J. P. Leal and N. Moreira, "Automatic Grading of Programming Exercises", Technical Report (DCC-98-4), University de Porto, Portugal, 1998.
19. U. V. Matt, "Kassandra: The Automatic Grading System", ACM SIGCUE Outlook, Vol. 22, Issue 1, pp 26-40, 1998.
20. D. Jackson, "A Semi-Automated Approach to Online Assessment", ACM SIGCSE Bulletin, Vol. 32, Issue 3, pp. 164-167, 2000.
21. A. Zeller, "Making Students Read and Review Code", ACM SIGCSE Bulletin, Vol. 32, Issue 3, pp. 89-92, 2000.
22. R. Saikkonen, L. Malmi, and A. Korhonen, "Fully Automatic Assessment of Programming Exercises", ACM SIGCSE Bulletin, Vol. 33, Issue 3, pages 133-136, 2001.
23. E. Odekirk-Hash and J. L. Zachary," Automated Feedback on Programs Means Students Need Less Help form Teacher", ACM SIGSCE Bulletin, Vol. 33, Issue 1, pp. 55-59, 2001.
24. th Australian Conference on Computing Education, Adelaide, Australia, Vol. 20, 2001, pp. 267-272.
25. C. Higgins, P. Symeonidis and A. Tsintsifas, "The Marking System for CourseMaster", ACM SIGCSE Bulletin, Vol. 34, Issue 3, pp. 46-50, 2002.
26. B. Cheang, A. Kurnia, A. Lim and W. C. Oon, "On automated grading of programming assignments in an academic institution", Computers & Education, Vol. 41, pp. 121-131, 2003.
27. J. A. Harris, E. S. Adams, and N. L. Harris, "Making Program Grading Easier (But Not Totally Automatic. Journal of Computing Sciences in Colleges", Vol. 20, Issue 1, pp. 248-261, 2004.
28. M. Blumenstein, S. Green, A. Nguyen, and V. Muthukkumarasamy, "GAME: A Generic Automated Marking Environment for Programming Assessment", Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC'04), Vol. 2, 2004, pp. 212-216.
29. N. Truong, P. Bancroft, and P. Roe, "Learning to Program Through the Web", ACM SIGCSE Bulletin, Vol. 37, Issue 3, pp. 9-13, 2005.
30. A. Allowatt and S. Edwards, "IDE Support for Test-Driven Development and Automated Grading in Both Java and C++", Proceeding of OOPSLA'05 Eclipse Technology Exchange Workshop, San Deigo, California, 2005, pp. 100-104.
31. S. C. Shaffer, "Ludwig: An Online Programming Tutoring and Assessment System", ACM SIGCSE Bulletin, Vol. 37, Issue 2, pp. 56-60, 2005.
32. P. Brusilovsky and S. Sosnovsky, "Individualized Exercise for Self-Assessment of Programming Knowledge: An Evaluation of QuizPACK", Journal on Educational Resources in Computing, Vol. 5, Issue 3, 2005.
33. J. Baldwin, E. Crupi, and T. Estrellado, "WeBWork for Programming Fundamentals. Proceedings of the 11th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, Bologna, Italy, 2006, pp. 361-361.
34. th International Symposium on Principles and Practice of programming in Java, Mannheim, Germany, 2006, pp. 208-211.
35. M. Choy, U. Nazir, C. K. Poon, and Y. T. Yu, "Experiences in Using an Automated System for Improving Students' of Computer Programming", Lecture Notes in Computer Science Learning. Springer Berlin/Heidelberg, 2005, pp. 267 - 272.
36. C. A. Higgins, G. Gray, P. Symeonidis, and A. Tsintsifas, "Automated Assessment and Experiences of Teaching Programming", Journal of Educational Resources in Computing, Vol. 5, Issue 3, Article 5, 2006.
37. K. A. Rahman, S. Ahmad, and M. J. Nordin, "The Design of an Automated C Programming Assessment Using Pseudo-code Comparison Technique", Proceedings of National Conference on Software Engineering and Computer Systems 2007, Pahang, Malaysia, 2007.
38. P. Nordquist, "Providing Accurate and Timely Feedback by Automatically Grading Student Programming Labs", Journal of Computing Sciences in Colleges, Vol. 23, Issue 2, pp. 16-23, 2007.
39. G. Li, W. Wu, Y. Sun, J. Wang, and T. Lai, "Transformation-based Assessment for C Programs", Proceedings of 9th International Symposium on Signal Processing and its Applications, Sharjah, U.A.E., 2007.
40. B. Auffarth, M. Lopez-Sanchez, J. C. Miralles, and A. Puig, "System for Automated Assistance in Correction of Programming Exercises (SAC)", In proceedings of the fifth CIDUI - V International Congress of University Teaching and Innovation, 2008.
41. D. Jiménez-González, C. Álvarez, D. López, J. M. Parcerisa, J. Alonso, C. Pérez, "Work in Progress-Improving Feedback Using an Automatic Assessment Tool", 38th ASEE/IEEE Frontiers in Education Conference, Saratoga Springs, NY, 2007.
42. G. Rossling and S. Hartte, "WebTask: Online Programming Exercises Made Easy", Proceedings of ITiCSE'08, Madrid, Spain, 2008, pp. 363.
43. H. Suleman, "Automatic Marking with Sakai", Proceedings of SAICSIT, South Africa, 2008, pp. 229-236.
44. A. Gerdes, J. Jeuring, and B. Heeren, "Using Strategies for Assessment of programming Exercises", Technical Report UU-CS-2009-031, Utrecht University, Dec 2009.
45. C. M. Tang, Y. T. Yua, and C. K. Poon, "An Approach Towards Automatic Testing of Student Programs Using Token Patterns", Proceedings of the 17th International Conference on Computers in Education, Hong Kong, 2009, pp. 118-190.
46. J. Sant, "'Mailing It In': Email-Centric Automated Assessment", Proceedings of ITiCSE'09, Paris, France, 2009, pp. 308-312.
47. K. A. Naudé, J. H. Greyling, and D. Vogts, "Marking student programs using graph similarity", Computers & Education, Vol. 54, pp. 545-561, 2010.
48. R. Ferguson and B. Korel, "The chaining approach for software test data generation", ACM Transactions on Software Engineering and Methodology, Vol. 5, No. 1, pp. 63-86, 1996.
49. nd Conference on Computer Science and Engineering in Linkoping, 1999, pp. 21-28.
50. S. Mahmod, "A Systematic Review of Automated Test Data Generation Techniques", Master Thesis, Blekinge Institute of Tech., Sweden, 2007.
51. P. McMinn, "Search-based Software Test Data Generation: A Survey", Soft. Testing, Verification & Reliability, Vol. 14, No.2, pp. 105-156, 2004.
52. M. J. Gallagher and V. Lakshmi-Narasimhan, "ADTEST: a test data generation suite for ada software systems", IEEE Transactions on Software Engineering, Vol. 23 (8), pp. 473-484, 1997.
53. S. Kirkpatrick, C. D. Gellat, and M. P. Vecchi, "Optimization by simulated annealing", Science, Vol. 220(4598), pp. 671-680, 1983.
54. M. Dorigo, G. D. Caro, and M. Samples, "Ant Algorithms: third international workshop, ANTS 2002 Brussels, Belgium, Sept 2002, Proceedings", Lecture Notes in CS, Vol. 2463, Springer-Verlag Berlin Heidelberg, 2002.
55. F. Glover, "Tabu Search Part I", ORSA Journal Computing, Vol.1, No. 3, pp. 190-206, 1989.
56. J. H. Holland, Adaptation in Natural and Artificial Systems, University of Michigan Press, Ann Arbor, 1975.
57. W. Miller W. and D. L. Spooner, "Automatic Generation of Floating-Point Test Data", IEEE Trans on Software Engineering, Vol. SE-2, No. 3, 1976.
58. L. A. Clarke, "A system to generate test data and symbolically execute programs", IEEE Trans. Software Eng., Vol. 2, Issue 3, pp. 215-222, 1976.
59. R. A. DeMillo and A. J. Offutt, "Constraint-Based Automatic Test Data Generation", IEEE Transactions on Software Engineering, Vol. 12, No. 9, pp. 900-909, 1991.
60. A. J. Offutt, Z. Jin and J. Pan, "The Dynamic Domain Reduction Procedure for Test Data Generation: Design and Algorithms", Technical Report ISSE, August 1994.
61. H. D. Chu, J. E. Dobson and I. C. Liu, "FAST: A Framework for Automating Statistical-based Testing", Software Quality Journal, Vol.6, pp. 13-36, 1997.
62. C. Michael and G. McGraw, "Opportunism and Diversity in Automated Software Test Data Generation", Technical Report Reliable Software Technologies, Sterling, VA, 8 Dec 1997.
63. N. Gupta, A. P. Mathur, and M. L. Soffa, "Automated Test Data Generation Using an Iterative Relaxation Method", ACM SIGSOFT Software Engineering Notes, Vol. 23, Issue 6, pp. 231-245, 1998.
64. A. Gotlieb, B. Botella and M. Reuher, "Automatic Test Data Generation using Constraint Solving Techniques", Proceeding of ISSTA 98, 1998, pp. 53-62.
65. A. Hajnal and I. Forgacs, "An Applicable Test Data Generation Algorithm for Domain Errors", Proceedings of the 1998 ACM SIGSOFT international symposium on Software testing and analysis, Folrida, USA, 1998, pp. 63-72.
66. B. Korel and A. M. Al-Yami, "Assertion-oriented automated test data generation", Proceedings of the 18th international conference on Software engineering, Berlin, Germany, 1996, pp. 71 - 80.
67. R. P. Pargas, M. J. Harrold and R. R. Peck, "Test-Data Generation Using Genetic Algorithms", Journal of Software Testing, Verification and Reliability, Vol. 9, Issue 4, pp. 263-282, 1999.
68. M. Roper, I. Maclean, A. Brooks, J. Miller and M. Wood, "Genetic Algorithms and the Automatic Generation of Test Data", 1999. Available: http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.52.2394. Retrieved on 23 Mac 2009.
69. A. Gargantini and C. Heitmeyer, "Using model checking to generate tests from requirements specifications", ACM SIGSOFT Software Engineering Notes, Vol. 24, Issue 6, pp. 146-162, 1999.
70. J. Offutt and S. Liu "Generating Test Data from SOFL Specifications", The Journal of Systems and Software. Vol. 49, No. 1, pp. 49-62, 1999.
71. C. C. Micheal, G. McGraw and M. A. Schatz, "Generating Software Test Data by Evolution", IEEE Trans on Software Engineering, Vol. 27, No. 12, pp. 1085-1109, 2001.
72. N. G. Tracey, "A Search-Based Automated Test-Data Generation Framework for Safety Critical Software", PhD Thesis, University of York, UK, 2000.
73. J. C. Lin and P. L. Yeh, "Automatic Test Data Generation for Path Testing Using GAs", Elsevier Information Sciences 131, 2001, pp. 47-64.
74. th IEEE International Conference on Automated SE, 2003, pp. 310-313.
75. I. Hermadi and M. A. Ahmed, "Genetic algorithm based test data generator", Appears in the 2003 Congress on Evolutionary Computation, Vol. 1, pp. 85-91, 2003.
76. J. Offutt, S. Liu, A. Abdurazik and P. Ammann, "Generating test data from state-based specifications", Software Testing, Verification And Reliability, Vol. 13, pp. 25-53, 2003.
77. N. T. Sy and Y. Deville, "Consistency Techniques for Inter-procedural Test Data Generation", Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering, Helsinki, Finland, 2003, pp. 108-117.
78. I. Hermadi, "Genetic algorithm based test data generator", Master Thesis, King Fahd University of Petroleum and Minerals, 2004.
79. P. McMinn and M. Holcombe, "Hybridizing Evolutionary Testing with the Chaining Approach", Lecture Note in CS, Springer Berlin/Heidelberg, 2004.
80. th IEEE International Conference on Tools with AI, 2004, pp. 174 - 182.
81. H. Gao, B. Feng, and L. Zhu, "A kind of SaaGA Hybrid Meta-heuristic Algorithm for the Automatic Test Data Generation", Proceedings of International Conference on Neural Net. & Brain, Vol. 1, 2005, pp. 13-15.
82. Y. Zhan Y., "A Search-Based Framework for Automatic Test-Set Generation for MATHLAB/Simulink Models", PhD Thesis, University of York, UK, 2005.
83. R. M. Hierons, "Avoiding Coincidental Correctness in Boundary Value Analysis", ACM Transactions on Software Engineering and Methodology, Vol. 15, No. 3, 2006, pp. 227-241.
84. T. Xie and J. Zhao, "A Framework and Tool Supports for Generating Test Inputs of AspectJ Programs", Proceedings of the 5th international conference on Aspect-oriented software development, Bonn, Germany, 2006, pp. 190-201.
85. nd IEEE International Conference on Software Maintenance, 2006, pp. 113 - 122.
86. K. Ayari, S. Bouktif, and G. Antoniol, "Automatic Mutation Test Input Data Generation via Ant Colony", Proceedings of the 9th annual conference on Genetic and Evolutionary Computation, London, England, 2007, pp. 1074 - 1081.
87. B. L. Li, Z. S. Li, J. Y. Zhang and J. R. Sun, "An Automated Test Case Generation Approach by Genetic Simulated Annealing Algorithm", Proceedings of the Third International Conference on Natural Computation, Vol. 4, 2007, pp. 106-111.
88. K. Z. Zamli, N. A. M Isa, M. F. J. Klaib and S. N. Azizan, "Tool for Automated Test Data Generation (and Execution) Based on Combinatorial Approach", International Journal of Software Engineering and Its Applications, Vol. 1, No.1, 2007.
89. J. H. Andrews, F. C. H. Li and T. Menzies, "Nighthawk: A Two-level Genetic-Random Unit Test Data Generator", Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, Georgia, USA, 2007, pp. 144.
90. P. M. S. Bueno, W. E. Wong, and M. Jino, "Automatic Test Data Genration Using Particle Systems", Proceedings of the 2008 ACM Symposium on Applied Computing, 2008, pp. 809-814.
91. F. Pinte, N. Oster, and F. Siagletti, "Techniques and Tools for the Automatic Generation of Optimal Test Data at Code, Model and Interface Level", Proceedings of ICSE'08, Leipzig, Germany, 2008, pp. 927-928.
92. A. A. Sofokleous and A. S. Andreou, "Automatic, evolutionary test data generation for dynamic software testing", The Journal of Systems and Software, Vol. 81, pp. 1883-1898, 2008.
93. M. Harman, F. Islam, T. Xie and S. Wappler, "Automated Test Data Generation for Aspect-Oriented Programs", Proceedings of the 8th ACM international conference on Aspect-oriented software development, Charlottesville, Virginia, USA, 2009, pp. 185-196.
94. Y. Cao, C. Hu and L. Li, "Search-based Multi-paths Test Data Generation for Structure-oriented Testing", Proceedings of the first ACM/SIGEVO Summit on Genetic and Evolutionary Computation, Shanghai, China, 2009, pp. 25-32.
95. P. R. Srivastava, P. Gupta, Y. Arrawatia and S. Yadav, "Use of Genetic Algorithm in Generation of Feasible Test Data", SIGSOFT Software Engineering Notes, Vol. 34, No. 2, pp. 1-4. 2009.
96. Z. Awedikian, K. Ayari and G. Antoniol, "MC/DC Automatic Test Input Data Generation", Proceedings of the 11th Annual conference on Genetic and evolutionary computation, Montreal, Québec, Canada, 2009, pp. 1657-1664.
97. M. Alshraideh, L. Bottaci and B. A. Mahafzah, "Using program data-state scarcity to guide automatic test data generation", Software Quality Journal, Vol. 18, No.1, pp. 109-144, 2010.
98. L. Malmi, V. Karavirta, A. Korhonen, J. Nikander, O. Seppala, and P. Silvasti, "Visual Algorithm Simulation Exercise System with Automatic Assessment: TRAKLA2", Informatics in Education, Vol. 3, Issue 2, pp. 267-288, 2004.
99. Z. Shukur, R. Romli, and A. B. Hamdan, "Skema Penjanaan Data dan Pemberat Ujian Berasaskan Kaedah Analisis Nilai Sempadan (A Schema of Generating Test Data and Test Weight Based on Boundary Value Analysis Technique), Technology Journal, Issue 42(D), June 2005, pp. 23-40.
100. P. Ihantola, "Automatic Test Data Generation for Programming Exercises with Symbolic Execution and Java PathFinder", Master Thesis of Helsinki University of Technology, 2006.