Automated metamorphic testing of variability analysis tools

Software Testing Verification and Reliability

Volumn 25, Issue 2, 2015, Pages 138-163

  Segura, Sergio ^a   Durán, Amador ^a   Sánchez, Ana B ^a   Le Berre, Daniel ^b   Lonca, Emmanuel ^b   Ruiz Cortés, Antonio ^a  

^a UNIVERSITY OF SEVILLE   (Spain)

^b UNIVERSITÉ D'ARTOIS   (France)

Author keywords

Automated testing; Metamorphic testing; Software testing; Software variability

Indexed keywords

APPLICATION PROGRAMS; AUTOMATION; BOOLEAN ALGEBRA; PROGRAM DEBUGGING; WELL TESTING;

AUTOMATED DETECTION; AUTOMATED TESTING; CONFIGURATION OPTIONS; METAMORPHIC RELATIONS; METAMORPHIC TESTING; SOFTWARE APPLICATIONS; SOFTWARE VARIABILITIES; VARIABILITY ANALYSIS;

SOFTWARE TESTING;

EID: 84924231632     PISSN: 09600833     EISSN: 10991689     Source Type: Journal    
DOI: 10.1002/stvr.1566     Document Type: Article

References (70)

1. Svahnberg M, van Gurp L, Bosch J. A taxonomy of variability realization techniques: research articles. Software Practice and Experience 2005; 35(8):705-754.
2. Debian 7.0 wheezy, Released May 2013, Accessed November 2013.
3. Eclipse marketplace. Available from: http://marketplace.eclipse.org/[last accessed November 2013].
4. García-Galán J, Rana O, Trinidad P, Ruiz-Cortés A. Migrating to the cloud: a software product line based analysis. In 3rd International Conference on Cloud Computing and Services Science (CLOSER'13), Aachen, Germany, 2013.
5. Kang K, Cohen S, Hess J, Novak W, Peterson S. Feature-oriented domain analysis (FODA) feasibility study. Technical Report CMU/SEI-90-TR-21, SEI, Pittsburgh, Pennsylvania, USA, 1990.
6. Pohl K, Bückle G, van der Linden F. Software Product Line Engineering: Foundations, Principles, and Techniques. Springer-Verlag: New York, USA, 2005.
7. Stoiber R, Glinz M. Supporting stepwise, incremental product derivation in product line requirements engineering. International Workshop on Variability Modelling of Software-intensive Systems 2010; 37:77-84.
8. Berger T, She S, Lotufo R, Wasowski A, Czarnecki K. Variability modeling in the real: a perspective from the operating systems domain. International Conference on Automated Software Engineering (ASE'10), Antwerp, Belgium, 2010; 73-82.
9. Le Berre D, Rapicault P. Dependency management for the eclipse ecosystem: eclipse p2, metadata and resolution. Proceedings of the 1st international Workshop on Open Component Ecosystems, IWOCE '09, ACM: New York, NY, USA, 2009; 21-30.
10. Müller C, Resinas M, Ruiz-Cortés A. Automated analysis of conflicts in WS-agreement documents. IEEE Transactions on Services Computing 2013; 7:530-544.
11. Jang M. Linux Annoyances for Geeks, O'Reilly, 2006.
12. FaMa tool suite. Available from: http://www.isa.us.es/fama/[last accessed November 2013].
13. Mendonca M, Branco M, Cowan D. S.P.L.O.T.: software product lines online tools. Companion to the 24th ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), ACM: Orlando, Florida, USA, 2009; 761-762.
14. Software product lines automated reasoning library (SPLAR). Available from: http://code.google.com/p/splar/[last accessed November 2013].
15. Veer B, Dallaway J. The eCos component writer's guide. Available from: http://ecos.sourceware.org/ecos [last accessed November 2013].
16. Debian reference guide. Available from: http://www.debian.org/doc/manuals/debian-reference/[Accessed November 2013].
17. Benavides D, Segura S, Ruiz-Cortés A. Automated analysis of feature models 20 years later: a literature review. Information Systems 2010; 35(6):615-636.
18. Weyuker E. On testing non-testable programs. The Computer Journal 1982; 25(4):465-470.
19. Chen T, Cheung S, Yiu S. Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01, University of Science and Technology: Hong Kong, 1998.
20. Chen T, Feng J, Tse T. Metamorphic testing of programs on partial differential equations: a case study. In Proceedings of the 26th International Computer Software and Applications Conference, Oxford, UK, 2002; 327-333.
21. Chen T, Huang D, Tse T, Zhou Z. Case studies on the selection of useful relations in metamorphic testing. In Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC), Madrid, Spain, 2004; 569-583.
22. Chan W, Cheung S, Leung K. A metamorphic testing approach for online testing of service-oriented software applications. International Journal of Web Services Research 2007; 4(2):61-81.
23. Segura S, Hierons R, Benavides D, Ruiz-Cortés A. Automated test data generation on the analyses of feature models: a metamorphic testing approach. International Conference on Software Testing, Verification and Validation, IEEE press: Paris, France, 2010; 35-44.
24. Segura S, Hierons R, Benavides D, Ruiz-Cortés A. Automatedmetamorphic testing on the analyses of feature models. Information and Software Technology 2011; 53:245-258.
25. Argelich L, Le Berre D, Lynce I, Silva J, Rapicault P. Solving Linux upgradeability problems using Boolean optimization. In Workshop on Logics for Component Configuration, Vol. 29 of EPTCS, Lynce I, Treinen R (eds). Edinburgh, UK, 2010; 11-22.
26. Treinen R, Zacchirol S. Common upgradeability description format (CUDF) 2.0. Technical Report 003, The Mancoosi project (FP7): Paris, France, 2009.
27. Le Berre D, Parrain A. On SAT technologies for dependency management and beyond. In First workshop on Analyses of Software Product Lines 2008; 2:197-200.
28. Mendonca M, Wasowski A, Czarnecki K. SAT-based analysis of feature models is easy. In Proceedings of the International Sofware Product Line Conference (SPLC), San Francisco, CA, USA, 2009.
29. Benavides D, Ruiz-Cortés A, Trinidad P. Automated reasoning on feature models. In 17th International Conference on Advanced Information Systems Engineering (CAiSE), vol. 3520, Lecture Notes in Computer Sciences. Springer- Verlag: Berlin Heidelberg, Germany, 2005; 491-503.
30. Wang H, Li Y, Sun J, Zhang H, Pan J. Verifying feature models using OWL. Journal of Web Semantics 2007; 5: 117-129.
31. Durán A, Benavides D, Segura S, Trinidad P, Ruiz-Cortés A. FLAME: FAMA formal framework (v 1.0). Technical Report ISA-12-TR-02, ETS Ingeniería Informítica. University of Seville: Seville, Spain, March 2012.
32. Mancoosi European research project. Available from: http://www.mancoosi.org/[last accessed November 2013].
33. Gebser M, Kaminski R, Schaub T. aspcud: a linux package configuration tool based on answer set programming. In Workshop on Logics for Component Configuration, Vol. 65, Drescher C, Lynce I, Treinen R (eds). EPTCS, 2011; 12-25. http://about.eptcs.org/.
34. Zhou Z, Huang D, Tse T, Yang Z, Huang H, Chen T. Metamorphic testing and its applications. Proceedings of the 8th International Symposium on Future Software Technology, Xi'an, China, 2004; 346-351.
35. Liu H, Kuo F, Towey D, Chen T. How effectively does metamorphic testing alleviate the oracle problem? IEEE Transactions on Software Engineering 2014; 40(1):4-22.
36. Mayer J, Guderlei R. An empirical study on the selection of good metamorphic relations. Proceedings of the 30th Annual International Computer Software and Applications Conference - Volume 01, IEEE Computer Society: Washington, DC, USA, 2006; 475-484.
37. Galindo J, Benavides D, Segura S. Debian packages repositories as software product line models. Towards automated analysis. In Proceedings of the 1st International Workshop on Automated Configuration and Tailoring of Applications (ACoTA), Antwerp: Belgium; 2010.
38. Segura S, Duran A, Sänchez A, Le Berre D, Lonca E, Ruiz-Cortés A. Automated metamorphic testing on the analysis of software variability. Technical Report ISA-2013-TR-03, ISA Research Group: Seville, Spain, 2013. Available from: http://www.lsi.us.es/segura/files/papers/segura13-TR-03.pdf.
39. Segura S, Galindo J, Benavides D, Parejo J, Ruiz-Cortás A. BeTTy: benchmarking and testing on the automated analysis of feature models. In Sixth International Workshop on Variability Modelling of Software-Intensive Systems (VaMoS'12), Eisenecker U, Apel S, Gnesi S (eds). ACM: Leipzig, Germany, 2012; 63-71.
40. p2cudf. Available from: http://wiki.eclipse.org/Equinox/p2/CUDFResolver [last accessed November 2013].
41. Le Berre D, Parrain A. The Sat4j library, release 2.2. Journal on Satisfiability, Boolean Modeling and Computation 2010; 7:59-64. system description.
42. aspcud. Available from: http://www.cs.uni-potsdam.de/wv/aspcud [last accessed November 2013].
43. Cudf-tools debian package. Available from: http://packages.debian.org/wheezy/cudf-tools [last accessed November 2013].
44. Lingeling SAT solver. Available from: http://fmv.jku.at/lingeling/[last accessed November 2013].
45. Eén N, Sörensson N. An extensible SAT-solver. In SAT, Vol. 2919 of Lecture Notes in Computer Science, Giunchiglia E, Tacchella A (eds). Springer: Berlin Heidelberg, Germany, 2003; 502-518.
46. Gebser M, Kaufmann B, Neumann A, Schaub T. Clasp: a conflict-driven answer set solver. In Lecture Notes in Computer Science, vol. 4483, Baral C, Brewka G, Schlipf JS (eds), LPNMR. Springer: Berlin Heidelberg, Germany, 2007; 260-265.
47. Biere A. Picosat essentials. JSAT 2008; 4(2-4):75-97.
48. Pipatsrisawat K, Darwiche A. A lightweight component caching scheme for satisfiability solvers. In SAT, vol. 4501 of, Marques-Silva J, Sakallah KA (eds), Lecture Notes in Computer Science. Springer: Berlin Heidelberg, Germany, 2007; 294-299.
49. Heule M. SmArT solving: tools and techniques for satisfiability solvers. PhD thesis, TU Delft, 2008.
50. Dubois O, Dequen G. A backbone-search heuristic for efficient solving of hard 3-SAT formulae. In IJCAI, Nebel B (ed.). Morgan Kaufmann: San Francisco, CA, USA, 2001; 248-253.
51. Brummayer R, Lonsing F, Biere A. Automated testing and debugging of SAT and QBF solvers. Proceedings of the 13th international conference on Theory and Applications of Satisfiability Testing, SAT'10, Springer-Verlag: Berlin, Heidelberg, 2010; 44-57.
52. Wohlin C, Runeson P, Hst M, Ohlsson MC, Regnell B, Wessln A. Experimentation in Software Engineering. Springer: Berlin Heidelberg, Germany, 2012.
53. Czarnecki K, Wasowski A. Feature diagrams and logics: there and back again. 11th International Software Product Line Conference (SPLC), IEEE Computer Society: Los Alamitos, CA, USA, 2007; 23-34.
54. Van Gelder A. Producing and verifying extremely large propositional refutations - have your cake and eat it too. Annals of Mathematics and Artificial Intelligence 2012; 65(4):329-372.
55. Brummayer R, Järvisalo M. Testing and debugging techniques for answer set solver development. Journal of Theory and Practice of Logic Programming 2010; 10(4-6):741-758.
56. Artho C, Biere A, Seidl M. Model-based testing for verification back-ends. 7th International Conference on Tests & Proofs, Springer: Budapest, Hungary, 2013.
57. Cook S, Mitchell D. Finding hard instances of the satisfiability problem: a survey. In Satisfiability Problem: Theory and Applications, volume 35 of Dimacs Series in Discrete Mathematics and Theoretical Computer Science. American Mathematical Society: Washington, DC, USA, 1997; 1-17.
58. Segura S, Benavides D, Ruiz-Cortés A. Functional testing of feature model analysis tools: a test suite. Software, IET 2011; 5(1):70-82.
59. Segura S, Parejo J, Hierons R, Benavides D, Ruiz-Cortés A. Automated generation of computationally hard feature models using evolutionary algorithms. Expert Systems with Applications 2014; 41(8):3975-3992.
60. Di Cosmo R, Vouillon J. On software component co-installability. 13th European conference on Foundations of Software Engineering, ESEC/FSE '11, ACM: New York, NY, USA, 2011; 256-266.
61. Artho C, Suzaki K, Di Cosmo R, Treinen R, Zacchiroli S. Why do software packages conflict? 9th IEEE Working Conference of Mining Software Repositories, Zurich, Switzerland, 2012; 141-150.
62. Kuo F, Zhou Z, Ma J, Zhang G. Metamorphic testing of decision support systems: a case study. Software, IET 2010; 4(4):294-301.
63. Liu H, Liu X, Chen T. A new method for constructing metamorphic relations. In 12th International Conference on Quality Software (QSIC), Xi'an, China, 2012; 59-68.
64. Wu P. Iterative metamorphic testing. 29th Annual International Computer Software and Applications Conference, Vol. 1, COMPSAC, Edinburgh, UK, 2005; 19-24.
65. Xie X, Wong W, Chen T, Xu B. Metamorphic slice: an application in spectrum-based fault localization. Information and Software Technology 2013; 55(5):866-879.
66. Anand S, Burke E, Chen TY, Clark J, Cohen M, Grieskamp W, Harman M, Harrold M, McMinn P. An orchestrated survey on automated software test case generation. Journal of Systems and Software August 2013; 86(8):1978-2001.
67. Chen T, Kuo F, Merkel R, Tse T. Adaptive random testing: the art of test case diversity. Journal of Systems and Software Jan. 2010; 83(1):60-66.
68. Cohen M, Dwyer M, Jiangfan S. Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. Software Engineering, IEEE Transactions on 2008; 34(5):633-650.
69. Nie C, Leung H. A survey of combinatorial testing. ACM Computing Surveys 2011; 43(2):11:1-11:29.
70. Utting M, Pretschner A, Legeard B. A taxonomy of model-based testing approaches. Software Testing Verification and Reliability 2012; 22(5):297-312.