Evolving readable string test inputs using a natural language model to reduce human oracle cost

Proceedings - IEEE 6th International Conference on Software Testing, Verification and Validation, ICST 2013

Volumn , Issue , 2013, Pages 352-361

  Afshan, Sheeva ^a   McMinn, Phil ^a   Stevenson, Mark ^a  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Crowd Sourced Human Study; Language Model; Search Based Testing; Strings

Indexed keywords

AUTOMATICALLY GENERATED; HUMAN READABILITY; HUMAN STUDY; LANGUAGE MODEL; NATURAL LANGUAGE MODEL; OPEN SOURCES; SEARCH-BASED TESTING; STRINGS;

COMPUTATIONAL LINGUISTICS; COSTS; SOFTWARE TESTING;

RESEARCH;

EID: 84883415791     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICST.2013.11     Document Type: Conference Paper

References (35)

1. M. Harman, S. G. Kim, K. Lakhotia, P. McMinn, and S. Yoo, "Optimizing for the number of tests generated in search based test data generation with an application to the oracle cost problem," International Workshop on Search-Based Software Testing (SBST 2010), pp. 182-191.
2. P. McMinn, M. Stevenson, and M. Harman, "Reducing qualitative human oracle costs associated with automatically generated test data," International Workshop on Software Test Output Validation (STOV 2010), pp. 1-4.
3. P. McMinn, "Search-based software test data generation: A survey," Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105-156, 2004.
4. M. Harman and P. McMinn, "A theoretical and empirical study of searchbased testing: Local, global and hybrid search," IEEE Transactions on Software Engineering, vol. 36, pp. 226-247, 2010.
5. P. Godefroid, N. Klarlund, and K. Sen, "DART: Directed Automated Random Testing," ACM SIGPLAN Notices, vol. 40, no. 6, pp. 213-223, June 2005.
6. N. Tillmann and J. de Halleux, "Pex-white box test generation for .NET," Tests and Proofs (TAP 2008), pp. 134-153.
7. H. Manning and H. Sch?utze, Foundations of Statistical Natural Language Processing. Cambridge, MA: MIT Press, 1999.
8. A. Lopez, "Statistical machine translation," ACM Computing Surveys, vol. 40, no. 3, pp. 8:1-8:49, 2008.
9. D. Jurafsky and J. Martin, Speech and Language Processing, 2nd ed. Pearson, 2009.
10. A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu, "On the naturalness of software," International Conference on Software Engineering (ICSE 2012), pp. 837-847.
11. S. Katz, "Estimation of probabilities from sparse data for the language model component of a speech recognizer," IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 35, no. 5, pp. 400-401, 1987.
12. A. Stolcke, "SRILM-an extensible language modeling toolkit," in International Conference on Spoken Language Processing, pp. 257-286, 2002.
13. H. Melvile, Moby Dick. Harper and Brothers, 1851.
14. A. Arcuri, "It does matter how you normalise the branch distance in search based software testing," International Conference on Software Testing, Verification and Validation (ICST 2010), pp. 205-214.
15. "Android code style guidelines, http://source.android.com/source/ codestyle. html#write-short-methods.
16. P. McMinn, "IGUANA: Input generation using automated novel algorithms. A plug and play research tool," Department of Computer Science, University of Sheffield, Tech. Rep. CS-07-14, 2007.
17. K. T. Stolee and S. Elbaum, "Exploring the use of crowdsourcing to support empirical studies in software engineering," International Symposium on Empirical Software Engineering and Measurement (ESEM 2010).
18. Z. P. Fry and W. Weimer, "A human study of fault localization accuracy," International Conference on Software Maintenance (ICSM 2010), pp. 1-10.
19. Z. P. Fry, B. Landau, and W. Weimer, "A human study of patch maintainability," International Symposium on Software Testing and Analysis (ISSTA 2012), pp. 177-187.
20. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng., "Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks," Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 254-263, 2008.
21. A. Kittur, E. H. Chi, and B. Suh, "Crowdsourcing user studies with mechanical turk," SIGCHI conference on Human Factors in Computing, pp. 453-456, 2008.
22. A. Vargha and H. Delaney, "A critique and improvement of the CL common language effect size statistics of McGraw and Wong," Journal of Educational and Behavioral Statistics, vol. 25, no. 2, pp. 101-132, 2000.
23. R. S. Boyer, B. Elspas, and K. N. Levitt, "SELECT-A formal system for testing and debugging programs by symbolic execution," International Conference on Reliable Software, pp. 234-244, 1975.
24. C. S. P?as?areanu, P. C. Mehlitz, D. H. Bushnell, K. Gundy-Burlet, M. Lowry, S. Person, and M. Pape, "Combining unit-level symbolic execution and systemlevel concrete execution for testing NASA software," International Symposium on Software Testing and Analysis (ISSTA 2008), pp. 15-26.
25. N. Li, T. Xie, N. Tillmann, J. de Halleux, and W. Schulte, "Reggae: Automated test generation for programs using complex regular expressions," International Conference on Automated Software Engineering (ASE 2009), pp. 515-519.
26. M. Alshraideh and L. Bottaci, "Search-based software test data generation for string data using program-specific search operators," Software Testing, Verification and Reliability, vol. 16, no. 3, pp. 175-203, 2006.
27. V. Ganesh, A. Kiezun, S. Artzi, P. J. Guo, P. Hooimeijer, and M. D. Ernst, "HAMPI: a string solver for testing, analysis and vulnerability detection," International Conference on Computer Aided Verification (CAV 2011), pp. 1-19.
28. P. Saxena, D. Akhawe, S. Hanna, F. Mao, S. McCamant, and D. Song, "A symbolic execution framework for JavaScript," IEEE Symposium on Security and Privacy, pp. 513-528, 2010.
29. P. McMinn, M. Shahbaz, and M. Stevenson, "Search-based test input generation for string data types using the results of web queries," International Conference on Software Testing, Verification and Validation (ICST 2012), pp. 141-150.
30. M. Shahbaz, P. McMinn, and M. Stevenson, "Automated Discovery of Valid Test Strings from the Web using Dynamic Regular Expressions Collation and Natural Language Processing," International Conference on Quality Software (QSIC 2012), pp. 79-88.
31. M. Bozkurt and M. Harman, "Automatically generating realistic test input from web services," International Symposium on Service-Oriented System Engineering (SOSE 2011), pp. 13-24.
32. G. Fraser and A. Zeller, "Exploiting common object usage in test case generation," International Conference on Software Testing, Verification and Validation (ICST 2011), pp. 80-89.
33. M. J. Harrold, R. Gupta, and M. L. Soffa, "A methodology for controlling the size of a test suite," ACM Transactions on Software Engineering and Methodology, vol. 2, no. 3, pp. 270-285, 1993.
34. J. Ferrer, F. Chicano, and E. Alba, "Evolutionary algorithms for the multiobjective test data generation problem," Software Practice and Experience, vol. 42, pp. 1331-1362, 2012.
35. G. Fraser and A. Arcuri, "Evolutionary generation of whole test suites," in International Conference on Quality Software (QSIC 2011), pp. 31-40.