An empirical analysis of flaky tests

Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering

Volumn 16-21-November-2014, Issue , 2014, Pages 643-653

  Luo, Qingzhou ^a   Hariri, Farah ^a   Eloussi, Lamyaa ^a   Marinov, Darko ^a  

^a University of Illinois at Urbana Champaign   (United States)

Author keywords

Empirical study; Flaky tests; Non determinism

Indexed keywords

OPEN SOURCE SOFTWARE; REGRESSION ANALYSIS; SOFTWARE DESIGN; SOFTWARE ENGINEERING; TESTING;

COMMON STRATEGY; EMPIRICAL ANALYSIS; EMPIRICAL STUDIES; NON DETERMINISM; OPEN SOURCE PROJECTS; REGRESSION TESTING; ROOT CAUSE; SOFTWARE CHANGE;

SOFTWARE TESTING;

EID: 84986877728     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2635868.2635920     Document Type: Conference Paper

References (37)

1. API design wiki - OrderOfElements. http://wiki.apidesign.org/wiki/OrderOfElements.
2. Android FlakyTest annotation. http://goo.gl/e8PILv.
3. Apache Software Foundation SVN Repository. http://svn. apache.org/repos/asf/.
4. Apache Software Foundation. HBASE-2684. https://issues.apache.org/jira/browse/HBASE-2684.
5. A. Bachmann, C. Bird, F. Rahman, P. T. Devanbu, and A. Bernstein. The missing links: bugs and bug-fix commits. In FSE, 2010.
6. E. T. Barr, T. Vo, V. Le, and Z. Su. Automatic detection of floating-point exceptions. In POPL, 2013.
7. J. Bell and G. Kaiser. Unit test virtualization with VMVM. In ICSE, 2014.
8. N. Bettenburg, W. Shang, W. M. Ibrahim, B. Adams, Y. Zou, and A. E. Hassan. An empirical study on inconsistent changes to code clones at the release level. SCP, 2012.
9. B. Daniel, V. Jagannath, D. Dig, and D. Marinov. ReAssert: Suggesting repairs for broken unit tests. In ASE, 2009.
10. O. Edelstein, E. Farchi, E. Goldin, Y. Nir, G. Ratsaby, and S. Ur. Framework for Testing Multi-Threaded Java Programs. CCPE, 2003.
11. E. Farchi, Y. Nir, and S. Ur. Concurrent bug patterns and how to test them. In IPDPS, 2003.
12. Flakiness dashboard HOWTO. http://goo.gl/JRZ1J8.
13. M. Fowler. Eradicating non-determinism in tests. http://goo.gl/cDDGmm.
14. P. Guo, T. Zimmermann, N. Nagappan, and B. Murphy. Characterizing and predicting which bugs get fixed: an empirical study of Microsoft Windows. In ICSE, 2010.
15. P. Gupta, M. Ivey, and J. Penix. Testing at the speed and scale of Google, 2011. http://goo.gl/2B5cyl.
16. V. Jagannath, M. Gligoric, D. Jin, Q. Luo, G. Rosu, and D. Marinov. Improved multithreaded unit testing. In FSE, 2011.
17. Jenkins RandomFail annotation. http://goo.gl/tzyC0W.
18. G. Jin, L. Song, W. Zhang, S. Lu, and B. Liblit. Automated atomicity-violation fixing. In PLDI, 2011.
19. JUnit and Java7. http://goo.gl/g4crZL.
20. F. Lacoste. Killing the gatekeeper: Introducing a continuous integration system. In Agile, 2009.
21. T. Lavers and L. Peters. Swing Extreme Testing. 2008.
22. Z. Li, L. Tan, X. Wang, S. Lu, Y. Zhou, and C. Zhai. Have things changed now?: An empirical study of bug characteristics in modern open source software. In ASID, 2006.
23. Y. Lin and D. Dig. CHECK-THEN-ACT misuse of Java concurrent collections. In ICST, 2013.
24. S. Lu, S. Park, E. Seo, and Y. Zhou. Learning from mistakes: A comprehensive study on real world concurrency bug characteristics. In ASPLOS, 2008.
25. P. Marinescu, P. Hosek, and C. Cadar. Covrig: A framework for the analysis of code, test, and coverage evolution in real software. In ISSTA, 2014.
26. A. M. Memon and M. B. Cohen. Automated testing of GUI applications: models, tools, and controlling flakiness. In ICSE, 2013.
27. J. Micco. Continuous integration at Google scale, 2013. http://goo.gl/0qnzGj.
28. B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of Unix utilities. CACM, 1990.
29. K. Muşlu, B. Soran, and J. Wuttke. Finding bugs by isolating unit tests. ESEC/FSE, 2011.
30. E. R. Murphy-Hill, T. Zimmermann, C. Bird, and N. Nagappan. The design of bug fixes. In ICSE, 2013.
31. Spring Repeat Annotation. http://goo.gl/vnfU3Y.
32. P. Sudarshan. No more flaky tests on the Go team. http://goo.gl/BiWaE1.
33. 6 tips for writing robust, maintainable unit tests. http://blog.melski.net/tag/unit-tests.
34. TotT: Avoiding flakey tests. http://goo.gl/vHE47r.
35. R. Tzoref, S. Ur, and E. Yom-Tov. Instrumenting where it hurts: An automatic concurrent debugging technique. In ISSTA, 2007.
36. G. Yang, S. Khurshid, and M. Kim. Specification-based test repair using a lightweight formal method. In FM, 2012.
37. S. Zhang, D. Jalali, J. Wuttke, K. Muslu, M. Ernst, and D. Notkin. Empirically revisiting the test independence assumption. In ISSTA, 2014.