Crossover Designs in Software Engineering Experiments: Benefits and Perils

IEEE Transactions on Software Engineering

Volumn 42, Issue 2, 2016, Pages 120-135

  Vegas, Sira ^a   Apa, Cecilia ^b   Juristo, Natalia ^a,c  

^a UNIVERSIDAD POLITÉCNICA DE MADRID   (Spain)

^b UNIVERSIDAD DE LA REPÚBLICA   (Uruguay)

^c UNIVERSITY OF OULU   (Finland)

Author keywords

carryover; controlled experiment; crossover design; data analysis; Experimental software engineering

Indexed keywords

DATA REDUCTION; SOFTWARE ENGINEERING;

CARRYOVER; CONTROLLED EXPERIMENT; CROSSOVER DESIGN; CROSSOVER EXPERIMENTS; EXPERIMENTAL SOFTWARE ENGINEERING; GOOD PRACTICES; RISK AVOIDANCE; SOFTWARE ENGINEERING EXPERIMENTS;

DESIGN;

EID: 84962866799     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2015.2467378     Document Type: Article

References (38)

1. S. Abrahao, C. Gravino, E. Insfran, and G. Scanniello, "Assessing the effectiveness of sequence diagrams in the comprehension of functional requirements: Results from a family of five experiments, " IEEE Trans. Softw. Eng., vol. 39, no. 3, pp. 327-342, Mar. 2013.
2. V. R. Basili and R. W. Selby, "Comparing the effectiveness of software testing strategies, " IEEE Trans. Softw. Eng., vol. SE-13, no. 12, pp. 1278-1296, Dec. 1987.
3. B. Beizer, Software Testing Techniques, 2nd ed. Boston, MA, USA: Int. Thomson Comput. Press, 1990.
4. D. Binkley, M. Davis, D. Lawrie, J. I. Maletic, C. Morrell, and B. Sharif, "The impact of identifier style on effort and comprehension, " Empirical Softw. Eng., vol. 18, pp. 219-276, 2014.
5. M. Ceccato, M. Di Penta, P. Falcarin, F. Ricca, M. Torchiano, and P. Tonella, "A family of experiments to assess the effectiveness and efficiency of source code obfuscation techniques, " Empirical Softw. Eng., vol. 19, pp. 1040-1074, 2014.
6. T. J. Cleophas. Human Experimentation. Methodologic Issues Fundamental to Clinical Trials. Norwell, MA, USA: Kluwer, 1999.
7. J. Cornfield and R. T. O'Neill, "Minutes of the food and drug administration, " Biostatics and Epidemiology Advisory Committee Meeting, Jun. 23, 1976.
8. G. D'Angelo, D. Potvin, and J. Turgeon, "Carryover effects in bioequivalence studies, " J. Biopharmaceutical Statist., vol. 11, pp. 27-36. 2001.
9. A. C. Dias-Neto and G. H. Travassos, "Supporting the combined selection of model-based testing techniques, " IEEE Trans. Softw. Eng., vol. 40, no. 10, pp. 1025-1041, Oct. 2014.
10. P. D. Ellis, The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results. Cambridge, U.K.: Cambridge Univ. Press, 2010.
11. J. L. Fleiss, "A critique of recent research on the two treatment crossover design, " Controlled Clinical Trials; vol. 10, pp. 237-244, 1989.
12. P. Freeman, "The performance of the two-stage analysis of twotreatment, two-period crossover trials, " Statist. Med., vol. 8, pp. 1421-1432, 1989.
13. O. S. Gómez, N. Juristo, and S. Vegas, "Understanding replication of experiments in software engineering: A Classification, " Inf. Softw. Technol., vol. 56, no. 8, pp. 1033-1048. 2014.
14. A. P. Grieve, "A Bayesian analysis of the two-period crossover design for clinical trials, " Biometrics, vol. 41, pp. 979-990, 1985.
15. J. E. Grizzle, "The two-period change-over design and its use in clinical trials, " Biometrics, vol. 21, no. 2, pp. 467-480, Jun. 1965.
16. "Guidance for industry E9 statistical principles for clinical trials, " U. S. Dept. Health and Human Services, Food and Drug Administration, Sep. 1998.
17. M. Hills and P. Armitage, "Two-period crossover clinical trial, " Brit. J. Clinical Pharmacol., vol. 8, pp. 7-20, 1979.
18. P. K. Ito, "Robustness of ANOVA and MANOVA test procedures, " in Handbook of Statistics, vol. 1, P. R. Krishnaiah, Ed. Amsterdam, The Netherlands: Elsevier, 1980, pp 199-236.
19. A. Jedlitschka, M. Ciolkowski, and D. Pfahl, "Reporting experiments in software engineering, " in Guide to Advanced Empirical Software Engineering, F. Shull, J. Singer, and D. I. K. Sjøberg, Eds. London, U.K.: Springer, 2008, pp. 201-228.
20. B. Jones and M. G. Kennard, Design and Analysis of Crossover Trials. London, U.K.: Chapman & Hall, 1989.
21. N. Juristo and A. M. Moreno, Basics of Software Engineering Experimentation. New York, NY, USA: Springer, 2001.
22. E. Kamsties and C. M. Lott, "An empirical evaluation of three defect-detection techniques, " in Proc. 5th Eur. Softw. Eng. Conf., 1995, pp. 362-383.
23. B. Kitchenham, J. Fry, and S. G. Linkman, "The case against crossover designs in software engineering, " in Proc. 11th Int. Workshop Softw. Technol. Eng. Practice, 2003, pp. 65-67.
24. T. Kosar, M. Mernik, and J. C. Carver, "Program comprehension of domain-specific and general-purpose languages: Comparison using a family of experiments, " Empirical Softw. Eng., vol. 17, pp. 276-304, 2012.
25. R. O. Kuehl. Design of Experiments: Statistical Principles of Research Design and Analysis, 2nd ed. Pacific Grove, CA, USA: Duxbury Thomson Learning, 2000.
26. R. Latorre, "Effects of developer experience on learning and applying unit test-driven development, " IEEE Trans. Softw. Eng., vol. 40, no. 4, pp. 381-395, Apr. 2014.
27. O. A. L. Lemos, F. C. Ferrari, F. F. Silveira, and A. Garcia, "Development of auxiliary functions: Should you be agile? An Empirical assessment of pair programming and test-first programming, " in Proc. 34th Int. Conf. Softw. Eng., Zurich, Switzerland, 2012, pp. 529-539.
28. G. Levine and S. Parkinson, Experimental Methods in Psychology. Hove, U.K.: Psychol. Press, 1993.
29. C. E. McCulloch, S. R. Searle, and J. M. Neuhaus, Generalized, Linear, and Mixed Models, 2nd ed. New York, NY, USA: Wiley-Interscience, 2008.
30. G. J. Myers, T. Badgett, and C. Sandler, The Art of Software Testing, 2nd ed. New York, NY, USA: Wiley-Interscience, 2004.
31. V. Pankratius, F. Schmidt, and F. Garretón, "Combining functional and imperative programming for multicore software: An empirical study evaluating scala and java, " in Proc. Int. Conf. Softw. Eng., Zurich, Switzerland, 2012, pp. 123-133.
32. M. Roper, M. Wood, and J. Miller, "An empirical evaluation of defect detection techniques, " Inf. Softw. Technol., vol. 39, pp. 763-775, 1997.
33. N. J. Salkind, Encyclopedia of Research Design, vol. 1. Newbury Park, CA, USA: SAGE, 2010.
34. S. Senn, Crossover Trials in Clinical Research. New York, NY, USA: Wiley, 2002.
35. W. R. Shadish, T. D. Cook, and D. T. Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference, 2nd ed. Boston, MA, USA: Cengage Learning, 2002.
36. M. Shepperd, D. Bowes, and T. Hall, "Researcher bias: The use of machine learning in software defect prediction, " IEEE Trans. Softw. Eng., vol. 40, no. 6, pp. 603-616, Jun. 1, 2014.
37. W. Trochim, J. P. Donnelly, and K. Arora, Research Methods: The Essential Knowledge Base. Boston, MA, USA: Cengage Learning, 2015.
38. C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslen, Experimentation in Software Engineering. New York, NY, USA: Springer, 2010.