On the difficulty of replicating human subjects studies in software engineering

Proceedings - International Conference on Software Engineering

Volumn , Issue , 2008, Pages 191-200

  Lung, Jonathan ^a   Aranda, Jorge ^a   Easterbrook, Steve ^a   Wilson, Greg ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

Empirical; Experience report; Human subjects; Replication

Indexed keywords

PROGRAMMING THEORY; COMPUTER SOFTWARE;

EMPIRICAL; EMPIRICAL RESULTS; EXPERIENCE REPORT; HUMAN SUBJECTS; INTRODUCTORY PROGRAMMING COURSE; MENTAL MODELS; REPLICATION;

SOFTWARE ENGINEERING; EXPERIMENTS;

EMPIRICAL; EMPIRICAL RESULTS; EXPERIENCE REPORT; HUMAN SUBJECTS; INTRODUCTORY PROGRAMMING COURSE; MENTAL MODEL; REPLICATION;

EID: 57349192830     PISSN: 02705257     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1368088.1368115     Document Type: Conference Paper

References (15)

1. V. R. Basili, F. Shull, and F. Lanubile. Building knowledge through families of experiments. IEEE Trans. Softw. Eng., pages 456-473, 1999.
2. A. Brooks, M. Roper, M. Wood, J. Daly, and J. Miller. Replication's role in software engineering. In F. Shull, J. Singer, and D. I. K. Sjøberg, editors, Guide to Advanced Empirical Software Engineering, pages 365-379. Springer, 2007.
3. J. Campbell. CSC108H1S final examination, 2006. http://eres.library. utoronto.ca.
4. M. E. Caspersen, J. Bennedsen, and K. D. Larsen. Mental models and programming aptitude. ITiCSE, 39:206-210, 2007.
5. S. Dehnadi and R. Bornat. The camel has two humps (working title), 2006. http://www.cs.mdx.ac.uk/research/PhDArea/saeed.
6. S. Dehnadi and R. Bornat. Re: The camel has two humps. Personal correspondence, 2007.
7. S. M. Easterbrook, J. Singer, M.-A. Storey, and D. Damian. Selecting empirical methods for software engineering research. In F. Shull, J. Singer, and D. I. K. Sjoberg, editors, Guide to Advanced Empirical Software Engineering, pages 285-311. Springer, 2007.
8. M. Jørgensen and D. I. K. Sjøberg. Generalization and theory building in software engineering research. In EASE'2004, pages 29-36. IEE Proceedings, 2004.
9. B. A. Kitchenham, T. Dybå, and M. Jøorgensen. Evidence-based software engineering. In ICSE '04: Proc. 26th Int. Conf. on Softw. Eng., pages 273-281, Washington, DC, USA, 2004. IEEE Computer Society.
10. J. E. McGrath. Methodology matters: doing research in the behavioral and social sciences. In R. M. Baecker, J. Grudin, W. Buxton, and S. Greenberg, editors, Human-Computer interaction: Toward the Year 2000, pages 152-169. Morgan Kaufmann Publishers, 1995.
11. F. Shull, V. Basili, J. Carver, J. C. Maldonado, G. H. Travassos, M. Mendonça, and S. Fabbri. Replicating software engineering experiments: Addressing the tacit knowledge problem. In ISESE '02: Proc. Int. Symp. on Empirical Softw. Eng., Washington, DC, USA, 2002. IEEE Computer Society.
12. J. A. Singer and N. G. Vinson. Ethical issues in empirical studies of software engineering. IEEE Trans. Softw. Eng., 28(12):1171-1180, 2002.
13. D. I. K. Sjøberg, T. Dybå, and M. Joørgensen. The future of empirical methods in software engineering research. In FOSE '07: 2007 Future of Software Engineering, pages 358-378, Washington, DC, USA, 2007. IEEE Computer Society.
14. D. I. K. Sjøberg, J. E. Hannay, O. Hansen, V. B. Kampenes, A. Karahasanovic, N.-K. Liborg, and A. C. Rekdal. A survey of controlled experiments in software engineering. IEEE Trans. Softw. Eng., 31(9):733-753, 2005.
15. R. Yin. Case Study Research: Design and Methods (3rd Edition). Sage, 2002.