On the dataset shift problem in software engineering prediction models

Empirical Software Engineering

Volumn 17, Issue 1-2, 2012, Pages 62-74

  Turhan, Burak ^a  

^a UNIVERSITY OF OULU   (Finland)

Author keywords

Dataset shift; Defect prediction; Effort estimation; Prediction models

Indexed keywords

DATA SETS; DEFECT PREDICTION; EFFORT ESTIMATION; ENGINEERING COMMUNITY; ENGINEERING PREDICTION; FAULT PREDICTION; IMBALANCED DATA; PREDICTION MODEL; PREDICTION MODELS; RESEARCH OUTCOME; SAMPLE SELECTION BIAS; SOFTWARE EFFORT; TEST DATA; TRAINING DATA;

MATHEMATICAL MODELS; SOFTWARE ENGINEERING;

DATA PROCESSING;

EID: 84857374320     PISSN: 13823256     EISSN: 15737616     Source Type: Journal    
DOI: 10.1007/s10664-011-9182-8     Document Type: Article

References (34)

1. Alpaydin E (2010) Introduction to machine learning, 2nd edn. The MIT Press, Cambridge, MA
2. A BakIr B Turhan A Bener 2010 A new perspective on data homogeneity in software cost estimation: a study in the embedded systems domain Softw Qual J 18 1 57 80 10.1007/s11219-009-9081-z
3. S Bickel M Brückner T Scheffer 2009 Discriminative learning under covariate shift J Mach Learn Res 10 2137 2155 2550104
4. Boehm B, Horowitz E, Madachy R, Reifer D, Clark BK, Steece B, Brown AW, Chulani S, Abts C (2000) Software cost estimation with Cocomo II. Prentice Hall, Englewood Cliffs, NJ
5. L Briand J Wust 2002 Empirical studies of quality models in object-oriented systems Adv Comput 56 97 166 10.1016/S0065-2458(02)80005-5
6. LC Briand WL Melo J Wust 2002 Assessing the applicability of fault-proneness models across object-oriented software projects IEEE Trans Softw Eng 28 706 720 10.1109/TSE.2002.1019484
7. Candela JQ, Sugiyama M, Schwaighofer A, Lawrence ND (eds) (2009) Dataset shift in machine learning. The MIT Press, Cambridge, MA
8. V Chandola A Banerjee V Kumar 2009 Anomaly detection: a survey ACM Comput Surv 41 3 15:1 15:58 10.1145/1541880.1541882
9. O Demirors C Gencel 2009 Conceptual association of functional size measurement methods IEEE Softw 26 3 71 78 10.1109/MS.2009.60
10. C Drummond RC Holte 2006 Cost curves: an improved method for visualizing classifier performance Mach Learn 65 1 95 130 10.1007/s10994-006-8199-5 (Pubitemid 44451195)
11. Guo P, Lyu MR (2000) Software quality prediction using mixture models with EM algorithm. In: Proceedings of the the first Asia-Pacific conference on quality software (APAQS'00). IEEE Computer Society, Washington, DC, USA, pp 69-78
12. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The WEKA data mining software: an update. SIGKDD explorations, vol 11/1
13. DJ Hand 2006 Classifier technology and the illusion of progress Stat Sci 21 1 1 15 2275965 10.1214/088342306000000060 (Pubitemid 44046906)
14. Huang J, Smola AJ, Gretton A, Borgwardt KM, Schšlkopf B (2006) Correcting sample selection bias by unlabeled data. Neural Information Processing Systems, pp 601-608
15. Y Jiang B Cukic Y Ma 2008 Techniques for evaluating fault prediction models Empir Soft Eng 13 5 561 595 10.1007/s10664-008-9079-3
16. Jiang Y, Cukic B, Menzies T (2008b) Cost curve evaluation of fault prediction models. In: Proceedings of the 19th int'l symposium on software reliability engineering (ISSRE 2008), Redmond, WA, pp 197-206
17. JW Keung BA Kitchenham DR Jeffery 2008 Analogy-X: providing statistical inference to analogy-based software cost estimation IEEE Trans Softw Eng 34 4 471 484 10.1109/TSE.2008.34
18. BA Kitchenham E Mendes GH Travassos 2007 Cross versus within-company cost estimation studies: a systematic review IEEE Trans Softw Eng 33 5 316 329 10.1109/TSE.2007.1001 (Pubitemid 46681200)
19. Kocaguneli E, Menzies T (2011) How to find relevant data for effort estimation? In: Proceedings of the 5th ACM/IEEE international symposium on empirical software engineering and measurement (ESEM'11)
20. Kocaguneli E, Gay G, Menzies T, Yang Y, Keung JW (2010) When to use data from other projects for effort estimation. In: Proceedings of the IEEE/ACM international conference on automated software engineering (ASE '10). ACM, New York, pp 321-324
21. Lin J, Keogh E, Lonardi S, Lankford J, Nystrom DM (2004) Visually mining and monitoring massive time series. In: Proceedings of 10th ACM SIGKDD international conference on knowledge and data mining. ACM Press, pp 460-469 (Pubitemid 40114956)
22. C Lokan T Wright PR Hill M Stringer 2001 Organizational benchmarking using the isbsg data repository IEEE Softw 18 26 32 10.1109/52.951491 (Pubitemid 32886797)
23. T Menzies O Jalali J Hihn D Baker K Lum 2010 Stable rankings for different effort models Autom Softw Eng 17 4 409 437 10.1007/s10515-010-0070-z
24. Menzies T, Turhan B, Bener A, Gay G, Cukic B, Jiang Y (2008) Implications of ceiling effects in defect predictors. In: Proceedings of the 4th international workshop on predictor models in software engineering (PROMISE '08). ACM, New York, pp 47-54
25. I Myrtveit E Stensrud M Shepperd 2005 Reliability and validity in comparative studies of software prediction models IEEE Trans Softw Eng 31 5 380 391 10.1109/TSE.2005.58 (Pubitemid 41046926)
26. Premraj R, Zimmermann T (2007) Building software cost estimation models using homogenous data. In: Proceedings of the first international symposium on empirical software engineering and measurement (ESEM '07). IEEE Computer Society, Washington, DC, USA, pp 393-400
27. M Shepperd G Kadoda 2001 Comparing software prediction techniques using simulation IEEE Trans Softw Eng 27 11 1014 1022 10.1109/32.965341 (Pubitemid 33106029)
28. H Shimodaira 2000 Improving predictive inference under covariate shift by weighting the log-likelihood function J Stat Plan Inference 90 2 227 244 1795598 10.1016/S0378-3758(00)00115-4
29. Storkey A (2009) When training and test sets are different: characterizing learning transfer. In: Quionero-Candela J, Sugiyama M, Schwaighofer A, Lawrence ND (eds) Dataset shift in machine learning, chapter 1. The MIT Press, Cambridge, MA, pp 3-28
30. M Sugiyama T Suzuki S Nakajima H Kashima P von Bünau M Kawanabe 2008 Direct importance estimation for covariate shift adaptation Ann Inst Stat Math 60 4 699 746 10.1007/s10463-008-0197-x
31. B Turhan T Menzies AB Bener J Di Stefano 2009 On the relative value of cross-company and within-company data for defect prediction Empir Softw Eng 14 5 540 578 10.1007/s10664-008-9103-7
32. Wieczorek I, Ruhe M (2002) How valuable is company-specific data compared to multi-company data for software cost estimation? In: Proceedings of the 8th international symposium on software metrics (METRICS '02). IEEE Computer Society, Washington, DC, USA, p 237
33. Zhang H, Sheng S (2004) Learning weighted naive Bayes with accurate ranking. In: Proceedings of the 4th IEEE international conference on data mining, pp 567-570 (Pubitemid 40731109)
34. Zimmermann T, Nagappan N, Gall H, Giger E, Murphy B (2009) Cross-project defect prediction. In: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering. ACM