An experimental investigation of the impact of aggregation on the performance of data mining with logistic regression

Information and Management

Volumn 42, Issue 5, 2005, Pages 695-707

  Fadlalla, Adam ^a  

^a Cleveland State University   (United States)

Author keywords

Aggregation; Area under the ROC curve; Data mining; Data warehouse; DJIA; Logistic regression; Model assessment; Model performance; Prediction; Predictive modeling; ROC

Indexed keywords

DATA PROCESSING; DATA WAREHOUSES; DECISION MAKING; FINANCE; MARKETING; MATHEMATICAL MODELS; REGRESSION ANALYSIS;

AREA UNDER THE ROC CURVE; DATA AGGREGATION; DOW JONES INDUSTRIAL AVERAGE (DJIA); LOGISTIC REGRESSION; MODEL ASSESSMENT; MODEL PERFORMANCE; PREDICTIVE MODELING; RECEIVER OPERATING CHARACTERISTIC (ROC);

DATA MINING;

EID: 18544363237     PISSN: 03787206     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.im.2004.04.005     Document Type: Article

References (49)

1. J.R. Beck, and E.K. Shultz The use of relative operating characteristic (ROC) curves in test performance evaluation Archives of Pathology and Laboratory Medicine 110 1986 13 20
2. E. Bendoly Theory and support for process frameworks of knowledge discovery and data mining from ERP systems Information & Management 40 2003 639 647
3. R.J. Brachman, T. Khabaza, W. Kloesgen, G. Piatetsky-Shapiro, and E. Simoudis Mining business databases Communications of the ACM 39 11 Nov 1996 41 48
4. A.P. Bradley The use of the area under the ROC curve in the evaluation of machine learning algorithms Pattern Recognition 30 7 1997 1145 1159
5. K.P. Burnham, and D.R. Anderson Model Selection and Inference 1998 Springer-Verlag New York
6. U. Fayyad G. Piatetsky-Shapiro P. Smyth R. Uthurusamy Advances in Knowledge Discovery and Data Mining 1996 AAAI/MIT Press Cambridge, MA
7. P.K. Chan, S.J. Stolfo, Learning with non-uniform class and cost distributions: effects and a distributed multi-classifier approach, Work Notes KDD-98 Workshop on Distributed Data Mining (1998) 1-9.
8. S. Chaudhuri, and U. Dayal An overview of data warehousing and OLAP technology SIGMOD Record 26 1 1997 1 10
9. J. Gray, S. Chaudhuri, A. Bosworth, A. Layman, D. Reichart, M. Venkatrao, F. Pellow, and H. Pirahesh Data cube: a relation aggregation operator generalizing group-by, cross-tab, and sub-totals Data Mining and Knowledge Discovery 1 1997 29 53
10. A. Gupta, and I.S. Mumick Maintenance of materialized views: problems, techniques, and applications IEEE Data Engineering Bulletin 18 2 1995
11. J.A. Hanley, and B.J. McNeil The meaning and use of the area under a receiver operating characteristic (ROC) curve Radiology 143 1 April 1982 29 36
12. J.A. Hanley Receiver operating characteristics (ROC) methodology: the state of the art Critical Reviews in Diagnostic Imaging 29 3 1989 307 335
13. J.A. Hanley, and B.J. McNeil A method of comparing the areas under receiver operating characteristic curves derived from the same cases Radiology 148 1 1983 839 843
14. M. Hernandez, and S. Stolfo Real-world data is dirty: data cleansing and the merge/purge problem Data Mining and Knowledge Discovery 2 1 1998 9 37
15. S.C. Hui, and G. Jha Data mining for customer service support Information & Management 38 2000 1 13
16. W.H. Inmon The data warehouse and data mining Communications of the ACM 39 11 Nov 1996 49 50
17. W.H. Inmon Building the Data Warehouse 1992 John Wiley
18. N. Japkowicz, Learning from imbalanced data sets: a comparison of various strategies, AAAI 2000 Workshop on Learning from Imbalanced Data Sets, Technical Report WS-00-05, July 2000.
19. A.D. Kester, and F. Buntix Meta-analysis of ROC curves Medical Decision Making 20 4 2000 430 439
20. M. Kubat, R. Holte, and S. Matwin Learning when negative examples abound Proceedings of the European Conference on Machine Learning ECML'97, Prague 1997 146 153
21. M. Kubat, S. Matwin, Addressing the curse of imbalanced training sets: one-sided selection, in: Proceedings of the 14th International Conference on Machine Learning, ICML'97, Nashville, TN, USA, pp. 179-186.
22. A.L. Lederer, and J.R. Smith Individual differences and decision-making using various levels of aggregation of information Journal of Management Information Systems 5 3 1988-1989 53 69
23. D. Lomet, J. Widom (Eds.), Special issue on materialized views and data warehousing, IEEE Data Engineering Bulletin 18 (2) 1995.
24. S. Madnick, X. Chen, J. Funk, and R. Wang Corporate household data: research directions Proceedings of AMCIS 2001 Boston, MA 2001
25. B.G. Malkiel A Random Walk Down the Wall Street 2000 W.W. Norton & Company New York
26. B.J. McNeil, and J.A. Hanley Statistical approaches to the analysis of receiver operating characteristic (ROC) curves Medical Decision Making 4 2 1984 137 150
27. M.E. Meredith, and A. Khader Divide and aggregate: designing large warehouses Database Programming and Design June 1996 24 30
28. C.E. Metz Basic principles of ROC analysis Seminars in Nuclear Medicine 8 4 1978 283 298
29. C.E. Metz, B.A. Herman, and C.A. Roe Statistical comparison of two ROC-curve estimates obtained from partially-paired datasets Medical Decision Making 18 1 January-March 1998 110 121
30. C.E. Metz Some practical issues of experimental design and data analysis in radiological ROC studies Investigative Radiology 24 1989 234 245
31. C.E. Metz ROC methodology in radiological imaging Investigative Radiology 21 9 1986 720 733
32. A.R. Montazemi, and S. Wang The effects of models of information presentation on decision-making Journal of Management Information Systems 5 3 1988-89 101 127
33. Oracle Data Warehousing Reference Manual, 2000, p. 92.
34. F. Provost, Machine learning from imbalanced data sets 101, http://www.stern.nyu.edu/~fprovost/Papers/skew.PDF.
35. S. Salzberg On comparing classifiers: a critique of current research and methods Data Mining and Knowledge Discovery 1 1999 1 12
36. I. Spiegler Technology and knowledge: bridging a generating gap Information & Management 40 2003 533 539
37. J.A. Swets, and R.M. Pickett Evaluation of Diagnostic Systems: Methods from Signal Detection Theory 1982 Academic Press New York
38. J.A. Swets, R.M. Dawes, and J. Monahan Psychological science can improve diagnostic decisions Psychological Science in the Public Interest 1 1 May 2000 1 26
39. J.A. Swets, R.M. Dawes, and J. Monahan Better decisions through science Scientific American 2000 82 87
40. M.L. Thompson, and W. Zucchini On the statistical analysis of ROC curves Statistics in Medicine 8 1989 1277 1290
41. U.S. EPA, Handbook for use of data from the national health and nutrition examination surveys (NHANES), National Center for Environmental Assessment, Washington, DC, EPA/600/R-02/044, 2003, p. 21.
42. A. Vavouras, S. Gatziu, and K.R. Dittrich The SIRIUS approach for refreshing data warehouses incrementally Proceedings of the BTW'99 1999 80 96
43. S. Walczak An empirical analysis of data requirements for financial forecasting with neural networks Journal of Management Information Systems 17 4 Spring 2001 203 222
44. A. Weigend Analyzing customer behavior at Amazon.com Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Washington, DC, USA 24-27 August 2003 5
45. G.M. Weiss, F. Provost, The effect of class distribution on classifier learning: an empirical study, Technical Report ML-TR-44, Department of Computer Science, Rutgers University, 2 August 2001, pp. 1-26.
46. G.M. Weiss, F. Provost, The effect of class distribution on classifier learning, Technical Report ML-TR-43, Department of Computer Science, Rutgers University, 11 January 2001, pp. 1-6.
47. J. Widom Research problems in data warehousing Conference on Information and Knowledge Management (CIKM) Baltimore, Maryland, USA 1995
48. M.C. Wu, and A.P. Buchmann Research issues in data warehousing Proceedings of the BTW'97 Ulm, Germany 1997 61 82
49. M.H. Zweig, and G. Campbell Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine Clinical Chemistry 39 4 1993 561 577