Active Learning-Based Pedagogical Rule Extraction

IEEE Transactions on Neural Networks and Learning Systems

Volumn 26, Issue 11, 2015, Pages 2664-2677

  Junque De Fortuny, Enric ^a   Martens, David ^b  

^a INSEAD   (France)

^b UNIVERSITY OF ANTWERP   (Belgium)

Author keywords

Active learning; comprehensibility; neural network; random forest (RF); rule extraction; Support vector machine (SVM)

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPLEX NETWORKS; DECISION TREES; EXTRACTION; NEURAL NETWORKS; SUPPORT VECTOR MACHINES;

ACTIVE LEARNING; CLASSIFICATION TECHNIQUE; COMPREHENSIBILITY; EMPIRICAL STUDIES; PEDAGOGICAL APPROACH; RANDOM FORESTS; RULE EXTRACTION; RULE EXTRACTION ALGORITHMS;

DATA MINING;

EID: 85027923673     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2015.2389037     Document Type: Article

References (61)

1. C. M. Bishop, "Neural networks: A pattern recognition perspective," Neural Netw., pp. 1-23, 1996.
2. V. N. Vapnik, The Nature of Statistical Learning Theory (Statistics for Engineering and Information Science), vol. 8. New York, NY, USA: Springer-Verlag, 1995.
3. T. G. Dietterich, "Ensemble methods in machine learning," in Multiple Classifier Systems. New York, NY, USA: Springer-Verlag, 2000.
4. S. Lessmann, B. Baesens, C. Mues, and S. Pietsch, "Benchmarking classification models for software defect prediction: A proposed framework and novel findings," IEEE Trans. Softw. Eng., vol. 34, no. 4, pp. 485-496, Jul./Aug. 2008.
5. R. Caruana, N. Karampatziakis, and A. Yessenalina, "An empirical evaluation of supervised learning in high dimensions," in Proc. 25th Int. Conf. Mach. Learn., Helsinki, Finland, 2008, pp. 96-103.
6. A. Niculescu-Mizil et al., "Winning the KDD cup orange challenge with ensemble selection," in Proc. JMLR Workshop Conf., vol. 7, Paris, France, 2009, pp. 23-34.
7. M. J. Pazzani, S. Mani, and W. R. Shankle, "Acceptance of rules generated by machine learning among medical experts," Methods Inf. Med., vol. 40, no. 5, pp. 380-385, 2001.
8. D. Martens, L. Bruynseels, B. Baesens, M. Willekens, and J. Vanthienen, "Predicting going concern opinion with data mining," Decision Support Syst., vol. 45, no. 4, pp. 765-777, Nov. 2008.
9. D. Martens, B. Baesens, T. Van Gestel, and J. Vanthienen, "Comprehensible credit scoring models using rule extraction from support vector machines," Eur. J. Oper. Res., vol. 183, no. 3, pp. 1466-1476, Dec. 2007.
10. T. Van Gestel and B. Baesens, Credit Risk Management: Basic Concepts: Financial Risk Components, Rating Analysis, Models, Economic and Regulatory Capital: Basic Concepts: Financial Risk Components, Rating Analysis, Models, Economic and Regulatory Capital. London, U.K.: Oxford Univ. Press, 2008.
11. N. S. Umanath and I. Vessey, "Multiattribute data presentation and human judgment: A cognitive fit perspective," Decision Sci., vol. 25, nos. 5-6, pp. 795-824, Sep. 1994.
12. M. Limayem and G. DeSanctis, "Providing decisional guidance for multicriteria decision making in groups," Inf. Syst. Res., vol. 11, no. 4, pp. 386-401, Dec. 2000.
13. G. L. Lilien, A. Rangaswamy, G. H. Van Bruggen, and K. Starke, "DSS effectiveness in marketing resource allocation decisions: Reality vs. perception," Inf. Syst. Res., vol. 15, no. 3, pp. 216-235, Sep. 2004.
14. V. Arnold, N. Clark, P. A. Collier, S. A. Leech, and S. G. Sutton, "The differential use and effect of knowledge-based system explanations in novice and expert judgment decisions," MIS Quart., vol. 30, no. 1, pp. 79-97, Mar. 2006.
15. U. Kayande, A. D. Bruyn, G. L. Lilien, A. Rangaswamy, and G. H. van Bruggen, "How incorporating feedback mechanisms in a DSS affects DSS evaluations," Inf. Syst. Res., vol. 20, no. 4, pp. 527-546, Dec. 2008.
16. Y. Kodratoff, "The comprehensibility manifesto," AI Commun., 1994.
17. M. W. Craven and J. W. Shavlik, "Extracting tree-structured representations of trained networks," in Advances in Neural Information Processing Systems, vol. 8. Cambridge, MA, USA: MIT Press, 1996, pp. 24-30.
18. D. Martens, T. Van Gestel, and B. Baesens, "Decompositional rule extraction from support vector machines by active learning," IEEE Trans. Knowl. Data Eng., vol. 21, no. 2, pp. 178-191, Feb. 2008.
19. B. Baesens, D. Martens, R. Setiono, and J. M. Zurada, "Guest editorial white box nonlinear prediction models," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2406-2408, Dec. 2011.
20. U. Fayyad, G. Piatetsky-Shapiro, and P. Smyth, "From data mining to knowledge discovery in databases," AI Mag., vol. 17, no. 3, pp. 37-54, 1996.
21. L. Breiman, "Statistical modeling: The two cultures," Statist. Sci., vol. 16, no. 3, pp. 199-231, 2001.
22. J. R. Quinlan, C4.5: Programs for Machine Learning. San Francisco, CA, USA: Morgan Kaufmann, 1993.
23. R. Andrews, J. Diederich, and A. B. Tickle, "Survey and critique of techniques for extracting rules from trained artificial neural networks," Knowl.-Based Syst., vol. 8, no. 6, pp. 373-389, Dec. 1995.
24. E. Strumbelj and I. Kononenko, "An efficient explanation of individual classifications using game theory," J. Mach. Learn., vol. 11, pp. 1-18, Mar. 2010.
25. M. Robnik-Sikonja and I. Kononenko, "Explaining classifications for individual instances," IEEE Trans. Knowl. Data Eng., vol. 20, no. 5, pp. 589-600, May 2008.
26. D. Martens and F. Provost, "Explaining data-driven document classifications," MIS Quart., vol. 38, no. 1, pp. 73-100, Mar. 2014.
27. G. P. J. Schmitz, C. Aldrich, and F. S. Gouws, "ANN-DT: An algorithm for extraction of decision trees from artificial neural networks," IEEE Trans. Neural Netw., vol. 10, no. 6, pp. 1392-1401, Nov. 1999.
28. O. Boz and D. Hillman, "Converting a trained neural network to a decision tree dectext-Decision tree extractor," Ph.D. dissertation, Lehigh Univ., Bethlehem, PA, USA, 2000.
29. H. Nuñez, C. Angulo, and A. Català, "Rule-based learning systems for support vector machines," Neural Process. Lett., vol. 24, no. 1, pp. 1-18, 2002.
30. Z.-H. Zhou, Y. Jiang, and S.-F. Chen, "Extracting symbolic rules from trained neural network ensembles," AI Commun., vol. 16, no. 1, pp. 3-15, Jan. 2003.
31. U. Markowska-Kaczmar and W. Trelak, "Extraction of fuzzy rules from trained neural network using evolutionary algorithm," in Proc. Eur. Symp. Artif. Neural Netw. (ESANN), 2003, pp. 149-154.
32. J. R. Rabunãl, J. Dorado, A. Pazos, J. Pereira, and D. Rivero, "A new approach to the extraction of ANN rules and to their generalization capacity through GP," Neural Comput., vol. 16, no. 7, pp. 1483-1523, 2004.
33. F. Chen, "Learning accurate and understandable rules from SVM classifiers," Ph.D. dissertation, Simon Fraser Univ., Burnaby, BC, Canada, 2004.
34. N. Barakat and J. Diederich, "Eclectic rule-extraction from support vector machines," Int. J. Comput. Intell., vol. 2, no. 1, pp. 59-62, 2005.
35. G. Fung, S. Sandilya, and R. B. Rao, "Rule extraction from linear support vector machines," in Neurocomputing (Lecture Notes in Computer Science), vol. 74. New York, NY, USA: ACM, 2005, pp. 107-112.
36. R. Setiono, B. Baesens, and C. Mues, "Risk management and regulatory compliance: A data mining framework based on neural network rule extraction," in Proc. Int. Conf. Inf. Syst., Milwaukee, WI, USA, Dec. 2006.
37. J. Huysmans, B. Baesens, and J. Vanthienen, "ITER: An algorithm for predictive regression rule extraction," in Proc. 8th Int. Conf. Data Warehousing Knowl. Discovery, vol. 4081. 2006, pp. 270-279.
38. N. H. Barakat and A. Bradley, "Rule extraction from support vector machines: A sequential covering approach," IEEE Trans. Knowl. Data Eng., vol. 19, no. 6, pp. 729-741, Jun. 2007.
39. M. A. H. Farquad, V. Ravi, and S. B. Raju, "Support vector regression based hybrid rule extraction methods for forecasting," Expert Syst. Appl., vol. 37, no. 8, pp. 5577-5589, Aug. 2010.
40. C.-T. Su and Y.-C. Chen, "Rule extraction algorithm from support vector machines and its application to credit screening," Soft Comput., vol. 16, no. 4, pp. 645-658, Sep. 2011.
41. J. Chorowski and J. M. Zurada, "Extracting rules from neural networks as decision diagrams," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2435-2446, Dec. 2011.
42. M. Craven and J. Shavlik, "Using sampling and queries to extract rules from trained neural networks," in Proc. 11th Int. Conf. Mach. Learn., New Brunswick, NJ, USA, 1994, pp. 37-45.
43. M. Craven and J. Shavlik, "Rule extraction: Where do we go from here?" in Proc. Mach. Learn. Res. Group Working Paper, 1999, pp. 1-6.
44. K. Dejaeger, J. Huysmans, C. Mues, J. Vanthienen, and B. Baesens, "An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models," Decision Support Syst., vol. 51, no. 1, pp. 141-154, Apr. 2010.
45. U. Markowska-Kaczmar and M. Chumieja, "Discovering the mysteries of neural networks," Int. J. Hybrid Intell. Syst., vol. 1, nos. 3-4, pp. 153-163, Dec. 2004.
46. V. V. Fedorov, Theory of Optimal Experiments. New York, NY, USA: Academic, 1972.
47. D. Cohn, L. Atlas, and R. Ladner, "Improving generalization with active learning," Mach. Learn., vol. 15, no. 2, pp. 201-221, May 1994.
48. B. D. Ripley, "Neural networks and related methods for classification," J. Roy. Statist. Soc. Ser. B (Methodological), vol. 56, no. 3, pp. 409-456, 1994.
49. D. F. Shanno, "Conditioning of quasi-Newton methods for function minimization," Math. Comput., vol. 24, no. 111, pp. 647-656, 1970.
50. H. Czichos, T. Saito, and L. E. Smith, Springer Handbook of Metrology and Testing. New York, NY, USA: Springer-Verlag, 2011.
51. F. E. Grubbs, "Procedures for detecting outlying observations in samples," Technometrics, vol. 11, no. 1, pp. 1-21, 1969.
52. W. W. Cohen, "Fast effective rule induction," in Proc. 12th Int. Conf. Mach. Learn., vol. 3. Tahoe City, CA, USA, 1995, pp. 115-123.
53. K. Duan, S. S. Keerthi, W. Chu, S. K. Shevade, and A. N. Poo, "Multicategory classification by soft-max combination of binary classifiers," in Proc. 4th Int. Workshop Multiple Classifier Syst., 2003, p. 160.
54. J. C. Platt, "Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods," Adv. Large Margin Classifiers, vol. 10, no. 3, pp. 61-74, 1999.
55. S. Rüping, "A simple method for estimating conditional probabilities for SVMs," Dept. Comput. Sci., Humboldt Univ. Berlin, Berlin, Germany, Tech. Rep., 2004.
56. L. Breiman, "Random forests," Mach. Learn., vol. 45, no. 1, pp. 5-32, 2001.
57. K. Bache and M. Lichman. (2013). UCI machine learning repository. School Inf. Comput. Sci., Univ. California, Irvine, CA, USA. [Online]. Available: http://archive.ics.uci.edu/ml
58. D. Martens, M. De Backer, R. Haesen, J. Vanthienen, M. Snoeck, and B. Baesens, "Classification with ant colony optimization," IEEE Trans. Evol. Comput., vol. 11, no. 5, pp. 651-665, Oct. 2007.
59. B. Baesens, R. Setiono, C. Mues, and J. Vanthienen, "Using neural network rule extraction and decision tables for credit-risk evaluation," Manage. Sci., vol. 49, no. 3, pp. 312-329, 2003.
60. J. Chorowski and J. M. Zurada, "Learning understandable neural networks with nonnegative weight constraints," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 1, pp. 62-69, Jan. 2015.
61. J. Demšar, "Statistical comparisons of classifiers over multiple data sets," J. Mach. Learn. Res., vol. 7, pp. 1-30, Jan. 2006.