Connectionist theory refinement: Genetically searching the space of network topologies

Journal of Artificial Intelligence Research

Volumn 6, Issue , 1997, Pages 177-209

  Opitz, David W ^a   Shavlik, Jude W ^b  

^a UNIVERSITY OF MONTANA   (United States)

^b University of Wisconsin Madison   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0001659341     PISSN: 10769757     EISSN: None     Source Type: Journal    
DOI: 10.1613/jair.368     Document Type: Article

References (101)

1. Ackley, D. (1987). A Connectionist Machine for Genetic Hillclimbing. Kluwer, Norwell, MA.
2. Ackley, D., & Littman, M. (1992). Interactions between learning and evolution. In Langton, C., Taylor, C., Farmer, C., & Rasmussen, S. (Eds.), Artificial Life II, pp. 487-509, Redwood City, CA. Addison-Wesley.
3. Ackley, D., & Littman, M. (1994). A case for Lamarckian evolution. In Langton, C. (Ed.), Artificial Life III, pp. 3-10, Redwood City, CA. Addison-Wesley.
4. Baffes, P., & Mooney, R. (1993). Symbolic revision of theories with M-of-N rules. In Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, pp. 1135-1140, Chambery, France. Morgan Kaufmann.
5. Bala, J., Huang, J., Vafaie, H., DeJong, K., & Wechsler, H. (1995). Hybrid learning using genetic algorithms and decision trees for pattern classification. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 719-724, Montreal, Canada. Morgan Kaufmann.
6. Baldwin, J. (1896). Physical and social heredity. American Naturalist, 30, 441-451.
7. Baum, E., & Haussler, D. (1989). What size net gives valid generalization? Neural Computation, 1, 151-160.
8. Baum, E., & Lang, K. (1991). Constructing hidden units using examples and queries. In Lippmann, R., Moody, J., & Touretzky, D. (Eds.), Advances in Neural Information Processing Systems, Vol. 3, pp. 904-910, San Mateo, CA. Morgan Kaufmann.
9. Belew, R. (1990). Evolution, learning and culture: Computational metaphors for adaptive search. Complex Systems, 4, 11-49.
10. Belew, R., Mclnerney, J., & Schraudolph, N. (1992). Evolving networks: Using the genetic algorithm with connectionist learning. In Langton, C., Taylor, C., Farmer, C., & Rasmussen, S. (Eds.), Artificial Life II, pp. 511-547, Redwood City, CA. Addison-Wesley.
11. Belew, R., & Mitchell, M. (1996). Adaptive Individuals in Evolving Populations: Models and Algorithms. Addison-Wesley, Massachusetts.
12. Berenji, H. (1991). Refinement of approximate reasoning-based controllers by reinforcement learning. In Proceedings of the Eighth International Machine Learning Workshop, pp. 475-479, Evanston, IL. Morgan Kaufmann.
13. Bergadano, F., Giordana, A., & Ponsero, S. (1989). Deduction in top-down inductive learning. In Proceedings of the Sixth International Workshop on Machine Learning, pp. 23-25, Ithaca, NY. Morgan Kaufmann.
14. Blanziere, E., & Katenkamp, P. (1996). Learning radial basis function networks on-line. In Proceedings of the Thirteenth International Conference on Machine Learning, pp. 37-45, Bari, Italy. Morgan Kaufmann.
15. Clocksin, W., & Mellish, C. (1987). Programming in Prolog. Springer-Verlag, New York.
16. Cohen, W. (1992). Compiling prior knowledge into an explicit bias. In Proceedings of the Ninth International Conference on Machine Learning, pp. 102-110, Aberdeen, Scotland. Morgan Kaufmann.
17. Danyluk, A. (1989). Finding new rules for incomplete theories: Explicit biases for induction with contextual information. In Proceedings of the Sixth International Workshop on Machine Learning, pp. 34-36, Ithaca, NY. Morgan Kaufmann.
18. Das, A., Giles, C., & Sun, G. (1992). Using prior knowledge in an NNPDA to learn context-free languages. In Hanson, S., Cowan, J., & Giles, C. (Eds.), Advances in Neural Information Processing Systems, Vol. 5, pp. 65-72, San Mateo, CA. Morgan Kaufmann.
19. Dean, T., & Boddy, M. (1988). An analysis of time-dependent planning. In Proceedings of the Seventh National Conference on Artificial Intelligence, pp. 49-54, St. Paul, MN. Morgan Kaufmann.
20. DeJong, K. (1975). An Analysis of the Behavior of a class of Genetic Adaptive Systems. Ph.D. thesis, University of Michigan, Ann Arbor, MI.
21. Dodd, N. (1990). Optimization of network structure using genetic techniques. In Proceedings of the IEEE International Joint Conference on Neural Networks, Vol. III, pp. 965-970, Paris. IEEE Press.
22. Donoho, S., & Rendell, L. (1995). Rerepresenting and restructuring domain theories: A constructive induction approach. Journal of Artificial Intelligence Research, 2, 411-446.
23. Drucker, H., Cortes, C., Jackel, L., LeCun, Y., & Vapnik, V. (1994). Boosting and other machine learning algorithms. In Proceedings of the Eleventh International Conference on Machine Learning, pp. 53-61, New Brunswick, NJ. Morgan Kaufmann.
24. Fahlman, S., & Lebiere, C. (1989). The cascade-correlation learning architecture. In Touretzky, D. (Ed.), Advances in Neural Information Processing Systems, Vol. 2, pp. 524-532, San Mateo, CA. Morgan Kaufmann.
25. Farmer, J., & Belin, A. (1992). Artificial life: The coming evolution. In Langton, C., Taylor, C., Farmer, J. D., & Rasmussen, S. (Eds.), Artificial Life II, pp. 815-840, Redwood City, CA. Addison-Wesley.
26. Flann, N., & Dietterich, T. (1989). A study of explanation-based methods for inductive learning. Machine Learning, 4, 187-226.
27. Fletcher, J., & Obradovic, Z. (1993). Combining prior symbolic knowledge and constructive neural network learning. Connection Science, 5, 365-375.
28. Forrest, S., & Mitchell, M. (1993). What makes a problem hard for a genetic algorithm? Some anomalous results and their explanation. Machine Learning, 13, 285-319.
29. Frean, M. (1990). The upstart algorithm: A method for constructing and training feedforward neural networks. Neural Computation, 2, 198-209.
30. Fu, L. (1989). Integration of neural heuristics into knowledge-based inference. Connection Science, 1, 325-340.
31. Ginsberg, A. (1990). Theory reduction, theory revision, and retranslation. In Proceedings of the Eighth National Conference on Artificial Intelligence, pp. 777-782, Boston, MA. AAAI/MIT Press.
32. Giordana, A., & Saitta, L. (1993). REGAL: An integrated system for relations using genetic algorithms. In Proceedings of the Second International Workshop on Multistrategy Learning, pp. 234-249, Harpers Ferry, WV.
33. Giordana, A., Saitta, L., & Zini, F. (1994). Learning disjunctive concepts by means of genetic algorithms. In Proceedings of the Eleventh International Conference on Machine Learning, pp. 96-104, New Brunswick, NJ. Morgan Kaufmann.
34. Goldberg, D. (1989). Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, Reading, MA.
35. Greene, D., & Smith, S. (1993). Competition-based induction of decision models from examples. Machine Learning, 13, 229-258.
36. Grefenstette, J., & Ramsey, C. (1992). An approach to anytime learning. In Proceedings of the Ninth International Conference on Machine Learning, pp. 189-195, Aberdeen, Scotland. Morgan Kaufmann.
37. Hansen, L., & Salamon, P. (1990). Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12, 993-1001.
38. Harp, S., Samad, T., & Guha, A. (1989). Designing application-specific neural networks using the genetic algorithm. In Touretzky, D. (Ed.), Advances in Neural Information Processing Systems, Vol. 2, pp. 447-454, San Mateo, CA. Morgan Kaufmann.
39. Hart, W. (1994). Adaptive Global Optimization with Local Search. Ph.D. thesis, University of California, San Diego.
40. Hassibi, B., & Stork, D. (1992). Second order derivatives for network pruning: Optimal brain surgeon. In Hanson, S., Cowan, J., & Giles, C. (Eds.), Advances in Neural Information Processing Systems, Vol. 5, pp. 164-171, San Mateo, CA. Morgan Kaufmann.
41. Hekanaho, J. (1996). Background knowledge in GA-based concept learning. In Proceedings of the Thirteenth International Conference on Machine Learning, pp. 234-242, Bari, Italy. Morgan Kaufmann.
42. Hinton, G., & Nowlan, S. (1987). How learning can guide evolution. Complex Systems, 1, 495-502.
43. Holder, L. (1991). Maintaining the Utility of Learned Knowledge Using Model-Based Control. Ph.D. thesis, Computer Science Department, University of Illinois at Urbana-Champaign.
44. Holland, J. (1975). Adaptation in Natural and Artificial Systems. University of Michigan Press, Ann Arbor, MI.
45. Janikow, C. (1993). A knowledge intensive GA for supervised learning. Machine Learning, 13, 198-228.
46. Judson, R., Colvin, M., Meza, J., Huffa, A., & Gutierrez, D. (1992). Do intelligent configuration search techniques outperform random search for large molecules? International Journal of Quantum Chemistry, 277-290.
47. Kibler, D., & Langley, P. (1988). Machine learning as an experimental science. In Proceedings of the Third European Working Session on Learning, pp. 1-12, Edinburgh, UK.
48. Kitano, H. (1990a). Designing neural networks using genetic algorithms with graph generation system. Complex Systems, 4, 461-476.
49. Kitano, H. (1990b). Empirical studies on the speed of convergence of neural network training using genetic algorithms. In Proceedings of the Eighth National Conference on Artificial Intelligence, pp. 789-795, Boston, MA. AAAI/MIT Press.
50. Kohavi, R., & John, G. (1997). Wrappers for feature subset selection. Artificial Intelligence.
51. Koppel, M., Feldman, R., & Segre, A. (1994). Bias-driven revision of logical domain theories. Journal of Artificial Intelligence Research, 1, 159-208.
52. Koza, J. (1992). Genetic Programming. MIT Press, Cambridge, MA.
53. Koza, J., & Rice, J. (1991). Genetic generation of both the weights and architectures for a neural network. In International Joint Conference on Neural Networks, Vol. 2, pp. 397-404, Seattle, WA. IEEE Press.
54. Krogh, A., & Vedelsby, J. (1995). Neural network ensembles, cross validation, and active learning. In Tesauro, G., Touretzky, D., & Leen, T. (Eds.), Advances in Neural Information Processing Systems, Vol. 7, pp. 231-238, Cambridge, MA. MIT Press.
55. Lacher, R., Hruska, S., & Kuncicky, D. (1992). Back-propagation learning in expert networks. IEEE Transactions on Neural Networks, 3, 62-72.
56. Le Cun, Y., Denker, J., & Solla, S. (1989). Optimal brain damage. In Touretzky, D. (Ed.), Advances in Neural Information Processing Systems, Vol. 2, pp. 598-605, San Mateo, CA. Morgan Kaufmann.
57. Litzkow, M., Livny, M., & Mutka, M. (1988). Condor - a hunter of idle workstations. In Proceedings of the Eighth International Conference on Distributed Computing Systems, pp. 104-111, San Jose, CA. Computer Society Press.
58. MacKay, D. (1992). A practical Bayesian framework for backpropagation networks. Neural Computation, 4, 448-472.
59. Maclin, R., & Shavlik, J. (1993). Using knowledge-based neural networks to improve algorithms: Refining the Chou-Fasman algorithm for protein folding. Machine Learning, 11, 195-215.
60. Mahoney, J. (1996). Combining Symbolic and Connectionist Learning Methods to Refine Certainty-Factor Rule-Bases. Ph.D. thesis, University of Texas, Austin, TX.
61. Mahoney, J., & Mooney, R. (1993). Combining connectionist and symbolic learning to refine certainty-factor rule-bases. Connection Science, 5, 339-364.
62. Mahoney, J., & Mooney, R. (1994). Comparing methods for refining certainty-factor rulebases. In Proceedings of the Eleventh International Conference on Machine Learning, pp. 173-180, New Brunswick, NJ. Morgan Kaufmann.
63. Masuoka, R., Watanabe, N., Kawamura, A., Owada, Y., & Asakawa, K. (1990). Neurofuzzy system - fuzzy inference using a structured neural network. In Proceedings of the International Conference on Fuzzy Logic & Neural Networks, pp. 173-177, Iizuka, Japan.
64. Michalski, R. (1983). A theory and methodology of inductive learning. Artificial Intelligence, 20, 111-161.
65. Miller, G., Todd, P., & Hegde, S. (1989). Designing neural networks using genetic algorithms. In Proceedings of the Third International Conference on Genetic Algorithms, pp. 379-384, Arlington, VA. Morgan Kaufmann.
66. Mitchell, M. (1996). An Introduction to Genetic Algorithms. MIT Press, Cambridge, MA.
67. Mitchell, T. (1982). Generalization as search. Artificial Intelligence, 18, 203-226.
68. Montana, D., & Davis, L. (1989). Training feedforward networks using genetic algorithms. In Proceedings of the Eleventh International Joint Conference on Artificial Intelligence, pp. 762-767, Detroit, MI. Morgan Kaufmann.
69. Moody, J. (1991). The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems. In Moody, J., Hanson, S., & Lippmann, R. (Eds.), Advances in Neural Information Processing Systems, Vol. 4, pp. 847-854, San Mateo, CA. Morgan Kaufmann.
70. Mozer, M. C., & Smolensky, P. (1989). Using relevance to reduce network size automatically. Connection Science, 1, 3-16.
71. Neri, F., & Saitta, L. (1996). Exploring the power of genetic search in learning symbolic classifiers. In IEEE Transactions on Pattern Analisys and Machine Intelligence.
72. Oliker, S., Furst, M., & Maimon, O. (1992). A distributed genetic algorithm for neural network design and training. Complex Systems, 6, 459-477.
73. Omlin, C., & Giles, C. (1992). Training second-order recurrent neural networks using hints. In Proceedings of the Ninth International Conference on Machine Learning, pp. 361-366, Aberdeen, Scotland. Morgan Kaufmann.
74. Opitz, D., & Shavlik, J. (1993). Heuristically expanding knowledge-based neural networks. In Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, pp. 1360-1365, Chambery, France. Morgan Kaufmann.
75. Opitz, D., & Shavlik, J. (1995). Dynamically adding symbolically meaningful nodes to knowledge-based neural networks. Knowledge-Based Systems, 8, 301-311.
76. Opitz, D., & Shavlik, J. (1996). Actively searching for an effective neural-network ensemble. Connection Science, 8, 337-353.
77. Ourston, D., & Mooney, R. (1994). Theory refinement combining analytical and empirical methods. Artificial Intelligence, 66, 273-309.
78. Pazzani, M., & Kibler, D. (1992). The utility of knowledge in inductive learning. Machine Learning, 9, 57-94.
79. Perrone, M. (1993). Improving Regression Estimation: Averaging Methods for Variance Reduction with Extension to General Convex Measure Optimization. Ph.D. thesis, Brown University, Providence, RI.
80. Provost, F., & Buchanan, B. (1995). Inductive policy: The pragmatics of bias selection. Machine Learning, 20, 35-61.
81. Quinlan, J. (1986). Induction of decision trees. Machine Learning, 1, 81-106.
82. Quinlan, J., & Cameron-Jones, R. (1995). Lookahead and pathology in decision tree induction. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1019-1024, Montreal, Canada. Morgan Kaufmann.
83. Richards, B., & Mooney, R. (1995). Automated refinement of first-order Horn-clause domain theories. Machine Learning, 19, 95-131.
84. Roscheisen, M., Hofmann, R., & Tresp, V. (1991). Neural control for rolling mills: Incorporating domain theories to overcome data deficiency. In Moody, J., Hanson, S., & Lippmann, R. (Eds.), Advances in Neural Information Processing Systems, Vol. 4, pp. 659-666, San Mateo, CA. Morgan Kaufmann.
85. Rumelhart, D., Hinton, G., & Williams, R. (1986). Learning internal representations by error propagation. In Rumelhart, D., & McClelland, J. (Eds.), Parallel Distributed Processing: Explorations in the microstructure of cognition. Volume 1: Foundations, pp. 318-363. MIT Press, Cambridge, MA.
86. Schaffer, C. (1993). Selecting a classification method by cross-validation. Machine Learning, 13, 135-143.
87. Schiffmann, W., Joost, M., & Werner, R. (1992). Synthesis and performance analysis of multilayer neural network architectures. Tech. rep. 16, University of Koblenz, Institute for Physics.
88. Scott, G., Shavlik, J., & Ray, W. (1992). Refining PID controllers using neural networks. Neural Computation, 5, 746-757.
89. Tishby, N., Levin, E., & Solla, S. (1989). Consistent inference on probabilities in layered networks, predictions and generalization. In International Joint Conference on Neural Networks, pp. 403-410, Washington, D.C. IEEE Press.
90. Towell, G. (1991). Symbolic Knowledge and Neural Networks: Insertion, Refinement, and Extraction. Ph.D. thesis, Computer Sciences Department, University of Wisconsin, Madison, WI.
91. Towell, G., & Shavlik, J. (1992). Using symbolic learning to improve knowledge-based neural networks. In Proceedings of the Tenth National Conference on Artificial Intelligence, pp. 177-182, San Jose, CA. AAAI/MIT Press.
92. Towell, G., & Shavlik, J. (1994). Knowledge-based artificial neural networks. Artificial Intelligence, 70, 119-165.
93. Turney, P. (1995). Cost-sensitive classification: Empirical evaluation of a hybrid genetic decision tree induction algorithm. Journal of Artificial Intelligence Research, 2, 369-409.
94. Waterman, D. (1986). A Guide to Expert Systems. Addison Wesley, Reading, MA.
95. Watrous, R., Towell, G., & Glassman, M. (1993). Synthesize, optimize, analyze, repeat (SOAR): Application of neural network tools to ECG patient monitoring. In Proceedings of the Symposium on Nonlinear Theory and its Applications, pp. 565-570, Honolulu, Hawaii.
96. Watson, J. D., Hopkins, N. H., Roberts, J. W., Argetsinger-Steitz, J., & Weiner, A. M. (1987). Molecular Biology of the Gene (Fourth edition). Benjamin/Cummings, Menlo Park, CA.
97. Weigend, A. (1993). On overfitting and the effective number of hidden units. In Proceedings of the 1993 Connectionist Models Summer School, pp. 335-342, Boulder, CO. Lawrence Erlbaum Associates.
98. Weigend, A., Huberman, B., & Rumelhart, D. (1990). Predicting the future: A connectionist approach. International Journal of Neural Systems, I, 193-209.
99. Whitley, D., Gordon, S., & Mathias, K. (1994). Lamarckian evolution, the Baldwin effect and function optimization. In Davidor, Y., Schwefel, H., & Manner, R. (Eds.), Parallel Problem Solving from Nature - PPSN III, pp. 6-15. Springer-Verlag.
100. Whitley, D., & Hanson, T. (1989). Optimizing neural networks using faster, more accurate genetic search. In Proceedings of the Third International Conference on Genetic Algorithms, pp. 391-396, Arlington, VA. Morgan Kaufmann.
101. Yao, X. (1993). Evolutionary artificial neural networks. International Journal of Neural Systems, 4, 203-221.