Connectionist Theory Refinement: Genetically Searching the Space of Network Topologies

By David W. Opitz, Jude W. Shavlik. 1997

An algorithm that learns from a set of examples should ideally be able to exploit the available resources of (a) abundant computing power and (b) domain-specific knowledge to improve its ability to generalize. Connectionist theory-refinement systems, which use background knowledge to select a neural network's topology and initial weights, have proven to be effective at exploiting domain-specific knowledge; however, most do not exploit available computing power. This weakness occurs because they lack the ability to refine the topology of the neural networks they produce, thereby limiting generalization, especially when given impoverished domain theories. We present the Regent algorithm which uses (a) domain-specific knowledge to help create an initial population of knowledge-based neural networks and (b) genetic operators of crossover and mutation (specifically designed for knowledge-based networks) to continually search for better network topologies. Experiments on three real-world domains indicate that our new algorithm is able to significantly increase generalization compared to a standard connectionist theory-refinement system, as well as our previous algorithm for growing knowledge-based networks.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Weight Agnostic Neural Networks (2019, background): “Our work has connections to existing work not only in deep learning, but also to various other ﬁelds: Architecture Search Search algorithms for neural network topologies originated from the ﬁeld of evolutionary computing in the 1990s [1, 8, 17, 25, 33, 40, 53, 59, 70, 71, 81, 86, 116, 117].”
Autonomous Evolution of Topographic Regularities in Artificial Neural Networks (2010, unknown): “…et al., 1996; Hornby & Pollack, 2002; Kitano, 1990; Koza & Rice, 1991; Lipson & Pollack, 2000; Montana & Davis, 1989; Moriarty & Miikkulainen, 1996; Opitz & Shavlik, 1997; Pujol & Poli, 1998; Sims, 1994; Srinivas & Paranaik, 1991; Stanley & Miikkulainen, 2002b; Yao, 1999; Zhang & Muhlenbein,…”
Artificial immune systems as a novel soft computing paradigm (2003, background): “For example, they can be used to evolve artificial neural network architectures (Harrald and Kamstra, 1997; Maniezzo, 1994; Friedrich and Moraga, 1996; Opitz and Shavlik, 1997) or to design fuzzy systems (Chan et al., 1997; Shi et al., 1999; Belarbi and Titel, 2000).”
Types of Cost in Inductive Concept Learning (2002, background): “Several researchers have examined ways of embedding domain knowledge in a learning algorithm (Opitz and Shavlik, 1997).”
Evolution of neural networks using Cartesian Genetic Programming (2010, background): “Direct encoding are generally thought of as better at fine tuning and generating compact architecture [1], [12], [25], [3], [19], [26], [14].”
Evolving Neural Networks through Augmenting Topologies (2002, methodology): “…appear in the phenotype (Angeline et al., 1993; Braun and Weisbrod, 1993; Dasgupta and McGregor, 1992; Fullmer and Miikkulainen, 1992; Krishnan and Ciesielski, 1994; Lee and Kim, 1996; Maniezzo, 1994; Opitz and Shavlik, 1997; Pujol and Poli, 1998; Yao and Liu, 1996; Zhang and M ¨ uhlenbein, 1993).”
Compositional Pattern Producing Networks : A Novel Abstraction of Development (2007, background): “It is fortuitous that CPPNs and ANNs are virtually the same from a structural perspective because many methods already exist for evolving ANN topology and weights [5, 32, 58, 61, 73, 83, 84].”
Compositional pattern producing networks: A novel abstraction of development (2007, methodology): “methods already exist for evolving ANN topology and weights [5, 32, 58 , 61, 73, 83, 84].”
Real-time neuroevolution in the NERO video game (2005, methodology): “For example, Samuel (1959) trained a computer to play checkers using a method similar to temporal difference learning (Sutton 1988) in the first application of machine learning (ML) to games.”