Harnessing Deep Neural Networks with Logic Rules
By Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard H. Hovy, Eric P. Xing. 2016
641 citations recorded by the source corpus.
Combining deep neural networks with structured logic rules is desirable to harness flexibility and reduce uninterpretability of the neural models. We propose a general framework capable of enhancing various types of neural networks (e.g., CNNs and RNNs) with declarative first-order logic rules. Specifically, we develop an iterative distillation method that transfers the structured information of logic rules into the weights of neural networks. We deploy the framework on a CNN for sentiment analysis, and an RNN for named entity recognition. With a few highly intuitive rules, we obtain substantial improvements and achieve state-of-the-art or comparable results to previous best-performing systems.
Subfields: Artificial Intelligence, Knowledge Graphs & Representation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Yoshua Bengio
- Jianfeng Gao
- Luke S. Zettlemoyer
- Yejin Choi
- Quoc V. Le
- Graham Neubig
In Knowledge Graphs & Representation
- Peter Clark
- Oyvind Tafjord
- A.S. d'Avila Garcez
- Song-Chun Zhu
- Quanshi Zhang
- Luciano Serafini
- Sebastian Riedel
- Tim Rocktaschel
Citations in the corpus
- Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning (2019, background): “Self-transfer with symbolic-knowledge distillation [23] is also useful as it can enhance several types of deep networks such as convolutional neural networks and recurrent neural networks.”
- Integrating Learning and Reasoning with Deep Logic Models (2019, background): “[15], Semantic-based regularization (SBR) [8] for kernel machines and Logic Tensor Networks (LTN) [9] for neural networks share the same basic idea of integrating logic reasoning and learning using a similar continuous relaxation of logic to the one presented in this paper.”
- Unsupervised Learning of Neural Networks to Explain Neural Networks (2018, background): “A new trend related to network interpretability is to learn networks with disentangled, interpretable representations [12, 31, 21].”
- Beyond Sparsity: Tree Regularization of Deep Models for Interpretability (2017, background): “Edge and node regularization is commonly used to improve prediction accuracy (Drucker and Le Cun 1992; Ochiai et al. 2017), and recently Hu et al. (2016) improve prediction accuracy by training neural networks so that predictions match a small list of known domain-specific first-order logic rules.”
- Interpretable Convolutional Neural Networks (2017, methodology): “[8] designed some logic rules for network outputs, and used these rules to regularize the learning process.”
- Visual interpretability for deep learning: a survey (2018, background): “To distill representations of a teacher network to a student network for sentiment analysis, [Hu et al., 2016] has proposed using logic rules of natural languages (e.”
- Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning (2019, methodology): “(Hu et al. 2016) proposed a combination of deep neural networks with logic rules by employing knowledge distillation procedure (Hinton, Vinyals, and Dean 2015) of transferring the learned tacit knowledge from larger neural network, to the weights of the smaller neural network in data-limited…”
- End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF (2016, methodology): “Huang et al. (2015) used BLSTM for word-level representations and CRF for jointly label decoding, which is similar to our model.”
- Syntax-Directed Variational Autoencoder for Structured Data (2018, background): “By addressing the most common semantics to harness the deep networks, it can greatly reshape the output domain of decoder (Hu et al., 2016).”
- Representation Stability as a Regularizer for Improved Text Analytics Transfer Learning (2017, neutral): “(Hu et al., 2016) very recently also looked at the problem of distilling logical rules into a neural network text analytics classifier.”
- Toward Controlled Generation of Text (2017, methodology): “We use a state-of-the-art sentiment classifier (Hu et al., 2016a) which achieves 90% accuracy on the SST test set, to automatically evaluate the sentiment generation accuracy.”
- Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes (2017, background): “Some applications include weakly supervised segmentation [44, 49] and detection [5], and rule-regularized training of neural networks [29].”
- Evidence Extraction for Machine Reading Comprehension with Deep Probabilistic Logic (2018, methodology): “It draws inspiration from recent work using logical rules to aid deep learning (Hu et al., 2016a,b).”
- Embedding Logical Queries on Knowledge Graphs (2018, background): “However, existing work in this area primarily focuses on using logical reasoning to improve edge prediction in knowledge graphs [14, 13, 27], for example, by using logical rules as regularization [15, 20, 35, 37].”
- A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “A novel contribution is that of using knowledge distillation [120] to acquire prior knowledge from internal and external”
- Neurosymbolic AI: the 3rd wave (2020, background): “…knowledge into the set of weights of a network (d’Avila Garcez and Zaverucha 1999; d’Avila Garcez et al. 2002, 2009; Ham-mer and Hitzler 2007; Hu et al. 2016) and systems that translate and encode symbolic knowledge into the network’s loss function (Serafini and d’Avila Garcez 2016; Marra…”