Just Add Functions: A Neural-Symbolic Language Model

By David Demeter, Doug Downey. 2019

11 citations recorded by the source corpus.

Neural network language models (NNLMs) have achieved ever-improving accuracy due to more sophisticated architectures and increasing amounts of training data. However, the inductive bias of these models (formed by the distributional hypothesis of language), while ideally suited to modeling most running text, results in key limitations for today's models. In particular, the models often struggle to learn certain spatial, temporal, or quantitative relationships, which are commonplace in text and are second-nature for human readers. Yet, in many cases, these relationships can be encoded with simple mathematical or logical expressions. How can we augment today's neural models with such encodings?In this paper, we propose a general methodology to enhance the inductive bias of NNLMs by incorporating simple functions into a neural architecture to form a hierarchical neural-symbolic language model (NSLM). These functions explicitly encode symbolic deterministic relationships to form probability distributions over words. We explore the effectiveness of this approach on numbers and geographic locations, and show that NSLMs significantly reduce perplexity in small-corpus language modeling, and that the performance improvement persists for rare tokens even on much larger corpora. The approach is simple and general, and we discuss how it can be applied to other word classes beyond numbers and geography.

Subfields: Artificial Intelligence, General Social Sciences, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luyu Gao
2. Graham Neubig
3. Pengfei Liu
4. Jamie Callan
5. Yiming Yang
6. Shuyan Zhou
7. Uri Alon
8. Aman Madaan

In General Social Sciences

1. Pengfei Liu
2. Jamie Callan
3. Graham Neubig
4. Yiming Yang
5. Luyu Gao

In Large Language Models

1. Graham Neubig
2. Pengfei Liu
3. Luyu Gao
4. Jamie Callan
5. Aman Madaan
6. Yiming Yang
7. Hinrich Schütze
8. Yoav Goldberg

In Natural Language Processing

1. Luyu Gao
2. Pengfei Liu
3. Graham Neubig
4. Shuyan Zhou
5. Uri Alon
6. Jamie Callan
7. Yiming Yang
8. Aman Madaan

Citations in the corpus

• PAL: Program-aided Language Models (2022, methodology): “For example, Cobbe et al. (2021) employ a calculator for arithmetic operations as a post hoc processing, and Demeter & Downey (2020) add specialized modules for generating cities and dates.”