Building Language Models for Text with Named Entities

By Md. Rizwan Parvez, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang. 2018

46 citations recorded by the source corpus.

Text in many domains involves a significant amount of named entities. Predicting the entity names is often challenging for a language model as they appear less frequent on the training corpus. In this paper, we propose a novel and effective approach to building a language model which can learn the entity names by leveraging their entity type information. We also introduce two benchmark datasets based on recipes and Java programming codes, on which we evaluate the proposed model. Experimental results show that our model achieves 52.2% better perplexity in recipe generation and 22.06% on code generation than state-of-the-art language models.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

In Information Extraction

Citations in the corpus

Unified Pre-training for Program Understanding and Generation (2021, background): “, 2020), Code completion (Li et al., 2018a; Hellendoorn and Devanbu, 2017; Parvez et al., 2018) are some of the tasks that are addressed with deep neural solution.”
Relational Memory-Augmented Language Models (2022, background): “, 2021), entity types (Parvez et al., 2018; Wang et al., 2018b) and entity descriptions (Bahdanau et al.”
CODIT: Code Editing With Tree-Based Neural Models (2018, methodology): “Generating source code using machine learning models has been explored in the past [31], [32], [33], [34].”
Strategies for Structuring Story Generation (2019, methodology): “Language models have been extended to model entities based on additional information, such as entity type (Parvez et al., 2018).”
RecipeNLG: A Cooking Recipes Dataset for Semi-Structured Text Generation (2020, methodology): “(Parvez et al., 2018) used a dataset of 100K recipes to build an LSTM-based discriminative language model for the task of named entity recognition.”
Latent Relation Language Models (2019, methodology): “…LMs exist, conditioning on a variety of information sources such as expert coreference annotations (Ji et al., 2017; Clark et al., 2018; Yang et al., 2017), entity annotations (Logan et al., 2019), deﬁnitions (Bah-danau et al., 2017), or keywords (Kiddon et al., 2016; Parvez et al., 2018).”
Code Prediction by Feeding Trees to Transformers (2020, background): “The most common neural technique for code prediction is Recurrent Neural Networks (RNNs) [4] and their variants [5]–[10], where the code represented as a linear sequence is fed as input to the model.”