Dependency Recurrent Neural Language Models for Sentence Completion

By Piotr Wojciech Mirowski, Andreas Vlachos. 2015

61 citations recorded by the source corpus.

Recent work on language modelling has shifted focus from count-based models to neural models. In these works, the words in each sentence are always considered in a left-to-right order. In this paper we show how we can improve the performance of the recurrent neural network (RNN) language model by incorporating the syntactic dependencies of a sentence, which have the effect of bringing relevant contexts closer to the word being predicted. We evaluate our approach on the Microsoft Research Sentence Completion Challenge and show that the dependency RNN proposed improves over the RNN by about 10 points in accuracy. Furthermore, we achieve results comparable with the state-of-the-art models on this task.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, methodology): “While there is no official dev/test split for this dataset, we followed previous work (Mirowski and Vlachos, 2015) and used the first 520 sentences for parameter tuning and the rest as the test set.”
Submix: Practical Private Prediction for Large-Scale Language Models (2022, methodology): “Such a query-answering API is useful for applications such as smart keyboard for auto-correction and text completion (Mirowski & Vlachos, 2015; Hertel, 2019).”
Generative Knowledge Transfer for Neural Language Models (2016, methodology): “Neural network based language models (LMs) are used in many fields such as speech recognition, chatbot, sentence completion and machine translation (Mikolov et al., 2010; Serban et al., 2015; Spithourakis et al., 2016; Mirowski & Vlachos, 2015; Cho et al., 2014).”
Incrementally Learning the Hierarchical Softmax Function for Neural Language Models (2017, methodology): “The natural future work is to extend our approach to other advanced NNLMs beyond CBOW and Skip-gram such as dependency RNN (Mirowski and Vlachos 2015) the LORELEI Contract HR0011-15-2-0025 with the US Defense Advanced Research Projects Agency (DARPA).”
Recurrent Memory Networks for Language Modeling (2016, result): “It is worth to mention that our LSTM baseline outperforms a dependency RNN making explicit use of syntactic information (Mirowski and Vlachos, 2015) and performs on par with the best published result (Mikolov et al.”
Embedding Logic Rules Into Recurrent Neural Networks (2019, background): “As shown in FIGURE 1, existing structured knowledge usually consists of a structured knowledge base [9], [10], social graph [11], and syntactic dependency [7].”