Online Representation Learning in Recurrent Neural Language Models

By Marek Rei. 2015

12 citations recorded by the source corpus.

We investigate an extension of continuous online learning in recurrent neural network language models. The model keeps a separate vector representation of the current unit of text being processed and adaptively adjusts it after each prediction. The initial experiments give promising results, indicating that the method is able to increase language modelling accuracy, while also decreasing the parameters needed to store the model along with the computation required at each step.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, methodology): “Rei (2015) developed a method which learns a context vector that can be adapted online, with an application to recurrent language models.”
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL (2018, methodology): “This is representative of commonly used dynamic evaluation methods (Rei, 2015; Krause et al., 2017; 2016; Fortunato et al., 2017).”
Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning (2018, methodology): “Dynamic evaluation algorithms (Rei, 2015; Krause et al., 2017; 2016; Fortunato et al., 2017), for example, learn an approximate global distribution at training time and adapt those model parameters at test time to fit the current local distribution via gradient descent.”
Fast Context Adaptation via Meta-Learning (2018, background): “Rei (2015) propose a similar approach in the context of neural language models, where a context vector represents the sentence that is currently being processed (see also the Appendix of Finn et al. (2017a)).”
Learning to Adapt: Meta-Learning for Model-Based Control (2018, background): “Dynamic evaluation algorithms [40, 22, 21, 16], for example, learn an approximate global distribution at training time and adapt those model parameters at test time to fit the current local distribution via gradient descent.”
CAML: Fast Context Adaptation via Meta-Learning (2018, background): “In the context of neural language models, Rei (2015) also considers adapting only context parameters in the inner loop, but they do not consider the beneﬁts of initialising and resetting their value to 0 .”