Incrementally Learning the Hierarchical Softmax Function for Neural Language Models
By Hao Peng, Jianxin Li, Yangqiu Song, Yaopeng Liu. 2017
79 citations recorded by the source corpus.
Neural network language models (NNLMs) have attracted a lot of attention recently. In this paper, we present a training method that can incrementally train the hierarchical softmax function for NNMLs. We split the cost function to model old and update corpora separately, and factorize the objective function for the hierarchical softmax. Then we provide a new stochastic gradient based method to update all the word vectors and parameters, by comparing the old tree generated based on the old corpus and the new tree generated based on the combined (old and update) corpus. Theoretical analysis shows that the mean square error of the parameter vectors can be bounded by a function of the number of changed words related to the parameter node. Experimental results show that incremental training can save a lot of time. The smaller the update corpus is, the faster the update training process is, where an up to 30 times speedup has been achieved. We also use both word similarity/relatedness tasks and dependency parsing task as our benchmarks to evaluate the correctness of the updated word vectors.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Xinchao Wang
- Mingli Song
- Jingwen Ye
- Jie Song
- Xingyi Yang
- R. Venkatesh Babu
- Naveen Venkat
- Ambareesh Revanur
Citations in the corpus
- Learning with Recoverable Forgetting (2022, positive): “Peng et al. [34] present to train the hierarchical soft-max function for deep language models for the new-coming tasks.”
- Class-Incremental Domain Adaptation (2020, background): “truly facilitate model upgrade (e.g. adding new classes to the recognition model) thereby having a limited practical use-case. Another line of work consists of Class-Incremental (CI) learning methods [4,28,38,42,53] which aim at adding new classes to a trained model while preserving the performance ”
- Diachronic word embeddings and semantic shifts: a survey (2018, methodology): “Among others, Peng et al. (2017) described a novel method of incrementally learning the hierarchical softmax function for the CBOW and Continuous Skipgram algorithms. In this way, one can update word embedding models with new data and new vocabulary much more efficiently, achieving faster training t”