Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning

By Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Brian MacWhinney, Chris Dyer. 2016

100 citations recorded by the source corpus.

We use Bayesian optimization to learn curricula for word representation learning, optimizing performance on downstream tasks that depend on the learned representations as features. The curricula are modeled by a linear ranking function which is the scalar product of a learned weight vector and an engineered feature vector that characterizes the different aspects of the complexity of each instance in the training corpus. We show that learning the curriculum improves performance on a variety of downstream tasks over random orders and in comparison to the natural corpus order.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learning to select data for transfer learning with Bayesian Optimization (2017, methodology): “Similar to Tsvetkov et al. (2016), we use Bayesian Optimization (Brochu et al., 2010), which has emerged as an efﬁcient framework to optimize any function.”
Learning Hierarchical Teaching Policies for Cooperative Agents (2019, background): “Curriculum learning [3, 14, 36], which progressively increases task difficulty, is also relevant.”
Choosing Transfer Languages for Cross-Lingual Learning (2019, methodology): “Tsvetkov et al. (2016b) used typological information in the target language as additional input to their model for phonetic representation learning.”
A Survey on Curriculum Learning (2021, methodology): “[50], #phrases [113] Complexity Text Parse tree depth [113] Complexity Text Nesting of operations [131] Complexity Programs Shape variability [6] Diversity Images Word rarity [50], [86] Diversity Text POS entropy [113] Diversity Text Mahalanobis distance [14] Diversity Tabular Cluster density [11], ”
Learning to Teach (2018, background): “• Data features, contain information for data instance, such as its label category (we use 1 of |Y | representations), (for texts) the length of sentence, linguistic features for text segments (Tsvetkov et al., 2016), or (for images) gradients histogram features (Dalal & Triggs, 2005).”
Learning What Data to Learn (2017, background): “…f(s): • Data features, contain information for data instance, such as its label category (we use 1 of |Y | representations), (for texts) the length of sentence, linguistic features for text segments (Tsvetkov et al., 2016), or (for images) gradients histogram features (Dalal & Triggs, 2005).”
Curriculum Learning: A Survey (2021, methodology): “[144] introduce Bayesian optimization to optimize curricula for word representation learning.”
An Empirical Exploration of Curriculum Learning for Neural Machine Translation (2018, methodology): “…that are easier to learn from before presenting difficult samples is an intuitively attractive idea, which has been applied in various ways in Machine Learning and Natural Language Processing tasks (Bengio et al., 2009; Tsvetkov et al., 2016; Cirik et al., 2016; Graves et al., 2017, inter alia).”
Edinburgh Research Explorer Understanding Learning Dynamics Of Language Models with SVCCA (unknown): “Inspiration for such a system can be taken from (Tsvetkov et al., 2016), which presents a system for dynamically selecting a training curriculum based on performance on a specific end task.”
Reinforcement Learning based Curriculum Optimization for Neural Machine Translation (2019, background): “Work on meta-learning a curriculum originated with Tsvetkov et al. (2016), who used Bayesian optimization to learn a linear model for ranking examples in a word-embedding task.”
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering (2020, methodology): “The competence-aware curriculum in our work is related to curriculum learning [7,53,55,20,51,44,21,47] and machine teaching [61,62,38,11,40,15,58].”
Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learning (2021, background): “example difficulty exists in the data and then leverage various heuristics to sort examples by difficulty and train models on progressively harder examples (Bengio et al., 2009; Tsvetkov et al., 2016; Weinshall et al., 2018).”
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020, background): “Most of these have been carefully selected manually, raising interesting connections to data selection (Section 5.2) and finding better curricula (Tsvetkov et al., 2016) to learn under domain shift (Ruder and Plank, 2017).”
AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning (2019, methodology): “In particular, Ruder and Plank (2017) used Bayesian optimization to select relevant training instances for transfer learning, and Tsvetkov et al. (2016) applied it to learn a curriculum for training word embeddings via reordering data.”