A Mutual Information Maximization Perspective of Language Representation Learning

By Lingpeng Kong, Cyprien de Masson d'Autume, Wang Ling, Lei Yu, Zihang Dai, Dani Yogatama. 2019

181 citations recorded by the source corpus.

We show state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word sequence (i.e., a sentence). Our formulation provides an alternative perspective that unifies classical word embedding models (e.g., Skip-gram) and modern contextual embeddings (e.g., BERT, XLNet). In addition to enhancing our theoretical understanding of these methods, our derivation leads to a principled framework that can be used to construct new self-supervised tasks. We provide an example by drawing inspirations from related methods based on mutual information maximization that have been successful in computer vision, and introduce a simple self-supervised objective that maximizes the mutual information between a global sentence representation and n-grams in the sentence. Our analysis offers a holistic view of representation learning methods to transfer knowledge and translate progress across multiple domains (e.g., natural language processing, computer vision, audio processing).

Subfields: Artificial Intelligence, Large Language Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Representation & Self-Supervised Learning

Citations in the corpus

The Effect of Natural Distribution Shift on Question Answering Models (2020, neutral): “InfoWord Large (Single) (Kong et al., 2019) https://worksheets.”
What makes for good views for contrastive learning (2020, background): “For example: luminance and chrominance decomposition [53], randomly augmenting an image twice [58, 7, 4, 23, 60, 50, 65, 67], using different time-steps of videos [42, 66, 46, 22, 21], patches of the same image [27, 42, 26], multiple sensory data [39, 9, 44], or text and its context [37, 59, 35, 32]”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “, 2013) and BERT-style embeddings can be viewed in terms of mutual information maximization (Kong et al., 2019), but the latter are contextualized.”
InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning (2020, background): “b) Contrastive learning: An emerging theme in recent state-of-the-art works in unsupervised representation learning [19]–[25] lies in taking a contrastive approach to maximizing the mutual information between encoded local and global features.”
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, methodology): “Kong et al. (2020) show that learning word embeddings (Mikolov et al., 2013) or contextual embeddings (Devlin et al., 2019) can also be uniﬁed under the framework of mutual information maximization.”
Self-Supervised Learning: Generative or Contrastive (2020, background): “In language pre-training, InfoWord [72] proposes to maximize the mutual information between a global representation of a sentence and n-grams in it.”
Pre-trained models for natural language processing: A survey (2020, neutral): “[90] applied DIM to language representation learning.”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “Examples include different frames of video [69, 112, 81, 37, 31, 38], point clouds [100], multiple sensory data [65, 18, 74], text and its context [63, 101, 59, 51], or video and language [84, 62, 56].”
ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning (2020, background): “Some focus on obtaining better span representations, including entity mentions, via span-based pre-training (Sun et al., 2019; Joshi et al., 2020; Kong et al., 2020; Ye et al., 2020).”
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations (2020, background): “In fact, Kong et al., 2020 demonstrate that the MLM and NSP objectives are also instances of contrastive learning.”
Contrastive Representation Learning: A Framework and Review (2020, methodology): “[61] showed that BERT or XLnet also maximise global-local mutual information, whereas the next sentence prediction pre-training task can be seen as constructing similarity pairs using the sequential coherence property.”
On Learning Universal Representations Across Languages (2020, background): “Kong et al. (2020) improved language representation learning by maximizing the mutual information between a masked sentence representation and local n-gram spans.”
S3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization (2020, methodology): “(Y|X). (1) Maximizing mutual information directly is usually intractable. Thus we resort to a lower bound on I(X,Y). One particular lower bound that has been shown to work well in practice is InfoNCE [10, 12, 25], which is based on Noise Contrastive Estimation (NCE) [2]. InfoNCE is defined as: E p(X”
An Unsupervised Sentence Embedding Method by Mutual Information Maximization (2020, methodology): “, 2018), which were demonstrated to be effective for unsupervised representation learning in various scenarios (Hjelm et al., 2019; Ji et al., 2019; Sun et al., 2020; Kong et al., 2020).”
Self-Alignment Pretraining for Biomedical Entity Representations (2020, methodology): “The MLM framework is similar to denoising autoencoders trying to reconstruct part(s) of the text (Kong et al., 2020; Lewis et al., 2020) and can be considered as a form of self-training.”