Multi-modal Self-Supervision from Generalized Data Transformations

By Mandela Patrick, Yuki M. Asano, Ruth C. Fong, João F. Henriques, Geoffry Zweig, Andrea Vedaldi. 2020

196 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, methodology): “Mildly jittering with SpecAugment [64] was not beneficial, and more aggressive augmentations were detrimental; in contrast with the findings of [65] where SpecAugment helped, presumably due to training on a relatively small dataset.”
What makes for good views for contrastive learning (2020, background): “For example: luminance and chrominance decomposition [53], randomly augmenting an image twice [58, 7, 4, 23, 60, 50, 65, 67], using different time-steps of videos [42, 66, 46, 22, 21], patches of the same image [27, 42, 26], multiple sensory data [39, 9, 44], or text and its context [37, 59, 35, 32]”
Support-set bottlenecks for video-text representation learning (2020, background): “The resulting noise contrastive representation associates audio and visual signals that come from the same source video, diﬀerentiating others (Patrick et al., 2020).”
Self-Supervised Pretraining of 3D Features on any Point-Cloud (2021, methodology): “These instance discrimination methods can be extended to multiple modalities [57, 62, 93].”
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, background): “…motivated by recent advances in large-scale language model (Artetxe et al., 2020) and multimodal pre-training (Lu et al., 2019; Miech et al., 2019; Patrick et al., 2020), we propose a multilingual multimodal pre-training (MMP) strategy to exploit the weak supervision from large-scale multilingual…”
ImageBind One Embedding Space to Bind Them All (2023, background): “Recently, many methods learn image features aligned with text [1, 31, 46, 60, 64, 65, 82, 83], audio [3, 4, 50, 55, 56, 70] etc.”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “Similarly, to combine audio and visual modalities for unsupervised learning, existing works exploit the natural audio-visual correspondence in videos to formulate various self-supervised signals, which predict the cross-modal correspondence [314], [315], align the temporally corresponding representa”
Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better (2021, unknown): “One of the classical example of efficient layers in the Vision domain are the Convolu-tional layers, which improved over Fully Connected (FC) layers in Vision models.”
Parameter Efficient Multimodal Transformers for Video Representation Learning (2020, methodology): “We also apply audio-visual synchronized temporal jittering (Patrick et al., 2020).”
A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, methodology): “The strongest previous approaches are using multi-modal input, Vision “V”, Audio “A”, Text “T”, to train a contrastive objective across modalities; XDC [3] performs DeepCluster [8] on (V+A), CVRL [71], GDT [68] and MMV [2] use an objective similar to SimCLR on (V), (V+A), and method pre-train backbo”
Hard Negative Mixing for Contrastive Learning (2020, background): “A number of works exploit the audio-visual nature of video [37, 1, 51, 14, 3] to learn visual representation, e.g . via learning intra-modality synchronization.”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “Examples include different frames of video [69, 112, 81, 37, 31, 38], point clouds [100], multiple sensory data [65, 18, 74], text and its context [63, 101, 59, 51], or video and language [84, 62, 56].”
Broaden Your Views for Self-Supervised Video Learning (2021, unknown): “Video and audio have been used as a rich source of self-supervision [5, 6, 7, 45, 57, 60, 63, 70].”
Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data (2021, methodology): “R3D-18 is an effective yet lightweight model for video recognition tasks, which has been shown effective for multi-modal self-supervised learning [36], allowing the use of a relatively larger batch size for contrastive learning.”