M6: A Chinese Multimodal Pretrainer

By Junyang Lin, Rui Men, An Yang, Chan Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia and 15 more. 2021

152 citations recorded by the source corpus.

In this work, we construct the largest dataset for multimodal pretraining in Chinese, which consists of over 1.9TB images and 292GB texts that cover a wide range of domains. We propose a cross-modal pretraining method called M6, referring to Multi-Modality to Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. We scale the model size up to 10 billion and 100 billion parameters, and build the largest pretrained model in Chinese. We apply the model to a series of downstream applications, and demonstrate its outstanding performance in comparison with strong baselines. Furthermore, we specifically design a downstream task of text-guided image generation, and show that the finetuned M6 can create high-quality images with high resolution and abundant details.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “Some prior works have chosen to use the mixture-of-experts (MoE) [32, 35, 61] technique to scale to larger model sizes more economically, producing large-scale models that selectively use a subset of its parameters in each forward pass.”
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “There have been increasing demand for training and scaling Transformers (Shoeybi et al., 2019; Raffel et al., 2019; Fedus et al., 2021; Conneau et al., 2019; Lin et al., 2021).”
CogView: Mastering Text-to-Image Generation via Transformers (2021, background): “(Paul Cézanne [17]) As contrastive self-supervised pretraining has revolutionized computer vision (CV) [24, 21, 8, 32], visual-language pretraining, which brings high-level semantics to images, is becoming the next frontier of visual understanding [38, 30, 39].”
Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, neutral): “DALL-E [48], CogView [13] and M6 [35] propose ARbased text-to-image frameworks.”
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, methodology): “Single-stream107 models (Chen et al., 2020; Li et al., 2019; Ramesh 108 et al., 2021; Lin et al., 2021; Kim et al., 2021a) 109 concatenate the patch-wise or regional visual fea- 110 tures and textual embeddings and feed them into a 111 single model.”
FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, background): “…which aligns the textual and visual information into a uniﬁed semantic space; (ii) VisualBERT (Li et al., 2019), UNITER (Chen et al., 2020), M6 (Lin et al., 2021), and DALL-E (Ramesh et al., 2021) employ LM-like objectives, including both masked LM (e.g., Masked Language/Region Modeling), and…”
Transformers in Vision: A Survey (2021, methodology): “Recently, M6 model family [129] has been pretrained on large-scale Chinese text and image data sources to learn models up to 100 billion parameters using an ensemble approach.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “Researchers have applied the masking strategies and the encoder-decoder architecture to adapt models to generation tasks [15, 17, 18, 22].”
Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning (2021, methodology): “• Mixture of Experts (MoE): Scaling the model size with Sparsely Gated Mixture-of-Experts (MoE), such as GShard[8], Switch Transformer[9], Wudao[10, 11] and M6[12] .”
ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, methodology): “[21] have proposed a cross-modal pre-training method called M6(Multi-Modality to Multi-Modality Multitask Mega-Transformer) including 100 billion parameters for unified pre-training on multiple modalities data.”
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2022, methodology): “Later, encoderdecoder architectures are introduced to additionally handle multimodal generation tasks with a causal language modeling loss (Li et al., 2021b; Lin et al., 2021; Cho et al., 2021; Ding et al., 2021; Li et al., 2022; Wang et al., 2022).”
DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis (2020, background): “Recently, some large transformer-based text-to-image methods [7,24,35] show excellent performance on complex image synthesis.”
A Survey of Transformers (2021, background): “, visual question answering [55, 75, 77, 125], visual commonsense reasoning [75, 125], caption generation [22, 81, 128], speech-to-text translation [46] and text-to-image generation [29, 81, 107].”
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (2024, unknown): “In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Lin et al., 2021; Du et al., 2022; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures.”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “50 M6 [103] arXiv-2021 image-text Trans LM The largest pretrained model in Chinese 100B -”
Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities (2023, background): “In recent years, researchers have shown considerable interest in vision-language learning (Su et al., 2019; Chen et al., 2020; Li et al., 2020; Zhang et al., 2021; Li et al., 2021b; Lin et al., 2021; Kim et al., 2021; Dou et al., 2022; Li et al., 2021a, 2022), especially in the development of multi-”
M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining (2021, methodology): “[19; 49; 47; 41] Sparse expert models with large model capacity are capable of fast training owing to the combination of data parallelism and expert parallelism [38; 17; 13; 35], and it is even accessible to train a 1-trillion-parameter transformer with no more than 500 GPUs [45].”
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond (2023, background): “…shown considerable interest in vision-language learning (Su et al., 2019; Chen et al., 2020; Li et al., 2020; Zhang et al., 2021; Li et al., 2021b; Lin et al., 2021; Kim et al., 2021; Dou et al., 2022; Zeng et al., 2021; Li et al., 2021a, 2022), especially in the development of multi-task…”