Training

By R. Sarre. 2020

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep learning (2021, methodology): “Large model training is generally trained in mixed precision, where the forward and backward propagation are done in FP16 and the parameter updates in FP32 [36].”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, neutral): “This idea has been extended in DeiT [61].”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, background): “Early efforts achieve this goal with fine-tuned generative models (Anaby-Tavor et al., 2020; Puri et al., 2020; Kumar et al., 2020; Lee et al., 2021).”
Understanding the Effects of RLHF on LLM Generalisation and Diversity (2023, methodology): “This is a set of inputs taken from a variety of open-source instruction following and dialogue training and evaluation datasets (Bai et al., 2022; Ganguli et al., 2022; Gudibande, 2023; Köpf et al., 2023; Wang et al., 2023; Zheng et al., 2023), curated by Dubois et al. (2023).”
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization (2023, methodology): “80.0 31.3 for various input resolutions starting 1024 1024 on an iPhone 12 Pro mobile WithTrain-TimeOverparam.”
Combating Noisy Labels by Agreement: A Joint Training Method with Co-Regularization (2020, methodology): “Since all datasets are clean except Clothing1M , following [28, 29], we need to corrupt these datasets manually by the label transition matrix Q, where given that noisy ˜ y is ﬂipped from clean y. Assume that the matrix Q has two representative structures: (1) Symmetry ﬂipping [36]; (2) Asymmetry…”
Activating More Pixels in Image Super-Resolution Transformer (2022, methodology): “6 shows the quantitative comparison of our approach and the state-of-the-art methods: EDSR [32], RCAN [68], SAN [8], IGNN [72], HAN [43], NLSN [42], RCAN-it [34], as well as approaches using ImageNet pre-training, i.e. , IPT [6] and EDT [27].”
PreDet: Large-scale weakly supervised pre-training for detection (2021, unknown): “In particular, [19, 12, 48, 67] showed that models trained from scratch achieve comparable performance with ImageNet-pretrained models.”
DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters (2020, methodology): “In addition, other features of DeepSpeed will be discussed such as mixed precision [2] support to take full advantage of NVIDIA’s Volta architecture [3].”