Efficient Rematerialization for Deep Networks

By Ravi Kumar, M. Purohit, Zoya Svitkina, Erik Vee, Joshua R. Wang. 2019

64 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “Both our pipelining approach and our SPMD approach make use of gradient checkpointing (Chen et al., 2016), which is also called rematerialization in certain literature (Kumar et al., 2019; Jain et al., 2020).”
Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality (2022, methodology): “Our key observation is that one can unroll the sampling chain of a diffusion model and use reparametrization trick (Kingma & Welling, 2013) and gradient rematerialization (Kumar et al., 2019a) to optimize over a class of parametric few-step samplers with respect to a global objective function.”
Dynamic Tensor Rematerialization (2020, methodology): “For example, the common LRU heuristic is “minimize 1/s(t),” the GreedyRemat heuristic from Kumar et al. [2019] is “minimize 1/m(t),” and the MSPS heuristic from Peng et al.”