PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

By Yanli Zhao, A. Gu, Rohan Varma, Liangchen Luo, Chien-chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer and 6 more. 2023

795 citations recorded by the source corpus.

It is widely acknowledged that large models have the potential to deliver superior performance across a broad range of domains. Despite the remarkable progress made in the field of machine learning systems research, which has enabled the development and exploration of large models, such abilities remain confined to a small group of advanced users and industry leaders, resulting in an implicit technical barrier for the wider community to access and leverage these technologies. In this paper, we introduce PyTorch Fully Sharded Data Parallel (FSDP) as an industry-grade solution for large model training. FSDP has been closely co-designed with several key PyTorch core components including Tensor implementation, dispatcher system, and CUDA memory caching allocator, to provide non-intrusive user experiences and high training efficiency. Additionally, FSDP natively incorporates a range of techniques and settings to optimize resource utilization across a variety of hardware configurations. The experimental results demonstrate that FSDP is capable of achieving comparable performance to Distributed Data Parallel while providing support for significantly larger models with near-linear scalability in terms of TFLOPS.

Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Hardware and Architecture, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Optimization & Training

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “To train quickly with large batch sizes, we use FSDP (Zhao et al., 2023).”
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models (2023, methodology): “OpenFlamingo-4B models were trained using model sharding with Fully Sharded Data Parallel [43]; other models were trained using only data parallel.”
InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, positive): “These include fully shared data parallelism (FSDP) [182] to partition model parameters across GPUs, data packing [13] to reduce padding tokens while balancing the token computation load across ranks for improved training efficiency, FP8 training based on DeepGEMM [68] and liger-kernel’s fused…”
YaRN: Efficient Context Window Extension of Large Language Models (2023, positive): “For s = 16 we fine-tuned for 400 steps with global batch size 64 using PyTorch [26] Fully Sharded Data Parallelism [42] and Flash Attention 2 [13] on the PG19 dataset [29] chunked into 64k segments bookended with the BOS and EOS token.”
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, methodology): “In this work, we train our model on TPU-V4 [53] with FSDP [141] using TorchXLA.”
LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, neutral): “(Zhao et al., 2023) or Zero-3 (Ra-jbhandari et al., 2020), on 32 H100 GPUs.”
MegaScale: Scaling Large Language Model Training to More Than 10, 000 GPUs (2024, methodology): “Inspired by PyTorch FSDP [15], the initial all-gather operation is pre-fetched at the beginning of each iteration, allowing it to overlap with data loading operations, effectively reducing the communication time by a factor of 1 / ( 2 ∗ vpp _ size ) .”
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, methodology): “First, the method relies heavily on the availability of open-source pre-training datasets and large language models.”
Scaling and evaluating sparse autoencoders (2024, neutral): “The very small number of layers creates a challenge for parallelism - it makes pipeline parallelism [Huang et al., 2019] and FSDP [Zhao et al., 2023] inapplicable.”
OpenVLA: An Open-Source Vision-Language-Action Model (2024, methodology): “…of a generic architecture, not custom-made for robot control, allows us to leverage the scalable infrastructure underlying modern VLM training [75–77] and scale to training billion-parameter policies with minimal code modifications, and (3) it provides a direct pathway for robotics to benefit…”
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (2024, methodology): “With these requirements, we implement our training codebase in PyTorch, using Fully Sharded Data Parallel (FSDP; Zhao et al., 2023) and BF16 mixed precision.”
Qwen-Image Technical Report (2025, positive): “Given the large parameter size of the Qwen-Image model, using FSDP (Zhao et al., 2023) alone is insufficient to fit the model on each GPU.”
Extending Context Window of Large Language Models via Positional Interpolation (2023, methodology): “, 2019) with Fully Sharded Data Parallel (Zhao et al., 2023) and Flash Attention (Dao et al.”
OLMo: Accelerating the Science of Language Models (2024, methodology): “We train our models using the ZeRO optimizer strategy (Rajbhandari et al., 2019) via PyTorch’s FSDP framework (Zhao et al., 2023), which reduces memory consumption by sharding the model weights and their corresponding optimizer state across GPUs.”