HybridFlow: A Flexible and Efficient RLHF Framework

By Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, Chuan Wu. European Conference on Computer Systems, 2024

2,000 citations recorded by the source corpus.

Reinforcement Learning from Human Feedback (RLHF) is widely used in Large Language Model (LLM) alignment. Traditional RL can be modeled as a dataflow, where each node represents computation of a neural network (NN) and each edge denotes data dependencies between the NNs. RLHF complicates the dataflow by expanding each node into a distributed LLM training or generation program, and each edge into a many-to-many multicast. Traditional RL frameworks execute the dataflow using a single controller to instruct both intra-node computation and inter-node communication, which can be inefficient in RLHF due to large control dispatch overhead for distributed intra-node computation. Existing RLHF systems adopt a multi-controller paradigm, which can be inflexible due to nesting distributed computation and data communication. We propose HybridFlow, which combines single-controller and multi-controller paradigms in a hybrid manner to enable flexible representation and efficient execution of the RLHF data flow. We carefully design a set of hierarchical APIs that decouple and encapsulate computation and data dependencies in the complex RLHF dataflow, allowing efficient operation orchestration to implement RLHF algorithms and flexible mapping of the computation onto various devices. We further design a 3D-HybridEngine for efficient actor model resharding between training and generation phases, with zero memory redundancy and significantly reduced communication overhead. Our experimental results demonstrate 1.53x~20.57× throughput improvement when running various RLHF algorithms using HybridFlow, as compared with state-of-the-art baselines. HybridFlow source code is available at https://github.com/volcengine/verl

Subfields: Artificial Intelligence, Information Systems and Management

View Paper: https://www.semanticscholar.org/paper/f2d0f3d47ae850f49a58f4977393bd0025af4bec

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tong Lu
2. Maosong Sun
3. Ning Ding
4. Yuan Yao
5. Zhaoyang Liu
6. Jianfeng Gao
7. Noah D. Goodman
8. Weijie Su

In Information Systems and Management

1. Maosong Sun

Citations in the corpus

• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (2025, positive): “For GRPO training, we set the policy LLM learning rate to 1e-6 and sample 5 responses per prompt, following the GRPO implementation in Verl (Sheng et al., 2024) 2 .”
• ToolRL: Reward is All Tool Learning Needs (2025, positive): “We conduct all RL experiments using the veRL framework (Sheng et al., 2024), adopting the GRPO algorithm detailed in the previous section.”
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (2025, positive): “Training details We adapt our training codebase from verl (Sheng et al., 2024) and follow the training recipe of DAPO (Yu et al., 2025), one of the state-of-the-art RL algorithms for LLMs.”
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, positive): “For the online stage, we use verl [111] as our codebase.”
• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (2025, positive): “…which is subsequently trained on the collected 10K math dataset using GRPO on 64 NVIDIA H800 80G GPUs for approximately 2 days in Verl training framework [52, 53], while following the two-stage PTST approach (note that we do not use the third stage checkpoints), i.e. , S was set to 2.”
• Group-in-Group Policy Optimization for LLM Agent Training (2025, neutral): “…episode relative advantages A E ( τ i ) via Equation (3) 12: Build step-level groups G S (˜ s ) via the anchor states 13: Compute step relative advantages A S ( a t ) via Equation (7) 14: // Policy update phase 15: Combine advantages: A ( a Update policy θ by maximizing objective J GiGPO ( θ )”
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025, positive): “We re-implement popular RL algorithms using the VeRL framework (Sheng et al., 2024) for fair comparison, including PPO (Schulman et al., 2017), GRPO (Shao et al., 2024), Reinforce++ (Hu, 2025), RLOO (Ahmadian et al., 2024), ReMax (Li et al., 2023), and DAPO (Yu et al., 2025).”
• Process Reinforcement through Implicit Rewards (2025, positive): “We use veRL (Sheng et al., 2024) to conduct experiments.”
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale (2025, positive): “We adopt the verl framework [20] for training.”
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs (2025, positive): “Our reinforcement learning experiments build on the VERL library (Sheng et al., 2024), utilizing the TinyZero (Pan et al., 2025) implementation.”
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild (2025, neutral): “We train our models using the verl (Sheng et al., 2024) framework.”
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models (2025, positive): “We start RL from the base models following the “Zero” setting (DeepSeek-AI et al., 2025) with the veRL framework (Sheng et al., 2024).”
• ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows (2025, unknown): “For reinforcement optimization (GRPO), we adopt the verl framework (Sheng et al., 2024), training for three epochs on top of the SFT checkpoint.”
• Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning (2026, unknown): “Group Name: A concise technical label for the approach.”
• Learning Adaptive Parallel Execution for Efficient Code Localization (2026, unknown): “The training framework is built on RLLM (Li et al., 2025), which leverages veRL (Sheng et al., 2024) for distributed RL. Sampling Configuration To ensure diversity in trajectory exploration, we set the sampling temperature to 0.7 during rollouts.”
• Agentic Reinforcement Learning for Real-World Code Repair (2025, unknown): “Since VerL cannot perform rollouts with GPT models, we report GPT results only for the full pipeline.”