EasyR1: An Efficient, Scalable, Multi-Modality RL Training Framework

Used by Amazon Web Services

This project is a clean fork of the original veRL project to support vision language models, we thank all the authors for providing such a high-performance RL training framework.

EasyR1 is efficient and scalable due to the design of HybirdEngine and the latest release of vLLM's SPMD mode.

Features

• Supported models

  • Llama3/Qwen2/Qwen2.5/Qwen3 language models
  • Qwen2-VL/Qwen2.5-VL/Qwen3-VL vision language models
  • DeepSeek-R1 distill models

• Supported algorithms

  • GRPO
  • DAPO
  • Reinforce++
  • ReMax
  • RLOO
  • GSPO
  • CISPO

• Supported datasets

  • Any text, vision-text dataset in a specific format

• Supported tricks

  • Padding-free training
  • LoRA training
  • Resuming from the latest/best checkpoint
  • Wandb & SwanLab & Mlflow & Tensorboard tracking

Requirements

Software Requirements

• Python 3.9+
• transformers>=4.54.0
• flash-attn>=2.4.3
• vllm>=0.8.3

We provide a Dockerfile to easily build environments.

We recommend using the pre-built docker image in EasyR1.

docker pull hiyouga/verl:ngc-th2.8.0-cu12.9-vllm0.11.0
docker run -it --ipc=host --gpus=all hiyouga/verl:ngc-th2.8.0-cu12.9-vllm0.11.0

If your environment does not support Docker, you can consider using Apptainer:

apptainer pull easyr1.sif docker://hiyouga/verl:ngc-th2.8.0-cu12.9-vllm0.11.0
apptainer shell --nv --cleanenv --bind /mnt/your_dir:/mnt/your_dir easyr1.sif

Use USE_MODELSCOPE_HUB=1 to download models from the ModelScope hub.

Hardware Requirements

* estimated

Method	Bits	1.5B	3B	7B	32B	72B
GRPO Full Fine-Tuning	AMP	2*24GB	4*40GB	8*40GB	16*80GB	32*80GB
GRPO Full Fine-Tuning	BF16	1*24GB	1*40GB	4*40GB	8*80GB	16*80GB
GRPO LoRA Fine-Tuning	AMP	1*12GB	1*24GB	2*32GB	2*80GB	4*80GB

[!NOTE] Use worker.actor.fsdp.torch_dtype=bf16 and worker.actor.optim.strategy=adamw_bf16 to enable bf16 training.

Tutorial: Run Qwen2.5-VL GRPO on Geometry3K Dataset in Just 3 Steps

Installation

git clone https://github.com/hiyouga/EasyR1.git
cd EasyR1
pip install -e .

GRPO Full Training

bash examples/qwen2_5_vl_7b_geo3k_grpo.sh

GRPO LoRA Training

bash examples/qwen3_vl_4b_geo3k_grpo_lora.sh

Merge Checkpoint in Hugging Face Format

python3 scripts/model_merger.py --local_dir checkpoints/easy_r1/exp_name/global_step_1/actor

[!TIP] If you encounter issues with connecting to Hugging Face, consider using export HF_ENDPOINT=https://hf-mirror.com.

If you want to use SwanLab logger, consider using bash examples/qwen2_5_vl_7b_geo3k_swanlab.sh.

Custom Dataset

Please refer to the example datasets to prepare your own dataset.

• Text dataset: https://huggingface.co/datasets/hiyouga/math12k
• Image-text dataset: https://huggingface.co/datasets/hiyouga/geometry3k
• Multi-image-text dataset: https://huggingface.co/datasets/hiyouga/journeybench-multi-image-vqa
• Text-image mixed dataset: https://huggingface.co/datasets/hiyouga/rl-mixed-dataset

How to Understand GRPO in EasyR1

• To learn about the GRPO algorithm, you can refer to Hugging Face's blog.

How to Run 70B+ Model in Multi-node Environment

1. Start the Ray head node.

ray start --head --port=6379 --dashboard-host=0.0.0.0

1. Start the Ray worker node and connect to the head node.

ray start --address=<head_node_ip>:6379

1. Check the Ray resource pool.

ray status

1. Run training script on the Ray head node only.

bash examples/qwen2_5_vl_7b_geo3k_grpo.sh

See the veRL's official doc for more details about multi-node training and Ray debugger.

Other Baselines

We also reproduced the following two baselines of the R1-V project.

• CLEVR-70k-Counting: Train the Qwen2.5-VL-3B-Instruct model on counting problem.
• GeoQA-8k: Train the Qwen2.5-VL-3B-Instruct model on GeoQA problem.

Performance Baselines

See baselines.md.

Awesome Work using EasyR1

• MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources. [](https://github.com/LengSicong/MMR1) [](https://arxiv.org/abs/2509.21268)
• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models. [](https://github.com/Osilly/Vision-R1) [](https://arxiv.org/abs/2503.06749)
• Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement. [](https://github.com/dvlab-research/Seg-Zero) [](https://arxiv.org/abs/2503.06520)
• MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse. [](https://github.com/PzySeere/MetaSpatial) [](https://arxiv.org/abs/2503.18470)
• Temporal-R1: Envolving Temporal Reasoning Capability into LMMs via Temporal Consistent Reward. [](https://github.com/appletea233/Temporal-R1) [](https://arxiv.org/abs/2506.01908)
• NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation. [](https://github.com/John-AI-Lab/NoisyRollout) [](https://arxiv.org/pdf/2504.13055)
• GUI-R1: A Generalist R1-Style Vision-Language Action Model For GUI Agents. [](https://github.com/ritzz-ai/GUI-R1) [](https://arxiv.org/abs/2504.10458)
• FAST-GRPO: Fast-Slow Thinking framework that dynamically adapts reasoning depth based on question characteristics. [](https://github.com/Mr-Loevan/FAST) [](https://arxiv.org/abs/2504.18458)
• R1-Track: Direct Application of MLLMs to Visual Object Tracking via Reinforcement Learning. [](https://github.com/Wangbiao2/R1-Track)
• VisionReasoner: Unified Visual Perception and Reasoning via Reinforcement Learning. [](https://github.com/dvlab-research/VisionReasoner) [](https://arxiv.org/abs/2505.12081)
• MM-UPT: Unsupervised Post-Training for Multi-Modal LLM Reasoning via GRPO. [](https://github.com/waltonfuture/MM-UPT) [](https://arxiv.org/pdf/2505.22453)
• RL-with-Cold-Start: Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start. [](https://github.com/waltonfuture/RL-with-Cold-Start) [](https://arxiv.org/pdf/2505.22334)
• ViGoRL: Grounded Reinforcement Learning for Visual Reasoning. [](https://github.com/Gabesarch/grounded-rl) [](https://arxiv.org/abs/2505.23678)
• Revisual-R1: Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning. [](https://github.com/CSfufu/Revisual-R1) [](https://arxiv.org/abs/2506.04207)
• SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward. [](https://github.com/kxfan2002/SophiaVL-R1) [](https://arxiv.org/abs/2505.17018)
• Vision-Matters: Simple Visual Perturbations Can Boost Multimodal Math Reasoning. [](https://github.com/YutingLi0606/Vision-Matters) [](https://arxiv.org/abs/2506.09736)
• VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use. [](https://github.com/VTOOL-R1/vtool-r1) [](https://arxiv.org/abs/2505.19255)
• Long-RL: Scaling RL to Long Sequences. [](https://github.com/NVlabs/Long-RL) [](https://arxiv.org/abs/2507.07966)
• EditGRPO: Reinforcement Learning with Post-Rollout Edits for Clinically Accurate Chest X-Ray Report Generation. [](https://github.com/taokz/EditGRPO)
• ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping. [](https://github.com/shawn0728/ARES) [](https://arxiv.org/abs/2510.08457)
• VPPO: Spotlight on Token Perception for Multimodal Reinforcement Learning. [](https://github.com/huaixuheqing/VPPO-RL) [](https://arxiv.org/abs/2510.09285)
• IE-Critic-R1: Advancing the Explanatory Measurement of Text-Driven Image Editing for Human Perception Alignment. [](https://github.com/Coobiw/IE-Critic-R1) [](https://arxiv.org/abs/2511.18055)
• OneThinker: All-in-one Reasoning Model for Image and Video. [](https://github.com/tulerfeng/OneThinker) [](https://arxiv.org/abs/2512.03043)
• MetaphorStar: Image Metaphor Understanding and Reasoning with End-to-End Visual Reinforcement Learning. [](https://github.com/MING-ZCH/MetaphorStar) [](https://arxiv.org/abs/2602.10575)

TODO

• Support ulysses parallelism for VLMs (middle priority).
• Support more VLM architectures.

[!NOTE] We will not provide scripts for supervised fine-tuning and inference in this project. If you have such requirements, we recommend using LlamaFactory.

Known bugs

These features are temporarily disabled for now, we plan to fix them one-by-one in the future updates.

• Vision language models are not compatible with ulysses parallelism yet.

Discussion Group

👋 Join our WeChat group.

FAQs

ValueError: Image features and image tokens do not match: tokens: 8192, features 9800

Increase the data.max_prompt_length or reduce the data.max_pixels.

RuntimeError: CUDA Error: out of memory at /workspace/csrc/cumem_allocator.cpp:62

Reduce the worker.rollout.gpu_memory_utilization and enable worker.actor.offload.offload_params.

RuntimeError: 0 active drivers ([]). There should only be one.

Uninstall deepspeed from the current python environment.

Citation

Core contributors: Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, Yuwen Xiong and Richong Zhang

We also thank Guangming Sheng and Chi Zhang for helpful discussions.

@misc{zheng2025easyr1,
  title        = {EasyR1: An Efficient, Scalable, Multi-Modality RL Training Framework},
  author       = {Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, Yuwen Xiong, Richong Zhang},
  howpublished = {\url{https://github.com/hiyouga/EasyR1}},
  year         = {2025}
}

We recommend to also cite the original work.

@article{sheng2024hybridflow,
  title   = {HybridFlow: A Flexible and Efficient RLHF Framework},
  author  = {Guangming Sheng and Chi Zhang and Zilingfeng Ye and Xibin Wu and Wang Zhang and Ru Zhang and Yanghua Peng and Haibin Lin and Chuan Wu},
  year    = {2024},
  journal = {arXiv preprint arXiv: 2409.19256}
}