RLVR GPU training costs, benchmarks, and pricing

Measured GRPO throughput (ROCm + veRL; short GSM8K rollouts)

The AMD ROCm walkthrough (April 2025) reports GRPO and PPO on GSM8K with veRL v0.3.0 on MI300X and H100—i.e. RLHF with rule-based rewards on math, which is the standard pattern when response lengths are short (512–1024 tokens in their table).¹ Those runs give a clean GPU and algorithm comparison (GRPO 1.7–2.5× faster than PPO in the same harness) before we extrapolate to regimes where long rollouts dominate cost.¹²

GSM8K in this stack: GSM8K is Cobbe et al.’s grade-school math word problems (arXiv:2110.14168). veRL’s GSM8K example scores completions with a simple rule (parse the answer after `####`, compare to the label)—a verifiable reward that fits short generations and aligns with how many math RL pipelines are built; veRL’s docs describe that agent under the broad RLHF umbrella.³⁸ The ROCm numbers are measured system throughputs for that recipe; the rest of the note scales from them toward long-CoT RLVR where decoding dominates.

Additional related measurements come from Yotta Labs on MI300X + veRL⁸ and the OpenRLHF paper.² This 7B measured slice is still where most public tok/GPU/s tables live—few comparable public rows exist for 14B, 32B, or 70B in the same reporting format.

These benchmarks used 8 GPUs per node with train_batch_size=1024.¹ Two critical patterns emerge: GRPO achieves 1.7–2.5× higher throughput than PPO because it eliminates the critic model entirely,⁹ and MI300X outperforms H100 by 1–23% across all configurations, likely due to its larger 192GB HBM3 memory enabling more efficient batching.¹⁸ Yotta Labs separately confirmed MI300X efficiency with veRL v0.5.0, achieving 14.01% training MFU for 7B GRPO with optimal TP=1 on a single MI300X, and near-linear scaling across data parallelism dimensions.⁸

The OpenRLHF paper provides a framework-level comparison: OpenRLHF completes one GSM8K GRPO epoch in 1,657 seconds versus TRL's 5,189 seconds — a 3.1× speedup — on identical hardware and hyperparameters.²

Estimated throughput across model sizes requires significant extrapolation

Published RLVR throughput data is almost exclusively at 7B scale. The estimates below combine the measured 7B baselines with a rough, non-cited vLLM-style inference scaling heuristic (7B-class vs 32B-class decode often differs by roughly ~5× on a single GPU in community reports; re-measure for your stack),⁴⁰ the GPTOSS-20B MoE training benchmark (~500–598 tok/s on 512 H800 GPUs with veRL+Megatron),¹⁰ and the OLMo 3 32B RL training disclosure showing inference-to-training compute ratios of 5–14×.¹¹

Critical caveat: These estimates assume short (1K) response lengths matching the benchmark conditions. At RLVR-typical response lengths of 8K–32K tokens, overall step throughput stays roughly proportional (more tokens produced per step), but each step takes 8–32× longer to complete because autoregressive generation scales linearly with output length. KV cache pressure at longer contexts can also force batch size reductions, further degrading effective throughput. The QeRL paper notes that typical RL training for reasoning models takes 20–100 hours on 8× H100 GPUs — suggesting effective throughput is significantly lower than these per-token rates imply when accounting for real-world RLVR conditions.¹²

Wall-clock training times from published RLVR runs

The table below compiles selected RLVR-style training runs with disclosed compute details. Dollar costs use illustrative $/GPU-hour assumptions where sources do not publish invoices (see ⁴²). Costs span from a third-party README ~$2.62 to order-of-magnitude ~$200,000, driven primarily by model scale and rollout length.

DeepSeek-R1-Zero represents the largest published RLVR run: 512 H800 GPUs for 198 hours training the 671B MoE model with GRPO.⁶⁷ The full DeepSeek-R1 pipeline (including V3 pre-training) consumed 2.788 million H800 GPU-hours at an estimated $5.58M (same order-of-magnitude $/GPU-hr caveat as ⁴²).⁷ At the other extreme, community GRPO-on-VLM writeups report ~30 minutes on 8 A100s with a README-derived total of ~$2.62 (GPU-hours × quoted rates)—use as an illustration, not a peer-reviewed benchmark.⁵ DeepScaleR's iterative context-length scaling approach (8K→16K→24K) reduced estimated compute from ~70K to just 3,800 A100-hours — demonstrating that curriculum-based context scaling is a critical cost optimization for RLVR.¹⁶

RLVR is faster than RLHF but far slower than SFT

GRPO eliminates the critic model that PPO requires,⁹ and practitioners often cite an order-of-magnitude ~40–50% memory savings versus four-model PPO+RM stacks—treat that band as a heuristic, not a controlled measurement in this note. The ROCm + veRL GRPO vs PPO comparison is measured: GRPO is 1.7–2.5× faster than PPO under the same settings.¹ Traditional PPO-based RLHF requires four large models in GPU memory simultaneously (policy, reference, reward model, critic), while GRPO needs only two (policy and reference).⁹ DAPO goes further by removing the KL penalty entirely, eliminating even the reference model.¹⁴

The throughput hierarchy is clear from measured data. GRPO on a 7B model achieves 1,544 tok/GPU/s versus PPO's 907 tok/GPU/s on an H100 — a 1.70× improvement.¹ On MI300X, GRPO reaches 1,748 tok/GPU/s versus PPO's 921 tok/GPU/s (1.90×).¹ One non-peer-reviewed Substack overview claims GRPO can reduce overall training cost to roughly 1/18 of some traditional PPO-style RL setups when memory savings enable larger batches—use as anecdotal context, not a universal ratio.⁹

However, RLVR's rule-based verification advantage (no neural reward model forward pass) is substantially offset by its long rollouts. The DC-SFT paper reports that in their VLM setting, SFT achieved about 4.9× higher training efficiency than GRPO (see their tables for exact comparisons—do not read this as a universal constant across tasks).¹⁹ Multiple sources confirm the 5–10× slowdown of online RL versus offline SFT, driven by autoregressive rollout generation consuming 70–90% of RL training time.³⁴²⁰ A single RLVR training step can take minutes to over an hour depending on model size and rollout length, compared to seconds for SFT. The OLMo 3 32B reasoner allocated 20 H100 nodes for inference alongside 8 for training — inference consumed 5–14× more compute than policy updates, with learner GPUs idle 75% of the time waiting for rollout data.¹¹

Framework landscape for RLVR training

Four major frameworks dominate RLVR training, each with distinct strengths. The table below reflects capabilities as of early April 2026.

OpenRLHF (widely used open-source RLHF/RLVR stack; EMNLP 2025) is the throughput leader for dense models up to 70B in the OpenRLHF team’s own comparisons, achieving 1.22–1.68× speedup over veRL in long-CoT RLVR settings across 1.5B–14B models with 1K–8K generation lengths.² Its Ray+vLLM+DeepSpeed architecture is battle-tested by Google, ByteDance, NVIDIA, and Tencent.²¹ The team explicitly recommends REINFORCE++-baseline for RLVR tasks due to its robustness across reward patterns.²

veRL (EuroSys 2025; large public footprint on GitHub) has broad algorithm support and large proven scale — the DAPO paper's results were produced using veRL,¹⁴ and it has been validated on DeepSeek-V3 671B and Qwen3-235B MoE models.²² Its Megatron backend enables expert parallelism essential for MoE training. Note that OpenRLHF's published speed advantage was measured against veRL v0.4.0; veRL has since released significant optimizations through v0.7.1 that may have closed this gap.²²²

TRL (Hugging Face; frequent releases) prioritizes accessibility over raw throughput.²³ Its GRPOTrainer requires minimal code and integrates natively with Hugging Face's ecosystem. It is ~3.1× slower than OpenRLHF on GRPO benchmarks,² making it best suited for prototyping and smaller-scale training. The Open-R1 and Mini-R1 reproduction projects both use TRL.¹⁸

NeMo RL (successor to deprecated NeMo-Aligner) is NVIDIA's enterprise offering, uniquely supporting end-to-end FP8 training and Megatron Core's full 3D parallelism suite.²⁴ It trained Nemotron 3 Nano with GRPO across multiple environments simultaneously, using up to 49K-token generation lengths.²⁵

Notable emerging frameworks include AReaL (Ant Group/Tsinghua), achieving 2.77× speedup via fully asynchronous RL,²⁰ and ROLL Flash (Alibaba), reaching 2.24× speedup on RLVR through queue scheduling and rollout-train decoupling.³

Cloud GPU pricing for RLVR workloads in April 2026

RLVR training demands multi-GPU clusters with high-bandwidth interconnect (NVLink intra-node, InfiniBand inter-node). Prices below are per GPU per hour, transcribed from provider websites and aggregators in March–April 2026 where footnoted.^2627282930 Rows without an inline footnote marker (e.g. some prepaid or bare-metal SKUs) should be spot-checked on the vendor site—live quotes move weekly.⁴¹

H100 SXM 80GB

H200 SXM 141GB

MI300X 192GB

A100 80GB SXM

For serious RLVR training at scale (multi-node with InfiniBand), Lambda 1-Click Clusters at $2.76/GPU/hr for H100 and DigitalOcean at $1.99/GPU/hr for MI300X represent the strongest price-performance combinations with production-grade interconnect.²⁸³¹ Vast.ai offers the lowest absolute prices but lacks guaranteed InfiniBand connectivity.²⁹

Cost per million tokens processed during RLVR training

The following calculations combine measured GRPO throughput at 7B scale (1K response length) with current GPU pricing.¹ These represent best-case costs — real-world RLVR with longer rollouts will have higher per-step costs (though similar per-token costs if memory permits maintaining batch size).

H100 SXM — 7B GRPO (1,544 tok/GPU/s = 5.56M tok/GPU/hr) ¹

MI300X — 7B GRPO (1,748 tok/GPU/s = 6.29M tok/GPU/hr) ¹

Estimated cost scaling by model size (H100, Lambda 1-Click ~$2.76/hr) ³¹

MI300X provides a 30–45% cost advantage over H100 for RLVR training at current market rates, combining higher throughput (+13% for 7B GRPO ¹) with lower pricing (DigitalOcean MI300X at $1.99 ²⁸ vs Lambda H100 at $2.76 ³¹). veRL documents strong MI300X support (ROCm integration and tuning notes).⁸ For other frameworks, confirm AMD paths in upstream docs before committing to a stack.

Long rollouts dominate RLVR cost structure

The defining characteristic of RLVR versus traditional RLHF is the length of generated rollouts. While RLHF typically generates 512–2048 token responses, RLVR reasoning chains routinely reach 8K–32K tokens, with production systems like DAPO using max_response_length of 20,480 tokens¹⁴ and NeMo RL's Nemotron training reaching 49K tokens.²⁵ Production RLVR workload characterization (PolyTrace) shows math reasoning tasks averaging ~9,839 output tokens per sample.⁴

Rollout generation consumes 70–90% of total RLVR training time.³⁴ This means the bottleneck is autoregressive decoding — a memory-bandwidth-bound operation that cannot be trivially accelerated by adding more compute. Each GRPO step generates G completions per prompt (typically G=8–64; DeepSeek-R1 used G=64 ⁶), multiplying the generation burden. For a concrete example from the HuggingFace "Keep the Tokens Flowing" analysis:²⁰ generating 512 rollouts at 8K tokens for a 32B model on 8 H100 inference GPUs takes approximately 7 minutes for generation alone, before any gradient computation.

The long-tail distribution of rollout lengths creates severe GPU idling. ROLL Flash found that the longest responses can exceed the median by over 20×, meaning most GPUs finish early and wait for stragglers in synchronous systems.³ The OLMo 3 32B reasoner's learner GPUs spent 75% of time idle waiting for inference data.¹¹ This has spawned several optimization approaches:

• Asynchronous training (ROLL Flash,³ AReaL,²⁰ PipelineRL) decouples generation from training, achieving 2.0–2.8× speedups by overlapping these phases
• Dynamic sampling (DAPO ¹⁴) filters prompts where all responses are correct or all wrong, avoiding wasted computation on zero-gradient batches — achieving the same performance with 1/3 the training steps
• Overlong reward shaping (DAPO ¹⁴) applies soft penalties for responses exceeding a threshold rather than hard truncation, preventing training instability
• Token-level policy gradient loss (DAPO,¹⁴ veRL ²²) averages loss across total tokens rather than per-sample-then-per-batch, preventing gradient dilution for long sequences
• Dr. GRPO's debiased advantage ¹⁷ removes variance normalization and length divisors from GRPO, eliminating bias toward shorter responses and providing unbiased policy gradients
• Clip-Higher (DAPO ¹⁴) uses asymmetric clipping (ε_low=0.2, ε_high=0.28) to preserve exploration by allowing more room for increasing low-probability tokens, combating entropy collapse; related asymmetric clipping ideas appear in VAPO.³⁷
• NAT (Not All Tokens are Needed) ³³ performs policy optimization on only ~50% of tokens from each rollout while computing rewards on full responses, reducing activation memory
• FP8 precision (JetRL ³⁴) achieves 1.07–1.33× rollout speedup, but naive mixed-precision (BF16 training + FP8 rollout) fails catastrophically at context lengths beyond 8K due to numerical precision mismatches
• Iterative context scaling (DeepScaleR ¹⁶) trains at 8K→16K→24K progressively, reducing total compute by ~18× versus training at maximum length from the start

KV cache memory scales linearly with sequence length. For a Qwen-32B model, hosting with full 131K context increases memory requirements from ~70GB to approximately 400GB. Frameworks address this through dynamic memory management (veRL's `free_cache_engine=True` offloads KV cache after rollout generation ³⁵) and PagedAttention (vLLM) for non-contiguous memory allocation. Prefix caching is particularly valuable for GRPO since all G completions per prompt share the same prompt prefix — SGLang's RadixAttention provides 3–5× cache hit improvement in this multi-completion scenario.³⁶

Data quality assessment and key caveats

Every number in this report carries a confidence level that readers should understand when making cost projections.

Measured benchmarks (high confidence): AMD ROCm publishes GRPO/PPO throughputs for veRL v0.3 on GSM8K with rule-based rewards and short completions (7B models, 8 GPUs per node)—a solid baseline for GPU and algorithm comparisons.¹ Yotta Labs reports complementary MI300X + veRL numbers.⁸ DeepSeek-R1-Zero's training configuration (512×H800, ~198 hours) was disclosed via Stanford FMTI and Nature supplementary materials.⁶⁷ OpenRLHF vs TRL timing comparisons come from the peer-reviewed EMNLP 2025 paper.²

Derived with caveats (medium confidence): Wall-clock times and GPU-hours for open-source reproductions (DeepScaleR,¹⁶ SimpleRL-Zoo,¹⁵ Dr. GRPO ¹⁷) come from GitHub READMEs, blog posts, and WandB logs — credible but not peer-reviewed. The 70–90% rollout time proportion is consistent across ROLL Flash,³ veRL,²² OpenRLHF,² and NAT ³³ papers. The RLVR vs SFT slowdown band (5–10×) combines DC-SFT’s in-paper ~4.9× VLM comparison¹⁹ with production-style reports.¹¹

Estimated with significant uncertainty (lower confidence): Throughput estimates for 14B, 32B, and 70B models are extrapolations from 7B measured data combined with a heuristic inference scaling ratio (not a single cited benchmark row).⁴⁰ Real-world throughput depends heavily on batch size, parallelism strategy, sequence length, and framework optimizations. The 14B–70B rows in the throughput and cost tables should be treated as rough order-of-magnitude guides, not precision benchmarks. Cloud GPU pricing fluctuates; spot/marketplace rates (especially Vast.ai ²⁹) can vary by 2–3× within a single week. Un-footnoted list prices should be reverified on provider pages.⁴¹ The OpenRLHF vs veRL framework comparison was published by the OpenRLHF team ² against an older veRL version (v0.4.0); veRL's subsequent optimizations through v0.7.1 may have changed this relationship.²²

Unresolved gaps: Few public tables report long-rollout RLVR throughputs (8K–32K tokens) with the same tok/GPU/s detail as short-completion 7B baselines; few RLVR-oriented throughput benchmarks in this note’s sense exist for H200 GPUs. No framework has published comparable benchmarks across all GPU types. Qwen/QwQ training compute has never been publicly disclosed. DAPO's total GPU-hours on 128×H20 were not reported.¹⁴ The interaction between long rollout lengths (8K–32K) and per-token throughput under GPU memory pressure lacks systematic benchmarking — current data either measures short (1K) rollouts or reports only wall-clock totals without per-token rates.

Conclusion

RLVR training costs are dominated by a single bottleneck: autoregressive rollout generation of long reasoning chains.³⁴ The algorithmic efficiency of GRPO over PPO (1.7–2.5× measured throughput advantage,¹ plus the heuristic “fewer resident models” memory story ⁹) is real but secondary to the 70–90% of wall-clock time spent generating 8K–32K token completions.³¹¹ The most impactful cost optimizations are therefore architectural — asynchronous training (2–2.8× speedup ³²⁰), dynamic sampling (3× step reduction ¹⁴), and iterative context scaling (18× compute reduction ¹⁶) — rather than hardware-level. At current market rates, MI300X at $1.99/GPU/hr with 13% higher GRPO throughput than H100 offers the best raw price-performance,¹²⁸ though H100's broader framework support and InfiniBand availability make it the safer choice for production runs. The framework choice matters enormously: OpenRLHF's 3.1× advantage over TRL and 1.2–1.7× over veRL v0.4 on identical hardware ² represents a larger throughput delta than any GPU generational improvement. For organizations planning RLVR training, the decision tree is: veRL or NeMo RL for MoE models above 200B,²²²⁴ OpenRLHF for dense models up to 70B where throughput is critical,² and TRL for rapid prototyping ²³ — then invest heavily in the async and dynamic sampling optimizations that cut wall-clock time by 2–3× regardless of hardware choice.