Paper Finds Filter Metric Mismatch in Group-Relative RL With Shaped Rewards

A new arXiv paper examines how group-relative policy optimization methods such as GRPO drop rollout groups that show no contrast, through a dynamic sampling step, while real implementations let users configure the filter metric. The authors identify and measure a mismatch between that metric and the predicate used to decide which groups to discard, describing resulting "phantom advantages" when rewards are shaped. The work argues the filter metric choice is safety-critical rather than a minor implementation detail.

ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR

A revised arXiv paper proposes ThinkPrior, a method for choosing cold-start prompts in reinforcement learning with verifiable rewards without running any rollouts first. The authors observe that the KL-free reward-advantage term used in group relative policy optimization depends on how much reward varies within a group of rollouts, which collapses when a prompt is uniformly easy or hard. They use difficulty priors to pick prompts that are likely to produce useful within-group variation, aiming to make early training more efficient.

Paper proposes averaging SFT and RL policy logits for LLM post-training

An arXiv paper introduces a post-training method that blends supervised fine-tuning with reinforcement learning by averaging the output logits of a frozen reference policy with those of the trainable policy. The technique is integrated into GRPO, letting a model retain its supervised behaviors while still improving on verifiable rewards during reinforcement learning.

Hugging Face guide: 100 GRPO steps improve structured outputs from a 350M model

A new Hugging Face tutorial demonstrates using GRPO, a reinforcement learning technique, to fine-tune a small 350M-parameter model so it reliably generates valid structured outputs such as JSON. The walkthrough shows that roughly 100 training steps are enough to meaningfully improve format adherence, and it includes code for reproducing the results with open-source tooling.

WHY IT MATTERS ↘Format adherence for structured outputs like JSON is a persistent production bottleneck, and showing that ~100 GRPO steps fix it on a 350M model means teams can handle such workloads with tiny, cheaply trainable local models instead of frontier APIs. That lowers inference costs and latency, enables on-device deployment, and reduces dependence on vendor-gated structured-output features.