A new arXiv paper analyzes the actor-only variants of Proximal Policy Optimization that are commonly used to post-train large language models. The authors derive non-asymptotic global convergence guarantees for the clipped PPO objective, addressing how the clipping mechanism affects optimization. The work offers theoretical grounding for a method widely deployed in practice.
An arXiv paper reports that reinforcement learning does not lift large language model performance evenly across a dataset. Gains are large on problems the model can already solve and much smaller on difficult ones, a pattern the authors call the Matthew Effect. The finding suggests current RL training methods may widen the gap between easy and hard tasks.
A revised arXiv paper introduces REGEN, a method that recycles replay data to distill specialized expert policies into a more general model using offline reinforcement learning. The approach targets the cost of scaling online RL, which is widely used to develop long-horizon reasoning and tool-use abilities in large language models. The v3 revision appears in both the cs.AI and cs.LG listings.
A new arXiv paper proposes an approach called on-policy self-distillation that acts as a "policy reheater" for reinforcement learning with verifiable rewards. The authors target entropy collapse, a failure mode where model policies become overly concentrated, cutting rollout diversity and weakening the learning signal. The method is presented as a way to keep exploration alive during RL training of large language models.
A revised arXiv preprint argues that the spectral norm used by the Muon optimizer is not a good measure of functional change for a language model's output head, because softmax removes scale invariance assumptions. The author proposes deriving an update geometry tailored to how that parameter block actually functions. The work is theoretical and has not been peer reviewed.
An arXiv paper introduces a post-training method that blends supervised fine-tuning with reinforcement learning by averaging the output logits of a frozen reference policy with those of the trainable policy. The technique is integrated into GRPO, letting a model retain its supervised behaviors while still improving on verifiable rewards during reinforcement learning.
A cross-listed arXiv paper revisits learned value models, which are often avoided in LLM reinforcement learning, and proposes using generative critics in their place. The approach targets the credit assignment problem by enabling fine-grained advantage estimation in the style of classical actor-critic methods during RL training.
A new arXiv preprint introduces BRACE, a method aimed at value-function staleness in asynchronous reinforcement learning. As training of language models increasingly relies on asynchronous setups, delays between acting and learning bias the critic toward outdated policies, while prior asynchronous-training fixes targeted only the actor. The proposed approach applies an anchored Bellman-residual correction to keep the critic aligned with the current policy.
A new arXiv paper analyzes EGGROLL, a method that makes evolution strategies viable for large language models by swapping dense Gaussian weight perturbations for low-rank Gaussian products, frequently of rank one. While this substitution reduces computation, the authors note it imposes severe geometric constraints on the perturbation space. The work aims to explain those limitations and propose improvements for scaling the approach.
A new arXiv preprint introduces a post-training method that uses solver feedback to guide self-distillation, aiming to improve how language models turn natural-language problem descriptions into operations research formulations. The approach is positioned as a way to go beyond training on verified answers alone when bootstrapping such models.