A new arXiv preprint introduces a federated fine-tuning method designed for mixture-of-experts large language models. The approach aims to adapt these sparse-activation models to specific tasks while keeping training distributed. The abstract frames the work as addressing efficiency and capacity trade-offs in MoE architectures.
A new arXiv paper introduces OpWeave, a system that breaks LLM inference into finer-grained operators rather than coarse stages, extending recent work that separates attention from FFN or MoE execution during decoding. The authors argue this operator-level disaggregation improves how workloads are matched to heterogeneous hardware during serving.
A new arXiv paper introduces STHMoE, a mixture-of-experts architecture that uses hypergraphs to coordinate heterogeneous dependencies in spatio-temporal traffic data. The method is designed to handle the non-stationary and structurally dynamic patterns produced by large networks of urban sensors. It targets LLM-based forecasting for intelligent transportation systems.
A revised arXiv preprint introduces a multimodal recommendation method that combines collaborative signals with visual and textual item features. It uses modality-guided mixture of structured experts, where entropy-based routing decides how much to rely on each evidence source per user-item interaction. Diagnostic probes trained on individual modalities are used to partition held-out interactions and guide the routing behavior.
A new arXiv preprint examines whether decoder-only language models built with mixture-of-experts layers can serve as effective first-stage retrievers. The authors note that retrieval quality typically rises with backbone size, but dense models force every query and document through all parameters. The work targets strong retrieval performance while limiting the computational cost per query.
A new arXiv preprint describes a key-value cache layout that is physically split so work can be divided between CPU and GPU during long-context inference with Mixture-of-Experts models. The approach targets the case where single-GPU memory is insufficient and the KV cache must spill into host memory, where the stored data either feeds attention on the GPU or supports computation on the CPU. The proposed format is designed to keep those two uses balanced rather than treating the spilled cache as a single monolithic block.
A new arXiv preprint proposes MoME, a technique that combines conditional memory — token-indexed embedding tables that give a model cheap parametric lookups — with sparse capacity ideas inspired by Mixture-of-Experts. The method aims to make those lookups context-aware by routing them selectively, as part of broader efforts to scale language models more efficiently.
A new arXiv preprint introduces a graph-guided mixture-of-experts approach for predicting tumor survival outcomes using multiple data modalities. The method aims to combine heterogeneous clinical inputs within a single predictive framework. The paper appears in the cs.LG category as a new submission.
Researchers released T1, a 122-billion-parameter Mixture-of-Experts model trained via reinforcement learning to act as an agent in terminal environments. The work targets long-horizon workloads such as software development and scientific research, where sustained multi-step command-line use matters. It is presented as part of a broader shift in agent design away from short, single-turn interactions.
A new arXiv paper proposes FluxMoE, a serving system that separates where mixture-of-experts weights live from GPU memory constraints. Existing inference engines keep every expert resident on GPUs, which competes for space with the key-value cache and limits throughput. The work targets higher-performance MoE inference by changing how expert residency is managed.
A new arXiv paper tackles a limitation in Mixture-of-Experts models, which typically activate a fixed number of experts for each input. The authors propose an inference method that remains distribution-consistent when the number of active experts varies dynamically. The approach aims to preserve efficient inference in large foundation models while allowing more flexible expert routing.
A new arXiv paper tackles a weakness in adapting Mixture-of-Experts models: routing layers can destabilise during supervised fine-tuning, causing rarely used experts to lose their specialised knowledge. The authors introduce a tuning approach designed to retain this long-tailed expert information and compare it with earlier anti-collapse techniques such as DenseMixer and ESFT. The work addresses a practical bottleneck for teams adapting large MoE models to downstream tasks.
A new technical report introduces Ling 2.0, a family of reasoning-oriented foundation models built on a unified Mixture-of-Experts architecture that spans from tens of billions up to one trillion parameters. The series is released as an open language foundation, with the stated goal of strengthening general reasoning ability across all model sizes.
A new paper examines how to shrink Mixture-of-Experts language models by pruning experts, targeting models specialized for particular domains. The authors find that earlier compression techniques break down badly when the trimmed models are tested on general-purpose benchmarks. The work aims to make these large, sparsely activated models easier to deploy without losing performance.
A new arXiv paper shows that removing one internally represented direction associated with refusals can disable a 320-billion-parameter mixture-of-experts model's ability to decline harmful requests. The technique, known as directional ablation, requires no gradient-based training or optimization—only a small set of contrastive examples to locate the direction. The authors argue this reveals that safety training in very large models may depend on a surprisingly brittle, low-dimensional mechanism.
A revised arXiv paper studies mixtures of experts, conditional mixture models in which both the mixing weights and component densities depend on predictors. The work develops regularized estimation methods with feature selection for mixtures of generalized linear experts, supporting regression, classification, and model-based clustering of heterogeneous data.
A new technical report on arXiv describes Palmyra x6, a large language model built to power agent-style workflows in business settings. The team started from a Mixture-of-Experts base model and applied a post-training technique called Anchored Supervised Fine-Tuning, using a small dataset of verified, synthetically generated tool-use examples. The release focuses on enabling the model to reliably call external tools across multi-step tasks.