A new arXiv preprint examines whether decoder-only language models built with mixture-of-experts layers can serve as effective first-stage retrievers. The authors note that retrieval quality typically rises with backbone size, but dense models force every query and document through all parameters. The work targets strong retrieval performance while limiting the computational cost per query.
A revised arXiv preprint introduces a multimodal recommendation method that combines collaborative signals with visual and textual item features. It uses modality-guided mixture of structured experts, where entropy-based routing decides how much to rely on each evidence source per user-item interaction. Diagnostic probes trained on individual modalities are used to partition held-out interactions and guide the routing behavior.
A new arXiv paper introduces STHMoE, a mixture-of-experts architecture that uses hypergraphs to coordinate heterogeneous dependencies in spatio-temporal traffic data. The method is designed to handle the non-stationary and structurally dynamic patterns produced by large networks of urban sensors. It targets LLM-based forecasting for intelligent transportation systems.
A new arXiv preprint introduces a federated fine-tuning method designed for mixture-of-experts large language models. The approach aims to adapt these sparse-activation models to specific tasks while keeping training distributed. The abstract frames the work as addressing efficiency and capacity trade-offs in MoE architectures.
A new arXiv preprint introduces a graph-guided mixture-of-experts approach for predicting tumor survival outcomes using multiple data modalities. The method aims to combine heterogeneous clinical inputs within a single predictive framework. The paper appears in the cs.LG category as a new submission.
North Small Translate is a new open-weight translation model that also follows instructions, described as being trained on the same base as Cohere's Command A Plus mixture-of-experts system with 25 billion active parameters. The authors position it as a cost-effective option for machine translation workloads that need instruction-following behaviour.
A new paper examines how to shrink Mixture-of-Experts language models by pruning experts, targeting models specialized for particular domains. The authors find that earlier compression techniques break down badly when the trimmed models are tested on general-purpose benchmarks. The work aims to make these large, sparsely activated models easier to deploy without losing performance.
A new arXiv paper tackles a limitation in Mixture-of-Experts models, which typically activate a fixed number of experts for each input. The authors propose an inference method that remains distribution-consistent when the number of active experts varies dynamically. The approach aims to preserve efficient inference in large foundation models while allowing more flexible expert routing.
A revised arXiv paper studies mixtures of experts, conditional mixture models in which both the mixing weights and component densities depend on predictors. The work develops regularized estimation methods with feature selection for mixtures of generalized linear experts, supporting regression, classification, and model-based clustering of heterogeneous data.
A new technical report introduces Ling 2.0, a family of reasoning-oriented foundation models built on a unified Mixture-of-Experts architecture that spans from tens of billions up to one trillion parameters. The series is released as an open language foundation, with the stated goal of strengthening general reasoning ability across all model sizes.
Researchers released T1, a 122-billion-parameter Mixture-of-Experts model trained via reinforcement learning to act as an agent in terminal environments. The work targets long-horizon workloads such as software development and scientific research, where sustained multi-step command-line use matters. It is presented as part of a broader shift in agent design away from short, single-turn interactions.