A newly revised arXiv preprint analyzes how quantization affects large language models deployed for machine translation in server settings. The work weighs reduced memory use and faster inference against the quality loss that compression can introduce. It aims to help practitioners pick quantization settings that balance latency and translation accuracy.
A new arXiv preprint introduces a method for having language models coordinate their own computation by exploiting semantic dependencies between generated tokens. The authors argue that standard autoregressive decoding is slow and leaves GPUs underused when batch sizes are small, and that their approach improves inference efficiency. The work is currently a research preprint and has not been peer reviewed or released as a product.
A new arXiv paper introduces FREDI, a security-focused wireless edge-intelligence framework for event-triggered inference across user devices, edge servers, and the cloud. It combines proportional-fair resource allocation with a dual-threshold early-exit scheme so that each user device can partially process inference locally before offloading. The work aims to balance fairness, latency, and efficiency in cooperative multi-layer edge deployments.
A new arXiv preprint examines Delayed Tensor Parallelism (DTP), a scheme that avoids the blocking all-reduce step in tensor-parallel Transformer inference by letting each device immediately add its partial output to the residual stream and broadcast it. The authors introduce affinity-aware sharding as a way to organize this communication and computation across devices. The work targets more efficient distributed inference for large Transformer models.
Researchers examined what happens when two single-GPU 27B vLLM inference replicas share a 256 GiB host-memory cache pool (LMCache), aiming to skip repeated prefill work as requests move between replicas. The paper reports both correctness problems in transferring cached state and performance limits tied to lost prefix locality. The authors argue that shared caching only pays off when state handoff is reliable and locality loss stays small.
A new arXiv paper proposes NeuroFlex, a scheme that lets artificial and spiking neural networks run together at the level of individual elements rather than whole layers or tiles. The authors argue this finer granularity avoids the idle hardware and wasted energy that hybrid accelerators suffer when workload traits change inside a layer, and that it does so without losing accuracy. The work targets sparse inference efficiency on specialized DNN accelerators.
A new arXiv paper investigates whether speculative decoding with the Orthrus architecture remains truly lossless when numerical precision is taken into account. Orthrus is a hybrid autoregressive-diffusion system that drafts several tokens at once and verifies them with a frozen autoregressive model, and its claimed exactness depends on the draft and verification steps producing identical results. The work analyzes how floating-point rounding in these computations can break that equivalence in practice.
A new arXiv preprint introduces discrete Beckmann transport models, a framework for language modeling and reasoning that avoids the multi-step sampling usually required by discrete diffusion and flow approaches. The authors note that existing methods compress sampling steps only by distilling a pretrained autoregressive teacher, which limits the student to the teacher's performance. Their approach instead aims to generate text in a single step without relying on that distillation ceiling.
A new arXiv paper argues that although a transformer model is trained to handle any prompt, each real-world deployment only receives a narrow slice of queries, such as delivery complaints for a support assistant or Python for a coding tool. Yet every deployment currently pays the same inference compute cost. The authors propose choosing inference depth based on the questions a given deployment actually asks, rather than applying one uniform setting across all uses.
A preprint describes a control mechanism for LLM agents that keeps small routing decisions computed on the GPU instead of sending them back to host memory for redispatch. The authors report faster execution than host-based dispatch across 36 tested placements. The work is positioned as a way to bound GPU opportunity and cut per-step communication overhead in agent control loops.
Researchers propose PEEK, a scheduling and cache eviction framework for LLM serving that handles both streaming and batch workloads. It builds an incremental radix tree over the pending request queue to guide KV cache decisions, with the paper concentrating on the online serving case. The approach is described as lightweight.
Researchers propose MANE, a multi-path adaptive network designed to improve split computing, where a small head model runs on a device and a larger tail model runs on an edge server. The approach targets efficient distributed inference by adapting how computation is divided between the device and the edge. It is described in a new arXiv preprint (2609.14660v1) listed under cross-submissions.
A new arXiv preprint describes a method for estimating single-sequence inference throughput in llama.cpp directly from GGUF file metadata. The authors use roofline-shaped predictors with quantization-specific scaling factors fitted on reference models. The approach was scored on 318 phase-depth measurements drawn from 53 host-file configurations across three systems.
A new arXiv paper introduces AdaVSkip, a technique that reduces the number of visual tokens processed at each transformer layer to lower the cost of multimodal large language model inference. Rather than only compressing tokens along the sequence dimension, the approach adapts skipping decisions per layer. The work targets efficiency gains without retraining the underlying model.
A revised arXiv preprint introduces a technique called post-reasoning that aims to raise the quality of large language model outputs produced without intermediate reasoning traces. The work targets the cost and latency that come from generating long chains of thought at inference time, arguing that many practical tasks do not need them. The authors present the approach as an add-on that improves non-reasoning models without extra expense.
Researchers propose MAPS, a scheduling framework designed to handle bursty large language model workloads on cloud infrastructure. The work targets memory-bound decode instances in prefill-decode disaggregated serving setups, where memory pressure limits throughput. It aims to improve scheduling decisions by predicting memory needs ahead of time.
A new arXiv paper systematically examines how much energy large language models consume when serving users in different languages. The authors frame the gap between language coverage and energy use as a "language-energy divide," suggesting that inference cost varies by language and is not well documented. The work aims to give a measurement basis for evaluating the efficiency of multilingual deployments.
A new arXiv paper introduces an adaptive sampling method for masked diffusion language models that decides which tokens to commit during each denoising step. The approach targets a known failure mode where individually confident positions become unsafe when decoded in parallel, aiming to preserve accuracy while still reducing the number of inference iterations. The work is a revision of an earlier preprint and has not been peer reviewed.
Analog accelerators promise better energy efficiency for machine learning on power-constrained devices, but their behavior is sensitive to temperature. This arXiv paper examines how thermal variation degrades inference accuracy in analog deep neural networks and proposes methods to mitigate those effects. The work targets deployment on mobile and embedded hardware.
A new arXiv paper introduces OpWeave, a system that breaks LLM inference into finer-grained operators rather than coarse stages, extending recent work that separates attention from FFN or MoE execution during decoding. The authors argue this operator-level disaggregation improves how workloads are matched to heterogeneous hardware during serving.
A new arXiv paper proposes Temporal Self-Distillation, a training method aimed at discrete diffusion language models that generate several tokens at once. Such models lose quality when too many tokens are decoded in parallel, and the technique is presented as a simple way to reduce that degradation. The approach targets faster inference without the accuracy drop that usually accompanies aggressive parallel decoding.
A new arXiv paper proposes Grouped Value Attention, a method that lowers the memory and read-bandwidth burden of the key-value cache during Transformer decoding. Unlike grouped-query attention, which still stores separate keys and values per shared head, the approach reconstructs keys on demand from stored values. This targets the KV cache, which the authors identify as a main bottleneck as sequence lengths grow.
Researchers have introduced HoneyRoute, an inference-serving layer that detects malicious requests before they reach a language model deployment. Traffic flagged as hostile is redirected to a dedicated honeypot model, protecting the production system while operators can study the attacker's behavior.
A new arXiv preprint describes a key-value cache layout that is physically split so work can be divided between CPU and GPU during long-context inference with Mixture-of-Experts models. The approach targets the case where single-GPU memory is insufficient and the KV cache must spill into host memory, where the stored data either feeds attention on the GPU or supports computation on the CPU. The proposed format is designed to keep those two uses balanced rather than treating the spilled cache as a single monolithic block.
A new arXiv paper proposes AdaFlash, a speculative decoding method that uses diffusion-based draft models distilled on-policy to speed up large language model inference. The approach adapts the drafting process rather than relying on a fixed draft model, aiming to improve acceptance rates during verification by the target model. It builds on prior work in this line, including DFlash, and appears as a revised submission.
A new arXiv paper proposes FluxMoE, a serving system that separates where mixture-of-experts weights live from GPU memory constraints. Existing inference engines keep every expert resident on GPUs, which competes for space with the key-value cache and limits throughput. The work targets higher-performance MoE inference by changing how expert residency is managed.
A new arXiv preprint argues that AI deployment performance depends on how compression methods, compiler transformations, and serving policies interact, rather than on model architecture alone. It notes that existing benchmarks often report latency and throughput under conditions that cannot be directly compared, which limits practical conclusions. The work appears to be a cross-listed submission surveying the inference deployment stack.
A newly posted arXiv paper introduces a technique for shrinking large language models by removing entire Transformer blocks without running any forward passes. Rather than gathering hidden states from calibration data, the method scores blocks using redundancy between their weights to decide which layers can be safely dropped. This avoids the extra compute that activation-based pruning approaches typically require.
A new arXiv paper presents SymbolicLight V2, a language model architecture that combines sparse, event-driven computation with conventional continuous-state processing. It extends the earlier spike-gated design by adding graded signed events at additional projection layers along with a softmax-free local attention mechanism. The work targets reduced energy consumption during language inference.
A new arXiv paper proposes EFQ-Softmax, a technique for quantizing the softmax step in Transformer attention without computing exponentials. Low-bit attention already speeds up inference by shifting matrix multiplications to FP8 or FP4 engines, but the softmax path often still needs higher precision for shifted-score exponentials and temporary probability storage. The work targets that remaining bottleneck in the attention pipeline.
Researchers introduce X-CoSD, a distributed inference framework in which a small on-device model drafts tokens and a larger server-side LLM verifies them. The method tackles the mismatch between the two models' vocabularies while cutting the communication cost between device and server. The work aims to make collaborative speculative decoding practical for latency-sensitive edge deployments.
A new arXiv paper examines the practice of cutting reasoning-model inference short by repeatedly sampling answers from a partial reasoning trace and stopping once the probes agree. The authors argue that this self-consensus approach is not a safe signal, since a model that appears settled may still change its final answer. The work also investigates whether any probing-based exit rule can be both reliable and genuinely token-saving.
A new arXiv paper tackles a limitation in Mixture-of-Experts models, which typically activate a fixed number of experts for each input. The authors propose an inference method that remains distribution-consistent when the number of active experts varies dynamically. The approach aims to preserve efficient inference in large foundation models while allowing more flexible expert routing.
A new arXiv paper addresses learning systems that must output a single prediction even when the set of acceptable outputs is disconnected or non-convex. The authors show that with squared loss, the Bayes-optimal conditional mean can fall outside the valid output region when the underlying distribution is ambiguous. They propose an equilibrium-based 'settling' procedure that returns a point estimate guaranteed to lie within the validity set.
A new arXiv paper proposes splitting a neural normalizing flow into components that expose a quantity close to a pivotal statistic when nuisance parameters are present. The approach requires only a sample generator from the target distribution rather than an explicit likelihood. This could simplify inference in settings where the likelihood is intractable but simulation is straightforward.
A new research paper applies likelihood-free inference to real2sim2real transfer for manipulating deformable linear objects such as cables using vision. By estimating a distribution over simulation parameters from black-box models, the method handles nonlinear and stochastic dynamics that are hard to model directly. A posterior-driven heuristic then adapts the inferred parameter support so control policies can generalize to varied deployment conditions.
Researchers used simulated galaxy catalogs from the CAMELS hydrodynamic simulations to infer the matter density parameter with machine learning models that embody different inductive biases. The paper evaluates how these architectural assumptions shape field-level likelihood-free inference of cosmological parameters.
A new arXiv paper shows that attention-based KV cache pruning methods break down on NoPE-MLA models, with H2O and SnapKV retrieving almost none of the injected needles at 8x compression. The proposed VestigeKV method instead relies on a vestigial branch within the NoPE-MLA cache that carries its own sparse-attention signal, allowing token selection before the future queries are known.
A new arXiv paper tackles the overhead that concatenated retrieved chunks create for KV caches in retrieval-augmented generation systems. The authors fine-tune a model to account for how retrieved passages are joined in the cache while also selectively recomputing cache entries where that still pays off. The work appears under cs.LG with cross-listings in cs.AI and cs.CL.
A new survey on arXiv examines mechanisms for reducing inference costs in video large language models, which pair video representations with pretrained LLMs to generate responses from text prompts. The paper addresses why video understanding remains computationally expensive and organizes existing efficiency techniques across video and audiovisual tasks.