#llm-inference

13 curated events

papersTODAY 04:00 UTC

WaterKron Method Ties Kronecker-Factored Hessian Choice to Information Theory for Quantization

A new preprint introduces WaterKron, a post-training quantization approach that pairs two-sided GPTQ with waterfilling-based scaling that varies by row and column, along with entropy coding. The authors also present FlipFlop Hessian, a way of selecting Kronecker-factored Hessian approximations that they ground in information-theoretic arguments. The work targets how such approximations should be chosen when compressing neural networks after training.

quantization/llm-inference/arxiv/compression

papersTODAY 04:00 UTC

Audit Compares ConFlayers and SWIFT for Periodic-Step Layer Skipping in LLM Inference

A three-seed, rigor-matched study audits two periodic-step, search-based layer-skipping methods, ConFlayers and SWIFT, which choose which transformer layers to run for a given input. The work also examines trained routing approaches as an alternative to search-based selection. The authors frame it as a reproducibility-focused comparison of efficiency techniques for LLM inference.

llm-inference/layer-skipping/efficiency/benchmark

papersTODAY 04:00 UTC

LayerRoute Combines Adaptive Layer-Skipping With LoRA for Efficient LLM Inference

A new arXiv paper introduces LayerRoute, a parameter-efficient technique that skips selected transformer layers at inference time to cut compute costs. Each layer uses a hard-gated routing decision trained with a straight-through estimator, and the model is jointly fine-tuned with LoRA so that output quality is largely retained. The authors position the approach as a way to speed up large language model inference without sacrificing accuracy.

research/efficiency/lora/llm-inference

papersTODAY 04:00 UTC

BOOST: Concurrent Host Memory and HBM Access to Speed Up LLM Inference

A new arXiv paper proposes BOOST, a technique that lets GPUs read from host memory and high-bandwidth memory at the same time during large language model inference. Because GPU memory capacity and bandwidth are key bottlenecks for throughput, the approach aims to use both memory tiers concurrently rather than treating host memory as a slow fallback. The work targets faster LLM serving on existing CPU-to-GPU interconnect hardware.

llm-inference/gpu-memory/hardware/performance

papersTODAY 04:00 UTC

Paper Proposes Reusing Rejected Tokens to Speed Up Speculative Decoding

A new arXiv paper introduces a technique called carryover drafting that reuses the model states computed for rejected draft tokens during speculative decoding. Because verification already produces representations for both accepted and rejected tokens, the method puts the discarded ones to further use instead of throwing them away. The goal is to improve the efficiency of LLM inference, where speculative decoding lets one target forward pass accept multiple drafted tokens at once.

speculative-decoding/llm-inference/efficiency/arxiv

papersTODAY 04:00 UTC

arXiv Paper Proposes Dynamic Semantic Extraction Framework for LLM Inference

A new arXiv preprint introduces a Dynamic Semantic Extraction and Inference (DSEI) framework aimed at reducing the memory overhead that comes with token-level inference in large language models. The approach compresses semantics dynamically so that inference can partly happen in latent space rather than relying solely on token-by-token processing. The authors present it as a step toward more computationally efficient LLM deployment.

llm-inference/efficiency/latent-space/arxiv

papersTODAY 04:00 UTC

SpliTEE combines trusted hardware with differentially private GPU offloading for LLM inference

A new paper proposes SpliTEE, a system that runs large language model inference partly on trusted hardware while outsourcing the rest to GPUs with differential privacy guarantees. The approach aims to keep user prompts confidential, addressing risks such as sensitive data being memorized during retraining by remote model providers. It targets a balance between privacy protection and inference performance.

privacy/llm-inference/differential-privacy/trusted-hardware

papersTODAY 04:00 UTC

Validating Hybrid-State Cache Recovery for GLM-5.3-Flash with vLLM and LMCache

A new arXiv paper investigates whether an external cache transfer that appears successful can still leave a hybrid language model resuming from an inconsistent internal state. The authors test the full 45-layer GLM-5.3-Flash model using the RedHatAI NVFP4 quantized checkpoint together with vLLM and LMCache under a four-way configuration. The work focuses on validating cache recovery correctness rather than raw throughput.

llm-inference/kv-cache/vllm/quantization

papersTODAY 04:00 UTC

AgentKV: Phase-Aware KV Eviction for Agentic LLMs

A new arXiv paper introduces AgentKV, a KV-cache eviction scheme designed for agentic serving, where token usage and decode-time bandwidth demands far exceed those of ordinary chatbot workloads. Existing eviction methods typically score cached keys against queries taken from the most recent tokens; AgentKV instead adapts this scoring to the distinct phases of agent execution. The approach targets both cache capacity pressure and decode throughput in long-running agent pipelines.

kv-cache/agents/llm-inference/efficiency

papersSEP 10 04:00 UTC

Osprey: Target-Agnostic Pre-training Builds Stronger Draft Models for Speculative Decoding

Researchers introduce Osprey, a pre-training approach for draft models used in speculative decoding that is not tied to any specific target model. Draft models are typically tuned to a single target's output distribution, causing acceptance rates to fall when workloads change, and target-agnostic pre-training aims to keep them robust. The work focuses on achieving faster and more stable inference for large language models.

speculative-decoding/llm-inference/efficiency/research

papersSEP 10 04:00 UTC

Study Proposes Maverick for Private, Verifiable LLM Inference via Matrix-Vector Delegation

A paper posted to arXiv introduces Maverick, a system designed to let users run LLM inference on external servers without exposing their inputs or blindly trusting the returned results. The method delegates the heavy matrix-vector multiplications that dominate transformer inference while adding privacy protections and a mechanism to confirm that computations were performed correctly. The authors frame the work as a step toward making private and verifiable inference practical for open-source models.

privacy/llm-inference/verifiability/open-source

papersSEP 12 04:00 UTC

Paper Proposes Predictive Cache Orchestration for LLM Accelerators

A new arXiv paper introduces DCO, a scheme that manages on-chip cache memory in AI accelerators by predicting what data large language models will need next. The approach aims to reduce reliance on complex hierarchical scratchpad memory designs that complicate software development. It targets efficiency gains for LLM inference hardware.

hardware/llm-inference/arxiv/efficiency

papersSEP 11 04:00 UTC

REAL-Q: End-to-End LLM Quantization via Dynamic Gradient Descent

A new arXiv paper presents REAL-Q, a post-training quantization method that adjusts a model's compressed weights end-to-end rather than optimizing each layer with its own solver. It replaces the single closed-form second-order solution used by current PTQ techniques with dynamic gradient descent. The work targets cheaper deployment of large language models under tight memory and compute limits.

quantization/llm-inference/efficiency/arxiv