#kv-cache

10 curated events

papersTODAY 04:00 UTC

arXiv Paper Proposes Partitioned KVCache Format to Balance CPU-GPU Load in MoE Inference

A new arXiv preprint describes a key-value cache layout that is physically split so work can be divided between CPU and GPU during long-context inference with Mixture-of-Experts models. The approach targets the case where single-GPU memory is insufficient and the KV cache must spill into host memory, where the stored data either feeds attention on the GPU or supports computation on the CPU. The proposed format is designed to keep those two uses balanced rather than treating the spilled cache as a single monolithic block.

inference/kv-cache/moe/arxiv

papersTODAY 04:00 UTC

Validating Hybrid-State Cache Recovery for GLM-5.3-Flash with vLLM and LMCache

A new arXiv paper investigates whether an external cache transfer that appears successful can still leave a hybrid language model resuming from an inconsistent internal state. The authors test the full 45-layer GLM-5.3-Flash model using the RedHatAI NVFP4 quantized checkpoint together with vLLM and LMCache under a four-way configuration. The work focuses on validating cache recovery correctness rather than raw throughput.

llm-inference/kv-cache/vllm/quantization

papersTODAY 04:00 UTC

arXiv Paper Casts LLM Prefix Sharing as a Sorting Problem

A new arXiv paper argues that when prompts are built from reusable components such as retrieved passages, tool definitions, or few-shot examples, the order in which those pieces are arranged determines how much KV cache computation can be reused. Since current serving systems only reuse cache on exact prefix matches, the work recasts the ordering decision as a sorting problem. No specific benchmark results are described in the provided abstract.

kv-cache/llm-serving/efficiency/research

papersTODAY 04:00 UTC

PEEK: Queue-Informed KV Cache Management for Online LLM Serving

Researchers propose PEEK, a scheduling and cache eviction framework for LLM serving that handles both streaming and batch workloads. It builds an incremental radix tree over the pending request queue to guide KV cache decisions, with the paper concentrating on the online serving case. The approach is described as lightweight.

llm-serving/kv-cache/inference/arxiv

papersTODAY 04:00 UTC

AgentKV: Phase-Aware KV Eviction for Agentic LLMs

A new arXiv paper introduces AgentKV, a KV-cache eviction scheme designed for agentic serving, where token usage and decode-time bandwidth demands far exceed those of ordinary chatbot workloads. Existing eviction methods typically score cached keys against queries taken from the most recent tokens; AgentKV instead adapts this scoring to the distinct phases of agent execution. The approach targets both cache capacity pressure and decode throughput in long-running agent pipelines.

kv-cache/agents/llm-inference/efficiency

papersTODAY 04:00 UTC

Grouped Value Attention Cuts KV Cache Cost via On-Demand Key Reconstruction

A new arXiv paper proposes Grouped Value Attention, a method that lowers the memory and read-bandwidth burden of the key-value cache during Transformer decoding. Unlike grouped-query attention, which still stores separate keys and values per shared head, the approach reconstructs keys on demand from stored values. This targets the KV cache, which the authors identify as a main bottleneck as sequence lengths grow.

kv-cache/efficiency/attention/inference

modelsSEP 10 12:40 UTC

DeepSeek releases V4.1-Flash multimodal model with reduced KV cache memory

DeepSeek has introduced V4.1-Flash, a multimodal model with 552 billion parameters, though only 16 billion are active per token. The release cuts KV cache memory requirements to a quarter of the previous generation. On the DeepSWE coding benchmark, it slightly outperforms Opus 5 and GPT-5.6 Sol.

deepseek/multimodal/kv-cache/benchmarks

papersSEP 10 04:00 UTC

VestigeKV paper uses NoPE-MLA's vestigial branch for KV cache compression

A new arXiv paper shows that attention-based KV cache pruning methods break down on NoPE-MLA models, with H2O and SnapKV retrieving almost none of the injected needles at 8x compression. The proposed VestigeKV method instead relies on a vestigial branch within the NoPE-MLA cache that carries its own sparse-attention signal, allowing token selection before the future queries are known.

kv-cache/efficiency/inference/research

papersSEP 10 04:00 UTC

Paper combines KV cache-aware fine-tuning with recomputation for RAG efficiency

A new arXiv paper tackles the overhead that concatenated retrieved chunks create for KV caches in retrieval-augmented generation systems. The authors fine-tune a model to account for how retrieved passages are joined in the cache while also selectively recomputing cache entries where that still pays off. The work appears under cs.LG with cross-listings in cs.AI and cs.CL.

rag/kv-cache/long-context/inference

papersSEP 10 04:00 UTC

KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints

A new arXiv paper tackles a limitation in LLM serving systems, which typically reuse KV caches only when shared text appears at the very start of a prompt. The authors target two workloads that break this rule: retrieval-augmented generation servers that assemble different retrieved chunks per request, and requests served across different model checkpoints. KVShareArena aims to expand cache reuse in these cases, reducing redundant computation and improving serving efficiency.

kv-cache/llm-serving/rag/efficiency