LIVE PULSE
5.1 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.8 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.6 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.3 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.8 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.4 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.4 Fine-Tuning Vision-Language Models with Listener Gaze for Referring Expressions — 1 src1.4 Perceptual Reality Transformer Explores What Illustrations Must Preserve — 1 src1.4 Study Analyzes Self-Reported Limitations in NLP Research — 1 src5.1 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.8 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.6 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.3 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.8 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.4 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.4 Fine-Tuning Vision-Language Models with Listener Gaze for Referring Expressions — 1 src1.4 Perceptual Reality Transformer Explores What Illustrations Must Preserve — 1 src1.4 Study Analyzes Self-Reported Limitations in NLP Research — 1 src
HEATPULSEAI MAGAZINES
FLIP · FOLLOW · SAVE
Log inSign up

KV cache

topic6 events
papersTODAY 04:00 UTC

BudgetBench: Budget-Tiered Protocol for Evaluating Memory in Local LLM Agents

Researchers introduce BudgetBench, a protocol and pilot test harness for assessing memory strategies in locally run large language model agents. The work treats active context as a limited resource, accounting for memory capacity, prefill latency, cache growth, and service targets when deciding how many input tokens a call can afford. It aims to give a structured, budget-aware way to compare agent memory approaches under constrained hardware and latency conditions.

agents/memory/local-llm/benchmarks
papersTODAY 04:00 UTC

Grouped Value Attention Cuts KV Cache Cost via On-Demand Key Reconstruction

A new arXiv paper proposes Grouped Value Attention, a method that lowers the memory and read-bandwidth burden of the key-value cache during Transformer decoding. Unlike grouped-query attention, which still stores separate keys and values per shared head, the approach reconstructs keys on demand from stored values. This targets the KV cache, which the authors identify as a main bottleneck as sequence lengths grow.

kv-cache/efficiency/attention/inference
papersTODAY 04:00 UTC

arXiv Paper Proposes Partitioned KVCache Format to Balance CPU-GPU Load in MoE Inference

A new arXiv preprint describes a key-value cache layout that is physically split so work can be divided between CPU and GPU during long-context inference with Mixture-of-Experts models. The approach targets the case where single-GPU memory is insufficient and the KV cache must spill into host memory, where the stored data either feeds attention on the GPU or supports computation on the CPU. The proposed format is designed to keep those two uses balanced rather than treating the spilled cache as a single monolithic block.

inference/kv-cache/moe/arxiv
papersTODAY 04:00 UTC

arXiv Paper Casts LLM Prefix Sharing as a Sorting Problem

A new arXiv paper argues that when prompts are built from reusable components such as retrieved passages, tool definitions, or few-shot examples, the order in which those pieces are arranged determines how much KV cache computation can be reused. Since current serving systems only reuse cache on exact prefix matches, the work recasts the ordering decision as a sorting problem. No specific benchmark results are described in the provided abstract.

kv-cache/llm-serving/efficiency/research
papersSEP 11 04:00 UTC

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

A new arXiv paper proposes FluxMoE, a serving system that separates where mixture-of-experts weights live from GPU memory constraints. Existing inference engines keep every expert resident on GPUs, which competes for space with the key-value cache and limits throughput. The work targets higher-performance MoE inference by changing how expert residency is managed.

moe/inference/efficiency/arxiv
papersSEP 10 04:00 UTC

Paper combines KV cache-aware fine-tuning with recomputation for RAG efficiency

A new arXiv paper tackles the overhead that concatenated retrieved chunks create for KV caches in retrieval-augmented generation systems. The authors fine-tune a model to account for how retrieved passages are joined in the cache while also selectively recomputing cache entries where that still pays off. The work appears under cs.LG with cross-listings in cs.AI and cs.CL.

rag/kv-cache/long-context/inference