papersTODAY 04:00 UTC
AgentKV: Phase-Aware KV Eviction for Agentic LLMs
A new arXiv paper introduces AgentKV, a KV-cache eviction scheme designed for agentic serving, where token usage and decode-time bandwidth demands far exceed those of ordinary chatbot workloads. Existing eviction methods typically score cached keys against queries taken from the most recent tokens; AgentKV instead adapts this scoring to the distinct phases of agent execution. The approach targets both cache capacity pressure and decode throughput in long-running agent pipelines.