Researchers propose PEEK, a scheduling and cache eviction framework for LLM serving that handles both streaming and batch workloads. It builds an incremental radix tree over the pending request queue to guide KV cache decisions, with the paper concentrating on the online serving case. The approach is described as lightweight.
Researchers propose MAPS, a scheduling framework designed to handle bursty large language model workloads on cloud infrastructure. The work targets memory-bound decode instances in prefill-decode disaggregated serving setups, where memory pressure limits throughput. It aims to improve scheduling decisions by predicting memory needs ahead of time.
A new arXiv paper argues that when prompts are built from reusable components such as retrieved passages, tool definitions, or few-shot examples, the order in which those pieces are arranged determines how much KV cache computation can be reused. Since current serving systems only reuse cache on exact prefix matches, the work recasts the ordering decision as a sorting problem. No specific benchmark results are described in the provided abstract.
A new arXiv paper introduces OpWeave, a system that breaks LLM inference into finer-grained operators rather than coarse stages, extending recent work that separates attention from FFN or MoE execution during decoding. The authors argue this operator-level disaggregation improves how workloads are matched to heterogeneous hardware during serving.
A new arXiv preprint argues that AI deployment performance depends on how compression methods, compiler transformations, and serving policies interact, rather than on model architecture alone. It notes that existing benchmarks often report latency and throughput under conditions that cannot be directly compared, which limits practical conclusions. The work appears to be a cross-listed submission surveying the inference deployment stack.
A new arXiv paper tackles a limitation in LLM serving systems, which typically reuse KV caches only when shared text appears at the very start of a prompt. The authors target two workloads that break this rule: retrieval-augmented generation servers that assemble different retrieved chunks per request, and requests served across different model checkpoints. KVShareArena aims to expand cache reuse in these cases, reducing redundant computation and improving serving efficiency.