Researchers have introduced HoneyRoute, an inference-serving layer that detects malicious requests before they reach a language model deployment. Traffic flagged as hostile is redirected to a dedicated honeypot model, protecting the production system while operators can study the attacker's behavior.
A new arXiv paper argues that existing LLM scheduling designs were built around human users and may not suit agentic workloads, where software agents issue the requests. The authors note that agents consume far more tokens per request and exhibit different traffic patterns than people. They propose SMetric, a session-centric scheduling approach intended to balance serving performance under these agent-driven conditions.
A new arXiv paper argues that when prompts are built from reusable components such as retrieved passages, tool definitions, or few-shot examples, the order in which those pieces are arranged determines how much KV cache computation can be reused. Since current serving systems only reuse cache on exact prefix matches, the work recasts the ordering decision as a sorting problem. No specific benchmark results are described in the provided abstract.
Researchers propose PEEK, a scheduling and cache eviction framework for LLM serving that handles both streaming and batch workloads. It builds an incremental radix tree over the pending request queue to guide KV cache decisions, with the paper concentrating on the online serving case. The approach is described as lightweight.
Researchers propose MAPS, a scheduling framework designed to handle bursty large language model workloads on cloud infrastructure. The work targets memory-bound decode instances in prefill-decode disaggregated serving setups, where memory pressure limits throughput. It aims to improve scheduling decisions by predicting memory needs ahead of time.
A new arXiv paper introduces OpWeave, a system that breaks LLM inference into finer-grained operators rather than coarse stages, extending recent work that separates attention from FFN or MoE execution during decoding. The authors argue this operator-level disaggregation improves how workloads are matched to heterogeneous hardware during serving.
A new arXiv paper tackles a limitation in LLM serving systems, which typically reuse KV caches only when shared text appears at the very start of a prompt. The authors target two workloads that break this rule: retrieval-augmented generation servers that assemble different retrieved chunks per request, and requests served across different model checkpoints. KVShareArena aims to expand cache reuse in these cases, reducing redundant computation and improving serving efficiency.
A new arXiv preprint examines how agentic LLM workflows, which alternate model turns with tool calls, are affected by the timing of when completed turns are dispatched. The authors argue that separating a turn's readiness from its release lets runtimes optimize for tail latency rather than simply pushing each turn out immediately. The work targets scheduling policies that reduce worst-case end-to-end completion times for these multi-step pipelines.