Researchers propose PEEK, a scheduling and cache eviction framework for LLM serving that handles both streaming and batch workloads. It builds an incremental radix tree over the pending request queue to guide KV cache decisions, with the paper concentrating on the online serving case. The approach is described as lightweight.
A new arXiv paper argues that although a transformer model is trained to handle any prompt, each real-world deployment only receives a narrow slice of queries, such as delivery complaints for a support assistant or Python for a coding tool. Yet every deployment currently pays the same inference compute cost. The authors propose choosing inference depth based on the questions a given deployment actually asks, rather than applying one uniform setting across all uses.
A newly revised arXiv preprint analyzes how quantization affects large language models deployed for machine translation in server settings. The work weighs reduced memory use and faster inference against the quality loss that compression can introduce. It aims to help practitioners pick quantization settings that balance latency and translation accuracy.
A new arXiv preprint argues that AI deployment performance depends on how compression methods, compiler transformations, and serving policies interact, rather than on model architecture alone. It notes that existing benchmarks often report latency and throughput under conditions that cannot be directly compared, which limits practical conclusions. The work appears to be a cross-listed submission surveying the inference deployment stack.