A new arXiv paper proposes FluxMoE, a serving system that separates where mixture-of-experts weights live from GPU memory constraints. Existing inference engines keep every expert resident on GPUs, which competes for space with the key-value cache and limits throughput. The work targets higher-performance MoE inference by changing how expert residency is managed.
A cross-listed arXiv paper introduces IBIB, a protocol for evaluating AI systems as they are actually deployed inside enterprises rather than as bare model checkpoints. The authors argue that real-world capability emerges from the combination of weights, serving configuration, precision, output contract, and harness, so scoring an advertised model name alone is a measurement error. After auditing 18 existing benchmarks and finding that every one grades model identifiers, the paper proposes routing-based measurement instead.
With TorchServe no longer maintained, teams must manage their own GPU inference stack. AWS's Ray Serve Deep Learning Container bundles the framework, GPU drivers, and serving layer in a pre-tested package. The post demonstrates deploying a vision-language model using this container.