LIVE PULSE
5.1 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.8 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.6 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.3 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.8 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.4 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.4 Fine-Tuning Vision-Language Models with Listener Gaze for Referring Expressions — 1 src1.4 Perceptual Reality Transformer Explores What Illustrations Must Preserve — 1 src1.4 Study Analyzes Self-Reported Limitations in NLP Research — 1 src5.1 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.8 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.6 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.3 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.8 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.4 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.4 Fine-Tuning Vision-Language Models with Listener Gaze for Referring Expressions — 1 src1.4 Perceptual Reality Transformer Explores What Illustrations Must Preserve — 1 src1.4 Study Analyzes Self-Reported Limitations in NLP Research — 1 src
HEATPULSEAI MAGAZINES
FLIP · FOLLOW · SAVE
Log inSign up

llm-inference

topic11 events
papersTODAY 04:00 UTC

AdaFlash: Adaptive Speculative Decoding with On-Policy Distilled Diffusion Drafters

A new arXiv paper proposes AdaFlash, a speculative decoding method that uses diffusion-based draft models distilled on-policy to speed up large language model inference. The approach adapts the drafting process rather than relying on a fixed draft model, aiming to improve acceptance rates during verification by the target model. It builds on prior work in this line, including DFlash, and appears as a revised submission.

speculative-decoding/diffusion/inference/research
papersTODAY 04:00 UTC

arXiv Paper Proposes Dynamic Semantic Extraction Framework for LLM Inference

A new arXiv preprint introduces a Dynamic Semantic Extraction and Inference (DSEI) framework aimed at reducing the memory overhead that comes with token-level inference in large language models. The approach compresses semantics dynamically so that inference can partly happen in latent space rather than relying solely on token-by-token processing. The authors present it as a step toward more computationally efficient LLM deployment.

llm-inference/efficiency/latent-space/arxiv
papersTODAY 04:00 UTC

SpliTEE combines trusted hardware with differentially private GPU offloading for LLM inference

A new paper proposes SpliTEE, a system that runs large language model inference partly on trusted hardware while outsourcing the rest to GPUs with differential privacy guarantees. The approach aims to keep user prompts confidential, addressing risks such as sensitive data being memorized during retraining by remote model providers. It targets a balance between privacy protection and inference performance.

privacy/llm-inference/differential-privacy/trusted-hardware
papersTODAY 04:00 UTC

Paper Proposes Reusing Rejected Tokens to Speed Up Speculative Decoding

A new arXiv paper introduces a technique called carryover drafting that reuses the model states computed for rejected draft tokens during speculative decoding. Because verification already produces representations for both accepted and rejected tokens, the method puts the discarded ones to further use instead of throwing them away. The goal is to improve the efficiency of LLM inference, where speculative decoding lets one target forward pass accept multiple drafted tokens at once.

speculative-decoding/llm-inference/efficiency/arxiv
papersTODAY 04:00 UTC

BOOST: Concurrent Host Memory and HBM Access to Speed Up LLM Inference

A new arXiv paper proposes BOOST, a technique that lets GPUs read from host memory and high-bandwidth memory at the same time during large language model inference. Because GPU memory capacity and bandwidth are key bottlenecks for throughput, the approach aims to use both memory tiers concurrently rather than treating host memory as a slow fallback. The work targets faster LLM serving on existing CPU-to-GPU interconnect hardware.

llm-inference/gpu-memory/hardware/performance
papersTODAY 04:00 UTC

Audit Compares ConFlayers and SWIFT for Periodic-Step Layer Skipping in LLM Inference

A three-seed, rigor-matched study audits two periodic-step, search-based layer-skipping methods, ConFlayers and SWIFT, which choose which transformer layers to run for a given input. The work also examines trained routing approaches as an alternative to search-based selection. The authors frame it as a reproducibility-focused comparison of efficiency techniques for LLM inference.

llm-inference/layer-skipping/efficiency/benchmark
papersTODAY 04:00 UTC

Study measures energy costs of multilingual LLM inference

A new arXiv paper systematically examines how much energy large language models consume when serving users in different languages. The authors frame the gap between language coverage and energy use as a "language-energy divide," suggesting that inference cost varies by language and is not well documented. The work aims to give a measurement basis for evaluating the efficiency of multilingual deployments.

energy/multilingual/inference/benchmarks
papersSEP 10 04:00 UTC

X-CoSD: Cross-Vocabulary Collaborative Speculative Decoding for Efficient LLM Inference

Researchers introduce X-CoSD, a distributed inference framework in which a small on-device model drafts tokens and a larger server-side LLM verifies them. The method tackles the mismatch between the two models' vocabularies while cutting the communication cost between device and server. The work aims to make collaborative speculative decoding practical for latency-sensitive edge deployments.

inference/edge-ai/efficiency/distributed-systems
papersSEP 10 04:00 UTC

Osprey: Target-Agnostic Pre-training Builds Stronger Draft Models for Speculative Decoding

Researchers introduce Osprey, a pre-training approach for draft models used in speculative decoding that is not tied to any specific target model. Draft models are typically tuned to a single target's output distribution, causing acceptance rates to fall when workloads change, and target-agnostic pre-training aims to keep them robust. The work focuses on achieving faster and more stable inference for large language models.

speculative-decoding/llm-inference/efficiency/research
productsSEP 9 07:10 UTC

Running LLMs in the browser: a new AI runtime for local inference

A German tech outlet begins a multi-part series on a new runtime that lets language models execute directly in the web browser rather than on remote servers. The approach keeps inference local, so applications can work offline and avoid per-request compute costs. Part one frames this shift as AI moving into the frontend, where models can respond to page context.

browser/local-inference/llm/webgpu