#long-context

8 curated events

papersTODAY 04:00 UTC

Self-Indexing Attention Unifies Token Retrieval Across Prefill and Decode

A new arXiv paper proposes a self-indexing attention mechanism for sparse long-context LLM inference. The authors note that current sparse methods typically rely on separate retrieval strategies for the prefill and decode stages, which prevents a single retrieval representation from being shared across the whole inference run. Their approach aims to make one retrieval representation reusable throughout, while remaining compatible with compression.

long-context/sparse-attention/inference-efficiency/llm

papersTODAY 04:00 UTC

Paper Tackles Lost-in-the-Middle Problem in Long-Text Generation

A new arXiv paper addresses how large language models tend to ignore information placed in the middle of long contexts, a problem studied mostly for retrieval tasks rather than long-input-to-long-output generation. The authors introduce a synthetic dataset and evaluation framework for this setting and propose a mitigation approach. The work is a revised cross-listing (v2) on arXiv cs.AI.

long-context/llm/evaluation/dataset

papersTODAY 04:00 UTC

Paper proposes content-based addressing to replace positional rotation in long-context attention

A new arXiv preprint argues that rotary position embeddings (RoPE) degrade as context windows grow, because the integer positions seen during inference differ from those used in training. The authors suggest determining attention rotation from content-based addressing rather than from a token's fixed index, aiming to remove this train-test mismatch. The work is a methodological proposal and cross-listing revision, not a released model or product.

long-context/position-embeddings/arxiv/attention

papersTODAY 04:00 UTC

Concertina: Adaptive Pipeline Parallelism for Long-Context LLM Training

A new arXiv paper introduces Concertina, a data-centric approach to pipeline parallelism aimed at making long-context LLM training more efficient. The authors argue that sequence parallelism carries heavy communication costs, and that pipeline parallelism's benefits depend on how the model and data are partitioned. Concertina adaptively splits pipeline stages to better handle long sequences on heterogeneous hardware.

long-context/training/pipeline-parallelism/distributed-training

papersTODAY 04:00 UTC

SpectralShift Method Extends Context Window of Gated DeltaNet Models

A new arXiv paper proposes SpectralShift, a technique that reparameterizes the spectral properties of gated DeltaNet layers to stretch their usable context window. The authors note that linear attention layers are increasingly used in place of softmax attention for long-context work, but existing extension methods typically rely on continued pretraining without altering the layer internals. Their approach instead modifies the layers themselves, aiming to make context extension more effective.

long-context/linear-attention/arxiv/efficiency

papersSEP 10 04:00 UTC

Paper combines KV cache-aware fine-tuning with recomputation for RAG efficiency

A new arXiv paper tackles the overhead that concatenated retrieved chunks create for KV caches in retrieval-augmented generation systems. The authors fine-tune a model to account for how retrieved passages are joined in the cache while also selectively recomputing cache entries where that still pays off. The work appears under cs.LG with cross-listings in cs.AI and cs.CL.

rag/kv-cache/long-context/inference

papersSEP 10 04:00 UTC

EvolveScaler paper generates evolving-context data with executable state machines

A new arXiv preprint, EvolveScaler, addresses situations where newer events in a long interaction can override or invalidate statements made earlier. The authors build synthetic datasets of such shifting information by pairing executable state machines with natural-language rendering, yielding material that tests how well models track what remains valid over time. The approach is aimed at benchmarking and training systems that must reason over dynamically changing contexts rather than static records.

long-context/synthetic-data/reasoning/evaluation

papersSEP 10 04:00 UTC

New Preprint Introduces ConvMem, a Convolutional Memory Method for Long-Context Reasoning

An arXiv preprint proposes ConvMem, a convolutional memory technique aimed at helping large language models reason over documents that exceed their fixed context windows. The approach builds on prior sequential strategies such as MemAgent, which extend usable context by processing text in stages. The paper was posted to the AI and computational linguistics categories on arXiv.

long-context/llm/memory/research