RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context Large Language Models

Large‑scale language models that can handle long contexts face a significant bottleneck during the pre‑fill phase, because dense self‑attention processes the entire input before generation begins. To cut this cost, researchers have proposed sparse block selection methods, but these can hide critical tokens amid many irrelevant ones—a problem the authors call “medium dilution.” The paper introduces RBS‑Attention, a sparse pre‑fill strategy that requires no extra training and combines two complementary selection branches. The first branch, based on centroids, captures the average relevance of blocks. The second, called the rescue branch, uses the maximum key radius of a block together with a distribution that depends on the prompt, layer, and head to spot blocks that might be undervalued. Each branch applies its own threshold, and the resulting masks are merged, allowing rescue blocks to be integrated without disrupting the regular execution of FlashAttention with sparse blocks.

Benchmarks on H100 GPUs show that RBS‑Attention speeds up pre‑fill attention by more than 20× compared to a standalone run and nearly 12× versus the vLLM implementation, while also reducing the time to the first token by almost six times for a 128 k‑token context. Accuracy loss is negligible, with a RULER score of 88.65 versus 89.52 for the dense method, and results on LongBench‑v2, InfiniteBench, and Video‑MME confirm the approach’s quality. This innovation matters because it lets large language models process far longer inputs faster and with lower resource consumption, making them more practical for tasks that require extensive text analysis, such as legal document review or large‑scale content generation.

Read the original article on arXiv cs.AI

This summary is an informational synthesis produced by dataqbs.com. All rights to the original content belong to its author and the cited media outlet. We act solely as curators of technology news and claim no authorship.