AI research lab Zyphra has introduced Zyphra Hierarchical Attention (ZHA), a novel sparse attention mechanism designed to enable true subquadratic compute scaling for long-context language models. While existing sparse attention methods reduce full attention calculations, their indexing steps often still scale linearly across the full context length—consuming up to 90% of layer processing times near 1 million tokens. ZHA addresses this bottleneck by structuring the key cache into a multi-level tree index.

By organizing keys into hierarchical summary chunks evaluated against learned reference queries, ZHA allows decoding models to selectively search tree branches instead of scanning the full context. In evaluations using a distilled Qwen3.6-35B model, ZHA retained over 93% accuracy on RULER-13 retrieval tests while reading small fractions of keys, matching dense attention baseline performance on complex reasoning tasks.

Zyphra’s analytical modeling projects speedups of 373x to 417x over dense attention at 1 billion tokens, with meaningful throughput gains beginning past 48K tokens. The technique relies on training reference query parameters while keeping base model weights frozen, offering a potential path to scale long-context serving economics for enterprise applications.

Why it matters

  • Offers a scalable compute reduction method for teams serving multi-hundred-thousand token context windows.

  • Enables practical long-context performance maintenance via lightweight post-training distillation without retraining base weights.

  • Tackles key-value memory indexing bottlenecks essential for subquadratic inference scalability.

Source: marktechpost.com