AFBS-BO automates the discovery of layer-specific sparse attention hyperparameters, making long-context acceleration 'plug-and-play.'
March 20, 2026
Original Paper
Self-Tuning Sparse Attention: Multi-Fidelity Hyperparameter Optimization for Transformer Acceleration
arXiv · 2603.18417
The Takeaway
It eliminates the manual grid search typically required to make sparse attention (like SpargeAttn) work on new models. By accelerating hyperparameter discovery by 3.4x, it makes it feasible to scale context windows on Llama-2-7B and similar models with minimal quality loss.
From the abstract
Sparse attention mechanisms promise to break the quadratic bottleneck of long-context transformers, yet production adoption remains limited by a critical usability gap: optimal hyperparameters vary substantially across layers and models, and current methods (e.g., SpargeAttn) rely on manual grid search to identify them. We propose AFBS-BO (Adaptive Fidelity Binary Search with Bayesian Optimization), a fully automated framework that discovers optimal layer- and head-specific hyperparameters witho