What comes after attention? This startup says it already knows.
7 relevance
Score Breakdown
technical depth 9
novelty 9
actionability 3
community 4
strategic 7
personal 8
Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.
Novel sparse-attention architecture with huge context window, highly technical.
Summary
Subquadratic's SubQ 1.1 Small model, using Subquadratic Sparse Attention (SSA), scales linearly with context length to 12 million tokens, achieving 64.5x less compute than dense attention at 1M tokens and near-perfect long-context retrieval (99.12% on RULER). It scores 85.4 on GPQA Diamond and 89.7 on LiveCodeBench, with third-party verification from Appen. The company plans to eventually move beyond attention architectures entirely.