SubQ 1.1 Small: 小型ながら高性能な長文検索モデル

DateJune 16, 2026The hardest enterprise AI problems share a common shape. They require reasoning over complete artifacts: entire codebases, document collections, contracts, financial filings.For years, the industry worked around this problem by building retrieval pipelines, chunking strategies, and agentic scaffolding — useful tools, but ultimately workarounds for context limitations of the model architecture. The underlying constraint was attention: compute that scales quadratically with context length, making direct reasoning over large artifacts prohibitively expensive.SubQ is built to remove that constraint. Today we're releasing the model card for SubQ 1.1 Small — the second iteration of our Subquadratic Sparse Attention (SSA) model, at the smallest size. We are in the process of deploying SubQ 1.1 Small with select design partners and plan to deploy a broader lineup of models ranging from 2M to 12M tokens later in the year.Key FeaturesNear-perfect long-context retrieval up to 12M tokens on the needle-in-a-haystack test, with up to nearly 1,000x attention compute reduction.A balance of long-context optimization and general reasoning ability, with strong performance retained across knowledge, coding, and non-coding enterprise agent benchmarks.At 1M tokens, SubQ 1.1 Small requires 64.5x less compute than dense attention and runs 56x faster than FlashAttention-2.These results reflect the scaling advantage that SSA's efficiency gains make possible.BenchmarksSubQ 1.1 Small was evaluated across five axes, covering long-context retrieval, context-length generalization, knowledge, coding, and long-horizon agentic tasks.Long-Context Retrieval & GeneralizationWe selected Needle-In-A-Haystack (NIAH) and Nvidia's RULER test because together they test whether the model can find a single fact buried deep in a large context, and whether it can connect the dots across that context.NIAH is the precision test. It places one retrievable fact at a controlled depth within a long context and asks the model to return it exactly. SubQ 1.1 Small scores near-perfect at 1M, 2M, 6M, and 12M tokens. The model was trained predominantly at 1M tokens yet the retrieval held near perfectly at 12x that length, despite compressing attention to just 0.13% of relationships. This generalization is a direct consequence of SSA routing attention based on content relevance rather than fixed positional patterns.RULER is the capability test. It's 13 tasks go beyond single-fact lookup to cover multi-hop variable tracing, frequency extraction, and aggregation across the full context using the kind of reasoning complete-artifact workloads actually require. SubQ 1.1 Small scores 99.12% at 128K.Multi-task retrievalRULER (128K)99.12%128KSingle-fact retrievalNeedle-in-a-haystack (1M–12M)100%1M100%2M98%6M98%12MGeneral Knowledge & ReasoningSubQ 1.1 Small balances long-context optimization with general reasoning ability without compromise. GPQA Diamond at 85.4% sits just below mid-tier frontier models and well above the smaller tier. LiveCodeBench at 89.7% pass@4 is close to the absolute frontier. AutomationBench Finance at 13% places SubQ 1.1 Small close to the strongest models on that benchmark, ahead of mid-tier and smaller baselines. Absolute scores remain low across all models on this benchmark.BenchmarkSubQ 1.1 SmallGPT-5.5Opus 4.8Sonnet 4.6GPT-5.4-miniGPT-5.4-nanoHaiku 4.5Graduate-level scienceGPQA Diamond · pass@185.493.29287.587.581.767.2Agentic financeAutomationBench13%18%16%8%0%n/r3%Competitive programmingLiveCodeBench v6 · pass@489.79292.288.978.678.269.7n/r = result not reported by the model providerEfficiencySSA replaces the O(n²) dense attention pass with a learned sparse formulation that scales linearly with context length. SSA's advantage over dense attention grows as context length increases. At 1M tokens, SubQ requires 64.5x fewer compute than dense attention and runs 56x faster than FlashAttention-2 on a single attention layer. In practice, this drastically changes the economics of long-context training and inference.A full breakdown of the mechanism and how it compares to FlashAttention, DeepSeek sparse attention, and recurrent architectures is in the Technical Report.SubQ uses 64.5x less compute than dense attention, and is 56× faster than FlashAttention-2 at 1M-token contextTrainingWe started with an existing open-weight frontier model, replaced dense attention with SSA, and built long-context capability through staged context extension (262K, 512K, 1M, 2M) followed by roughly one trillion tokens of continued pretraining on naturally long artifacts: books, documents, and repository-scale code.The strongest lever we found for improving long-context retrieval was long-context continued pretraining, made possible by the efficiency of the SSA algorithm. The 12M generalization result reflects both factors: SSA's selection criterion is independent of absolute position, and the capability to use that generalization reliably develops through training on long data.Additionally, we ran more than one hundred experiments across six to seven model generations to get the balance of capabilities between long- and short-context tasks right. That kind of iteration is only possible because SSA enabled our team to run multi-million-token experiments as a standard procedure rather than a rare event, making the research loop more efficient.Use CasesSubQ is designed for workloads that require reasoning over information distributed across the artifact without fragmentation. Here are just a few of the use cases from our initial research:Financial analysis and due diligence. Filings, earnings reports, contracts, and internal records are only meaningful in combination. SubQ reasons across the full collection rather than summarizing each document in isolation.Legal and contract work. A contract may define a term on page 2, qualify it on page 12, and carve out an exception on page 46. Retrieval finds the sentence but loses the relationships. SubQ holds the whole document and reasons across it directly.Software engineering. Codebases distribute logic across files, modules, and dependencies in ways that short-context models can't hold at once. SubQ loads an entire repository into a single context window, enabling architecture-level reasoning, cross-file refactoring, and dependency tracing in one pass. We believe there will be significant value for long-context models in planning, review, and long-horizon memory within coding.What's NextWe'll be kicking off with the first cohort of design partners in the next few weeks, with broader rollout through the quarter and general model releases by end of year.

※ 著作権に配慮し、引用は冒頭3段落までです。続きは元記事をご覧ください。