inference economics
Training gets the headlines, but the money is in inference — the large majority of AI compute spend, paid again on every request. And the workloads growing fastest are exactly the ones standard attention punishes most.
Training gets the headlines, but the money is in inference — the large majority of AI compute spend, paid again on every request. And the workloads growing fastest are exactly the ones standard attention punishes most.
The story everyone tells about AI cost is a training story: the enormous one-time run, the cluster, the months. It's a real cost — but it's the wrong one to budget around. Training a model is something you do once. Inference — running the model — is something you do on every single request, forever. That's where the spend actually accumulates.
Industry estimates bear this out. Inference is projected to grow from about half of all AI compute in 2025 to roughly two-thirds in 2026 (Deloitte), and it can reach 80–90% of the lifetime cost of a production AI system, precisely because it runs on every request rather than once. The AI inference market alone is estimated near $106B in 2025 and projected toward $255B by 2030 (MarketsandMarkets). The one-time cost gets the attention; the recurring one gets the invoice.
Here's the part that should worry anyone forecasting an AI budget: the fastest-growing inference workloads are the ones standard attention handles least efficiently — the long-document ones. Contracts. Case law. Patient histories. Regulatory filings. Whole codebases.
Standard attention makes these expensive by construction. Its cost grows with the square of the context length, and the key–value cache it builds devours GPU memory. Double the document, roughly quadruple the attention cost. So the workloads people most want to point AI at — "read this entire thing and reason over it" — are precisely the ones that scale the worst on the bill. Demand is climbing fastest where the unit economics are worst.
That gap is the whole game. If long-context inference stays super-linear, the most valuable AI workloads stay the most expensive ones, indefinitely.
The fix isn't a bigger GPU or a cheaper token — it's changing how the cost scales.
We wrote last week about treating a document's context as a durable asset: index it once, reuse it across every chat, agent, and API call, with no prefill to redo. That's the reuse half of the story. The other half is the scaling: lab358 converts the open-weight model you already use into a natively sparse, top-k attention architecture served by approximate nearest-neighbor retrieval, so long-context inference scales sub-quadratically instead of with the square of the context. The documents that are most expensive today are the ones that see the biggest change.
Put the two together — reuse a document's context instead of recomputing it, and pay sub-quadratically for the context you do compute — and the long-document workloads that break a standard inference budget become the ones that hold up best under it.
lab358 is coming two ways — the same converted model and the same OpenAI-compatible API either way:
If long-context inference is a line item you're trying to forecast, join the lab358 Cloud waitlist — or talk to us about self-hosting.
Investors: hello@lab358.ai.