Loading…
Research
lab358 serves the open-weight model you already use, exactly as its authors published it, and makes its memory durable: a document is processed once into a stored form and retrieved on demand rather than re-read on every call. Reach is a property of the model you pick — see its model card. Here is the published work we build on, what we integrate that is new, and how the approach differs from the common alternatives.
The idea
In a conventional stack, a long document is pushed through the prompt on every request and re-read from scratch — and the window it has to fit inside has a fixed ceiling. lab358 changes what the model is given: a document is processed once into stored memory, and a later request draws on that stored memory directly rather than re-reading the document.
The search over stored memory is exact, and the attention over what it returns is the model's own, unchanged. A request draws on the documents you attach to it; the model attends over their stored memory, and over recent context, merged into a single result — the same attention its authors shipped, with nothing substituted for it. And because stored memory does not have to sit in GPU memory to be usable, how much a model can draw on is not decided by what fits alongside it on one device. How far it reaches is then a property of the model itself — see its model card.
A document's key–value memory is computed once and reused wherever it is needed — the same precomputed memory serves every later request against it, with no re-encoding.
Lineage
lab358 is an integration of ideas from a rich body of published research. A selection of the work most directly related to how our attention and retrieval are built:
Bertsch, et al. — NeurIPS 2023, arXiv:2305.01625
An early, clean statement of the core idea: let a model reach into an index for the entries it needs instead of holding everything in one window.
Read the paper →Yao, et al. — EuroSys 2025 (best paper), arXiv:2405.16444
Fuses key–value caches that were computed independently, one per chunk, by selectively recomputing a small fraction of tokens to restore the attention that crossing a chunk boundary would otherwise lose — the result that made precomputed-and-reused chunk memory a serving technique rather than an approximation.
Read the paper →Hu, et al. — ICML 2025, arXiv:2410.15332
Names and formalises position-independent caching: a chunk's stored memory should not be bound to the place it happened to occupy when it was computed. It also locates the accuracy cost in the per-chunk attention sink rather than in the positions themselves.
Read the paper →Acharya, et al. — arXiv:2411.17116, 2024
Splits a long context into blocks attended independently, then reconstructs the result by merging each block's softmax statistics rather than its full cache — the operation that lets work computed apart be combined without re-reading everything.
Read the paper →Yang, et al. — ICLR 2025, arXiv:2502.05431
Encodes contexts in parallel and shows what has to be corrected afterwards for the merged answer to hold up — a shared prefix, an attention temperature, and an explicit reweighting of each part's contribution. The clearest statement that combining separately-encoded context is a question of weighting, not of concatenation.
Read the paper →A selection, not the full bibliography. Each ingredient above is well established in the literature; what lab358 contributes is the integration.
What's different
Each piece has strong prior art. Bringing them together into a single serving stack — around a model we do not touch — is where lab358 is distinctive.
We serve the open-weight model exactly as its authors published it. What changes is what the model is given to read: a document is processed once into stored memory and retrieved on demand, instead of being pushed through the prompt again on every call. No conversion, no retraining, no fine-tune — so the model's published evaluations remain its evaluations.
Retrieval systems usually buy speed by narrowing the field first: an approximate search proposes a shortlist, and the answer can only ever be as good as that shortlist. lab358 does not narrow it. A request draws on the documents you attach to it, and the question is scored exactly against their stored memory — so nothing is missed because a candidate list was too small.
Stored memory does not have to sit in GPU memory to be usable, so how much a model can draw on is not decided by what fits alongside it on one device. Recent context stays immediately at hand; the rest is retrieved when it is wanted.
A document's key–value memory is computed a single time and reused across chats, agents, and users — no re-encoding and no prefill to redo, even days apart. Reuse is a first-class product primitive, not an accident of caching.
Approach comparison
A comparison of techniques, not products. Each row describes a way of giving a model access to more context, and where it runs into a wall.
The resolution
A document is processed once into stored memory. A request draws on the documents you attach to it; the question is scored exactly against their stored memory, and the model attends over it directly, merged with recent context into one result.
The result: Nothing is dropped for falling outside a window, and nothing is re-encoded: memory is retrieved into attention directly, and precomputed document memory is shared across uses. How far a given model reaches is a property of that model — see its model card.
The research, in a product you can run — hosted on lab358 Cloud or in your own AWS account.