Latency is a feature: making completions feel instant
The engineering behind sub-100ms next-edit predictions on files with thousands of lines.
A completion that arrives in 400ms is a different product than one that arrives in 80ms. Above roughly 150ms, suggestions interrupt typing instead of anticipating it, and users learn to ignore them. This post covers the three layers of engineering that get Arshas predictions under 100ms end to end.
Layer one: don’t send what the server already knows
The dominant cost in naive implementations is payload. Re-sending a 3,000-line file on every keystroke burns time on serialization, transport, and prompt processing before the model does any useful work.
Arshas maintains a session on the inference side keyed to your editing context. After the first request, the client sends only deltas: the edit you just made, updated cursor position, new diagnostics. Server-side, the prompt cache is reused across keystrokes, so the model re-processes only the changed suffix. Median request payload dropped from ~40KB to under 1KB.
Layer two: predict before the keystroke
The second trick is speculation. While you type, the client watches for high-probability trigger points — end of a statement, a newline after a function signature, an accepted previous prediction — and issues the request before you pause. If you keep typing something incompatible with the speculative result, it’s discarded; if you pause, the prediction is already there.
Speculation converts network latency into wasted-but-cheap compute. Roughly 60% of shown predictions were requested before the pause that displayed them.
Layer three: a model sized to the job
Next-edit prediction doesn’t need a frontier model — it needs a fast model with the right context. Predictions run on a small model distilled specifically from accepted-edit data, with the codebase index supplying the cross-file signal that raw model size would otherwise have to compensate for.
The distilled model is roughly 20× cheaper to serve than the chat model and, on the narrow task of next-edit prediction, accepted at a higher rate. Specialization beats scale when the task is this well-defined.
The result
P50 end-to-end latency — keystroke to rendered ghost text — sits at 71ms, with P95 under 160ms. At that speed predictions stop feeling like a feature and start feeling like the editor is simply ahead of you. That’s the bar: latency isn’t a performance metric here, it’s the product.