Upstream contribution · #2346
Memoize SpecPrefill scoring by prompt hash
Byte-identical prompts were re-scored every time. The input was cached (draft KV); the output — the selected tokens — was not.
Sparse prefill uses a cheap scorer to estimate which prompt tokens matter, then fully prefills only those. Scoring is deterministic for a fixed input, yet the server re-ran it for byte-identical prompts — production logs show the same twenty-thousand-token prompt scored four times within ninety seconds.
The interesting part is that no existing cache can be made to skip it. After a run, the cache holds sparse KV keyed by which tokens were selected; using it to skip scoring would require knowing the selection first, and the selection is what scoring produces. The only way out is to key on something already in hand beforehand — a hash of the raw tokens.
As of July 27, 2026, this pull request remains open and unmerged; GitHub is authoritative for its live status.