Memoize SpecPrefill scoring by prompt hash

Sparse prefill uses a cheap scorer to estimate which prompt tokens matter, then fully prefills only those. Production logs show the same twenty-thousand-token prompt scored four times within ninety seconds, at roughly 4.1 seconds each. That symptom is real, and this pull request set out to key the selection by a hash of the raw tokens so the repeated scoring could be skipped.

The premise I submitted it on was that scoring is deterministic for a fixed input. That premise was wrong. In review the maintainer pointed out that the scorer is stochastic: _lookahead_decode samples at temp=0.6, top_p=0.95 with no fixed seed, so score → select is not the pure function I described in the module docstring. Memoizing by hash therefore does not transparently cache a deterministic result — it freezes one sampled selection and reuses it for every later identical prompt. The selected 11744/23444 line in the logs proves nothing about positions either, since keep_n = ceil(n_chunks * keep_pct) fixes that count regardless of which chunks win. I accept both points.

The other half of the maintainer's argument is that the hit condition is too narrow: a multi-turn conversation going from A-B-C-D to A-B-C-D-E changes the hash, and the existing draft KV cache already reuses the shared prefix. On that half I offered one correction: SpecPrefill requests never populate the target prefix cache — scheduler.py:8891 drops raw_cache, and scheduler.py:5554 does the same on the boundary-store fallback — so a byte-identical retry re-enters with the full prompt and re-scores, which is the mechanism behind those four repeated scorings. It does not disturb his conclusion, though: retries themselves are not common.

As of August 1, 2026, this pull request remains open and unmerged, and the maintainer has explicitly recommended against merging it. It stays in this record because it documents a mistaken judgment, not because it still looks promising. The next step is not to reinforce this cache but to break down the 4.1 seconds itself: after a draft-cache hit covering twenty-two thousand tokens, the remaining cost is the lookahead decode plus _compute_importance over the full K cache — and every request pays that, including the multi-turn path a selection cache can never help. This pull request will be rewritten or closed. GitHub is authoritative for its live status.