For greedy decoding, equality with the target's chosen token can be a valid way to advance. Under temperature or nucleus sampling it is a different question. The target is a probability distribution, not one designated correct token. If the draft proposes a less likely but valid target token, an argmax check rejects it too often. And after rejection, sampling from the unmodified target distribution double-counts probability mass already admitted through accepted drafts. Fluent example outputs do not test distributional correctness.

Here is the first-principle rule for one position. Let p be the target distribution after the requested sampling transform, and q the draft distribution at the same prefix. Draw x from q. Accept x with probability min(1, p(x)/q(x)). If rejected, draw the replacement from the normalized positive part of p minus q. The accepted path contributes min(p(x), q(x)) probability mass to x. The residual contributes p(x) minus that mass. Together they give p(x). The original speculative decoding paper develops this exact sampling construction. Its speed results are for the studied implementations, not an assurance about this fleet.

Speculative sampling accepts a draft token or corrects the first rejection
Only verified accepted tokens become output. The remaining draft suffix is discarded after the first rejection.

For several draft positions, each target distribution is conditioned on the actually accepted prefix. At the first rejection, discard later draft tokens and correct that position. If all four pass, sample the extra token from the target distribution at the resulting prefix. Apply temperature, top-p and other sampling rules consistently to the p and q used by the acceptance calculation. The two models must agree on token identity. A tokenizer mismatch or an incorrect prefix position breaks the comparison before the acceptance formula even matters.

I would validate the implementation with a tiny vocabulary where we can enumerate p and q, including a draft that never proposes a token the target likes. Then compare empirical output frequencies with ordinary target sampling over many seeds, and separately test greedy equivalence, stop tokens and partial streaming. Draft tokens must not be shown to a user or parsed as a tool call before verification. If the draft version changes mid-request, either pin it or retain the exact q used for every proposal and prove the implementation still conditions correctly. Pinning is the simpler operational contract. Once correctness passes, the cost question decides whether to keep it.