When does accepted speculative decoding translate into higher throughput and lower latency?

Speculative decoding is a technique that uses a cheap drafter to propose the next n tokens while the target model verifies them in one pass. If enough draft tokens are accepted, we save target-model decode steps, improving latency and throughput.

Image source: Z.ai GLM 5.2.

Training the drafter with Megatron

During training, one large advantage of multi-token prediction is that the drafter does not need to be a separate model. It can be added as a small prediction layer on top of an existing checkpoint and trained alongside the main objective. In Megatron, this is enabled with --mtp-num-layers 1; the additional layer uses the model’s hidden states to predict future tokens.

Training is straightforward for many open source models: add the MTP loss to the standard language-modeling loss with a relatively small weight. This lets the auxiliary layer learn the target model’s generation patterns without materially changing the behavior of the base model. Because the target model verifies every speculative token at inference time, the goal is not for the MTP layer to be independently correct. It only needs to make proposals the main model is likely to accept.

This also makes MTP practical for post-training. It can be introduced during fine-tuning or in a short continuation run, rather than requiring a new pre-training pipeline.

Inference

Speculative decoding is not a silver bullet. Because we need verification work done by the target model (essentially a pre-fill), at a certain batch size, the verification can provide more overhead than just decoding from the target model. We ran this experiment for ourselves on different workloads:

When does accepted speculative decoding translate into higher throughput and lower latency? When does a large target batch make verification the bottleneck?

The setup

• A 9B model on one H100, served with vLLM.
• No MTP: ordinary autoregressive decoding.
• Main concurrency sweep: three proposed draft tokens per step from the checkpoint’s MTP head.
• MTP-depth sweep: 1, 2, 4, 8, and 16 proposed tokens at fixed concurrency 32.
• Concurrency: 1, 10, 24, 32, 40, 64, 80, 128, and 256.
• Three task shapes: text regions (text), form regions (key-value), and tables. All three use image inputs and require OCR output.
• Each workload ran independently for a fixed 120-second measurement window.
• Acceptance uses vLLM’s cumulative process counters, snapshotted before and after each workload.
• Every reported workload completed with zero request errors.

Baseline: no MTP

The no-MTP run tells us what batching alone can do. Throughput and request latency rise strongly with concurrency.

MTP improves over the baseline up to a certain batch size

With three MTP steps, accepted draft tokens let the model advance several output positions per target-model verification instead of one. MTP depth is fixed at three, so acceptance stays roughly constant as concurrency changes. What changes is the cost of target verification as the batch grows: at low and moderate concurrency, verification is still cheap enough that those skipped sequential decode steps translate directly into higher throughput and lower latency, especially for the longer key-value and table outputs.

The story is different at higher concurrencies:

However, text recovers at very high concurrency, where its short-output shape can still exploit the larger batch: MTP does not have one concurrency threshold. Its benefits depend on request shape, output length, and how efficiently the target model verifies the resulting batch.

Why not propose more tokens?

Acceptance is what turns speculative work into speed: every accepted draft token removes a sequential target-model decode step. But tokens farther into the future are harder to predict. The useful quantity is therefore not how many tokens we propose, but how many output positions each target verification actually advances.

At concurrency 32, one- and two-token drafts are accepted almost entirely and advance about 2.0 and 2.9 tokens per verification pass, respectively. Four-token drafts advance roughly 3.2–4.0 tokens. Beyond that, the accepted span stops growing: 8- and 16-token drafts still advance only about 3.1–4.4 tokens, while draft-token acceptance falls to 27–42% and then 13–20%.

Those extra speculative positions add drafting and verification work without removing more target decode steps. That is why deeper MTP eventually reverses the gain. From one to 16 steps, table throughput falls from 2,486 to 607 tok/s while mean latency rises from 9.1 to 35.3 seconds; key-value throughput falls from 1,541 to 414 tok/s while latency rises from 3.8 to 12.8 seconds.

The bottom line: generation versus drafting latency

We enabled vLLM’s named profiler scopes to separate target forward/verification from MTP drafting.

From concurrency 128 to 256, drafting time rises 42%, from 62ms to 88ms. Target forward/verification rises almost 3×, from 427ms to 1,272ms. Therefore, the limitation at high concurrency is the high verification cost on the larger target batch.¹

Takeaway

MTP wins when accepted draft tokens remove more sequential decode work than drafting and batched verification add:

• Speculation depth determines the useful accepted span. At concurrency 32, that span grows to only about 3–4 tokens per verification pass and then plateaus. Deeper proposals are mostly rejected, reducing throughput and increasing latency.
• Concurrency determines the cost of each verification. With depth fixed at three and acceptance nearly flat, MTP wins at low and moderate concurrency, loses across all workloads around 64–80, and becomes workload-dependent at 128–256.
• Workload shape determines the operating point. Longer key-value and table outputs gain most while accepted speculation is cheap, but they also degrade most when deeper drafts or larger target batches add unproductive work.
• Verification is the dominant pressure at high concurrency.

Tune MTP depth around the accepted span first, then compare MTP with ordinary decoding at the serving concurrency you expect. Acceptance is the mechanism; throughput and latency are the outcomes.

Engineering by Jean Ghislain Billa and the Reducto ML team. Edited by Palak Agarwal.

If these types of problems excite you, apply to join our team.