addresses the fundamental challenge of slow, sequential token generation in language model inference. DFlash speculative decoding support for CPUs was recently enabled in vLLM v0.25.0. In our testing with Qwen3.5-9B on an r8i AWS instance, powered by Intel® Xeon® 6 processors with Performance-cores, DFlash increased average token generation throughput to 3.92x that of the autoregressive baseline at concurrency 1—a 74% cost reduction per generated token.
This post explains what speculative decoding is, and how DFlash in particular allows you to speed up your AI workloads on CPU. Our configurations are detailed at the end of this post if you’d like to follow along and reproduce the results. (Full disclosure: the author is affiliated with Intel.)
Contents
What is Speculative Decoding?
A typical autoregressive decoder produces one token, appends it to the context, and runs again to predict the next token. No kernel optimization can parallelize across a dependency that has not been resolved yet, so a 500-token response requires about 500 dependent cycles. Each token generation step requires the transfer of the entire model’s parameters from memory to the processor’s compute units. This makes the operation memory bound and leaves those compute units mostly idle at low concurrency. Speculative decoding works around that serial dependency without changing the model’s output distribution. A lightweight draft model proposes several future tokens. The larger target model checks all of them in a single pass, accepts the longest valid prefix, and corrects the first miss, and—if every token is accepted—generates one bonus token. When the draft proposals are high-quality and fast, each expensive target pass commits several tokens instead of one. Unlike lossy optimization techniques such as quantization, speculative decoding is a lossless acceleration since rejection sampling recovers the target distribution.
This technique is especially relevant to inference on Xeon. At small batch sizes, decode often spends much of its time moving model weights from memory for very little work per weight. Verification turns per-token matrix-vector operations that lean on Intel® Advanced Vector Extensions 512 (Intel® AVX-512) into matrix-matrix operations that Intel® Advanced Matrix Extensions (Intel® AMX) accelerates: the target weights, once loaded, are reused across several candidate positions. The catch is that speculation adds a drafter and widens verification. It only pays off when the accepted work exceeds that overhead: we’re trading spare compute for saved memory bandwidth; when there is no spare compute, the trade becomes a loss.
DFlash: Block Diffusion for Drafting Plus Target KV Injection
DFlash is a novel speculative decoding method developed by Z Lab. Instead of generating draft tokens serially as in some older algorithms, the DFlash speculator predicts a block in one pass with a small block-diffusion drafter. It also injects hidden features from the target model into the draft model’s KV cache, which improves draft quality without requiring the drafter to reconstruct the full context by itself.
The acceptance rate depends on your model and is typically higher for more structured prompts such as code or math compared to conversation. If the target model is domain-specific, the acceptance rate drops on unrelated prompts. You should not expect to see much performance improvement, if any, when benchmarking a coder model on a conversational dataset.
The drafter predicts all masked positions in one forward pass, with bidirectional attention inside the block. The number of masked positions—num_speculative_tokens in the command below—is a server parameter, and its optimal value depends on several factors, including the target and draft models, the benchmark dataset, and maximum concurrency as well as your hardware.
Let’s see how to boost a model’s performance with DFlash using vLLM with an additional configuration flag:
docker run --rm
--name vllm-cpu-server
--network host --ipc host --security-opt seccomp=unconfined --cap-add SYS_NICE
-e VLLM_TARGET_DEVICE=cpu
-e VLLM_CPU_KVCACHE_SPACE=40
-v ~/.cache/huggingface:/root/.cache/huggingface
vllm/vllm-openai-cpu:latest
Qwen/Qwen3.5-9B
--dtype bfloat16
--trust-remote-code
--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-9B-DFlash", "num_speculative_tokens": 15}'
The effect is easy to see in a streamed response. With the same prompt and deterministic decoding settings, DFlash completes the response substantially faster while producing the same output.
Some recent models such as Muse Glimmer ship with a drafter model. If that’s not the case for your use case, start with Z Lab’s DFlash collection to explore drafters for popular models. For additional options, see Red Hat AI’s model collection, which includes models trained with the Speculator library. Below are the speedup results for Qwen3.5-9B at concurrency 1 and output length 128 compared to the autoregressive baseline.[i] Token generation cost reduction is measured at fixed concurrency on the same instance, so it does not depend on the hourly rate. The average is taken across three datasets for programming (HumanEval), mathematics (GSM8K), and multi-turn conversational questions (MT-Bench).
| Dataset | Baseline tok/s | DFlash tok/s | DFlash Speedup | Acceptance Length | Cost Reduction |
|---|---|---|---|---|---|
| GSM8K | 9.95 | 41.42 | 4.16x | 7.08 | 75.98% |
| HumanEval | 9.88 | 39.41 | 3.99x | 6.93 | 74.93% |
| MT-Bench | 9.95 | 35.89 | 3.61x | 6.01 | 72.28% |
| Average | 9.93 | 38.91 | 3.92x | 6.67 | 74.40% |
Understanding Speculative Decoding Metrics
vLLM reports several statistics that describe how effectively the target model accepts DFlash’s proposals. Consider the HumanEval run from Table 1, with num_speculative_tokens set to 15:
Acceptance rate (%): 39.56
Acceptance length: 6.93
Drafts: 381
Draft tokens: 5715
Accepted tokens: 2261Drafts is the number of speculative blocks verified by the target model: 381 verification steps carried this run’s 2560 output tokens, which the autoregressive baseline would have needed about 2560 sequential passes to generate. Because each of the 381 blocks contained 15 proposals, there are 15 x 381 = 5715 draft tokens. Of those proposals, the target accepted 2261, producing the reported acceptance rate 2261/5715 = 39.56%. Finally, each round (or block) produced an average of 2261/381 = 5.93 accepted draft tokens. Verification also emits one target token, either as a correction, or as a bonus when the block is fully accepted. The resulting acceptance length is therefore 5.93 + 1 = 6.93.
A word of caution is in order. For fixed-length drafting, as used by DFlash, the overall acceptance rate is the average of the per-position values below, so it typically falls as the block grows longer, even when throughput improves. Use acceptance length rather than acceptance rate to compare progress per verification round, but choose the block size from measured throughput or latency. Also, speculative decoding counters can include tokens accepted in a final block but discarded when a request reaches its output length limit, so the token count implied by these counters may exceed the number returned to clients.
The per-position statistics show how far proposals typically survive into the block:
Position 0: 85.83%
…
Position 4: 51.44%
…
Position 14: 11.81%These are survival (or acceptance) probabilities. Position n being accepted means the target accepted the complete speculative prefix through position n, so the column is decreasing by construction and sums to the accepted draft tokens per round (593.44% = 5.93). Dividing consecutive values gives the conditional acceptance rate at each position. A collapsing conditional rate indicates that later proposals are increasingly unlikely to survive.
An average acceptance length of 6.93 = 5.93 + 1 does not imply that the block should be shortened to six tokens as the distribution is skewed: 14% of blocks have every proposal rejected and 12% are accepted in full. Doing so would eliminate all the rounds in which the target could accept longer prefixes. Also, we cannot increase the block size indefinitely because the rejected tail positions still consume drafting and verification resources. The optimal block size balances useful progress against total round cost:
Both terms in the numerator generally grow with the block size while the denominator saturates as per-position survival probabilities typically approach zero, so the achievable gain is bounded.
Acceptance statistics help explain performance, but they do not determine the optimum by themselves. To tune num_speculative_tokens, benchmark several block sizes on the intended model, dataset, hardware, and concurrency, then select the value producing the best throughput or latency.
Key Takeaways
- Speculative decoding does not make an autoregressive dependency disappear; it moves uncertain future work into a cheaper parallel proposal path and lets the target validate several positions at once, with rejection sampling ensuring a bad proposal costs time, not output quality.
- DFlash improves the proposal path in two complementary ways: Block diffusion replaces several serial draft invocations with one parallel block pass, and KV injection gives every draft layer direct access to the target’s contextual representation, raising acceptance without turning the drafter into another large language model. That combination is a natural fit for low-batch CPU inference, where target weight movement dominates and wider verification improves weight reuse.
The payoff compounds in agentic workloads, where a model is called repeatedly across multi-step loops and every decode step counts which is why speculative decoding underpins Intel’s open source agentic stacks like the Intel® AI for Enterprise Agent Toolkit and Intel® AI SuperClaw.
Work on parallel drafting is advancing rapidly beyond DFlash. DSpark adds a semi-autoregressive correction stage and variable-length verification, while DFlash 2 uses a lightweight path selector to trace a coherent path through each position’s top candidates and local convolutions to reduce draft-accuracy decay near the end of each block. Keep an eye on newer speculative decoding methods as support reaches CPU inference frameworks.
Acknowledgments
The author would like to thank Alex Sin, Eric Petit, Eze Lanza, and Pradeep Surabhi for their review and feedback on this post.
References
Notices and Disclaimers
Performance varies by use, configuration, and other factors. Learn more at www.Intel.com/PerformanceIndex.
Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. See backup for configuration details. No product or component can be absolutely secure.
Your costs and results may vary.
Intel technologies may require enabled hardware, software, or service activation.
© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
[i] Configurations: 1-node, Amazon EC2 r8i.16xlarge, 1x Intel(R) Xeon(R) 6975P-C, 32 cores, Unknown TDP, HT On, Turbo On, Total Memory 512GB (1x512GB DDR5 7200MT/s [Unknown]), BIOS 1.0, microcode 0x1000434, 1x Elastic Network Adapter, 1x 400G Amazon Elastic Block Store, Ubuntu 24.04.4 LTS, 7.0.0-1009-aws, vLLM 0.26.1rc1.dev124+gb88916617. Test by Intel as of July 2026.