The faster engine won the short test and made a long conversation twelve times slower

We spent a day tuning a large language model on a single M4 Max laptop. Behind it sits a question more and more clients are asking us: is running on your own hardware a serious option, or a hobby?

The useful answer is not a tokens-per-second figure. It is one configuration decision: the engine that won the short-prompt benchmark left the second turn of a long conversation twelve times slower, because it destroyed the reuse of the already-processed prompt. Plus a retraction of our earlier block-size claim, wrong in every part, including which document we attributed it to.

How to read these numbers

Absolute figures are not clean: other GPU-using processes ran on the machine throughout and the hardware was thermally loaded. A control run on identical settings drifted 34 percent. Only ratios measured by interleaving survive that, variant A and variant B alternating and taken as a median, because the drift then hits both variants and cancels in the ratio.

The measurements come from two sessions on two checkpoints, meaning two saved copies of the same model's weights at different precision (quantisation stores weights in fewer bits, at some cost in fidelity). The prefix-caching result comes from the 4-bit copy we deployed, while the block-size sweep and the long-context runs come from an earlier 8-bit one. We also never recorded the version of MLX, Apple's compute library for its own chips, and that is where the cause of the block-size effect sits.

The machine and the model

M4 Max, 40-core GPU, 128 GB of RAM, 546 GB/s of memory bandwidth, oMLX 0.6.4 as the server, opencode as the client. The model is a mixture of experts: instead of one large network it holds many smaller sub-networks and a router picks a few of them per token, so of 35 billion parameters about 3 billion are active for any given token.

The decision that mattered: the faster engine wiped out prefix caching

Speculative decoding in one paragraph. Decode is slow because it is sequential: one token, one full read of the weights. So you add a small draft model that guesses several tokens ahead, and the large model checks the whole proposal in a single pass, accepting it up to the first disagreement. The output is identical to running without speculation, so it is a pure speed trick. Block size is how many tokens the draft proposes per round, and accept length is how many of them get through on average.

Our server offers two ways of doing it. DFlash runs a separate draft model, loaded through its own pipeline. MTP, short for multi-token prediction, uses a small extra head trained into the model itself and runs inside the main engine.

On short prompts DFlash simply wins: 137 against 100 tokens per second, single runs. Pick by that benchmark and you pick DFlash. So we ran the shape of workload we actually have: one 33k-token prompt, two consecutive turns on it. The thing under test is prefix caching, the mechanism that keeps the computed state of the start of a conversation so the second turn does not process the earlier prompt again from scratch.

The second turn is 12x faster without the faster engine. With DFlash on, every turn pays a full prefill from scratch and the server's cached_tokens counter reads zero.

The cause is structural rather than configurational. DFlash brings its own loading pipeline and a private cache cut off from the main one, while MTP runs inside the batching engine, so prefix caching is left untouched. There is no middle setting either. So we run MTP with DFlash disabled, knowingly losing the isolated benchmark.

This is a fact about oMLX 0.6.4, not about speculative decoding and not about Apple Silicon. What transfers is the method: the speculation path and the prefix cache are two separate mechanisms that can be built so they do not compose, and a single-turn benchmark will never see it.

We raised the context window to 262144 tokens while we were there. Prefill runs at about 350 tok/s, so the window genuinely works, but later turns are only cheap with DFlash off.

Speculation beats quantisation, and the two stack

Interleaved measurement, median of four rounds, a coding prompt, 800 output tokens, on the 4-bit and 6-bit checkpoints.

Speculation is worth 2.35x to 2.55x, and the entire 6-bit to 4-bit question is worth 18 to 28 percent. If you only ship one, do not ship quantisation. They do combine, though: 6-bit without speculation to 4-bit with it comes out at 3.01x against a naive product of 3.26x, which is 92 percent. Have both, in that order.

The mechanism for the shortfall is in the last row. Checking several candidates in one pass reuses a single read of the weights, so speculation lifts decode off the memory-bandwidth limit, and once bandwidth stops binding, shrinking the weights buys less. Checking a wide block is not free either: on MLX with quantized weights the cost of a pass grows faster than the number of tokens checked.

Block size: the retraction, and three sources we were conflating

An earlier version ran under the headline that the paper's recommendation is backwards, attributed that recommendation to a README, and quoted a table that came from neither. Here is what each source actually says.

The paper. arXiv 2602.06036, DFlash: Block Diffusion for Flash Speculative Decoding, Chen, Liang and Liu. It contains no block-size recommendation for concurrency 1, meaning a server handling one request at a time. Its own ablation says the opposite of what we accused it of: large blocks raise verification cost where compute is the bottleneck.

The model card. The benchmark table comes from the Hugging Face card for z-lab/Qwen3.6-35B-A3B-DFlash, measured on an NVIDIA B200 under SGLang. The card gives block 8 as the recommended default, and says only that 16 gives longer accept lengths and strong concurrency-1 throughput. The example launch command on that same card uses block 8.

The repository README. This is the source that actually covers our case, and we found it after making the claim. z-lab/dflash, README at commit 07ebd93db9f4, 18 August 2026, three weeks before our runs: "For quantized targets or drafts, use block_size <= 5: MLX's current quantized matmul kernel becomes less efficient at larger verify widths". In plain terms: if the target or the draft is quantized and you are on MLX, keep the block at 5 or below.

We ran a quantized model on MLX. Upstream guidance for our exact case already pointed below 8. Our measurement agrees with it and reverses nothing.

What limits it: the 8-bit checkpoint rather than the 4-bit one we deployed, one prompt, 1200 output tokens, one run per configuration. Block 16 came out 17 percent behind block 8, but that gap is smaller than the drift on this machine, so all we claim is that the direction matches upstream. Only adaptive ran at both sizes, and we never tested 5, 4 or 2.

How our measurement differs from the upstream table

A draft proposal is only accepted when it matches what the main model would have picked, so swapping greedy decoding for sampling on its own changes the quantity being measured. Nothing in this comparison isolates the hardware.

The mechanism we gave was wrong too. We explained the effect by the B200's compute headroom against the Mac's, which was a hypothesis with no measurement behind it. Upstream attributes it to the quantized matmul kernel in MLX, a software cause that a patch release can fix. So we are deleting the generalisation we built on it, that datacenter speculative-decoding benchmarks do not transfer to Apple Silicon. On top of that, z-lab ships its own oMLX fork for DFlash and we ran the standard one.

Accept length is where the headroom is

From the server's telemetry: acceptance rate 0.547 at 2.21 tokens accepted per round, and 0.618 at 2.62 across all 41 requests. A later reading gave 3.53, so the value depends on the prompt. That is 2.2 to 3.5 against the 5.35 upstream reports for block 8 on HumanEval, and we did not isolate the cause of the gap.

At an accept length of 5 or 6 the ceiling would be close and a draft fitted to our work would buy nothing. At 2.2 the gap is real, and every extra accepted token per round multiplies straight into throughput. That is an argument for training our own draft, not a plan: the published recipes need NVIDIA hardware. And honestly: accept length has not been measured for the MTP path, the one we run.

Machine drift killed one of our results

We also tested two runtime parameters: the decode mode, and whether prefill is chunked. Four runs of the same test, tens of minutes apart:

The last row restores the first row's settings, and the numbers did not come close to 34.2 s and 103.8 tok/s. The whole effect was drift on a machine that was heating up. Testing in blocks, we would have had a confident paragraph here recommending two setting changes, and it would have been false.

Hence the method on every comparison: A, B, A, B alternating, median rather than mean, and always a control run back to the starting configuration. The 6-bit against 4-bit comparison shows it works: the 6-bit variant fell from 123 to 104 across rounds, and the 4-bit variant won every round individually.

The configuration we settled on

• 35B mixture of experts with about 3 billion active parameters, 4-bit, pinned version

• MTP draft at the default block size, DFlash disabled on purpose

• Prefix cache in RAM: 24 GB, write-through. It had been set to zero, so it ran through SSD only

• Context 262144 server side, 65536 client side, 8192 output

• Sampling: temperature 0.6, top_p 0.95, top_k 20

• Decode mode and chunked prefill: defaults, question unresolved

What we did not measure

• The MLX version. Since the block-size effect comes from a kernel, every sentence about block size is version dependent.

• Block sizes below 8, even though upstream says 5 or less for a quantized model on MLX.

• Repeats in the block-size sweep: one run per configuration, no measure of variance.

• Accept length on the MTP path, which is the one we deployed.

• Output quality at 4-bit in real work.

What follows if you are considering your own hardware

Optimise the number of tokens per pass before you optimise the size of the weights. Speculation gave us more than 2x, quantisation a fifth or so.

Measure two turns, not one. The configuration that won the short-prompt benchmark left the second turn of a 33k conversation 12x slower, and you only see it by reading the cache counter after that second turn.

Before you look at the headline table, find the guidance for your own case. Ours was published three weeks before we measured, in a README rather than on the model card.

If you are weighing up inference on your own hardware and want someone to measure it on your workload rather than someone else's, get in touch.