Qwen 3.8-27B on an M1 Ultra: From 12.1 to 20.3 tok/s by Measuring Everything

I had been circling this kind of experiment for a while. Back in March, working with another AI in my autonomous refinement lab, we traced a sub-20 tok/s wall for 30B models on an M4 Max to bandwidth starvation from OS-level paging — and that investigation left a playbook: powermetrics first, per-cluster sampling, memory pinning before blame. The run this time started from pure curiosity: I had heard about Qwen3.8's optimizations, and I wanted to know whether it would run fast out of the box on this machine with this much RAM. Only mid-experiment did the bandwidth-bound reality make itself felt. What followed was a sprint of controlled experiments we designed together — Kimi drafted the hypotheses and read the hardware counters at my side while I kept my hands on the metal. Each experiment was built to acquit or convict a single layer of the stack. The wins came from 2 places that local-inference folklore under-rates: CPU scheduler hints and speculative decoding. The 2 fashionable alternatives — a GGUF build served by llama.cpp, and a mixture-of-experts model that looked unbeatable on paper — both lost on measurement. This article is the full evidence trail we built together, because the method turned out to be worth more than any single number.

The serving path had 3 hops: my workstation, an SSH-based exec relay, and the model server on the Studio's loopback. Blaming the relay would have been easy, so we measured it first. The server's own timings said 12.1 tok/s decode; a raw in-process generation benchmark, with no server and no relay at all, produced 11.5 tok/s. The transport was costing about 25% through per-connection overhead, and the model itself was the slow layer.

One transport bug was worth fixing anyway, and it is the kind of detail that costs hours if you have never seen it: a Python TCP relay that uses a buffered read(65536) will deadlock against a chatty local protocol, because the buffered reader waits for a full buffer or EOF before returning. Switching to read1(), which returns after a single underlying read, made the relay behave. The symptom looked like a network stall; the cause was stdio semantics.

With transport acquitted, we worked through the usual sysctl layer. Raising the GPU wired-memory limit (iogpu.wired_limit_mb) changed nothing with 128 GB of RAM free, so we reverted it. The Python process was native arm64, MLX reported the GPU as its default device, and powermetrics showed the GPU at 54-62% active residency drawing 20-23 W during decode. Every suspect in this round walked free — which was itself the useful result, because it pointed the investigation at the CPU.

Reading powermetrics per CPU cluster instead of per machine exposed the first real finding. During decode, the efficiency clusters ran at 75-93% busy while the performance clusters idled at 1-12%. Anything launched over SSH inherits a quality-of-service class that macOS reads as low-priority work, and the scheduler was honoring that hint by keeping a latency-critical workload on the slow cores.

The fix was 1 line of ctypes, executed before the model weights load:

import ctypes
ctypes.CDLL("libSystem.B.dylib").pthread_set_qos_class_self_np(0x21, 0)  # QOS_CLASS_USER_INTERACTIVE

That pin, plus loading the model's text tower through mlx-lm instead of the full vision stack, moved raw decode from 11.5 to 15.1 tok/s — a 31% gain from scheduler placement and a leaner loader, with zero change to the model. We later learned the pin's reach has a limit: it moves the calling thread, and MLX's worker threads keep their own scheduling class. For this model the main thread carried enough of the work to matter.

The largest single win came from a feature the base model already owned. Qwen3.8 ships with a multi-token-prediction head — a small auxiliary module that drafts several tokens ahead — and the MLX converter quietly drops those 15 tensors during conversion. A community package re-hosts the head as a standalone 253 MB drafter, which let me pair it back with the model it was trained alongside.

The serving pattern is draft-then-verify: the drafter proposes a few tokens, the full model checks them in 1 pass, and accepted tokens all count. With --draft-model and --draft-kind mtp on the server, draft acceptance measured 94% on realistic coding and chat prompts, and decode went from 15.1 to 20.3 tok/s server-side — 13.4 tok/s end-to-end through the relay, up from 9.0. The GPU told the confirming story: 77% active residency, all of it at the full 1296 MHz, drawing 48 W, with the performance cluster 97% busy. Prefill improved in the same upgrade, from 14.7 tok/s to 18-55 depending on prompt shape.

Community posts put llama.cpp with a speculative draft at 25-32 tok/s for this model class on M1 Ultra, comfortably ahead of our MLX numbers, so we gave the challenger a fair ring: the official Q4_K_M GGUF, a matching MTP-only draft model, and a current llama.cpp build with Metal acceleration confirmed active. The GGUF lane measured 9.2 tok/s baseline and 12.2 with the draft — about 40% behind the MLX stack it was meant to dethrone.

llama.cpp remains excellent engineering; the gap likely lives in how each runtime's Metal kernels handle this checkpoint on this chip. The durable lesson is simpler: a number somebody else measured on their machine, their build, and their checkpoint is a hypothesis about yours. The 18 GB of challenger weights left the machine the same afternoon, and the 50 MB llama.cpp binary stayed for future rematches. I have written about this rule before as benchmark-driven development — it carried SEOReport from heuristics to a product — and here it saved a migration.

The last challenger had the strongest theory — the same common knowledge we had set out to test at the start. Dense models decode slowly on Apple Silicon because every generated token pays to read nearly all the weights; a mixture-of-experts model with 35B total parameters activates only about 3B per token, so on a bandwidth-bound machine its decode should run several times faster than a dense 27B — every token reads about 11% of the weights. We downloaded the 4-bit MoE and its matching MTP drafter, warmed the server, and measured 14.7 tok/s raw: the same speed as the dense model it was supposed to humiliate. With speculative decoding it reached 19 tok/s on realistic prompts and 26.3 on highly predictable text — a wash against the dense model's 20.3, with no quality argument to break the tie.

powermetrics identified the regime in 1 sample. During MoE decode the GPU sat at 0-3% active residency while the efficiency clusters ground away at 60-90%. The model spends its per-token budget on CPU-side work, and op-level timing showed why: each dispatched operation costs 50-100 microseconds of overhead, and this hybrid architecture — GatedDeltaNet linear attention plus 256 experts behind a small 2048-wide hidden state — issues roughly 78 operations per layer across 40 layers. At batch size 1, the GPU finishes each tiny kernel before the CPU can queue the next one. The machine was dispatch-bound, and dispatch-bound workloads gain nothing from reading fewer weights per token.

graph TB
    A[Slow decode  
at batch 1] --> B{GPU busy  
during decode?}
    B -->|yes| C[Bandwidth-bound  
fewer bytes win]
    B -->|no| D[Dispatch-bound  
fewer ops win]
    C --> E[MoE helps  
smaller quant helps]
    D --> F[Speculative decoding  
helps most]

    style E fill:transparent,stroke:#10B981,stroke-width:2px
    style F fill:transparent,stroke:#3B82F6,stroke-width:2px

To close the loop, we proved the GPU path itself was healthy with a synthetic hog: an 8192³ matrix multiply loop hit 10.8 TFLOPS in bfloat16 with the GPU at 97-100% residency and 68 W. External evidence agreed with the local diagnosis — independent estimates put this MoE class near 21.7 tok/s on an M2 Ultra, and an OpenVINO issue documents the same model losing to a dense 8B on backends with weaker dispatch paths. The dense 27B with its MTP drafter kept the production slot, and 38.5 GB of MoE weights were deleted the same evening.

The machine now serves Qwen 3.8-27B at 20.3 tok/s server-side, a 68% improvement over where the day started, and the more durable yield is a checklist we will reuse on every future box: