Zero TVM Documentation

Quick start

No install. Just open the chat page. Weights load from your browser cache (if you've used WebLLM before) or download fresh from HuggingFace (~2 GB).

How it works

The engine has three phases per generated token:

1. Prefill — process the prompt in chunks (per token on specs that cannot chunk), building up the KV cache
2. First decode — the last prefill step produces the first generated token
3. Decode loop — each step takes the previous token as input, runs the full forward pass with the KV cache providing attention context, produces the next token

Each forward pass runs 10 kernel roles through 32 transformer layers — 260 dispatches per token on the default path — split-K attention has been on by default since 2026-07-27 and adds a combine dispatch per layer; ?splitk=0 runs the 228-dispatch reference chain — then reads one i32 token ID back from the GPU. For comparison, WebLLM's TVM-generated decode path fires 342 dispatches per token — around 11 distinct shaders on that path, of 85 captured across a whole session.

Architecture overview

The engine’s main source files:

Weight loader

Weights are stored in MLC's ndarray-cache.json format — an index file listing every parameter, which shard binary it lives in, its byte offset, and byte size. The loader reads this index then fetches each referenced shard.

Fetch priority

1. OPFS — where this engine caches; a returning visitor loads from here
2. Browser Cache API — read-only leftover from prior WebLLM sessions
3. HuggingFace — direct HTTPS fetch from huggingface.co/mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC

Parameter naming (MLC format)

MLC uses non-standard parameter names. The actual names in the cache:

Tokenizer

A hand-written BPE tokenizer in TypeScript. No SentencePiece WASM, no HuggingFace tokenizers bundle. Reads tokenizer.json directly.

Key steps

• Pre-tokenization — Metaspace: spaces become ▁, words are split on whitespace
• BPE encoding — merge pairs by rank from the merge table in tokenizer.json
• Special tokens — <|system|>, <|user|>, <|assistant|>, <|end|>
• Chat template — Phi-3 format applied by buildChatPrompt()

Phi-3 chat template

<|system|>
You are a helpful assistant.<|end|>
<|user|>
What is the capital of Australia?<|end|>
<|assistant|>

Stop tokens: 2 (EOS), 32000 (<|end|>), 32007 (<|endoftext|>).

KV cache

Uses a paged KV cache layout. Memory is divided into fixed-size pages (16 slots each) and a page table maps logical positions to physical pages. This is not vLLM-style relocatable blocks: K is RoPE’d before the cache write on every path, so a cached page is valid only at the positions it was written at. It is a prefix pool, and cannot become a block pool without moving RoPE.

Each layer has its own GPUBuffer for KV pages. The page table is a simple identity mapping (page i → physical page i) for single-sequence inference.

Decode loop

Each call to decodeToken(tokenId, position) submits one command encoder with the full forward pass:

// Per-token GPU state written via writeBuffer
B.inputIds   ← [tokenId]          // i32
B.posMap     ← [position]         // i32
B.pageIndptr ← [0, nnzPages]      // page range
B.lengthInfo ← [position+1, 0, 0] // seq length

// Forward pass (one command encoder)
embedding(B.residual)             // token → hidden state
rmsNorm(B.hidden1, B.residual)    // initial norm

for L in 0..32:
  // QKV matmul + RoPE + KV-append in ONE dispatch (M4 fusion)
  qkvFused(B.qOut, kvPages[L], B.hidden1)
  attention(B.attnOut, B.qOut, kvPages[L])
  int4Matmul(B.hidden2, B.attnOut)               // O projection
  addNorm(B.hidden2, resIn → B.hidden1, resOut)  // residual + RMSNorm, ping-pong
  // Gate + Up + SiLU + mul + Down in ONE dispatch
  fusedFfn(B.hidden2, B.hidden1)
  addNorm(B.hidden2, resIn → B.hidden1, resOut)  // residual + RMSNorm, ping-pong

int4Matmul(B.logits, B.hidden1)   // LM head
argmax(B.tokenOut, B.logits)      // → next token ID

Ping-pong residual buffers

WebGPU's validation rules forbid binding the same buffer as both read and read_write in the same dispatch. The add_norm shader needs to read the old residual and write the new one.

Solution: two residual buffers that alternate each dispatch.

let resIn  = B.residual   // ping (starts with embedding)
let resOut = B.residual2  // pong (uninitialized)

// Each add_norm:
dispatch(addNorm, [delta, resIn, gamma, hidden1, resOut])
[resIn, resOut] = [resOut, resIn]  // swap — O(1), no GPU copy

WGSL Kernel Roles

All shaders live in src/compiler/shaders/, implementing 10 distinct kernel roles. The rest are tiled and subgroup variants of the same role, selectable at runtime via URL flags. The compiler compiles them all at startup into GPUComputePipeline objects.

Binding convention: @group(0) always. Binding indices are zero-based and match the order you pass buffers to bg(device, pipeline, [...bufs]).

Phi-3 model constants

export const PHI3 = {
  D:        3072,   // hidden dimension
  HEADS:    32,     // attention heads
  HEAD_DIM: 96,     // D / HEADS
  LAYERS:   32,     // transformer layers
  FFN:      8192,   // FFN intermediate dimension
  VOCAB:    32064,  // vocabulary size
  PAGE_SIZE:16,     // KV cache slots per page
  MAX_PAGES:257,    // max pages (≈ 4096 context)
}

Q4F16 quantization format

MLC's Q4F16_1 format packs 8 int4 values into each uint32. Scales are stored as float16 with group_size=32 — one scale per 32 weights.

// Extract nibble for element i within a u32
let nibble = (packed_u32 >> (i * 4)) & 0xF;

// Dequantize: center around 0, multiply by scale
let value = f16(i32(nibble) - 7) * scale;

Weight shapes in Q4F16 (for Phi-3-mini):

Qwen3-4B (?model=qwen3)

The engine is parameterized over a ModelSpec, and a v1 Qwen3-4B (q4f16_1) port ships alongside Phi-3. Append ?model=qwen3 to zero-tvm.html or validate.html — it works on the live site (weights stream from HuggingFace, ~2.3 GB), and node scripts/download-weights.mjs --model qwen3 primes the local dev mirror. Phi-3 stays the default; all existing URLs keep their exact behavior.

What the port exercises that Phi-3 doesn't:

Measured 2026-07-30 on an Apple M2 Max under the corrected protocol (same session, identical local weight bytes, both engines paying a full prefill on every run): Zero-TVM 59.85 tok/s total (TTFT 453 ms, decode 75.49) vs WebLLM 0.2.84's prebuilt Qwen3-4B at 45.46 tok/s total (self-reported decode 47.77) — +31.7% total, +58.0% decode. Unlike the Phi-3 headline, these figures are static dated text, not synced from bench/results.json; the machine-readable record is bench/results/qwen3-4b.json.

Qwen3.5-4B hybrid (?model=qwen35)

The third model, and the first hybrid architecture on the engine: 24 gated-DeltaNet (linear-attention) layers interleaved with 8 gated full-attention layers (attention on every 4th layer). To our knowledge this is the first hand-written-kernel int4 implementation of a gated-DeltaNet hybrid running in a browser. Append ?model=qwen35 to zero-tvm.html or validate.html; node scripts/download-weights.mjs --model qwen35 primes the local dev mirror (~2.6 GB).

What the hybrid adds over Qwen3-4B:

Measured 2026-07-30 on an Apple M2 Max under the corrected protocol (same session, identical local weight bytes, both engines paying a full prefill on every run): Zero-TVM 65.28 tok/s total (TTFT 171 ms, decode 73.30) vs WebLLM 0.2.84's prebuilt Qwen3.5-4B at 32.56 tok/s total (self-reported decode 34.32) — +100.5% total, +113.6% decode. This is the one model where first-token latency is roughly a wash rather than a loss (our 171 ms against an implied ~0.2 s from WebLLM's 175–177 tok/s prefill). Static dated text, not synced from bench/results.json; the machine-readable record is bench/results/qwen35-4b.json.

Qwen3.6-35B-A3B MoE (?model=qwen36q3 / ?model=qwen36)

The fourth model, shipped 2026-08-05, and three firsts at once: the first sparse MoE (256 experts, top-8 plus a shared expert on every layer), the first MLX-format checkpoint (affine w = s·q + b, group 64, per-tensor biases, loaded by byte range — a 5.3 GB safetensors shard is never one ArrayBuffer), and the first model here with no WebLLM build to benchmark against. ?model=qwen36q3 is the 3-bit-expert build (~16.4 GB, needs ~20 GB free RAM, ~66 tok/s on a quiet 32 GB M2 Max); ?model=qwen36 is full 4-bit (~19.5 GB, needs ~24 GB free RAM). The MoE block runs in 7 dispatches with the expert index in grid z; every layer is validated against mlx_lm's own modules (npm run test:kernels:real). Full engineering notes live in the repo: README, BENCH.md, CLAUDE.md.

Port to Phi-4-mini or Qwen3

Both are available as MLC Q4F16 packages. The Qwen3 port has since landed (see Qwen3-4B above) — these are the steps it followed, and the recipe for the next model:

1. Update PHI3 constants in compiler.ts — D, HEADS, HEAD_DIM, LAYERS, FFN, VOCAB
2. Check parameter names — fetch ndarray-cache.json and log all keys. Update weight-loader.ts candidates to match
3. Check for GQA — if KV heads ≠ Q heads (grouped-query attention), the attention shader needs a small change to repeat KV heads
4. Update chat template — each model has its own special tokens and prompt format
5. Update HuggingFace base URL — change PHI3_MODEL_BASE in weight-loader.ts

Local weight serving

Run the download script once to save all shards locally. Subsequent page loads are instant (served from localhost, no network).

node scripts/download-weights.mjs

# Downloads to: .weights-local/Phi-3-mini-4k-instruct-q4f16_1-MLC/
# Served at:    /local-weights/Phi-3-mini-4k-instruct-q4f16_1-MLC/
# Size:         ~2 GB

Debugging tips

All output is <unk>

• Weight version mismatch — ndarray-cache.json offsets don't match shard content
• Buffer aliasing — add_norm dispatched with same buffer as both @1 and @4
• Wrong rope binding order — check @0=q_out, @1=k_out, @2=v_out, @3=qkv, @4=posMap

Garbage / repetitive output

• Rope bindings are in the wrong order (this was our bug — garbage like -,unlintzegesenma)
• Wrong uniform values for a shader (K_groups, N, etc.)

WebGPU validation error about aliasing

• Same buffer bound as read_write and read in one dispatch
• Fix: use ping-pong buffers. Never bind B.residual as both @1 and @4 to add_norm

Model not loading (Weight not found)

• Log all available keys: the weight loader prints them to console on load
• MLC names differ from HuggingFace standard names (transformer.h.N.mixer.* not model.layers.N.self_attn.*)

Bugs we fixed (and how)

vs WebLLM

Head-to-head on Phi-3-mini-4k-instruct Q4F16_1, same weights, same session, same browser (Chrome 150 with WebGPU), Apple M2 Max, WebLLM 0.2.80 (the Qwen pairs below ran against 0.2.84) — npm run bench, 128-token target × 5 runs, median, 2026-07-30 corrected protocol (latest run recorded in bench/results.json). Every run pays a full prefill on both sides, and both metrics are reported: total is wall-clock throughput including prefill, decode excludes it.

The corrected protocol (2026-07-30) — and what it invalidated

The advantage grows with architecture recency

Monotonic on both metrics across the three models then shipped, all measured 2026-07-30 on the same machine (Qwen3.6-35B-A3B came later and has no WebLLM baseline to pair against — see BENCH.md):

The reading that fits: compiler stacks have had less time to tune newer architectures, so there is more headroom for a hand-written kernel set to take. That is an observation across three points, not a proven law — one machine, one browser, one day, no mechanism isolated, no control for how differently each model stresses the two engines — and the baseline is not constant across the rows (WebLLM 0.2.80 for Phi-3, 0.2.84 for both Qwen rows). Worth testing on a fourth model.

Where we lose: time to first token on short prompts

How the result flipped (22% behind → +16% total / +31% decode ahead)

An earlier head-to-head (M2 Pro, 2026-06) read 42.14 vs ~51.5 tok/s — Zero-TVM 22% behind. Both the hardware and the engine changed since, so the two Zero-TVM numbers are not a pure optimization delta; the same-session, same-machine WebLLM number is the valid comparator for the current Zero-TVM median (both in the table above). What changed in the engine:

• Correctness fixes. f32 accumulation in the fused FFN, a workgroup-barrier fix in attention, and a decode off-by-one fix — the old number understates even the old hardware.
• vec4 loads, now default. Re-declaring weight + activation buffers as array<vec4<u32>> in the int4 matmuls and qkv_fused measured +7.1% on M2 Max; opt out with ?vec4=0 / ?vec4qkv=0.
• Tiled + subgroup kernels. The 4-row tiled subgroup FFN and the _sg matmul/attention/argmax variants are the default path on Apple sg32 hardware.

Still default since 2026-07-27 (`?splitk=0` to disable): split-K attention (?splitk=N, ~+3% at short context, needs a long-context A/B); the best measured opt-in config of that era (?vec4=1&vec4qkv=1&splitk=8, 68.36 tok/s on 2026-07-25) is now simply the default path. Falsified and not shipped: FFN prologue fusion (?fuseprologue=1, −13.7% on M2 Max). Dispatch overhead was never the gap — Zero-TVM already submitted fewer dispatches per token (260 vs 342) when it was slower. Full A/B tables in BENCH.md.

The memory-bandwidth ceiling

Phi-3-mini Q4F16_1 touches ~2.09 GB of weights per decode token (the weight-shape table above, summed: 32 layers plus the LM head; the embedding contributes one row). On M2 Max's 400 GB/s memory bus that's ~5.2 ms/token, or ~191 tok/s theoretical max. Zero-TVM's measured decode-only median sits at roughly 44% of that ceiling (36% on total, which pays prefill too); WebLLM's at roughly 33% decode-only / 31% total (exact medians in the table above). Neither engine can exceed the ceiling without changing weight layout or quantization (int8 KV shaves a bit, which is why it's exposed behind a flag).