Flash-Next MTP for llama.cpp

Speculative decoding, a quality-biased IQ4_NL quantization, and how we verified both — for the Qwen3.8-Flash-Next (qwen4exp) family.

fork branchnoonr48/llama.cpp · flashnext-mtp

modeljackasda211233/Qwen3.8-Flash-Next-Uncensored-IQ4_NL (Hugging Face)

The model
What we optimized for
How the quantization was prepared
Testing methodology
Quality results
Performance engineering
A measurement confound we retracted
Iterations & dead ends
What's supported
Hardware note & PCIe caveat
Acknowledgments
License

The model

Qwen3.8-Flash-Next is a large hybrid architecture ("qwen4exp" in llama.cpp): most layers use gated-delta-net (GDN) linear attention with a small number of full-attention layers using query-sparsity attention (QSA), plus a Mixture-of-Experts feed-forward with a per-layer token embedding (PLE) table. It ships with a NextN/MTP head — a 1-layer multi-token-prediction draft that enables speculative decoding.

The quantized release documented here starts from orcarouter/Qwen3.8-Flash-Next-Uncensored — an uncensored (abliterated) release of the model, Apache-2.0 — pinned to a specific verified revision. We repacked it with the MTP head split out as a sidecar draft file and produced a quality-biased IQ4_NL quantization of the backbone. The full chain (pinned revisions, tensor manifests, integrity checks) ships with the model as MODEL_PROVENANCE.json.

What we optimized for

This quant was built for long-session agentic and creative use, not for leaderboard chasing. The workloads that drove every decision:

Heavy reasoning

Long multi-step thinking traces; chain quality must survive quantization.

Character & story

Roleplay and long-form narrative: voice consistency, continuity, nuance.

Planning

Multi-phase plans with state carried across long contexts.

Games

Rules systems, simulation logic, and long interactive sessions.

Architecture / CAD

Spatial reasoning, precise geometry, structured output discipline.

3D-spatial

Object relationships, transforms, scene descriptions.

Science

Quantitative reasoning with correct units, notation, and calibration.

Vision

Image input via the stock projector at full precision, high-resolution.

Because these workloads punish subtle degradation more than benchmark prompts do, the acceptance gate was a replicated-loss criterion over a balanced suite (below), not an average score.

How the quantization was prepared

Source verification. An initial NVFP4-quantized derivative was evaluated and rejected during provenance verification. We settled on the verified raw source above and pinned its revision; every downstream artifact traces back to that pin.
Backbone repack. The 32 MTP-head tensors were removed from the backbone (1,224 backbone tensors retained) and re-exported as a sidecar draft GGUF, so the draft serves as the speculative-decoding model. Tensor order, KV sections, and the QSA compress-ratio layout were restored against the source model config — verified by name/shape/dtype checks on all tensors and value sampling at start/middle/end positions (3,360 sampled values matched).
Importance-matrix calibration. An imatrix was built from a multi-domain calibration corpus weighted toward the target workloads above (926 entries).
Quantization. IQ4_NL backbone with sensitive tensors held at higher precision. Final tensor mix: 754 IQ4_NL, 46 Q5_K, 3 Q6_K, 32 Q8_0, 388 F32, 1 F16.
Gating. Every stage wrote hash-stamped logs and manifests; the pipeline gate rechecks stamps and coverage before anything is deployable.

Testing methodology

The primary acceptance gate was a balanced evaluation with symmetric all-seed rules: three variants — this quant, the base model, and the source Q8_0 reference — ran the identical suite under identical samplers and prompts, across multiple seeds per suite (36 suite-seed runs total). A quality loss only counts if it replicates across seeds; single-seed dips are recorded as noise. Scoring uses a 10-point rubric judged blind per axis (quality, thinking-mode, capability under thinking, capability standalone).

Needle recall at 25k+ tokens with randomized codes — pass required under the deployed speculative-decoding path.
Vision probes with the stock projector (image grounding and OCR-style checks).
Perplexity on a disjoint public-domain holdout (H.G. Wells, The Time Machine — deliberately disjoint from the calibration corpus) as a secondary sanity check.
Sampling discipline. Thinking-mode settings follow the model author's recommended configuration (temperature 1.0, top-p 0.95, top-k 20); we probed alternatives before adopting them. All performance A/Bs use fresh prompts so prompt-checkpoint caching cannot contaminate cold-prefill numbers.

Quality results

How these were tested (details in methodology): three variants ran the identical suite — same prompts, same samplers, same seeds — across 36 suite-seed runs. Each response is scored on a 10-point rubric judged blind per axis; the numbers below are mean scores. Seed-to-seed spread on this suite is on the order of a few tenths of a point, so differences of that size between variants are within the margin of error — treat the three variants as statistically close, not as a ranking.

The claim we stand behind is not "beats the base model" — it is zero replicated quality losses: across every axis and seed, no loss against either reference replicated, i.e. no noticeable quality loss at less than half the file size of the Q8_0 reference (119 GB vs 188 GB). The quant doing as well as its source Q8_0 and the base is the result; the small mean differences are noise, not signal.

Mean rubric scores across the balanced suite (36 suite-seed runs, symmetric rules). Differences of this magnitude sit within the suite's seed-to-seed margin of error — read this as "the IQ4_NL tracks its references", not as a ranking. Thinking-mode probes: 10/10 for all three variants. Capability-under-thinking: 7.67 (this quant) vs 8.00 (base) — no replicated loss; the standalone-capability axis was seed-unstable for all variants (4.00 vs 5.33 means, not replicated across seeds) and is recorded as drift, not as damage.

Variant	Size	Mean score	Thinking	Zero replicated losses
This IQ4_NL	119 GB	6.33	10/10	—
Base model	~340 GB	6.00	10/10	reference
Source Q8_0 reference	188 GB	5.67	10/10	reference

Means differ by fractions of a point — within the suite's margin of error. The takeaway is parity, not superiority.

How to read this: a 10-point-rubric mean difference of a few tenths between variants is within this suite's margin of error — we do not claim the quant outperforms the base or the source. What the evaluation establishes is narrower and more useful: under identical prompts, samplers, and seeds, the IQ4_NL produced no replicated quality losses against either reference — no noticeable quality loss — at 63% of the Q8_0 reference's size and about a third of the base model's. If your use case needs a stronger guarantee, run your own suite against the references; the methodology above is the recipe.

Performance engineering

Measurement conditions, stated precisely: prefill numbers come from fresh 25k-token prompts (no cache reuse). Decode numbers are 600-token continuous generations; the draft-depth sweep specifically used a single fixed short chat prompt (≈40 tokens) — so its context stayed under ~700 tokens throughout. Decode slows as the KV cache fills, so treat all decode figures here as short-context results; deep-context decode on a 262k-context server will run lower. Sampling pinned at the model author's thinking-mode settings (temperature 1.0, top-p 0.95, top-k 20). All runs: llama.cpp layer-split across a mixed 7-GPU set (1× RTX 5090 + 3× RTX 3090 + 3× RTX 5060 Ti), 262,144-token context, fp32 KV. See the hardware note for the PCIe caveat.

MTP draft alone vs MTP combined with an n-gram draft. The combiner runs the 1-layer MTP head and an n-gram mapper together; both axes improve. Needle and vision probes pass under the combined path.

Deeper draft chains from a 1-layer head decay acceptance monotonically — depth 1 is optimal for this architecture. Bars span each depth's full observed min–max range (600-token generations on a fixed ≈40-token chat prompt: context under ~700 tokens — short-context decode; expect lower tok/s as context fills).

Serving configuration that produced these numbers (see the fork README for flags): --spec-type draft-mtp,ngram-map-k with the sidecar draft, flash attention on, and the full 262k context with fp32 KV.

A measurement confound we retracted

Our first MTP+n-gram measurement showed a +92% prefill gain. It was wrong: the benchmark reused prompts, and the server's persistent prompt-checkpoint cache answered most of the "cold" prefill from disk. A reviewer wave caught the inconsistency, and a controlled A/B with fresh prompts on both sides re-baselined the real effect: +3% prefill, +8% decode — the numbers published above. We kept the prompt-checkpoint feature (it is a genuine win for repeated real-world prompts) but banned prompt reuse from every cold-prefill benchmark after that. If you benchmark this stack: always use fresh prompts.

Iterations & dead ends

Step	What we tried	Outcome
Source selection	NVFP4-quantized derivative	Rejected during provenance verification; switched to the pinned raw source.
Draft layout	MTP head in-file vs sidecar	Sidecar adopted (backbone loads on stock loaders; draft used only for speculative decoding). Unsloth draft-head-only layout also supported in the fork.
Quant variants	Recipe A/B with different sensitive-tensor sets	Variant B (more tensors held at higher precision) won the balanced eval; final mix published above.
GPU topology	12 → 9 → 7 GPU layer-split	Smaller fast-mix beat larger pools (9 GPU: 51.2 t/s; 7 GPU: 54.4 t/s baseline) — inter-GPU hops, not compute, were the wall.
MTP depth	n1 / n2 / n3	n1 optimal; deeper chains monotonically worse (chart above).
Spec combiners	MTP alone vs +n-gram	Combined wins both axes (+8% decode, +3% prefill, fresh-prompt A/B).
Tensor-split serving	Splitting individual layers across GPUs for prefill throughput	Research branch, not shipped. A corruption bug in the split path was root-caused (full-attention layers on the composite device) and fixed, but the end-to-end path is still gated behind a needle-recall pass; not part of this release.

What's supported

The fork is upstream llama.cpp plus a scoped delta — every model and quant type stock llama.cpp runs, the fork runs unchanged.
No custom kernel required. The model runs in standard llama.cpp — qwen4exp architecture support is upstream, so any recent stock build loads the backbone. (Unlike our earlier Qwen3.6 RYS GGUFs, which required a custom loader.) The fork branch is optional: it adds MTP speculative decoding and the performance paths above.
Flash-Next (qwen4exp) models of any provenance get MTP speculative decoding when run on the fork: draft weights can come from in-file MTP tensors, a sidecar draft GGUF (our layout), or draft-head-only GGUFs in the unsloth layout.
Any standard GGUF quant of Flash-Next works (Q8_0, Q4_K_M, IQ family, …); ours is one specific quality-biased IQ4_NL build.
Server improvements apply to all models: persistent prompt-checkpoint save/restore and speculative-decoding rollback checkpointing.
What the model repo provides: backbone GGUF (119 GB), MTP draft sidecar (4 GB), F16 vision projector (1 GB), and the provenance manifest.

Hardware note & PCIe caveat

All performance numbers on this page come from a personal mixed-GPU workstation (RTX 5090 + RTX 3090s + RTX 5060 Tis). Important caveat: its PCIe links run at Gen3 with 4 lanes. Multi-GPU layer-split inference moves activations between cards every layer, so interconnect bandwidth is a first-order term for decode speed here. On a Gen4 or Gen5 system (or fewer, faster cards with NVLink), you can reasonably expect better numbers than ours — treat our speeds as a conservative floor, not a ceiling.

Acknowledgments

Alibaba's Qwen team — the Flash-Next architecture and base model.
orcarouter — the uncensored release this quantization is derived from, and the direct source of the verified weights.
ggml-org/llama.cpp — the runtime everything here builds on, including upstream qwen4exp architecture support.
flash-linear-attention — the GDN l2norm reference the normalization fix followed.
unsloth — the draft-head-only GGUF layout the loader also accepts.
H.G. Wells (1866–1946) — the public-domain holdout corpus.

License

Model weights chain: Apache-2.0 (inherited from the base release; see MODEL_PROVENANCE.json in the model repo). Fork code: MIT, inherited from llama.cpp.