Serve 9 users at 128K where fp16 serves 2.

KV cache and VRAM are the bottleneck.

At long context, KV cache size determines how many users you can serve concurrently and what models fit on your hardware. Most teams are forced to choose: shorter context, smaller model, or more GPUs. fraQtl removes that choice.

Fewer GPUs. Longer context. Lower cost.

At long context, KV cache is what caps users-per-GPU and context length. fraQtl compresses it while keeping retrieval intact — longer context, more concurrent users, fewer GPUs, on the hardware you already have. The receipts below are the argument.

The numbers behind the pilot.

All numbers traceable to the public Hugging Face sidecar repos and the benchmark receipts — and reproducible on your own hardware.

Receipts available per lane — maturity varies by lane. The vLLM speed receipt, D1 KV receipt, and Qwen artifact are public today.

The fraQtl runtime reads its packed KV pages straight into the tensor cores — no reconstruction step. In real vLLM serving on one A100-80GB (CUDA graphs on), single-user decode runs at 95–99% of fp16 speed at 8K–32K — and 1.32× faster than fp16 at 128K; under concurrent load the memory advantage becomes a throughput advantage: 2.1× fp16 aggregate tokens/sec at 128K context per user. Two models, one kernel, zero per-model changes. Every number retrieval-verified — never “lossless.”

Each model gets a sidecar from the factory (<1h, weights untouched) and runs on the identical membrane kernel. Same command, same recipe — three architectures, one pattern: ~2.4× fp16 capacity, weight-floor parity at short context, and the win grows where context grows.

MODEL	KV CAPACITY VS FP16	VS FP8	8K DECODE	32K DECODE	128K DECODE	NEEDLE CELLS
Qwen3-4B-Instruct-2507 2025 · native 256K	2.39×	1.21×	92% fp16	97% fp16	1.34× fp16 · 95% fp8	189/189
Llama-3.1-8B-Instruct native 128K	2.35×	1.20×	96% fp16	97% fp16	1.23× fp16 · 1.00× fp8	90/90
Mistral-7B-v0.3 native 32K	2.42×	1.21×	95% fp16	99% fp16	n/a — collapses at 128K in all arms incl. fp16	126/126

Needle cells = exact-match passkey grids, 7 depths × 3 keys per context (Llama 5 × 3), all three arms. Capacity = measured KV pool tokens, 8K config. Cells are receipts; blanks are runs not done, not failures.

REAL VLLM SERVING · ONE A100-80GB

CUDA GRAPHS ON · TEMPERATURE 0 · FP8 ALWAYS SHOWN

Alone, fraQtl matches fp16 — and beats it at 128K. Under load, the advantage grows.

At one user, weights dominate and everyone is within a few percent. Add users and fp16 runs out of KV memory first, fp8 second — fraQtl keeps serving: parity with fp8 at 1 user grows to 1.4× fp8 at 6 users.

CELL	FRAQTL	FP16	FP8 KV	NEEDLES
Mistral-7B · 8K · batch 1 · decode tok/s	85.7	90.39	90.05	✓
Mistral-7B · 32K · batch 1 · decode tok/s	76.84	77.98	82.81	✓
Qwen3-4B-2507 · 128K · batch 1 · decode tok/s	69.5	52.59	72.7	✓
Mistral-7B · KV pool tokens · one A100	997,200	412,544	825,104	—
Qwen3-4B-2507 · users @128K each · bit-plane V2 (Aug 14)	9	2	5	✓ 9/9
Qwen3-4B-2507 · aggregate tok/s · bit-plane V2 at 9 users	134.1	66.6	117.6 (peak, at 4)	✓
Qwen3-4B-2507 · users @128K ctx each · V1 layout (Jul 05)	6	2	5	✓ every user
Qwen3-4B-2507 · aggregate tok/s @128K · V1 clean max (Jul 05)	140.9	66.6	105.0	✓
Qwen3-4B-2507 · 32K · 24 concurrent users · aggregate tok/s	429.7	246.0	432.3	✓ 24/24

JULY V1 LADDER · 128K/USER · EACH CONFIG AT ITS CLEAN MAX · THE PER-LAYOUT BANDWIDTH ROOF

fraQtl

140.9 tok/s · 6 users

fp8 KV

105.0 tok/s · 5 users

fp16

66.6 tok/s · 2 users

Prefill at 128K: ~3,950 tok/s — at parity with fp16, faster than fp8. Every user needle-tested at their own depth, every rung of the ladder.

RETRIEVAL VERIFICATION · THE HONESTY LAYER

We never say “lossless.” We say retrieval-verified: needle-in-a-haystack passkey grids — 7 depths × 3 keys per context, exact-match gated — run on every arm of every receipt. Qwen3-4B-2507: 189/189 cells across 8K/32K/128K. Mistral-7B-v0.3: 126/126 cells across 8K/32K. Llama-3.1-8B: 90/90 cells across 8K/32K — 405/405 total — plus the Aug 14 nine-user run: 9/9 per-user passkeys at depths spanning 0–100%. Same kernel for all three models, zero per-model kernel changes — a new model's sidecar builds in under an hour.

Framing: BF16/FP16 is the quality reference; public Q4 is the practical size baseline teams already deploy. Matched Q4_K_M bakeoff in progress — results published when locked. Method: vLLM (real paged serving), CUDA graphs on, prefix caching off, two-pass subtractive decode timing, temperature 0, exact-match passkey retrieval per arm; concurrency cells give every user a full 128K context (130,048-token prompt + 1,024-token decode) with an independent passkey at a distinct depth. fp8 = fp16 weights + fp8-E4M3 KV (FlashInfer) — the strong baseline; weights identical across all arms. fraQtl = rank-protected eigenbasis KV read fragment-native into the tensor cores, zero reconstruction. 128K receipts run with chunked prefill disabled (known perf bug under chunking, fix in progress; stock arms unaffected). A batch-routing bug that dropped needles at batch ≥2 was found 07-03, root-caused, fixed, and fully re-receipted — pre-fix numbers were never published. No 256K claims yet: capacity verified, quality gate open. Qwen3-4B KV pool: 845,136 tokens vs fp16 361,776 (fp8 709,488) — 2.3× capacity. Aug 14 nine-user cells use bit-plane V2 KV tiering (+45% KV pool vs V1); the July V1-layout receipt (6 users, 140.9 aggregate) remains the per-layout bandwidth roof — V2 trades ~5% aggregate for 3 more users. Per-user decode at 9 users is 14.9 tok/s (fp16: ~33, for its 2 users). All cells reproducible from the public Hugging Face sidecar repos (huggingface.co/fraQtl) with fraqtl_repro_receipts.py; run IDs available on request.

RECEIPT · LLAMA.CPP 128K MISTRAL-7B

Standard low-bit KV loses retrieval at long context. fraQtl doesn't.

Mistral-7B at 128K context in llama.cpp. Same model, same prompts — only the KV-cache representation changes. Q8 and Q4 reduce memory but give up needle retrieval; fraQtl D1 reaches memory below Q8 with retrieval intact.

VRAM · Single A100-80GB

QWEN 3.6 35B-A3B

FP16 OOMs at 64K. fraQtl runs 128K with 28 GB headroom.

Same model. Same hardware. 8× the context. The difference between needing 1 GPU and needing 2.

CONTEXT	FP16 BASELINE	FRAQTL	HARDWARE
16K	~71 GB	25.6 GB	Both fit on 1× A100-80GB
64K	82.9 GB → OOM	36.8 GB	FP16 needs 2 GPUs; fraQtl 1
128K	85+ GB	51.7 GB	FP16 needs 2 GPUs; fraQtl 1 · 28 GB free

FP16 64K = 82.86 GB measured (OOMs on 80 GB ceiling). 16K and 128K FP16 figures are model + KV-cache extrapolations — conservative lower bounds.
fraQtl numbers measured on a single A100-80GB with the public artifact at huggingface.co/fraQtl/Qwen3.6-35B-A3B-compressed.

Bring your model. Keep the savings.

Send us your model and a workload sample. We calibrate, deliver a compressed artifact + benchmark report + integration path for your deployment stack. If the numbers don't move, no commitment.

Free technical pilot for the first 5 qualified design partners.

Compressed models, ready to load.

Everything shipped, dated, latest first — runtime integrations, receipts, and public artifacts on HuggingFace. Reproducible numbers throughout.

2026 · AUG 14 RECEIPT

Nine 128K users on one A100 (bit-plane V2 KV) — 134.1 aggregate tok/s, 2× fp16 throughput at 4.5× its users, 9/9 per-user needles, +45% KV pool Lead receipt ↓

2026 · JUL 22 HI-FI GGUF

Gemma-4 Hi-Fi GGUFs: E2B-it + 26B-A4B-it — calibrated GGUFs with KLD receipts, in the Hi-Fi Quants collection HuggingFace →

2026 · JUL 10 RECEIPT

Third architecture receipted: Llama-3.1-8B — 2.35× fp16 KV capacity, 1.23× fp16 decode @128K, 90/90 needle grids — model matrix complete, three architectures on one unchanged kernel See the matrix ↓

2026 · JUL 07 REPRO KIT

KV sidecar repro kit public — Mistral-7B + Qwen3-4B sidecar repos + SM80 runtime wheel; fraqtl_repro_receipts.py reproduces every cell on a rented A100 HuggingFace →

2026 · JUL 05 RECEIPT

128K concurrency receipt lands (vLLM) — 6 users at 128K context each on one A100 — 140.9 vs fp16’s 66.6 aggregate tok/s (2.1×), every user’s needle retrieved Lead receipt ↓

2026 · MAY RUNTIME

llama.cpp 128K KV integration (D1) — Mistral-7B at 128K below Q8’s live VRAM with 5/5 retrieval where Q8 drops to 1/5 — the original moat receipt D1 receipt ↓

What's shipping vs what's in the lab.

Two lanes. Public means measured, reproducible, and safe to deploy today. Research means active work we're not making customer claims on yet — listed so you can see where the substrate is heading.

Measured

D1 long-context KV

Mistral-7B 128K in llama.cpp: below Q8 live VRAM, 5/5 retrieval. The llama.cpp lane receipt above.

Measured · new

vLLM full-speed KV

95–99% of fp16 decode at 2.42× KV capacity, needle intact. the lead receipt above.

Public artifact

Qwen 3.6 35B-A3B

Compressed weight artifact on Hugging Face. 128K on 1× A100, MMLU 82.24% vs 82.40% FP16.

Quality anchor

Hi-Fi calibration

FP16-reference calibration recipe behind the public artifact — quality and size, no speed claim.