TEST A · CONCURRENT USERS AT 128KMEASURED 2026-10-31users at 128K on one H100, where FP8 KV fits 9
1.31× faster per user than vLLM’s native 4-bit KV, using 3% more memory per token. Accuracy: no significant difference (mean KL, 30 documents); equivalence not shown.
| KV cache | Users that fit | Per-user decode | Accuracy vs FP8 KV |
|---|---|---|---|
| fraQtl Membrane | 17 | ≈12 tok/s at 17 users | Slightly below (mean KL 0.075 vs 0.066) |
| FP8 KV | 9 | ≈19 tok/s at 9 users | Reference |
- Accuracy slightly below FP8 KV (mean KL 0.075 vs 0.066, 30 docs, p=0.009).
- Per-user speed equals FP8 at 9 users each; at 17 users each user decodes slower (≈12 vs ≈19 tok/s).
- This is a capacity result: more users at a lower per-user speed, not more users at the same speed.
TEST A2 · SECOND MODEL · GEMMA-4-E4B
FrogNano-4B
LFM2-1.2B