Inference engine for DFM Mimir v1 (danish-foundation-models/DFM-Mimir, HRM-Text
architecture, Danish Foundation Models / SDU) on consumer Ampere GPUs — the 4× RTX 3060
12 GB box that also runs zaphira.
Mimir is 1.8B parameters that behave like an 8B-class inference problem: two 16-layer modules applied 8 times per token (3.9B parameter-reads), 128 virtual layers of KV cache, and a PrefixLM mask that no Ampere attention kernel ships.
Status: M0–M6 complete.
frigg servespeaks the OpenAI API and works against the official Python SDK. int8 decode runs at 46.7 tok/s single stream (3.8× HF eager, measured at a fixed 65 °C — see Finding 12); GSM8K-200 scores 92.0% on bf16 and 91.0-91.5% on int8 against the paper's 89.9 — the range is the elementwise fusion, which is one question either way (Finding 7b). Four replicas serve 405.9 tok/s aggregate over 32 streams with zero errors; one replica does 141 tok/s over 8 streams, up from 22.8 when M4 first measured it — batched CUDA graphs and a batched W8A16 kernel, neither of which was worth much without the other (Finding 8). M0's job was to turn every assumption in the spec into a measured fact before writing a single kernel; this README is the record of what M0 and M1 found. Everything below was measured on this machine unless it says derived.
- The machine
- The model, as verified
- Finding 1:
token_type_idsis the whole ballgame - Finding 2: vLLM does run on Ampere
- Finding 3: it is launch-bound and bandwidth-bound
- Finding 4: "bit-identical" is not a well-defined target
- Finding 5: self-speculative decoding is dead on arrival
- Finding 6: the tokenizer depends on how you name the model
- Finding 7: two real speedups that did not pay for themselves
- Finding 7b: the same fusion, re-measured per build
- Finding 8: quantization's win inverts under load
- Finding 9: the recurrence schedule is not a dial
- Finding 10: the server was silently ignoring
temperature - Finding 11: the four-GPU flag had never worked
- Finding 12: the benchmark numbers are a function of GPU temperature
- Quickstart
- Limitations
- How it works
- Where the spec was wrong
- Baselines
- Reproducing everything
- Repository layout
- Licences
4× RTX 3060 12 GB (GA106, SM 8.6, 3 MB L2), Threadripper PRO 3945WX, 128 GB DDR4, Fedora, NVIDIA driver 595.80. Toolchain: Python 3.12 / uv, torch 2.11.0+cu128, triton 3.6.0, transformers 5.16.1.
uv run frigg bench vram, all four cards:
| GPU | D2D copy GB/s | fused read GB/s | GEMV GB/s |
|---|---|---|---|
| 0 | 333.1 | 341.5 | 338.7 |
| 1 | 333.1 | 341.4 | 338.7 |
| 2 | 333.9 | 342.3 | 339.7 |
| 3 | 333.0 | 341.4 | 338.7 |
The cards are within 0.3% of each other. The GEMV number (a [1,1536] × [1536,262144]
bf16 matrix-vector — Mimir's actual lm_head shape) is 94% of the 360 GB/s nominal
figure, and it is the ceiling that matters: at batch 1 every decode GEMM degenerates
to this shape. All ceilings below use the slowest card's 338.7 GB/s, not the nominal 360.
Bytes per decoded token = core weights × 8 runs + lm_head + KV read. Derived from the
measured bandwidth, not measured end to end.
| Build | @512 ctx | @2048 | @4096 |
|---|---|---|---|
| bf16 weights, bf16 KV | 9.06 GB → 37.4 tok/s | 10.27 GB → 33.0 | 11.88 GB → 28.5 |
| int8 weights, int8 KV | 4.53 GB → 74.7 | 5.15 GB → 65.8 | 5.96 GB → 56.8 |
| int4 core + int8 vocab, int8 KV | 2.57 GB → 131.8 | 3.18 GB → 106.4 | 4.00 GB → 84.6 |
Frigg's engineering target is ≥ 80% of these.
Every number here was read from the downloaded artifacts in M0, not from the paper.
Full detail and provenance in docs/MODEL.md; the VERIFY log in §10
is complete.
| Revision read | 2844f0178e695d7d9ce182cb660671fd34c76ce5, lastModified 2026-08-20 |
hidden_size / heads / head_dim |
1,536 / 12 / 128 — MHA, not GQA (num_key_value_heads: 12) |
| Layers | num_hidden_layers: 16 per module; 16 H + 16 L = 32 blocks |
H_cycles / L_cycles |
2 / 3 → 8 module runs per token (L L L H L L L H) |
| Virtual attention layers | 128 = 16 × 2 × (3+1). The config class rewrites num_hidden_layers 16 → 128 on load |
intermediate_size |
4,096, SwiGLU |
| Norm | RMSNorm, no learnable weight anywhere, eps 1e-6, pre-norm + per-module final norm |
| RoPE | theta 10,000, rope_type: default, no scaling → 4,096 is a hard wall |
embedding_scale |
39.191835884530846 — bit-exactly sqrt(1536), and exactly 1/initializer_range |
| Vocab | 262,144 (Gemma-4), untied lm_head. The two vocab matrices are 45% of all params |
| Checkpoint | packed layout, 131 tensors, all BF16, 3,573,566,200 bytes (accounting exact to the byte) |
z_L_init |
present as model.z_L_init [1536], stored bf16. No zH_init — z_H starts from the scaled embeddings |
| Zero-count check | 0 norm tensors, 0 bias tensors |
| Parameters | 1,786,775,040 = 981,467,136 core + 2 × 402,653,184 vocab + 1,536 |
| Token | id | Token | id | |
|---|---|---|---|---|
<pad> |
0 | <|tool> / <tool|> |
46 / 47 | |
<eos> |
1 | <|tool_call> / <tool_call|> |
48 / 49 | |
<bos> |
2 | <|tool_response> / <tool_response|> |
50 / 51 | |
<unk> |
3 | <|"|> (string delimiter) |
52 | |
<|turn> |
105 | \n / \n\n |
107 / 108 | |
<turn|> (generation EOS) |
106 |
vocab_size == len(tokenizer) == 262,144 (no added tokens). fix_mistral_regex: true.
model_max_length is 1e30, so the 4,096 limit is entirely Frigg's to enforce.
The template emits <bos> itself — never add it again. Assistant renders as model:
<bos><|turn>system
Du er Mimir, en hjælpsom dansk assistent.<turn|>
<|turn>user
Hvad er 2+2?<turn|>
<|turn>model
4<turn|>
<|turn>user
Og gange 3?<turn|>
<|turn>model
Tools render inside the system turn, with JSON types upper-cased and strings delimited
by the <|"|> token rather than ASCII quotes:
<|tool>declaration:get_weather{description:<|"|>Get the current weather in a city<|"|>,
parameters:{properties:{city:{description:<|"|>City name<|"|>,type:<|"|>STRING<|"|>}},
required:[<|"|>city<|"|>],type:<|"|>OBJECT<|"|>}}<tool|>
This is the finding the rest of the project rests on.
HF's HrmTextModel.forward turns token_type_ids into a per-position block id and
ORs a bidirectional overlay onto the causal mask:
block_sequence_ids = torch.where(token_type_ids == 1, 0, -1)
mask = causal OR blockwise_overlay(block_sequence_ids)
# overlay unmasks (q, kv) iff q_group == kv_group and q_group >= 0Four consequences, all covered by tests/test_prefixlm_mask.py (CPU, runs in CI):
- Only the literal value
1marks a prefix token.0,2,-1and omission are all plain causal — silently, with no error and no warning. The check is not "did I passtoken_type_ids", it is "are they equal to 1". onesover the whole input → fully bidirectional. At prefill the input is the prompt, so this is exactly the PrefixLM prefill mask.oneson the first P positions only → the training-time mask: a bidirectional P×P block plus causal response rows. So the multi-token verification mask is expressible through this interface.- It is honoured only on the first forward (
past_key_values is None or not initialized), sogenerate()takes it once and ignores it during decode — which is the correct semantics, not a bug.
24 golden prompts × 3 variants, bf16, greedy, GPU 0:
| Comparison | Result |
|---|---|
ones vs causal — greedy output differs |
19 / 24 prompts (11 diverge at the first generated token) |
ones vs causal — same argmax at the last prompt row |
15 / 24 |
zeros vs omitted — same argmax |
23 / 24 (they are the same mask) |
zeros vs omitted — identical fp32 logit bytes |
0 / 24 (see Finding 4) |
generate()'s first token == uncached prefill argmax |
72 / 72 |
Running Mimir without the prefix mask is not a subtle quality regression — it changes
the answer on ~79% of prompts. This is the single most likely way to build a fast engine
that is confidently wrong, and it is why tests/golden/hf_reference.json records all
three variants rather than just the right one.
M6 put a price on it (docs/RESEARCH.md E4c). Same weights, same tokenizer, same
sampling, same problems — only the prompt's attention mask differs:
| prompt mask | GSM8K (60 items) | invalid answers |
|---|---|---|
| bidirectional, as trained | 88.3% | 0 |
causal, i.e. ignoring prefix_lm |
50.0% | 4 |
38.3 points — 43% of the model's accuracy — and it starts emitting answers with no parseable number in them at all. No error, no warning, fluent Danish either way.
The spec listed this as an open question (Q4) and assumed the answer was probably no, since the paper's evaluations ran on vLLM + FlashAttention 4 (Hopper/Blackwell).
It runs. vLLM 0.28.0, HrmTextForCausalLM, FLASH_ATTN backend, CUDA graphs
captured, 33.0 tok/s single-stream greedy — 2.7× HF eager. Three things had to
be right, each of which looks like "unsupported" if you stop at the first traceback:
VLLM_USE_FLASHINFER_SAMPLER=0is mandatory here. FlashInfer JIT-compiles its sampling kernels at startup using the system/usr/local/cudanvcc (CUDA 12.6.3), which refuses this host's gcc 16.2.1. This is unrelated to the paper's note that the FlashInfer attention backend was unusable for Mimir.ninjamust also be on PATH.- Default memory settings OOM in the sampler, not the model. vLLM upcasts logits to
fp32; at 262,144 vocab a 256-row warm-up batch is a 256 MiB allocation on an already
85%-full card.
max_num_seqs=8,gpu_memory_utilization=0.75fixes it. This is the "vocab matrices are 45% of params" problem biting a production engine. - KV is the binding constraint. vLLM reports a 5,024-token GPU KV cache and "maximum concurrency for 4,096 tokens per request: 1.23x". That is the 128-virtual-layer tax measured on a real engine: ~9.5 GB of the card buys barely more than one full-context request. It is the strongest quantitative argument for Frigg's int8 KV work in M3.
Two independent confirmations of Finding 1 fell out of this:
- vLLM's greedy output matches HF-with-
token_type_ids=1on 21/24 prompts and 94.3% of decode steps, but HF-without on only 5/24. Two separate implementations agree on the prefix mask. (The 3 divergences are late-position FA2-vs-SDPA drift.) - vLLM auto-disables chunked prefill and prefix caching, reason: "model has non-causal attention layers" — independently confirming that a PrefixLM prefix cannot be cached or chunked across turns.
One thing vLLM cannot do: its PrefillPrefixLMAttention gets bidirectionality by reusing
the encoder-only wrapper (causal=False on every metadata build) with
attn_type=DECODER. That makes the whole prefill bidirectional and cannot express a
partial-prefix mask — so it cannot do multi-token verification. Frigg's
token_type_ids-shaped mask can.
So why Frigg? Not "because nothing else runs" — that premise is now dead. Because vLLM's 33 tok/s is ~26% of the derived bf16 ceiling, it cannot do int4 or int8 KV for this model, and it can hold 1.23 full-context requests per card. Those are the gaps.
torch.profiler over one cached decode step of the HF oracle counts 5,454 CUDA
kernels per decoded token. The spec estimated ~1,700; the real figure is 3.2× that.
Roughly 256 of the excess are CatArrayBatchedCopy calls from HF's DynamicCache
growing K and V once per virtual layer per step — a class of kernel a static
preallocated cache removes outright.
At the measured 5.0 µs per launch:
| ms/token | launch-bound ceiling | |
|---|---|---|
| eager, 5,454 kernels | 27.5 | 36.4 tok/s |
| one CUDA graph replay | 5.0 | 198.5 tok/s |
The eager launch ceiling (36.4) sits essentially on top of the bf16 bandwidth ceiling (37.4 @512 ctx). Eager bf16 decode is launch-bound and bandwidth-bound simultaneously, so neither number alone explains anything.
The scheduling consequence is sharp: quantization before CUDA graphs would measure nothing. An int8 build's 74.7 tok/s ceiling and an int4 build's 131.8 tok/s ceiling are both far above what eager PyTorch can issue. Any int8 number benchmarked without graphs would be a measurement of the launch path, not of the quantization. This independently confirms the roadmap's M2-before-M3 ordering.
For reference, HF eager actually achieves 12.20 tok/s — a third of its own launch ceiling — because those 5,454 kernels also carry Python-side dispatch. Summed GPU kernel time for one decode step is 38.3 ms against ~82 ms of wall time, so more than half of HF's decode time is CPU gap, not GPU work.
The spec's NF1 asks for torch.equal on logits against HF. M0 found that the oracle is
not bit-identical to itself, twice over:
Same mask, different representation. Passing token_type_ids of all zeros and
omitting it entirely produce the same semantic mask (plain causal), but the fp32 logit
bytes differ on 24 of 24 prompts. Omission makes create_causal_mask return None
so SDPA takes its is_causal=True fast path; zeros materialise a dense additive mask and
take a different kernel. On 3 of 24 prompts the difference flips a near-tie argmax.
Same math, different GEMM shape. lm_head applied to one sliced row is not
bit-identical to lm_head applied to a batch containing that row: cuBLAS picks a
different kernel and bf16 logits differ by up to 0.031 absolute.
So NF1 is only meaningful against one pinned SDPA call shape and one pinned row count. Concretely, for M1/M2:
- The exact path must reproduce the oracle's call, not just its math: same mask dtype,
same mask shape, same
attn_implementation. frigg paritymust compare a decode-row logit against a decode-row logit, never against a row sliced out of a prefill batch, or it will report a spurious mismatch.- The golden digests in
tests/golden/hf_reference.jsonare valid only at the row count that produced them (1 row, from a full prefill forward). - M2's graph-vs-eager test must keep the
lm_headbatch shape identical on both sides.
A related trap: a standalone AutoConfig leaves _attn_implementation unset, and
create_causal_mask then returns None for every input, silently. Pin it to sdpa
when building masks outside a loaded model.
docs/RESEARCH.md E2 proposed cycle-split self-speculative decoding:
the first H-cycle (L0 L1 L2 H0) is a complete sub-model, so lm_head on z_H after
cycle 1 gives draft logits at half the weight traffic. The pre-test measures the
acceptance ceiling before anything is built — a forward hook on the HF oracle, one
teacher-forced pass per golden prompt under the real PrefixLM mask, 1,322 response
positions.
| Metric | Measured | Needed |
|---|---|---|
| Greedy agreement, cycle-1 draft vs final | 55.1% | ≥ 70% to be viable at all |
| Top-5 overlap | 34.4% | — |
| Final greedy token inside draft's top-5 | 73.0% | — |
| E[accepted] per block at k = 4 | 1.12 | ≥ 2.5 (the go/no-go rule) |
NO-GO. 55.1% is below the 70% floor and well below the ~75% break-even the cost model
gives for k=4. At 1.12 expected accepted tokens per block the scheme would be slower
than plain decode. The cheaper (1,1) draft variant is strictly further from the final
state, so it cannot rescue this. E2 is a published negative result and is out of the M6
plan; --spec-decode self will not be implemented.
The more interesting number: mean ‖z_H⁽²⁾ − z_H⁽¹⁾‖ / ‖z_H⁽²⁾‖ = 1.16
(1.11–1.27 by prompt). The second H-cycle moves the high-level state by more than the
state's own magnitude. The two cycles are not successive refinements of one answer —
they compute substantially different states. That answers experiment E4e for free, and
makes a strong prediction for E1 (recurrence scaling): reducing H_cycles should degrade
sharply rather than gracefully, and the schedule is not a smooth compute dial downward.
E1 is now the more valuable M6 experiment and should be run first.
Found in M1 while chasing a decode mismatch that turned out not to be a decode bug at
all. tokenizer_config.json sets fix_mistral_regex: true, which asks transformers to
swap the on-disk pre-tokenizer for a corrected regex Split. Whether it does so depends
on how you name the model:
AutoTokenizer.from_pretrained("danish-foundation-models/DFM-Mimir")
# pre_tokenizer = Split(" ", MergedWithPrevious) UNFIXED
AutoTokenizer.from_pretrained("/path/to/that/exact/snapshot")
# pre_tokenizer = Sequence([Split(<mistral regex>, Isolated),
# Split(" ", MergedWithPrevious)]) fixedSame files, same TokenizersBackend, both reporting
init_kwargs["fix_mistral_regex"] == True, no warning in either direction. Passing
fix_mistral_regex=True explicitly does not change the hub-id result — the flag is only
read inside a branch that path has already skipped. The on-disk tokenizer.json holds
the unfixed pre-tokenizer, so the fix is a load-time patch, not a file.
The corrected form is the intended one, on the DFM team's own authority: their fork's
notes say "The export declares fix_mistral_regex=true. vLLM applies Transformers'
corrected pre-tokenizer", and their converter persists the flag deliberately after
finding that "loading with and without the flag produced different token IDs".
It is not cosmetic — 14 of the 24 golden prompts tokenize differently, +33 tokens overall:
"Den danske sommer er kort, men"
fixed (correct) : ... '▁kort' ',' '▁' 'men' 9 tokens
hub id (wrong) : ... '▁kort' ',' '▁men' 8 tokens
An unfixed tokenizer feeds the model out-of-distribution ids on most Danish text, and nothing anywhere says so. This is the same shape of failure as Finding 1: a default that is silently wrong, in a component everybody assumes is settled.
Frigg loads through frigg/tokenizer.py, which applies the patch from any path and then
verifies the ids against known-good probes rather than trusting the loader. The M0
golden files were regenerated, because they had been produced through a hub-id load. The
M0 parity conclusions are unaffected — both sides of every comparison used the same
ids, so token_type_ids, the mask semantics, the eager tok/s and the vLLM agreement all
still stand — but the recorded prompt ids and continuations were for the wrong
tokenization.
M2 produced three optimizations. One was kept, two were measured and shelved — and the reason is the same each time: a kernel-level win is not an end-to-end win, and quality has to be measured on the same run as speed.
| optimization | kernel-level | end to end | greedy agreement | kept? |
|---|---|---|---|---|
| CUDA-graph capture | — | 2.08x | 100.00% | yes |
| Fast prefill (drop the all-true mask) | 2.0x on attention | 1.24x | 93.15% | no |
| Elementwise fusion (RoPE/SwiGLU/gate) | ~5x fewer launches | 1.06x | 97.39% | opt-in |
Fast prefill. In an ordinary prefill the whole call is prefix, so the PrefixLM mask is
entirely True — passing no mask is mathematically identical and lets SDPA use its flash
kernel instead of reading a 16.8 MB boolean mask. That is a clean 2x per virtual layer at
P = 4096. But attention is only part of prefill (the linears dominate), so 2x becomes
1.24x; and the 0.3% per-layer numerical difference compounds over 128 virtual layers into
a 1.9% logit error that costs 7% of decode steps. Half a second of prefill is not
worth that.
Elementwise fusion. The decode profile showed 3,643 elementwise kernels costing 17% of the step, RoPE worst at ~1,536 launches per token. Three triton kernels recover about a fifth of it — +6.4% throughput — for 2.6% of greedy steps, because they accumulate in fp32 where the torch chain stays in bf16. Better trade, still a trade, so it is a flag rather than a default. Worth revisiting in M3, where quantization error will dwarf it.
Neither is a failure of the optimization; both are a failure of the assumption that a faster kernel is a faster engine. Measuring the two together is what caught it.
Finding 7 shelved the elementwise fusion on a bf16 measurement. Re-run once the batched kernel, batched graphs and bucket selection had changed what fraction of a step is elementwise — and, this time, on every build rather than one:
| build | fused | B1 tok/s | B8 tok/s | free-running agreement |
|---|---|---|---|---|
| bf16 | off | 27.21 | 158.5 | 100.00% (24/24) |
| bf16 | on | 28.85 | 165.0 | 99.28% (23/24) |
| int8 | off | 42.55 | 150.8 | 91.14% (20/24) |
| int8 | on | 48.11 | 169.9 | 97.64% (22/24) |
| int4 | off | 30.99 | 96.0 | 74.37% (12/24) |
| int4 | on | 32.51 | 99.2 | 74.22% (12/24) |
Finding 7 was right, for the build it measured. bf16 reproduces its trade exactly (twice, to the decimal): ~6% faster, and the oracle stops being bit-exact. On a quantized build the trade is different — int8 gains 13% and 6.5 points of agreement — because the fused kernels carry an fp32 accumulator where the unfused chain rounds through bf16. That rounding is noise the bf16 path does not want and the quantized path benefits from.
So fusion is now on by default for quantized builds and off for bf16, which is what
is_quantized decides. The exact path stays exact.
The token-agreement gain did not show up in task accuracy. GSM8K-200 on int8, run back to back:
| accuracy | wall | |
|---|---|---|
| unfused | 91.5% (183/200) | 18.5 min |
| fused | 91.0% (182/200) | 13.5 min |
One question, on a 200-item set whose standard error is about 2 points — indistinguishable. Free-running token agreement against the bf16 oracle moved 6.5 points and GSM8K moved by less than noise in the other direction. Agreement with an oracle's tokens is a much weaker proxy for quality than it looks, which is worth remembering given how often this project reaches for it. The fusion is kept for the 13% and the 27% shorter eval, not for a quality claim it cannot support.
The load test contradicted an assumption, and the build matrix then found the real shape of it — which is not the one I first wrote down.
Decode, aggregate tok/s at 512 context (frigg bench matrix, 2026-08-30):
| build | batch 1 | batch 4 | batch 8 |
|---|---|---|---|
| bf16 | 26.6 | 47.5 | 95.1 |
| int8 | 41.0 | 32.9 | 65.3 |
At batch 1 int8 wins by 1.5×. At batch 4 that reverses: bf16 is faster than int8, and it stays ahead at batch 8. So for a loaded server today, bf16 weights are the better choice, and the int8 checkpoint is for single-stream work.
The cause is specific, and it is not "batching is hard". bf16 goes through cuBLAS, which reads each weight matrix once for the whole batch — exactly what batching is for. The W8A16 path had no such kernel, and neither of its two routes amortized: looping the gemv reads the weight once per row, so aggregate stays flat; dequant + cuBLAS reads it once but writes 3.57 GB of bf16 to do so.
Chasing this also found a plain defect: GEMV_MAX_ROWS was defined and never used, so
every batched step dequantized all 128 matrices plus the 805 MB lm_head — about 9 GB
of needless traffic, 25 times a second.
Superseded. This section previously read: "a proper batched W8A16 kernel is the single highest-value piece of work left", and that a first
tl.dotattempt had been thrown away at 3× slower. The second half is still true. The prediction was wrong, and the way it was wrong is the interesting part.
frigg/kernels/w8a16_batched.py now exists: one tl.dot program per (row-tile, N-tile,
K-split), BLOCK_N=128 with split-K reducing through a partials buffer rather than
atomics, so a captured graph replays bit-identically. The first attempt failed because it
inherited the gemv's tile_for (BLOCK_N 16–32) — far too small for the tensor cores on
1,536-wide outputs. Sizing the tile for tl.dot instead of for the gemv is the whole fix.
Its cost is nearly constant in M, which is exactly what was missing (128 matrices +
lm_head, GPU 0, isolated):
| rows | batched | loop the gemv | dequant + cuBLAS |
|---|---|---|---|
| 1 | 26.5 ms | 19.5 ms | 69.2 ms |
| 2 | 25.8 | 37.3 | 68.4 |
| 4 | 25.9 | 72.0 | 68.5 |
| 8 | 26.7 | 141.9 | 68.7 |
| 16 | 27.5 | 281.8 | 69.0 |
It is also numerically free: batched greedy output is identical to single-stream at batch 2, 4 and 8 (39/39, 122/122, 236/236 tokens), identical with the kernel disabled, and 8 identical prompts in 8 slots produce 8 identical rows.
And in a real batch-4 decode step it does what it promises. torch.profiler, GPU time:
| with kernel | without | |
|---|---|---|
| total GPU per step | 43.3 ms | 86.2 ms |
the 128 matmuls + lm_head |
27.1 ms (62%) | 69.9 ms (81%) |
decode attention (_stage1) |
7.25 ms | 7.21 ms |
The decode step now does half the GPU work. Throughput moved 5%. frigg bench matrix
puts int8 batch 4 at 31.2 → 32.9 tok/s, and the server at 8 slots at 42.3 → 44.0 tok/s.
The reason is in engine.py:292: GraphedDecoder raises NotImplementedError for
batch != 1. Only single-stream decode is CUDA-graph captured. Every batch > 1 step —
the entire reason slots exist — runs eager, at ~1,700 launches per token. A batch-4 step
takes 121 ms of wall clock to do 43.3 ms of GPU work: 78 ms of pure launch gap. Halving
GPU work inside a step that is 64% launch overhead buys almost nothing. Finding 3 said
this model is launch-bound before it is bandwidth-bound; that is still the binding
constraint at batch > 1, and it is the reason the batch-1 graphed path (38.9 tok/s) beats
the batch-4 eager one per stream.
So the corrected verdict: the batched kernel was worth writing — it is 2.7× on its own terms, it removes ~9 GB/step of dequant traffic, and it is a prerequisite — but it was not the highest-value work. Batched CUDA-graph capture is, and until it lands, int8 under load stays behind bf16 and the kernel's 2× sits unclaimed.
This is the fifth time the same root cause has bitten, in a new direction: small matrices on this model defeat optimizations that assume bandwidth is the constraint (Findings 3, 7, 8). The new wrinkle is that it also defeats measurement — the isolated microbenchmark predicted 146 tok/s aggregate at batch 4 and delivered 32.9. Amdahl's law was the missing term, and only a profile of the real step, not the kernel, showed it.
GraphedDecoder now captures any batch, so §8b's diagnosis could be tested directly.
The cache's cursors were already per-slot device tensors, so the body barely changed;
what batching adds is that a replay steps every slot and the host can no longer hold
one back, so the cursors are clamped inside the captured region. Without that, a short
slot runs on while a long one hits the wall, write_pos walks past max_seq, and
write_decode's index_copy_ writes outside the slab — corrupting another slot's rows
silently rather than raising.
The full 2×2 at int8, 512 context, batch 4 (frigg bench matrix):
| eager | CUDA graph | |
|---|---|---|
| dequant + cuBLAS | 31.2 tok/s | 48.7 (1.56×) |
| batched W8A16 | 32.9 (1.05×) | 85.6 (2.74×) |
Neither change alone was worth much; together they are worth 2.74×, which is more than their product (1.05 × 1.56 = 1.64). They are not independent — each one's benefit is capped by the other's bottleneck. Removing launch overhead does nothing while the GPU is busy dequantizing 3.57 GB per step; halving GPU work does nothing while the step is 64% launch gap. The isolated microbenchmark that said the kernel was 2.7× was right about the kernel and useless as a prediction of throughput, and the profile that explained it only became actionable once the other half was fixed too.
Decode, aggregate tok/s at 512 context, everything on:
| build | batch 1 | batch 4 | batch 8 |
|---|---|---|---|
| bf16 | 26.2 | 92.8 | 164.2 |
| int8 | 39.1 | 85.6 | 146.7 |
| int4 | 26.4 | 48.4 | 89.8 |
bf16 still wins at batch > 1, so the headline of this finding survives: int8 is a single-stream win (1.5×) and a batched loss. The reason has changed, though. It is no longer that the W8A16 path fails to amortize — it does now — but that at batch 4–8 the step is compute-bound, where cuBLAS on the tensor cores beats a triton int8 kernel that has to dequantize on the fly. Bandwidth is what int8 buys, and bandwidth stops being the constraint as the batch grows.
The server, one replica, 8 slots, int8, 180 s:
| streams | decode | requests | errors | aggregate | TTFT p50 / p95 |
|---|---|---|---|---|---|
| 8 | eager | 100 | 0 | 44.0 tok/s | 0.31 s / 1.19 s |
| 8 | graphed | 255 | 0 | 119.5 tok/s | 0.20 s / 0.79 s |
| 16 | graphed | 262 | 0 | 115.2 tok/s | 5.77 s / 7.93 s |
2.7× aggregate, and 2.5× the requests served. Graphed completions are byte-identical
to --no-graphs ones, and tests/test_graph_vs_eager.py checks replay-equals-eager and
replay-equals-single-stream at batch 2, 4 and 8. The 16-stream row shows the new ceiling
is ~117 tok/s at 8 slots: past the slot count, more clients still only buy latency.
Taken together the server went from 22.8 tok/s (Finding 8 as first written) to 119.5, a 5.2× improvement, from three changes of which the one predicted to matter most mattered least on its own.
The 4-slot saturation reported below was blamed on the kernel. It was not the kernel. 8 streams, 180 s, one replica, int8:
| slots | max tokens | batched kernel | requests | errors | aggregate | TTFT p50 / p95 |
|---|---|---|---|---|---|---|
| 4 | 96 | no (original) | 64 | 0 | 22.8 tok/s | 12.3 s / 15.9 s |
| 4 | 96 | yes | 68 | 0 | 23.0 tok/s | 11.8 s / 15.5 s |
| 8 | 128 | no | 97 | 0 | 42.3 tok/s | 0.31 s / 1.17 s |
| 8 | 128 | yes | 100 | 0 | 44.7 tok/s | 0.30 s / 1.18 s |
Three separate 8-slot runs with the kernel gave 44.0, 44.3 and 44.7 tok/s, so its ~4%
edge over the 42.3 control is real but only a few times the run-to-run spread — read it
as "a few percent", not a precise figure. The slot change is not close: with 4 slots
serving 8 clients, half the wall clock was queueing, not decoding — TTFT p50 of 12.3 s
says so plainly. Doubling the slots nearly doubled aggregate throughput and cut
TTFT p50 by 40×, and it would have done so without the kernel. --kv-slots 4 was simply
the wrong default for 8 clients on a card with room for 8.
HRM's premise is that recurrence decouples parameters from compute, which invites the obvious question: can you turn the schedule up at inference for more quality, or down for speed? Nobody had measured it for Mimir. The answer is no, in both directions, and the failure is not graceful.
Held-out NLL, teacher-forced, against the trained (2, 3):
| H | L | runs | NLL | ΔNLL | tok/s |
|---|---|---|---|---|---|
| 1 | 1 | 2 | 7.102 | +6.867 | 48.7 |
| 1 | 3 | 4 | 6.244 | +6.009 | 26.1 |
| 2 | 1 | 4 | 0.451 | +0.217 | 26.1 |
| 2 | 2 | 6 | 0.295 | +0.060 | 17.8 |
| 2 | 3 | 8 | 0.235 | — | 13.5 |
| 2 | 4 | 10 | 0.253 | +0.018 | 10.9 |
| 2 | 6 | 14 | 0.254 | +0.019 | 7.8 |
| 3 | 3 | 12 | 1.767 | +1.532 | 9.0 |
| 4 | 3 | 16 | 5.149 | +4.914 | 6.7 |
H_cycles is an architectural constant, not a knob. One H-cycle destroys the model
(6.2–7.1 nats, and no amount of L compensates). Three is worse than halving L. Four is
nearly as bad as one. Only 2 works, and not by a little.
L_cycles is a shallow dial whose optimum is exactly the trained value. Spending 75%
more compute (L=6: 14 runs against 8) makes it slightly worse. There is no test-time
scaling to be had.
This confirms a prediction M0 made before any of the engine existed. The E2 pre-test measured ‖z_H⁽²⁾ − z_H⁽¹⁾‖ / ‖z_H⁽²⁾‖ = 1.16 — the second H-cycle moves the high-level state by more than the state's own magnitude — and concluded the cycles are not successive refinements but a fixed two-stage computation. A two-stage computation breaks if you run one stage and breaks if you run three. It does.
The practical reading: H_cycles and L_cycles belong to the checkpoint, not to runtime
tuning. An engine that defaults them differently produces a model that is fluent and
wrong — the same failure mode as
token_type_ids and
fix_mistral_regex. That
is now three separate silent-default traps in one 1.8B model.
frigg serve accepted temperature, top_p, seed, n, stop, frequency_penalty
and logprobs, and acted on none of them. The decode step was a bare argmax, so a
client asking for temperature: 1.0 got greedy text, with a 200 and no warning.
sampling.py had existed since M1 with a complete Sampler; frigg run used it; the
server simply never wired it in, and no test asked whether it had.
That is the worst failure shape in this project so far. Every other bug here produced a
wrong number or a crash. This one produced plausible output, which is why it survived
a whole milestone with test_server.py green — the tests checked the response schema,
and a greedy completion has exactly the right schema.
Slots in one batch belong to different requests, so every parameter is per-slot:
BatchedSampler keeps temperature[B], top_k[B], top_p[B], noise_scale[B] and a
[B, vocab] noise buffer as static device tensors, which is the signature
docs/ARCHITECTURE.md §5 specified a milestone earlier. Greedy is not a special
case: a greedy slot gets temperature = 1, noise_scale = 0, which reduces
argmax(logits/T + s·g) to argmax(logits) exactly, so the body stays branch-free while
a batch mixes greedy and sampled slots — which it does, constantly.
Sampling is captured as a second CUDA graph rather than folded into the first, and the replica picks per step. Exact top-p costs a full vocabulary sort and the noise fill another draw, and the eval and benchmark paths are greedy; they should not pay for a feature they do not use. Greedy throughput after the change is 132.8 tok/s aggregate over a 120 s run, against 119.5 over the 180 s run before it — run length and prompt mix move this by ~10%, so that is "no regression", not a speedup.
The obvious way to make top-p cheap is to prefilter to a top-K shortlist and sort that. Before doing it I measured the thing it assumes — that Mimir's tail is negligible:
| T | p(top-1) | tokens for 95% mass | tokens for 99% | mass outside top-64 |
|---|---|---|---|---|
| →0 | 1.000 | 1 | 1 | 0.0000 |
| 0.5 | 0.967 | 1 | 2 | 0.0000 |
| 0.7 | 0.926 | 2 | 3 | 0.0002 |
| 1.0 | 0.805 | 1,897 | 44,554 | 0.028 |
| 1.3 | 0.501 | 133,417 | 214,950 | 0.316 |
| 2.0 | 0.035 | 237,254 | 255,622 | 0.935 |
A 262,144-token vocabulary has a very long tail, and temperature exposes it violently: between T=0.7 and T=1.0 the nucleus goes from 2 tokens to 1,897. At T=1.3 a top-1024 prefilter would discard almost the entire 95% nucleus and silently sample from a distribution that is not the one requested.
So the shortcut was wrong, and it would have been wrong invisibly. A full sort of
[8, 262144] costs 1.16 ms against a ~55 ms decode step — 2%. Frigg sorts. Exact
top-p, no shortlist, no accuracy question to answer.
On this model, temperature above ~1.0 needs top_p. Bare temperature: 1.3 puts
32% of the mass outside the top 64 tokens and the output degrades into other scripts
within a sentence — that is the model and its vocabulary, not the sampler. The sampler is
verified against the tempered softmax directly: empirical frequencies match
softmax(logits/T) within 0.03 over 4,000 draws at T = 0.7, 1.0 and 2.0.
Unimplemented parameters (frequency_penalty, presence_penalty, logprobs,
logit_bias) now return 400 with a reason instead of being dropped. Their defaults
still pass, because stock clients send them constantly.
n is supported (non-streaming, up to 8) as one request per choice, with seeds derived
from the base seed so the choices differ while staying reproducible; n > 1 with
stream is refused rather than interleaved onto one SSE stream. user is accepted and
ignored on purpose: SPEC FR14 wanted it as a session key, but a PrefixLM prompt
re-prefills every turn, so there is no session state for a router to be sticky to — see
the dated correction in docs/SPEC.md. Ignoring user changes nothing observable, which
is what separates it from the temperature case above.
The last of SPEC FR5. A variable-length token history cannot be gathered inside a
captured graph, so the penalty rides on a per-slot [B, vocab] boolean mask of what that
slot has already produced — 2 MB at 8 slots, and it turns the penalty into two
elementwise ops. docs/MODEL.md §8: the mask covers the response only, so it starts
empty after prefill and is never seeded with prompt tokens; it is cleared whenever a slot
is assigned, since slots are reused across requests.
Both graphs update the mask, not just the sampling one, so a slot that acquires a penalty
part-way through still sees the whole response behind it. A penalty forces the sampling
path even at temperature 0 — the penalty is applied inside select — so such a request
pays for a vocabulary sort it does not otherwise need. Plain greedy still skips all of it.
The half of the CTRL rule that is easy to invert: negative logits are multiplied, not divided. Divide a negative logit and the "penalty" makes a repeat more likely. There is a test built on values where the two rules disagree.
Measured through the API on a prompt that invites repetition ("gentag ordet 'hej' mange gange"), unique-word ratio:
repetition_penalty |
output | unique ratio |
|---|---|---|
| 1.0 | Hej, hej, hej, hej, hej, hej, … (27 words) |
0.07 |
| 1.5 | Hej, hej! Hej igen. |
1.00 |
| 2.0 | Hej, hej! Hejsa. |
1.00 |
stop is now implemented rather than refused, and it has one requirement that is easy to
miss: a partial match cannot be streamed. With stop: ["END"] and a model part-way
through emitting E, N, D, the naive loop sends E and then N before it knows they
are the stop string — and a token already streamed cannot be recalled. So the emittable
text ends before any suffix that is a proper prefix of some stop string, and "abcNE"
holds back the E too, because "abcNE" + "ND" contains END.
Wiring it up produced a second bug worth recording, because it is the same shape as
Finding 10 itself. The non-streaming path cut the text before the tool-call parser and
the streaming path cut it after; the parser strips whitespace (tools.py:196), so the
same request returned "hi " streamed and "hi" whole. Both now parse then cut, and a
test asserts the two agree character for character across four cases. Nothing about that
divergence was visible in either path alone — only in comparing them.
Verified against the real model, where stop strings straddle token boundaries: with the
model producing "København\nAarhus\nOdense", stop: ["Aarhus"] yields "København\n",
finish reason stop, the stop string absent, the result a prefix of the untruncated text,
and streamed output identical to non-streamed.
--devices 0,1,2,3 is in the quickstart, in CLAUDE.md, and is the reason this machine
has four cards. It failed on startup for every replica after the first, and had done
since the server was written, because nothing had ever run it — every load test in M4 and
since used --devices 0. The error names neither the cause nor the device:
ValueError: Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)
Two independent causes, and the misleading "cpu tensor?" is a red herring in both:
- Triton launches against the current CUDA device, not the device its argument
pointers live on. Once a kernel has been compiled on cuda:0, launching it with cuda:1
pointers while cuda:0 is still current fails. A replica runs on its own thread, and a
fresh thread points at cuda:0 whatever its model's device is. The fix belongs in
Mimir, not the call sites: a model now sets its own device on entry, so it is correct whatever the ambient device is. - Graph capture needs an explicitly device-bound stream. Without one, capturing on a
non-current device records an empty graph — torch warns, nothing errors, and the
replay silently does nothing.
bench launchhit exactly this in M0 and fixed it the same way; the lesson did not transfer because the two pieces of code never met.
Once it ran, the four-replica number is worth having and is below the prediction
docs/BENCHMARKS.md had been carrying:
Alternating arms, every GPU cooled to 60 °C before each — the method Finding 12 says is the minimum for a valid comparison here:
| replicas | streams | aggregate | requests | errors |
|---|---|---|---|---|
| 1 | 8 | 140.8 tok/s | 220 | 0 |
| 4 | 32 | 405.9 tok/s | 689 | 0 |
| 1 | 8 | 141.5 | 221 | 0 |
| 4 | 32 | 405.8 | 688 | 0 |
2.9× on four GPUs, not the "close to linear" the docs predicted — 72% of it. The arms reproduce to within 0.5% once each starts from the same temperature, which is what makes the ratio trustworthy rather than the spread being hidden.
Corrected 2026-08-31. This first read 3.2–3.3×, from 393.3 tok/s over a 123.6 control — measured consecutively, the four-replica arm first and the single-replica control second on a GPU 0 that had just been driven hard. Finding 12, written hours later, is precisely the argument that such a comparison is biased rather than noisy, and the bias runs against the second arm: the control was understated by 14% (123.6 against 141 from a cold start), which inflated the ratio. Publishing a number and then publishing the reason it was wrong is not a good look, so:
bench loadnow takes--cooldown-cand recordsgpu_c_start/gpu_c_end, and the figures above are alternated arms from a matched 60 °C.
The replicas share no GPU state, so the ~28% shortfall from linear is host-side: four replica threads each doing per-step Python, plus the HTTP layer and the load generator on one CPU. That is consistent with the numbers, not measured. The replicas share no GPU state, so the shortfall is almost certainly host-side: four replica threads doing per-step Python, plus the HTTP layer and the load generator, on one CPU. The router itself is fine — completions came out 196 / 192 / 207 / 204.
The general point is the one M0 was built around and this still got wrong: a documented flag that no test and no benchmark exercises is an untested flag, however prominent it is in the docs. Four GPUs are the premise of the whole project and the path that uses them all had never been run once.
This project's rule is "honest benchmarks: every number from a run on this machine with the reproducing command next to it". That turns out not to be sufficient, because running the same command twice does not give the same number, and the difference is not noise.
Same cell (int8, 512 context, batch 8), five separate invocations back to back, then a cooldown:
| run | GPU 0 | aggregate | spread within the run |
|---|---|---|---|
| 1 | warm | 186.5 tok/s | ±0.2% |
| 2 | 185.0 | ±0.3% | |
| 3 | 174.5 | ±4.0% | |
| 4 | 159.9 | ±2.3% | |
| 5 | 85 °C | 152.7 | ±1.5% |
| after cooling to 57 °C | 191.7 | ±0.1% |
26% between the coldest and hottest reading of identical code, monotonic, fully recovered by cooling. A GA106 in a four-card chassis heat-soaks in minutes.
Two things follow, and this repo had already been caught by both:
- Repeating the measurement inside one process measures almost nothing. It gives
±0.1–4% and looks reassuring. The uncertainty that matters is between invocations and
is an order of magnitude larger. (Worse:
bench matrixhad been returning a single sample while labelling it "median of 3" — now fixed, though the thermal state dominates either way.) - A consecutive before/after is biased, not just noisy — the first arm gets a cooler card. Alternating the arms, which Finding 7b and the bucket-selection measurement do, is the minimum for a valid comparison here. Two conclusions in this project were nearly drawn from consecutive single runs.
The published comparison table was biased by this, not merely noisy. A sweep heats the
card as it goes — 70 °C on the first cell, 93 °C from the third — and builds are swept in
order, so whichever ran last was measured ~20 °C hotter. Same code, one uncontrolled sweep
against one with --cooldown-c 65:
| cell | uncontrolled | all at 65 °C | |
|---|---|---|---|
| bf16 B8 @512 (first) | 166.7 | 167.0 | unchanged |
| int8 B8 @512 | 157.2 | 190.7 | +21% |
| int4 B1 @512 (last) | 26.0 | 32.7 | +26% |
int4 was penalised by a quarter for the crime of being swept last, which made it look level with bf16 at batch 1 when it is in fact clearly faster (32.7 vs 26.6). The standing advice not to use int4 survives — it is slower than int8 and far less accurate — but it survives on the reasons that were not distorted.
bench matrix now records the GPU temperature with every cell and takes --cooldown-c to
equalise it; the table above is produced at a fixed 65 °C. A figure without a temperature
beside it cannot be compared with another figure. The generated table above carries
the within-run spread, which is the small number — treat cross-commit comparisons of a
single figure as worth very little on this hardware.
You need a CUDA GPU with at least 12 GB, Python 3.12 and uv. Nothing is gated: the model repo is public and no token is required.
git clone https://github.com/TheRootKIDD/frigg && cd frigg
uv sync
# 1. Measure the machine first — every ceiling in these docs comes from these two.
uv run frigg bench vram # ~340 GB/s per RTX 3060
uv run frigg bench launch # ~5 us per kernel launch
# 2. Build the int8 checkpoint (7 s, 3.57 GB -> 1.79 GB) and verify it.
uv run frigg quantize danish-foundation-models/DFM-Mimir \
~/.cache/frigg/mimir-int8 --method int8 --check
# 3. Generate.
uv run frigg run ~/.cache/frigg/mimir-int8 \
-p "Hvad er hovedstaden i Danmark?" --max-new 64
# 4. Serve, one replica per GPU.
uv run frigg serve ~/.cache/frigg/mimir-int8 \
--devices 0,1,2,3 --kv-slots 4 --kv-dtype int8 --port 8000Then point any OpenAI client at it:
from openai import OpenAI
c = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
print(c.chat.completions.create(model="dfm-mimir",
messages=[{"role": "user", "content": "Hvad er 12 gange 8?"}]).choices[0].message.content)frigg serve prints a memory budget per replica at startup and refuses to start a
configuration that does not leave 512 MB of headroom, so a bad --kv-slots fails in
seconds rather than OOM-ing under load. Binding anything other than loopback requires
--api-key. See deploy/ for a systemd unit and firewall guidance.
uv run pytest -q # the CPU-only subset also runs in CI
uv run pytest -q -m parity # against HF transformers; needs a GPU
uv run frigg parity # 24 golden prompts, bit-exactness report
uv run frigg eval gsm8k-200 --ckpt ~/.cache/frigg/mimir-int8Read these before deploying anything.
- 4,096 tokens, hard. The RoPE tables are built for 4,096 positions and the config has no scaling. Requests past it are rejected with the counts, never truncated.
- Every turn re-prefills the whole conversation. PrefixLM prefix K/V depend on all
prompt tokens, so there is no cross-turn cache to warm and long chats cost linearly
more per turn. That is the model, not the engine — and it is measured, not assumed:
appending a turn causally instead is 2.3× faster and costs 33.3 GSM8K points
(
docs/RESEARCH.mdE3). --kv-dtype int8is a capacity flag, not a speed flag. It is ~13% slower at 512 context, ~13% faster at 4096, and it is the only way to run batch 4 at full context on a 12 GB card at all (70.4 tok/s aggregate, against "does not fit" for bf16 KV). Quality cost is +0.0001 nats. BENCHMARKS §5.10.- int8 is a single-stream win and a batched loss. It is 1.5x at batch 1 and behind bf16 from batch 4 up, because batched decode is compute-bound and int8 buys bandwidth. Pick weights by expected concurrency. Finding 8.
--kv-slotsno longer costs you when idle. Graphs are captured per bucket and the replica replays the smallest that covers the active slots, so an over-provisioned replica is ~2x faster for a lone client than it was (BENCHMARKS §5.8). More clients than slots still only buys latency: the ceiling is ~117 tok/s at 8 slots.- Unimplemented API parameters return 400, not silence:
frequency_penalty,presence_penalty,logprobs,logit_bias. Their defaults pass. Implemented:temperature,top_p,top_k,seed,repetition_penalty,stop(4 strings), andn(non-streaming, ≤ 8). - Temperature above ~1.0 needs
top_pon this model — a 262k vocabulary puts 32% of the mass outside the top 64 tokens at T=1.3 (Finding 10). - Prefill blocks decode on its replica, ~3 s for a 4K prompt.
- Danish and English only, and the model has no safety alignment. Treat output as raw model text.
- LAN only. An API key and nothing else — no TLS, no rate limiting, no tenancy.
- int4 is implemented but not recommended on this model: slower than int8 and 11x
less accurate. Use
--method int8.
frigg/
config.py ModelSpec — every shape, validated; refuses what it cannot run
checkpoint.py Weights — packed or split safetensors, bf16, shape-checked
tokenizer.py the corrected pre-tokenizer, applied and verified (Finding 6)
layers.py RMSNorm · RoPE · gated MHA · SwiGLU · pre-norm block
prefixlm.py the three masks: prefill · decode · verify
kvcache.py one static slab, 128 virtual layers, rewind and fork
model.py the recurrence, lm_head, prefill/decode/generate
sampling.py greedy · temperature · top-k/p · repetition penalty · stops
One forward. Embeddings are looked up and scaled by sqrt(1536); that is z_H.
z_L starts from the learned z_L_init buffer. Then the fixed schedule runs — for each
of 2 H-cycles, 3 L-module passes over z_L + z_H, then one H-module pass over
z_H + z_L. Eight 16-layer module runs, 3.9B parameter-reads, one token. lm_head is
applied to z_H from the final H run.
Masks. Prefill is bidirectional over the whole prompt. Decode passes no mask — one query against the whole cache, which is what the reference does and what keeps the bits identical. A run of response tokens gets a bidirectional prefix block plus a causal block, which is the mask the model was trained with.
Cache. One preallocated slab of [128, heads, 4096, 128] for K and V — 3.22 GB at
full context. Every module run writes its own virtual layer at the same offset; the length
advances once per forward. rewind_to_prefix() makes "regenerate" free, and fork()
copies just the prompt for n > 1. Static addresses are the point: M2 captures the whole
decode step in a CUDA graph, which the launch ceiling
says is worth ~5x.
Parity. The exact path is checked against HF HrmTextForCausalLM on 24 golden
prompts, at three levels: torch.equal on the full [1, T, 262144] prefill logits, the
sha256 digests recorded in tests/golden/, and the greedy continuations token for token.
Prefill is bit-identical. Cached decode is bit-identical for ~128 steps and then drifts by
at most one ulp, which matters only where the reference itself has an exact tie — the
tolerance, its cause and its bound are in docs/MODEL.md §11.
uv run frigg run -p "Hvad er hovedstaden i Danmark?" --max-new 64
uv run frigg run --chat conversation.jsonl --json
uv run frigg parity --layersEvery correction is dated in docs/MODEL.md §11 with the old claim
marked rather than deleted.
| # | Spec said | Actually |
|---|---|---|
| 1 | Repo is gated; mimir-license-v1.0-research-model-license |
Not gated (gated: false), plain Apache-2.0 in metadata, card and LICENSE blob alike |
| 2 | Native hrm_text landed in transformers 5.12.x |
5.9.0 (bisected: 5.8.1 lacks it, 5.9.0 has it) |
| 3 | ~1,700 kernels per decoded token | 5,454 (3.2×) |
| 4 | H_sliding_window/L_sliding_window are null |
The keys do not exist. Validation must accept missing, not require present-and-null |
| 5 | (unmentioned) | config.json declares no dtype — from_pretrained without an explicit dtype= loads this bf16 checkpoint as fp32, 7.1 GB and different numerics |
| 6 | NF1: "bit-identical logits against HF" | Not well defined — see Finding 4 |
| 7 | vLLM on SM 8.6: unknown, probably not | Runs, 33.0 tok/s |
| 8 | Cache as cache.L[k] / cache.H[i] arrays |
HF uses one flat interleaved array of 128 slots: slot(h,l,layer) = (h*(L_cycles+1)+l)*16 + layer, so L0 L1 L2 H0 L3 L4 L5 H1 occupy 0-15, 16-31, 32-47, 48-63, 64-79, 80-95, 96-111, 112-127 |
| 9 | "load with AutoTokenizer.from_pretrained and verify a Danish sentence" |
Not sufficient — the result depends on hub id vs local path, see Finding 6 |
| 10 | §4: RMSNorm is F.rms_norm(x, (1536,), eps) |
It is NanoChatRMSNorm — an explicit fp32 upcast around x.pow(2).mean(-1), a different kernel and not bit-identical |
| 11 | §4: RoPE is "fp32 math, cast to bf16" | The table is fp32; it is cast to bf16 before use, so the rotation runs in bf16 |
| 12 | §8: the reference passes an additive float mask | It passes a boolean mask — and at decode, no mask at all |
Smaller things worth knowing for M1/M4: apply_chat_template(tokenize=True) returns a
BatchEncoding in transformers 5.16, not a list of ids; and loading the oracle with
device_map= pulls in accelerate unnecessarily — .to(device) is enough.
Everything in this table is produced by frigg bench matrix and written here by
frigg bench render-readme — no number below is typed by hand.
| build | weights | decode B1 @512 | decode B1 @4096 | decode B4 @512 | decode B4 @4096 | decode B8 @512 | decode B8 @4096 | prefill 1024 | prefill 4096 |
|---|---|---|---|---|---|---|---|---|---|
| bf16 | 3.57 GB | 26.6 ±0% | 21.6 ±1% | 94.3 ±0% | n/f | 167.0 ±0% | n/f | 1897 | 1428 |
| int8 W / int8 vocab | 1.79 GB | 46.7 ±0% | 33.4 ±0% | 111.0 ±0% | n/f | 190.7 ±0% | n/f | 1723 | 1403 |
| int4 W / int8 vocab | 1.33 GB | 32.7 ±0% | 25.5 ±1% | 55.5 ±0% | n/f | 103.3 ±0% | n/f | 1769 | 1409 |
Decode figures are aggregate tok/s (all slots together), greedy, with the decode step CUDA-graph captured at every batch size; ± is half the observed min-max spread over the repeats, not a confidence interval — treat differences smaller than it as noise. Prefill is tok/s for a prompt of that length. n/f means the configuration does not fit on a 12 GB card — batch slots of bf16 KV at 4096 tokens is 3.22 GB each. Method: greedy, 8 warm-up + 64 timed tokens, median of 3 runs. The per-cell spread is WITHIN one process and is small; the real uncertainty is thermal drift ACROSS runs (BENCHMARKS §5.12), so gpu_c is recorded too.
Full methodology in docs/BENCHMARKS.md §5. Greedy, 32 warm-up
tokens then 256 timed, median of 3.
¹ with VLLM_USE_FLASHINFER_SAMPLER=0 and reduced batch settings; see Finding 2.
² the one exception is inside the documented decode tolerance (docs/MODEL.md §11):
a one-ulp drift that flips a position where the reference's top two logits are equal.
HF at 2048 ctx: 11.66 tok/s decode, 1,239 tok/s prefill. HF numbers were re-measured after Finding 6; they were 12.20 / 1,626 on the mis-tokenized prompts. vLLM aggregate over a 24-prompt batch: 177.7 tok/s.
Frigg's eager decode is a shade above HF's and about a third of the 37.4 tok/s bf16 bandwidth ceiling — which is what Finding 3 predicts for a launch-bound path. Closing that gap is M2's job, not a tuning exercise.
uv sync --extra dev
uv run pytest -q # the CPU-only subset also runs in CI
uv run pytest -q -m parity # needs GPU 0 + the downloaded checkpoint
uv run frigg bench vram # -> benchmarks/<host>-<date>/vram.json
uv run frigg bench launch # -> benchmarks/<host>-<date>/launch.json
# the generated decode table. --cooldown-c is not optional for a fair comparison:
# without it the build swept last is measured ~20 C hotter (Finding 12).
uv run frigg bench matrix --cooldown-c 65 \
--builds danish-foundation-models/DFM-Mimir,~/.cache/frigg/mimir-int8,~/.cache/frigg/mimir-int4
uv run frigg bench render-readme
# int8 KV is a capacity flag; sweep it to see the rows bf16 KV cannot run at all
uv run frigg bench matrix --kv-dtypes bf16,int8 --contexts 4096 --batches 1,2,4
# four replicas, one per GPU
uv run frigg serve ~/.cache/frigg/mimir-int8 --devices 0,1,2,3 --kv-slots 8
uv run frigg bench load --url http://127.0.0.1:8000 --streams 32 --duration-s 180
# regenerate the golden reference files (~7 min on GPU 0)
uv run python -m frigg.golden --timing
# the E2 pre-test
uv run python -m frigg.research.e2_pretest --out benchmarks/<host>-<date>/e2_pretest.json
# the vLLM probe (isolated venv — it needs its own torch build)
cat benchmarks/vllm-probe/README.mdThe model repo is public, so no token is needed. Reference training code (read-only, never copied into Frigg) is cloned next to the repo:
git clone --depth 1 https://github.com/schneiderkamplab/HRM-Text ../HRM-Texttests/golden/ holds prompts.jsonl (24 Danish/English prompts: QA, math, code,
grammar, summarisation, multi-turn, system prompts, tool use, raw completions) plus
config.json, tokenizer.json, safetensors_layout.json, chat_template.json and
hf_reference.json.
hf_reference.json stores, per prompt and per token_type_ids variant, a sha256 over
the exact fp32 bytes of the last-prompt-row logits, the top-64 (id, logit) pairs,
summary statistics, and the greedy continuation (256 tokens for the ones variant, 64
for the two wrong-oracle variants). The digest rather than the raw rows because 24
prompts × 3 variants × 262,144 fp32 would be 75 MB of binary in git, and a digest is
exactly the torch.equal pass/fail that NF1 asks for.
docs/ MODEL.md ARCHITECTURE.md ROADMAP.md BENCHMARKS.md RESEARCH.md SPEC.md
src/frigg/
artifacts.py config / safetensors header / tokenizer fact extraction
reference.py the HF oracle, loaded the way parity requires
golden.py regenerates tests/golden/hf_reference.json
bench/ vram, launch, env capture
research/ e2_pretest.py
cli.py frigg quantize | run | serve | bench | parity | eval
tests/ test_cli.py test_artifacts.py test_prefixlm_mask.py test_bench.py golden/
benchmarks/ per-machine snapshots + the vLLM probe
Milestones M1–M6 fill in config.py, checkpoint.py, layers.py, prefixlm.py,
kvcache.py, model.py, engine.py, sampling.py, chat.py, quantize/, kernels/
and server/ — see docs/ROADMAP.md.
The engine is Apache-2.0 (see LICENSE).
The weights are also Apache-2.0, verified in M0 by reading the LICENSE blob of
danish-foundation-models/DFM-Mimir at revision 2844f017: it is the canonical
Apache-2.0 text with the appendix filled in "Copyright © 2026 Danish Foundation
Models" — no added clause, no research-only restriction — and the repo is not gated,
so no token or conditions-acceptance is needed. The project spec had assumed a
mimir-license-v1.0-research-model-license; that was wrong.
Practical consequence: quantized derivative checkpoints produced by frigg quantize
may be redistributed, provided they carry the Apache-2.0 licence, a NOTICE stating
that the weights were modified, and attribution to Danish Foundation Models.
Mimir is Danish and English only and has no safety alignment ("not specifically aligned for safety" per the model card). Treat its output as raw model text.
- Mimir model card — https://huggingface.co/danish-foundation-models/DFM-Mimir
- Mimir v1 technical report — https://arxiv.org/abs/2608.13517
- DFM fork of HRM-Text (the training code) — https://github.com/schneiderkamplab/HRM-Text
- zaphira, the companion engine — https://github.com/TheRootKIDD/zaphira