Summary

deepseek_v4.c has had COLI_MODEL_MIRROR and O_DIRECT expert reads for many
releases. deepseek_v41.c -- its direct successor, and the largest container the
project ships at 510 GB -- never got either. That is the "a mechanism reached one
engine and never its siblings" shape, twice, in the engine whose docs already say the
expert stream "is most of the engine".

Three changes, all about the same fact -- an expert is read once and never wanted
again:

COLI_MODEL_MIRROR / COLI_DISK_WEIGHTS / SNAP_MIRROR register byte-identical
read-only copies on other drives and split the experts across them with a flat-index
hash (the clustering fix deepseek_v4.c documents, including why an XOR of two small
ids clustered the hot subset onto one replica). st_mirror_add already refuses a copy
whose size or safetensors header differs, so every data offset stays valid on any
replica, and a partial mirror keeps the shards it does not carry on the primary.

COLI_MODEL_DIRS is the other half of the same idea and the cheaper one: the
container split across drives as distinct shards, no second copy. At 510 GB that
is the only multi-drive layout most people can build -- two ~256 GB drives instead of
one 512 GB -- and GLM, V4 and Kimi K3 all accept it while V4.1 did not. One line
(st_init -> st_init_multi), plus the docs and a gate.

V41_DIRECT (default on, as COLI_V4_DIRECT is in V4) reads the expert tensors
with O_DIRECT. The engine already calls posix_fadvise(DONTNEED) on every expert
after reading it, which is the engine's own admission that a buffered read pays for
the bytes twice and then discards the cache it paid for.

The shared primitive, and why it is not a bounce buffer

st_read_range_rep() in st.h is the replica-aware, O_DIRECT-capable range read.
O_DIRECT constrains address, offset and length to the block size, and a safetensors
range begins wherever the writer put it. deepseek_v4.c and colibri.c solve that by
reading into an aligned scratch buffer and memmove-ing the payload out; on a turn
that moves 88.5 GB of experts, that copy is most of what the change was for. This does
the opposite: the caller reserves ST_DIRECT_ALIGN bytes of scratch behind the
destination, the read starts at the enclosing block boundary, one aligned pread
carries the block-aligned bulk and one short buffered pread carries the sub-block
tail. The payload lands exactly where the slot wants it and nothing is copied after.

The alignment is verified rather than assumed: a caller that did not reserve the
prefix, a platform without O_DIRECT, a replica whose twin failed to open, and a
device that refuses the call all fall back to the buffered path, which reads the same
bytes.

tests/test_st_range_rep.c drives that contract directly -- multi-block ranges,
sub-block tails, the EOF clamp where the aligned window would run past the end, a
replica read, a divergent replica (refused), a replica with no copy of the shard
(stays on the primary), and a caller that reserved nothing. Guard bytes past the
payload prove the window arithmetic never writes outside it.

Measurements, and the mistake I made measuring them

Read the correction first, because it is bigger than the result. I first measured
this A/B with the engine's default OpenMP team, which on a 208-logical-CPU host is
pathological -- 93% of cycles inside libgomp, a turn that takes 409.9 s instead of
37.2 s (that is a separate defect, PR #1517). It said the disk phase is 5% of a turn
and this work is worth ~2.5%. Re-run with the team sized:

208-core Xeon 8555C, released 510.3 GB checkpoint, cold cache before every run, arms
interleaved, 3 runs per arm, medians, every run token-exact 16/16 in both regimes.

So O_DIRECT is -48% of the disk phase and +63% of the decode rate, and the split
is -21% and +17%. Validated manifests: dsv41-direct.json (+41% whole-turn),
dsv41-mirror.json (+12%).

Three things in that table are the point, not the percentages:

• The engine now reaches the device's own ceiling. 6.17 GB/s against the 6.34 GB/s
  c/iobench measures one nvme0n1 at depth 4. M3 clears it at 7.83, which is only
  possible if the split is genuinely using two devices.
• M2 is the control. Both copies behind the md0 RAID0 pair is indistinguishable
  from one NVMe, so the variable is the number of independent filesystems, not the
  number of drives. Two NVMe behind one device add nothing here; a second filesystem
  adds 28% of read rate.
• The probe is not better than a pinned split. MP lands between the two with more
  variance (52/52/51% of bytes to the mirror, disk 12.81 / 13.44 / 13.72 s). On a pair
  of equal drives COLI_DISK_WEIGHTS=1,1 is the better choice; the docs already
  recommend that without saying why.

Full record with both regimes: docs/experiments/dsv41-expert-io-2026-09-14.md. Raw
per-run engine stderr: dsv41-expert-io-2026-09-14-raw.txt.

I kept the 208-thread numbers in the record rather than deleting them. They are real
measurements of a real configuration, the phase breakdown looks entirely reasonable in
both, and that is exactly why the wrong ratio was easy to publish. The record now leads
with a correction block.

Validation

• make -C c check -- full suite
• the tiny oracle token-exact at caps 1/2/8/49, with a mirror, with
  COLI_MODEL_DIRS, and for both V41_DIRECT arms
• a CI step holding the mirror, the split and the buffered path to the same token
  stream as the primary, and asserting a divergent mirror copy is refused rather
  than trusted
• the released 510.3 GB checkpoint end to end; all 88 files verified against the
  upstream manifest; 36 runs across both regimes, every one token-exact
• not applicable -- no CUDA changes

Compatibility

• default CPU build remains dependency-free -- open flags, one posix_memalign
  per cache slot, and a header-local helper
• no model files, generated binaries or benchmark artifacts are included
• defaults unchanged in observable behaviour: V41_DIRECT=1 reads the same bytes
  by a different path, COLI_MODEL_MIRROR/COLI_MODEL_DIRS are opt-in, and every
  arm of the A/B is token-exact against the same stream
• Windows/macOS: st_direct_fd_rep returns the compat_open_direct twin, the
  alignment check falls back to buffered where O_DIRECT does not exist, and
  posix_memalign is compat.h's _aligned_malloc shim

Reviewer notes

• st.h gains the primitive rather than deepseek_v41.c owning it, because V4 and GLM
  need exactly it and both currently carry their own copy-and-memmove version.
  Migrating them is a separate, riskier change in much larger files.
• COLI_MODEL_DIRS came along because it is the same user problem with a cheaper
  answer; say the word if you would rather split it out.
• This PR and PR fix(omp): size the OpenMP team in the four engines that never did #1517 are independent. fix(omp): size the OpenMP team in the four engines that never did #1517 is the CPU-side fix that makes this one
  measurable; neither needs the other to be correct, though the numbers in this record
  were taken with OMP_NUM_THREADS=32 on the command line.