Make the benchmark honest, then make insert and erase faster by martinus · Pull Request #213 · martinus/unordered_dense · GitHub
Skip to content

Make the benchmark honest, then make insert and erase faster - #213

Merged
martinus merged 14 commits into
mainfrom
claude/realistic-string-keys
Sep 2, 2026
Merged

Make the benchmark honest, then make insert and erase faster#213
martinus merged 14 commits into
mainfrom
claude/realistic-string-keys

Conversation

@martinus

@martinus martinus commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Fourteen commits. Three fix the benchmark, and each of those uncovered a result the next ones act on.

What the map gains

Paired with scripts/ab/run.sh, 20 epochs, honest keys, against main:

workload change 95% CI
build64 1.11x faster [1.11 .. 1.12]
ie64 1.07x faster [1.07 .. 1.08]
iestr 1.04x faster [1.04 .. 1.04]
buildstr 1.02x faster [1.02 .. 1.03]
findstr 1.01x faster [1.01 .. 1.01]
find64, it64, itstr unchanged

Against boost::unordered_flat_map with the same hash, build64 goes from 1.42x behind to 1.27x
and ie64 from 1.29x to 1.20x. buildstr is 1.25x ahead.

No hash value changes except for keys of 97 to 192 bytes (below). No API changes.

The benchmark: three things it could not see

String keys were all exactly 200 bytes (d3e5ee4). Real string keys are short and vary. One
fixed length made the hash's length dispatch perfectly predictable -- 0.01 mispredictions per hash
against 0.31 on mixed lengths -- and put every key on the heap. Keys now run 8 to 135 bytes,
skewed short; the value stays in the first 8 bytes, so every checksum is unchanged.

A third of the string workloads was std::string::assign (83f285f). Giving keys varying
lengths meant building them through libstdc++ in the timed loop: perf put 17.9% of findstr
in _M_replace and 15.8% in the memmove under it, against 3.3% in the key comparison. Buffers
are prepared once, one per length; a key costs 8 bytes again. findstr loses 25% of its
instructions.

Integer keys hashed to a lattice (801e4e5). insert_erase and iterate used small
sequential values as keys, and the hash of one is a multiply whose top bits walk a lattice: 10000
keys in 16384 buckets landed at most one to a bucket, 39% of buckets empty where uniform is
54%. Nothing collided, so the probe never probed and the shifts never shifted -- 82% of erases
moved nothing, against 58% for the same values as strings -- and both vector shifts below first
read as 1% losses on ie64. The value now goes through a 64-bit bijection; checksums unchanged.

Plus build (1a1a995): nothing in the score grew a table, and growth is 31-52% of building
one. It joins the score, which is eight workloads now. Scores are not comparable with before.

The hash

  • Short path returns instead of falling through (457c9d6): seed, a, b were phi
    nodes at a shared tail, so the constant seed could not fold. 36 -> 24 instructions for keys of
    16 bytes or less, on both compilers. No value changes.
  • Six-lane block enters at >192, not >96 (f6300d0): it sets up three extra
    accumulators, so it has to run twice to pay, and 97-192 byte keys ran it exactly once. 23.4 ->
    21.8 cycles at 100 bytes, unchanged at 400 and 1000. This changes hash values for keys of 97 to
    192 bytes; hash_golden.cpp exists to make that a decision, and its values are regenerated.

Insert and erase: one branch per bucket was a coin flip

Inserting into a reserved table cost 52 cycles and 0.61 mispredictions against boost's 23 and
0.10. The hardware attributed them all to the robin-hood shift loop: measured over 8M inserts,
73% shift nothing, 11% shift one, the rest tail off. place_and_shift_up now reads four buckets,
gets the four answers as one mask, and blends -- the same trade the probe made (c777625).
Mispredictions 0.61 -> 0.24, cycles 52 -> 46. erase_and_shift_down is the mirror image, with
the same numbers (e2171bd): 0.60 -> 0.26, 64 -> 55.

Two other shapes lost: settling two slots with scalar cmovs (still a branch that mispredicts as
often), and a two-bucket vector version (its second slot is still a coin flip). They're under the
dead ends in CLAUDE.md, along with two rehash ideas that did nothing.

Verification

  • Every leg of CI, plus locally clang 22 / gcc 16 / unity / ASan+UBSan and both 32-bit flavours.
  • Two mutation sweeps of the vector shifts: 138 and 167 mutants. Every survivor is triaged
    lane by lane in the commit messages; all are equivalent (the sentinel guards, and blend-table
    lanes the blend never reads). A wrong lane anywhere else fails the new erase_churn test within
    300 cycles by name -- that test churns a fixed-size table with no rehash, the one regime where a
    bucket the shift leaves behind survives long enough to matter.
  • hashstr resolves a 5% change cleanly but reported a real 2.6% one the wrong way twice: a tight
    loop with nothing else in it is layout-sensitive. Documented in scripts/ab/README.md; check a
    small hash change against the map workloads.

martinus and others added 14 commits September 2, 2026 13:40
Every string key in the benchmark was exactly 200 bytes. Real string keys are identifiers, field
names or paths: mostly short, occasionally long, never all the same length. One fixed length hid
two things at once.

The length dispatch of the hash was perfectly predicted, so the benchmark could not see what a
mixed-length workload pays for it -- measured separately, wyhash costs 0.31 branch mispredictions
per hash on lengths spread over 4 to 200 bytes and 0.01 on a fixed length. And at 200 bytes every
key went on the heap, because none of them could fit inside a std::string.

The lengths now run from 8 to 135 bytes, skewed towards short: a quarter are 16 bytes or less, a
third are 17 to 48, a quarter are 49 to 96 and a sixth are longer. That covers all four paths the
hash has, weighted the way a workload weighs them, and a quarter of the keys are short enough to
need no allocation at all.

The value still goes in the first 8 bytes, so two values still give two different keys and every
checksum in the benchmark is unchanged. Scores before and after this commit are not comparable.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The hash was 42-45% of them at a fixed 200 bytes. At the mixed lengths the keys now have it is
32-35%, measured the same way: against an 8 byte hash of the same keys, findstr goes 261 to 170 ms
and iestr 252 to 172 ms. A lookup is ~224 instructions and ~116 cycles, of which the hash is ~60
and ~41, for keys averaging ~50 bytes.

The per-workload table keeps its string rows and says which keys they were measured with, because
they are what the SSE2 probe work was judged on.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A lookup is ~224 instructions and only ~60 of them are the hash. A change that touches one length
range is diluted further by the share of keys in that range, and then the score cannot resolve it:
the change in the next commit is worth 1.2x at 16 bytes and under a percent on `findstr`.

`hashstr` hashes 10000 prebuilt keys of the lengths the string workloads use, so the hash is the
whole measurement. The keys are built before the loop, and there is one set for every caller
rather than one per instantiation -- the A/B harness runs two hashes interleaved, and a megabyte
of keys each measures the cache instead of the hash. That mistake cost a run: with a set each, a
5% difference read as 0.4% the other way.

`bench_hash_string_udm` runs the same workload standalone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Keys of 16 bytes or less set `a` and `b` and fell through to the `mix` at the end of the function,
which the long path reaches too. That makes `seed`, `a` and `b` values of two paths at once, so the
compiler cannot fold this path's constant seed into the mix and has to materialise it. Returning
here instead costs the short path 24 instructions where falling through cost 36.

Measured per hash, against this commit's parent (clang 22, 40M hashes, hardware counters):

| key length | before | after |
|---|---|---|
| 8 bytes | 8.09 cycles, 36.1 instructions | 6.93, 24.1 |
| 12 bytes | 8.15, 36.1 | 6.98, 24.1 |
| 16 bytes | 9.14, 38.1 | 7.60, 26.1 |
| 24 bytes | 12.05, 50.2 | 12.16, 47.2 |

gcc 16 is 27.3 instructions before and 24.3 after. Nothing above 16 bytes changes path, and no
hash value changes anywhere: this is the same arithmetic.

Paired against the parent with scripts/ab/run.sh, 20 epochs: `hashstr` 1.05x faster, 95% CI
[1.05 .. 1.06]. The score workloads do not resolve it -- `iestr` and `findstr` both come back at
1.00x with an interval of [1.00 .. 1.00] -- because only a quarter of their keys are short enough
to take this path.

The `else` had to go with it, or the path after a return is an else after a return.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`spread * spread` is 64 bits wide, so the length was too, and `std::string::assign` takes a
size_type that is 32 bits on a 32 bit target. clang's -Wshorten-64-to-32 caught it and gcc's
-Wconversion did not, so only the two clang 32 bit legs of CI went red.

The value is at most 8 + 127 either way.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The hash is 64 bits wide whatever size_t is, so adding it into a size_t loses half of it on a
32 bit target. MSVC caught it as C4244; clang's -Wshorten-64-to-32 does not look at a compound
assignment, so the clang 32 bit legs went green on it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Giving the keys varying lengths meant `set_key` had to change the length of a `std::string`, and
`assign` is an out-of-line call into libstdc++ followed by a generic `memmove` of ~50 bytes. `perf`
on `findstr`, by symbol:

| symbol | share |
|---|---|
| `wyhash::hash` | 24.3% |
| `find_50` (the loop and the inlined lookup) | 33.7% |
| `std::string::_M_replace` | 17.9% |
| `__memmove_avx512_unaligned_erms` | 15.8% |
| `memcmp`, the key comparison | 3.3% |

A third of the workload was making the key and a thirtieth was comparing it. What the score is
supposed to compare is the map.

The buffers are now prepared once, one per length, and a key costs the 8 bytes of its value again,
as it did when every key was 200 bytes. `findstr` loses 25% of its instructions and `iestr` 21%,
and the profile is 44% hash, 42% loop and lookup, 6% comparison. The keys keep the lengths and the
values they had, so every checksum is unchanged.

The buffers are shared, so a key is valid only until the next call for the same length. Every
workload here uses a key and is done with it, and `set_key`'s out parameter is gone so that a
caller cannot hold one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The numbers in these two files were measured when a third of the string workloads was
`std::string::assign`. With that gone: a lookup is ~167 instructions and ~110 cycles, not ~224 and
~116, and hashing is 33-38% of the workload rather than 32-35%. Measured the same way, against an
8 byte hash of the same keys: findstr 167 to 108 instructions and 110 to 68 cycles, iestr 187 to
121 and 120 to 80.

Also what `hashstr` is worth. It resolved the short-path return, which was 5%, cleanly. It reports
the six-lane threshold of the next commit as 1.01x *slower*, twice, where the same two headers
built into separate binaries come out 2.6% faster in five pairs out of five and both map workloads
resolve 1.01-1.02x faster. A loop that tight is more sensitive to which of the two hashes gets the
better code layout than to a few percent of hashing, so a small result there needs a second
opinion from the map workloads.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Six lanes cost three more accumulators to set up and three xors to fold back, so the block has to
run more than once to pay for them. Entering it at `i > 96` meant exactly one iteration for every
key from 97 to 192 bytes, which never can. It now starts at `i > 192`.

Cycles per hash, by key length, clang 22:

| length | at 96 | at 192 |
|---|---|---|
| 100 bytes | 23.4 | 21.8 |
| 150 bytes | 29.3 | 28.2 |
| 400 bytes | 61.3 | 60.5 |
| 1000 bytes | 143.8 | 143.5 |

Better where the block could only ever run once, and unchanged where it genuinely iterates, so
nothing above 192 bytes gives anything up.

Paired against this commit's parent with scripts/ab/run.sh, 20 epochs: `findstr` 1.01x faster,
95% CI [1.01 .. 1.01]; `iestr` 1.02x faster, [1.01 .. 1.02]; `itstr` unchanged, which it should be
because iterating does not hash. `hashstr` disagrees at 1.01x slower, and the commit before this
one says why it is the measurement to distrust here.

This changes the hash of keys from 97 to 192 bytes and nothing else: 97, 128 and 192 are the only
golden values that move. That is what `hash_golden.cpp` exists to make a decision rather than an
accident, so its values are regenerated here. 288 joins them, because the length that leaves the
counter exactly on the inner bound is no longer 192.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`insert_erase` draws its keys from a range that grows to 20000, so its map hovers at ~10k entries
and doubles its bucket array about a dozen times in eight million operations. Growth was a rounding
error there and it is not one in general: building a map of a million entries costs 52% more than
building the same map after `reserve` for `uint64_t` keys and 31% more for strings, all of it
rehashing. A map that grew badly would have scored the same as one that grew well.

`build` inserts 200000 distinct keys into a fresh map, so the bucket array doubles all the way up.
It joins the score, which is now eight benchmarks rather than six, and `build64`/`buildstr` join
the A/B harness.

It found something on its first run. Paired against boost::unordered_flat_map with the same hash,
`build64` is the weakest workload here: 6.65 ms against 4.70. Growth is not all of it -- inserting
into a table that already reserved the room costs 52.4 cycles and 101.5 instructions against
boost's 22.6 and 73.5, with 0.611 branch mispredictions per insert against 0.097. Measured over
8 million inserts, 73% of them shift no bucket and 11% shift one, so "is this bucket occupied" is a
coin flip. boost never moves an element once placed. That is the price of the distance ordering,
and three attempts to reduce it are in CLAUDE.md under the dead ends.

Scores from before this commit are not comparable with scores after it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The robin hood shift asks "is this bucket occupied" once per bucket, and the answer is a coin flip:
measured over 8 million inserts into a table at its load factor, 73% shift nothing, 11% shift one,
and the rest tail off. That cost 0.61 branch mispredictions per insert, against 0.10 for
boost::unordered_flat_map, which never moves an element once placed, and it was most of why
`build64` ran 1.42x slower than boost.

`place_and_shift_up` now takes the same trade the probe took. It loads four buckets, gets the
four "empty" answers as one mask, builds the contents as if every bucket moved one along with its
distance one more, and blends: buckets up to and including the first empty one take the moved
contents, the rest keep theirs. No decision per bucket. The one branch left is "did the run end
inside these four", which it does 91% of the time; a run of four or more, and the last three
buckets before the sentinels, take the loop as before.

Per reserved insert of a uint64_t key, hardware counters over 8 million:

| | before | after |
|---|---|---|
| branch mispredictions | 0.610 | 0.240 |
| cycles | 52.4 | 46.3 |
| instructions | 101.4 | 126.3 |

Paired with scripts/ab/run.sh, 20 epochs: `build64` 1.10x faster, 95% CI [1.10 .. 1.11];
`buildstr` 1.03x, [1.02 .. 1.03]; `iestr` 1.02x, [1.02 .. 1.02]; `ie64` 1.01x *slower*,
[1.01 .. 1.01] -- its map is small and often under its load factor, so fewer inserts shift at all
and the extra instructions cost what the mispredictions saved. `find64`, `findstr` and `it64` do
not run this code and come back within a percent, which is code layout. Against boost, `build64`
goes from 1.42x behind to 1.29x.

Two other shapes lost to this one. Settling the first two buckets with conditional moves in scalar
code does not remove the branch -- `a == 0 || b == 0` is two of them and a bitwise or is one that
mispredicts as often (0.608). A two bucket vector version is 49.7 cycles against 46.3: its second
slot is still a coin flip. Both are under the dead ends in CLAUDE.md.

The bound `place + 3 <= mask` keeps the four wide store off the sentinels, and a mutation sweep of
this diff (138 mutants: 67 caught by a test, 26 refused by the compiler, 38 hung a probe, 7
survived) found that every way of getting it wrong is equivalent: a sentinel inside the window reads
as occupied and is written back as it was, and the three sentinels are exactly the padding the
window needs. The seventh survivor is the size of the blend table, which has an unused row then.
The comment says so, so that nobody looks for the missing test.

The vector path needs SSE2 and the standard bucket in a contiguous container, which is what
`has_simd_scan` already says; everything else keeps the loop. It compiles for 32 bit x86 with and
without SSE2 under clang and gcc, and every intrinsic in it is SSE2 and present on MSVC x86.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
clang-tidy-18 rejects a C array for the blend table, so it is a std::array of std::array now;
the rows are still 32 bytes apart and the outer alignas(16) still makes both halves of every row an
aligned load. And clang-format 21.1.8 breaks two of the long lines after the `=` rather than inside
the call, which the local run had reported and I had cut off by reading only its last line.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`insert_erase` and `iterate` draw their values from a range that grows to 20000, and the value was
the key. The hash of a `uint64_t` is one multiply, and the top bits of a multiple of a small
integer walk a lattice rather than fall at random. Measured, 10000 such keys in 16384 buckets:

| keys | chi-square z | most in one bucket | empty buckets |
|---|---|---|---|
| the string keys of the same values | -0.37 | 6 | 54.1% |
| a uniform hash would give | 0 | ~6 | 54.3% |
| the integer keys | **-73.2** | **1** | **39.0%** |

At most one key per bucket, in a table three fifths full. Nothing collided, so the probe never
probed and the shifts never shifted: 82% of erases in `ie64` moved no bucket, against 58% in
`iestr` for the very same values. Two changes to the shift loops read as 1% losses on `ie64` and
were wins on every other table, which is how this was found.

The value now goes through a 64 bit bijection -- a multiply, a xor-shift, a multiply -- before it
becomes a key. It is still distinct for every distinct value, so every checksum is exactly what it
was, and `ie64` now sees the same table `iestr` does: 57.8% of its erases move nothing, the shift
runs average 2.04, and the same 10000 keys land with z = 0.19 and 54.3% of buckets empty.

Sequential integers are a real workload, and this hash happens to be perfect on them. A benchmark
that rewards that is measuring the input, not the map; `find64` and `build64` already used random
keys, and now all four do. Scores from before this commit are not comparable with scores after it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The shift down after an erase asks "is the next bucket displaced" once per bucket, and the answer
is the same coin flip the insert's shift was: measured over 8 million erases from a table at its
load factor, 73% move nothing, 11% move one, and the rest tail off. That cost 0.60 branch
mispredictions per erase against 0.10 for boost::unordered_flat_map, which never moves an element.

`erase_and_shift_down` now reads the four buckets after the victim, gets the four "displaced"
answers as one mask -- a distance of two or more, which is the value shifted right by nine being
nonzero, so no compare that a distance of 2^23 could confuse -- builds the contents as if every
bucket moved one back with its distance one less, and blends: buckets below the run take that,
the bucket where the run ends empties, and the rest keep theirs. A run of four or more, and the
last four buckets before the sentinels, take the loop as before.

Per erase of a uint64_t key from a reserved table, build subtracted, hardware counters over
8 million, a fresh erase order every repetition:

| | before | after |
|---|---|---|
| branch mispredictions | 0.598 | 0.259 |
| cycles | 63.6 | 54.5 |
| instructions | 106.4 | 132.7 |

The gain grows with the chains there are to shift: a table erased from full to empty is 27%
faster at 10000 entries and 7% at 200000. Paired with scripts/ab/run.sh, 20 epochs, against the
parent: `ie64` 1.02x faster, 95% CI [1.01 .. 1.02]; `iestr` 1.02x, [1.01 .. 1.02]; `it64` and
`rhit64` unchanged. And against the commit before the vector insert, so both shifts together:
`ie64` 1.07x, [1.07 .. 1.08]; `build64` 1.11x; `iestr` 1.04x; `buildstr` 1.02x. Against boost,
`build64` goes from 1.42x behind to 1.27x and `ie64` from 1.29x to 1.20x.

Both shifts first read as ~1% *losses* on `ie64`. That was the benchmark: its integer keys hashed
to a lattice with no chains to shift, which the previous commit fixes and explains.

A mutation sweep of this change, 167 mutants: 55 caught by a test, 36 refused by the compiler,
60 hung the suite, 16 survived. Every survivor is equivalent, and each was checked lane by lane:
four ways of getting the sentinel guard wrong, which the sentinels themselves make harmless; the
unused row of the blend table and the lane of the fourth bucket, which never moves; and the
lanes the blend never reads -- the index field of the bucket being emptied, and the first bucket
of the "keep the old contents" mask, whose old contents are built as zero. A wrong lane anywhere
else fails `erase_churn` within 300 cycles.

That test is new. Every other churn test grows its map, and growth rebuilds the buckets from the
values, which is exactly what hides a bucket the shift left behind. This one reserves so that the
bucket count is fixed and asserted to stay so, fills to a load that shifts often, and churns two
hundred thousand times asking for every live key along the way. 17 ms; 1.1 s under the sanitizers.

A harness that erases the same keys in the same order every repetition reports 0.003
mispredictions per erase and no gain from any of this, because the predictor learns the order.
That one is in scripts/ab/README.md so it is not measured again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@martinus martinus changed the title Give the string benchmark realistic key lengths, and make the short hash path cheaper Make the benchmark honest, then make insert and erase faster Sep 2, 2026
@martinus
martinus merged commit adc537d into main Sep 2, 2026
32 checks passed
@martinus
martinus deleted the claude/realistic-string-keys branch September 2, 2026 16:55
@martinus martinus mentioned this pull request Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant