{{ message }}
Make the benchmark honest, then make insert and erase faster - #213
Merged
Conversation
Every string key in the benchmark was exactly 200 bytes. Real string keys are identifiers, field names or paths: mostly short, occasionally long, never all the same length. One fixed length hid two things at once. The length dispatch of the hash was perfectly predicted, so the benchmark could not see what a mixed-length workload pays for it -- measured separately, wyhash costs 0.31 branch mispredictions per hash on lengths spread over 4 to 200 bytes and 0.01 on a fixed length. And at 200 bytes every key went on the heap, because none of them could fit inside a std::string. The lengths now run from 8 to 135 bytes, skewed towards short: a quarter are 16 bytes or less, a third are 17 to 48, a quarter are 49 to 96 and a sixth are longer. That covers all four paths the hash has, weighted the way a workload weighs them, and a quarter of the keys are short enough to need no allocation at all. The value still goes in the first 8 bytes, so two values still give two different keys and every checksum in the benchmark is unchanged. Scores before and after this commit are not comparable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The hash was 42-45% of them at a fixed 200 bytes. At the mixed lengths the keys now have it is 32-35%, measured the same way: against an 8 byte hash of the same keys, findstr goes 261 to 170 ms and iestr 252 to 172 ms. A lookup is ~224 instructions and ~116 cycles, of which the hash is ~60 and ~41, for keys averaging ~50 bytes. The per-workload table keeps its string rows and says which keys they were measured with, because they are what the SSE2 probe work was judged on. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A lookup is ~224 instructions and only ~60 of them are the hash. A change that touches one length range is diluted further by the share of keys in that range, and then the score cannot resolve it: the change in the next commit is worth 1.2x at 16 bytes and under a percent on `findstr`. `hashstr` hashes 10000 prebuilt keys of the lengths the string workloads use, so the hash is the whole measurement. The keys are built before the loop, and there is one set for every caller rather than one per instantiation -- the A/B harness runs two hashes interleaved, and a megabyte of keys each measures the cache instead of the hash. That mistake cost a run: with a set each, a 5% difference read as 0.4% the other way. `bench_hash_string_udm` runs the same workload standalone. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Keys of 16 bytes or less set `a` and `b` and fell through to the `mix` at the end of the function, which the long path reaches too. That makes `seed`, `a` and `b` values of two paths at once, so the compiler cannot fold this path's constant seed into the mix and has to materialise it. Returning here instead costs the short path 24 instructions where falling through cost 36. Measured per hash, against this commit's parent (clang 22, 40M hashes, hardware counters): | key length | before | after | |---|---|---| | 8 bytes | 8.09 cycles, 36.1 instructions | 6.93, 24.1 | | 12 bytes | 8.15, 36.1 | 6.98, 24.1 | | 16 bytes | 9.14, 38.1 | 7.60, 26.1 | | 24 bytes | 12.05, 50.2 | 12.16, 47.2 | gcc 16 is 27.3 instructions before and 24.3 after. Nothing above 16 bytes changes path, and no hash value changes anywhere: this is the same arithmetic. Paired against the parent with scripts/ab/run.sh, 20 epochs: `hashstr` 1.05x faster, 95% CI [1.05 .. 1.06]. The score workloads do not resolve it -- `iestr` and `findstr` both come back at 1.00x with an interval of [1.00 .. 1.00] -- because only a quarter of their keys are short enough to take this path. The `else` had to go with it, or the path after a return is an else after a return. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`spread * spread` is 64 bits wide, so the length was too, and `std::string::assign` takes a size_type that is 32 bits on a 32 bit target. clang's -Wshorten-64-to-32 caught it and gcc's -Wconversion did not, so only the two clang 32 bit legs of CI went red. The value is at most 8 + 127 either way. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The hash is 64 bits wide whatever size_t is, so adding it into a size_t loses half of it on a 32 bit target. MSVC caught it as C4244; clang's -Wshorten-64-to-32 does not look at a compound assignment, so the clang 32 bit legs went green on it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Giving the keys varying lengths meant `set_key` had to change the length of a `std::string`, and `assign` is an out-of-line call into libstdc++ followed by a generic `memmove` of ~50 bytes. `perf` on `findstr`, by symbol: | symbol | share | |---|---| | `wyhash::hash` | 24.3% | | `find_50` (the loop and the inlined lookup) | 33.7% | | `std::string::_M_replace` | 17.9% | | `__memmove_avx512_unaligned_erms` | 15.8% | | `memcmp`, the key comparison | 3.3% | A third of the workload was making the key and a thirtieth was comparing it. What the score is supposed to compare is the map. The buffers are now prepared once, one per length, and a key costs the 8 bytes of its value again, as it did when every key was 200 bytes. `findstr` loses 25% of its instructions and `iestr` 21%, and the profile is 44% hash, 42% loop and lookup, 6% comparison. The keys keep the lengths and the values they had, so every checksum is unchanged. The buffers are shared, so a key is valid only until the next call for the same length. Every workload here uses a key and is done with it, and `set_key`'s out parameter is gone so that a caller cannot hold one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The numbers in these two files were measured when a third of the string workloads was `std::string::assign`. With that gone: a lookup is ~167 instructions and ~110 cycles, not ~224 and ~116, and hashing is 33-38% of the workload rather than 32-35%. Measured the same way, against an 8 byte hash of the same keys: findstr 167 to 108 instructions and 110 to 68 cycles, iestr 187 to 121 and 120 to 80. Also what `hashstr` is worth. It resolved the short-path return, which was 5%, cleanly. It reports the six-lane threshold of the next commit as 1.01x *slower*, twice, where the same two headers built into separate binaries come out 2.6% faster in five pairs out of five and both map workloads resolve 1.01-1.02x faster. A loop that tight is more sensitive to which of the two hashes gets the better code layout than to a few percent of hashing, so a small result there needs a second opinion from the map workloads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Six lanes cost three more accumulators to set up and three xors to fold back, so the block has to run more than once to pay for them. Entering it at `i > 96` meant exactly one iteration for every key from 97 to 192 bytes, which never can. It now starts at `i > 192`. Cycles per hash, by key length, clang 22: | length | at 96 | at 192 | |---|---|---| | 100 bytes | 23.4 | 21.8 | | 150 bytes | 29.3 | 28.2 | | 400 bytes | 61.3 | 60.5 | | 1000 bytes | 143.8 | 143.5 | Better where the block could only ever run once, and unchanged where it genuinely iterates, so nothing above 192 bytes gives anything up. Paired against this commit's parent with scripts/ab/run.sh, 20 epochs: `findstr` 1.01x faster, 95% CI [1.01 .. 1.01]; `iestr` 1.02x faster, [1.01 .. 1.02]; `itstr` unchanged, which it should be because iterating does not hash. `hashstr` disagrees at 1.01x slower, and the commit before this one says why it is the measurement to distrust here. This changes the hash of keys from 97 to 192 bytes and nothing else: 97, 128 and 192 are the only golden values that move. That is what `hash_golden.cpp` exists to make a decision rather than an accident, so its values are regenerated here. 288 joins them, because the length that leaves the counter exactly on the inner bound is no longer 192. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`insert_erase` draws its keys from a range that grows to 20000, so its map hovers at ~10k entries and doubles its bucket array about a dozen times in eight million operations. Growth was a rounding error there and it is not one in general: building a map of a million entries costs 52% more than building the same map after `reserve` for `uint64_t` keys and 31% more for strings, all of it rehashing. A map that grew badly would have scored the same as one that grew well. `build` inserts 200000 distinct keys into a fresh map, so the bucket array doubles all the way up. It joins the score, which is now eight benchmarks rather than six, and `build64`/`buildstr` join the A/B harness. It found something on its first run. Paired against boost::unordered_flat_map with the same hash, `build64` is the weakest workload here: 6.65 ms against 4.70. Growth is not all of it -- inserting into a table that already reserved the room costs 52.4 cycles and 101.5 instructions against boost's 22.6 and 73.5, with 0.611 branch mispredictions per insert against 0.097. Measured over 8 million inserts, 73% of them shift no bucket and 11% shift one, so "is this bucket occupied" is a coin flip. boost never moves an element once placed. That is the price of the distance ordering, and three attempts to reduce it are in CLAUDE.md under the dead ends. Scores from before this commit are not comparable with scores after it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The robin hood shift asks "is this bucket occupied" once per bucket, and the answer is a coin flip: measured over 8 million inserts into a table at its load factor, 73% shift nothing, 11% shift one, and the rest tail off. That cost 0.61 branch mispredictions per insert, against 0.10 for boost::unordered_flat_map, which never moves an element once placed, and it was most of why `build64` ran 1.42x slower than boost. `place_and_shift_up` now takes the same trade the probe took. It loads four buckets, gets the four "empty" answers as one mask, builds the contents as if every bucket moved one along with its distance one more, and blends: buckets up to and including the first empty one take the moved contents, the rest keep theirs. No decision per bucket. The one branch left is "did the run end inside these four", which it does 91% of the time; a run of four or more, and the last three buckets before the sentinels, take the loop as before. Per reserved insert of a uint64_t key, hardware counters over 8 million: | | before | after | |---|---|---| | branch mispredictions | 0.610 | 0.240 | | cycles | 52.4 | 46.3 | | instructions | 101.4 | 126.3 | Paired with scripts/ab/run.sh, 20 epochs: `build64` 1.10x faster, 95% CI [1.10 .. 1.11]; `buildstr` 1.03x, [1.02 .. 1.03]; `iestr` 1.02x, [1.02 .. 1.02]; `ie64` 1.01x *slower*, [1.01 .. 1.01] -- its map is small and often under its load factor, so fewer inserts shift at all and the extra instructions cost what the mispredictions saved. `find64`, `findstr` and `it64` do not run this code and come back within a percent, which is code layout. Against boost, `build64` goes from 1.42x behind to 1.29x. Two other shapes lost to this one. Settling the first two buckets with conditional moves in scalar code does not remove the branch -- `a == 0 || b == 0` is two of them and a bitwise or is one that mispredicts as often (0.608). A two bucket vector version is 49.7 cycles against 46.3: its second slot is still a coin flip. Both are under the dead ends in CLAUDE.md. The bound `place + 3 <= mask` keeps the four wide store off the sentinels, and a mutation sweep of this diff (138 mutants: 67 caught by a test, 26 refused by the compiler, 38 hung a probe, 7 survived) found that every way of getting it wrong is equivalent: a sentinel inside the window reads as occupied and is written back as it was, and the three sentinels are exactly the padding the window needs. The seventh survivor is the size of the blend table, which has an unused row then. The comment says so, so that nobody looks for the missing test. The vector path needs SSE2 and the standard bucket in a contiguous container, which is what `has_simd_scan` already says; everything else keeps the loop. It compiles for 32 bit x86 with and without SSE2 under clang and gcc, and every intrinsic in it is SSE2 and present on MSVC x86. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
clang-tidy-18 rejects a C array for the blend table, so it is a std::array of std::array now; the rows are still 32 bytes apart and the outer alignas(16) still makes both halves of every row an aligned load. And clang-format 21.1.8 breaks two of the long lines after the `=` rather than inside the call, which the local run had reported and I had cut off by reading only its last line. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`insert_erase` and `iterate` draw their values from a range that grows to 20000, and the value was the key. The hash of a `uint64_t` is one multiply, and the top bits of a multiple of a small integer walk a lattice rather than fall at random. Measured, 10000 such keys in 16384 buckets: | keys | chi-square z | most in one bucket | empty buckets | |---|---|---|---| | the string keys of the same values | -0.37 | 6 | 54.1% | | a uniform hash would give | 0 | ~6 | 54.3% | | the integer keys | **-73.2** | **1** | **39.0%** | At most one key per bucket, in a table three fifths full. Nothing collided, so the probe never probed and the shifts never shifted: 82% of erases in `ie64` moved no bucket, against 58% in `iestr` for the very same values. Two changes to the shift loops read as 1% losses on `ie64` and were wins on every other table, which is how this was found. The value now goes through a 64 bit bijection -- a multiply, a xor-shift, a multiply -- before it becomes a key. It is still distinct for every distinct value, so every checksum is exactly what it was, and `ie64` now sees the same table `iestr` does: 57.8% of its erases move nothing, the shift runs average 2.04, and the same 10000 keys land with z = 0.19 and 54.3% of buckets empty. Sequential integers are a real workload, and this hash happens to be perfect on them. A benchmark that rewards that is measuring the input, not the map; `find64` and `build64` already used random keys, and now all four do. Scores from before this commit are not comparable with scores after it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The shift down after an erase asks "is the next bucket displaced" once per bucket, and the answer is the same coin flip the insert's shift was: measured over 8 million erases from a table at its load factor, 73% move nothing, 11% move one, and the rest tail off. That cost 0.60 branch mispredictions per erase against 0.10 for boost::unordered_flat_map, which never moves an element. `erase_and_shift_down` now reads the four buckets after the victim, gets the four "displaced" answers as one mask -- a distance of two or more, which is the value shifted right by nine being nonzero, so no compare that a distance of 2^23 could confuse -- builds the contents as if every bucket moved one back with its distance one less, and blends: buckets below the run take that, the bucket where the run ends empties, and the rest keep theirs. A run of four or more, and the last four buckets before the sentinels, take the loop as before. Per erase of a uint64_t key from a reserved table, build subtracted, hardware counters over 8 million, a fresh erase order every repetition: | | before | after | |---|---|---| | branch mispredictions | 0.598 | 0.259 | | cycles | 63.6 | 54.5 | | instructions | 106.4 | 132.7 | The gain grows with the chains there are to shift: a table erased from full to empty is 27% faster at 10000 entries and 7% at 200000. Paired with scripts/ab/run.sh, 20 epochs, against the parent: `ie64` 1.02x faster, 95% CI [1.01 .. 1.02]; `iestr` 1.02x, [1.01 .. 1.02]; `it64` and `rhit64` unchanged. And against the commit before the vector insert, so both shifts together: `ie64` 1.07x, [1.07 .. 1.08]; `build64` 1.11x; `iestr` 1.04x; `buildstr` 1.02x. Against boost, `build64` goes from 1.42x behind to 1.27x and `ie64` from 1.29x to 1.20x. Both shifts first read as ~1% *losses* on `ie64`. That was the benchmark: its integer keys hashed to a lattice with no chains to shift, which the previous commit fixes and explains. A mutation sweep of this change, 167 mutants: 55 caught by a test, 36 refused by the compiler, 60 hung the suite, 16 survived. Every survivor is equivalent, and each was checked lane by lane: four ways of getting the sentinel guard wrong, which the sentinels themselves make harmless; the unused row of the blend table and the lane of the fourth bucket, which never moves; and the lanes the blend never reads -- the index field of the bucket being emptied, and the first bucket of the "keep the old contents" mask, whose old contents are built as zero. A wrong lane anywhere else fails `erase_churn` within 300 cycles. That test is new. Every other churn test grows its map, and growth rebuilds the buckets from the values, which is exactly what hides a bucket the shift left behind. This one reserves so that the bucket count is fixed and asserted to stay so, fills to a load that shifts often, and churns two hundred thousand times asking for every live key along the way. 17 ms; 1.1 s under the sanitizers. A harness that erases the same keys in the same order every repetition reports 0.003 mispredictions per erase and no gain from any of this, because the predictor learns the order. That one is in scripts/ab/README.md so it is not measured again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fourteen commits. Three fix the benchmark, and each of those uncovered a result the next ones act on.
What the map gains
Paired with
scripts/ab/run.sh, 20 epochs, honest keys, againstmain:Against
boost::unordered_flat_mapwith the same hash,build64goes from 1.42x behind to 1.27xand
ie64from 1.29x to 1.20x.buildstris 1.25x ahead.No hash value changes except for keys of 97 to 192 bytes (below). No API changes.
The benchmark: three things it could not see
String keys were all exactly 200 bytes (
d3e5ee4). Real string keys are short and vary. Onefixed length made the hash's length dispatch perfectly predictable -- 0.01 mispredictions per hash
against 0.31 on mixed lengths -- and put every key on the heap. Keys now run 8 to 135 bytes,
skewed short; the value stays in the first 8 bytes, so every checksum is unchanged.
A third of the string workloads was
std::string::assign(83f285f). Giving keys varyinglengths meant building them through libstdc++ in the timed loop:
perfput 17.9% offindstrin
_M_replaceand 15.8% in thememmoveunder it, against 3.3% in the key comparison. Buffersare prepared once, one per length; a key costs 8 bytes again.
findstrloses 25% of itsinstructions.
Integer keys hashed to a lattice (
801e4e5).insert_eraseanditerateused smallsequential values as keys, and the hash of one is a multiply whose top bits walk a lattice: 10000
keys in 16384 buckets landed at most one to a bucket, 39% of buckets empty where uniform is
54%. Nothing collided, so the probe never probed and the shifts never shifted -- 82% of erases
moved nothing, against 58% for the same values as strings -- and both vector shifts below first
read as 1% losses on
ie64. The value now goes through a 64-bit bijection; checksums unchanged.Plus
build(1a1a995): nothing in the score grew a table, and growth is 31-52% of buildingone. It joins the score, which is eight workloads now. Scores are not comparable with before.
The hash
457c9d6):seed,a,bwere phinodes at a shared tail, so the constant seed could not fold. 36 -> 24 instructions for keys of
16 bytes or less, on both compilers. No value changes.
>192, not>96(f6300d0): it sets up three extraaccumulators, so it has to run twice to pay, and 97-192 byte keys ran it exactly once. 23.4 ->
21.8 cycles at 100 bytes, unchanged at 400 and 1000. This changes hash values for keys of 97 to
192 bytes;
hash_golden.cppexists to make that a decision, and its values are regenerated.Insert and erase: one branch per bucket was a coin flip
Inserting into a reserved table cost 52 cycles and 0.61 mispredictions against boost's 23 and
0.10. The hardware attributed them all to the robin-hood shift loop: measured over 8M inserts,
73% shift nothing, 11% shift one, the rest tail off.
place_and_shift_upnow reads four buckets,gets the four answers as one mask, and blends -- the same trade the probe made (
c777625).Mispredictions 0.61 -> 0.24, cycles 52 -> 46.
erase_and_shift_downis the mirror image, withthe same numbers (
e2171bd): 0.60 -> 0.26, 64 -> 55.Two other shapes lost: settling two slots with scalar cmovs (still a branch that mispredicts as
often), and a two-bucket vector version (its second slot is still a coin flip). They're under the
dead ends in CLAUDE.md, along with two rehash ideas that did nothing.
Verification
lane by lane in the commit messages; all are equivalent (the sentinel guards, and blend-table
lanes the blend never reads). A wrong lane anywhere else fails the new
erase_churntest within300 cycles by name -- that test churns a fixed-size table with no rehash, the one regime where a
bucket the shift leaves behind survives long enough to matter.
hashstrresolves a 5% change cleanly but reported a real 2.6% one the wrong way twice: a tightloop with nothing else in it is layout-sensitive. Documented in
scripts/ab/README.md; check asmall hash change against the map workloads.