Eight tests were removed on 2026-09-05 (501dacb0) because they failed intermittently under sanitizer-lane load and the troubleshooting cost per failure had exceeded what the coverage was returning. The decision was deliberate and I think correct on its own terms — but 573 assertions went with them, and this issue is the record of what, why, and how to get the valuable parts back.

Registered tests: 345 → 337.

What each test was validating

What we observed trying to validate them

The investigation is why the removal happened, and it is the part worth not repeating.

The failing test differs on every run. Three configurations, five distinct tests:

That is a lane that fails somewhere under load, not eight broken tests.

Local reproduction does not track CI. radiomodel_pan_id_mapping_test failed 4/4 on CI and 2/36 locally — and 0/6 at --cpus=2, so constraining CPUs made it less likely, not more. radio_capability_gating_test failed 2/2 on CI and passed in every local configuration tried: native RelWithDebInfo, ASan+UBSan Debug, CI's exact env via ctest, and the full suite run in order to test 272.

Two fixes were attempted and measured. test_event_loop_test's cross-thread QSignalSpy race went 30/30 → 0/30 with a queued-connection spy, confirmed absent in CI. hl2_spectrum_rate_test was fixed on the second attempt (the first looked perfect at 32 cores and failed 4/4 on CI) by feeding at a fixed cadence instead of the nominal sample rate: 0/8 failures under TSan at --cpus=2, 0.0% spread across spans. Both fixes worked. Both are moot now that the tests are gone.

spectral_nr_test was not flaky — it was slow. It passed at 274.33 s against ctest's 300 s cap in one run and timed out in the next. The June-era version of the test runs against today's SpectralNR.cpp in 0.035 s, so NR2 did not regress; the test grew 287 → 2645 lines. Cost is broad (32 sections, ~0.5 s each), so there was no hot spot to excise; an opt-in split bought only 27%, and FFTW_MEASURE → FFTW_ESTIMATE changed nothing (16.39 s vs 16.35 s).

One measurement of mine was wrong and is worth flagging. An early profile attributed 15.5 of 16.5 s to four checks. That profiler read stdout through a pipe, where printf is block-buffered, so arrival time said nothing about print time. Re-run under stdbuf -o0 the cost was evenly spread. Whole-process time had disagreed with the profiler all along and was right.

Two real defects this surfaced — deleting the tests does not delete these

1. Hl2Spectrum calls FFTW's non-thread-safe planner with no lock, while WdspChannel::open() does the same on another thread during every HL2 connect. AnanSpectrum, the sibling class, takes WdspChannel::fftwSetupLock() for exactly this reason. Fix in fix(hl2): serialise Hl2Spectrum's FFTW use. Principle VIII. #5424 — which now merges with nothing covering it, because the test that proved it was one of the eight.
2. WDSP leaks five semaphores per TX channel open — createWaitable (wdsp_port.c:67) via CreateSemaphore ← create_calcc (calcc.c:361) ← create_txa ← OpenChannel. Caught by wdsp_channel_test's runLeakChecked harness, which was not removed.

Suggested course of action

Ordered by value per effort. The goal is not to restore eight files; it is to get the assertions back without the load sensitivity that made them a liability.

1. Restore radio_capability_gating_test's capability assertions first. 301 checks, the largest single loss, and most of them are pure table assertions with no threads, no timing and no event loop — they cannot be load-sensitive. Split the file: keep the capability-table, band-ceiling and CI-V frame assertions as a socket-free unit test; drop or quarantine the parts that build a RadioModel and wait for it to settle. Note it was failing a genuine assertion when removed — FAIL: dial lock sends CI-V 16 50 01, seen both on 2026-08-31 and on 2026-09-04. That is currently unwatched and may be a live defect.

2. Restore tci_protocol_test's malformed-trx assertion. One assertion, guarding a class that has shipped three times. Its failure mode was a single processEvents() where the same file uses bounded settle loops nine times elsewhere — convert to the settle idiom and it stops being load-sensitive. Cheapest real-coverage win here.

3. Restore spectral_nr_test minus the four slow scenarios. ~176 checks run in ~0.9 s; the four DSP-convergence scenarios are what cost 15.5 s. Bring back the fast set as the registered test and put the slow four behind an option()-gated target with the # not registered: <reason> marker AGENTS.md recognises.

4. Re-land the two fixes that were already proven, if their tests come back: the queued-connection spy (30/30 → 0/30) and the fixed-cadence rate delivery (0/8 at --cpus=2). Both are on the branch history behind #5424 and both were measured, not assumed.

5. Fix the lane, not the tests. The strongest observation here is that the failing test is different every run. Until that is addressed, any restored test is a candidate for the same fate. Worth investigating whether the sanitizer lane should run with a longer per-test timeout, less concurrency, or a retry-once-then-fail policy that distinguishes a genuine failure from a load artifact.

6. Do not restore map_live_update_test, asr_remote_backend_test, test_event_loop_test or radiomodel_pan_id_mapping_test as they were. 70 checks between them, all with genuine load sensitivity in their current form, and two of them (asr_remote_backend_test, map_live_update_test) were never diagnosed to a root cause. If their behaviour matters, they want rewriting against a deterministic seam rather than restoring.

Refs #5423, #5424, #5408, #5410, #5254. Removal commit 501dacb0.