ci: time-balanced test shards planned per run, race and coverage on Linux only by peterkc · Pull Request #941 · zzet/gortex · GitHub
Skip to content

ci: time-balanced test shards planned per run, race and coverage on Linux only - #941

Open
peterkc wants to merge 4 commits into
zzet:mainfrom
peterkc:ci/time-balanced-shards
Open

peterkc wants to merge 4 commits into
zzet:mainfrom
peterkc:ci/time-balanced-shards

Conversation

@peterkc

@peterkc peterkc commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes #938. CI on main runs the whole suite in one job per OS and takes about 43 minutes. This PR splits the large packages into shards balanced by measured test time and runs macOS and Windows without -race and coverage. A plan job sets the shard counts on every run from timings saved by earlier main runs. On my fork the same CI took 12.4 minutes.

Diagrams
Before (main)
  test (ubuntu)  go test -race -coverprofile ./...    ~43 min
  test (macos)   go test -race -coverprofile ./...
  test (windows) go test -v ./...

After
  plan           timings: newest main cache, else scripts/ci-*-timings.json
    |            shards per package = ceil(total / 10 min), macOS <= 5 jobs
    v
  test (ubuntu)  -race + coverage: store, indexer, mcp shards + remaining
  test-platform  no race (macOS, Windows): indexer shards, mcp, remaining
    |            each job: go test -list -> assign by time -> Go checks -> run
    v
  timings        main pushes only: merge per-test times into the cache  <-- new

Changes

  • Carries Improve incremental indexing and resumable contract analysis #806's scripts/ci-test-partition.py, its tests and its ci.yml layout unchanged, so this does not wait for Improve incremental indexing and resumable contract analysis #806.
  • Carries 79bebd9 from Improve incremental indexing and resumable contract analysis #806, the Windows read-denial test fixture. Without it, three internal/indexer tests fail on Windows when they run in shards.
  • macOS and Windows run the full suite without -race and coverage. Linux keeps both. Coverage was already uploaded only from Linux.
  • Shards are balanced by per-test times. The plan job picks the counts (10 minutes of test time per shard, at most 5 macOS jobs) and passes them to the test matrices with fromJSON.
  • On main pushes, shard jobs record per-test times and a timings job merges them into the Actions cache. PRs read the newest entry. The committed timing files are the fallback, then today's split by count.
  • A race build cache is restored by every Linux job and saved only on main pushes. A test-partition job runs the script's unit tests.

Testing

  • All tests pass (go test -race ./...)

  • New tests added for new functionality

  • Benchmarks run if performance-relevant

  • Fork CI on this branch plus one fork-only commit that lets PR runs save timings, because my fork's main never runs CI. Run 39523017380 passed every job.

  • That run took 12.4 minutes and 150 runner-minutes. main's own run 39482173339 took 42.8 minutes and 108 runner-minutes. More jobs repeat setup and build, so the cost is higher.

  • python3 -m unittest scripts/test_ci_test_partition.py passes, also with TEST_WINDOWS=true TEST_RACE=false. I did not run go test -race ./... locally; CI runs the full suite.

  • Upstream run 39542412520 on this PR started cold: main has saved no timings or race build cache yet, so the plan used the committed timing files and every Linux job compiled from scratch. Every job that ran passed.

  • That run took 37.9 minutes and 224 runner-minutes over 16 test jobs (Linux 8, macOS 4, Windows 4). Some test jobs waited up to 29.9 minutes for a runner. Steady-state numbers need the first main push after merge.

  • Not run on the fork: the main-push path (timing merge, cache saves). The fork ran it on PR runs through the fork-only commit.

  • Check names change when shard counts change. That is fine while main has no required checks; with required checks, an aggregator job would be needed.

Checklist

  • Code follows existing patterns in the codebase
  • No unnecessary abstractions added
  • Language extractor includes Meta["methods"] for interfaces (if applicable)
  • Methods have EdgeMemberOf edges to their containing type (if applicable)

zzet and others added 4 commits October 20, 2026 16:34
Carry the partition workflow and both scripts from zzet#806 unchanged.
Original work by Andrey Kumanyaev in d4698c8, c61c05a, 1eeeae0,
8e438dd, 4c6b803, 18eb3a5 and 839b3de.
Run the full suite without race on macOS and Windows. Balance Linux
race and platform indexer shards with measured per-test durations.
Restore the race cache everywhere; warm and save only on main pushes.

Give rounded-zero tests a minimum scheduling weight and prefer fewer
cases when predicted shard times tie. Keep exact census coverage and
Windows command splitting, and reject POSIX -run arguments above
120 KiB before matcher or test execution.

The timing files come from fork CI run 39516010765 on main's own tests.
Plan bounded shard counts from cached timings or committed fallback
files without running Go. Share the exact inputs with every test job.
Target 10-minute shards for every OS, with at most five macOS jobs
and the previous counts on invalid input.

On main pushes, collect completed shard PASS results and merge them
against the timings used by that run. Average old and measured values,
and retain old names when any expected family shard is missing. Cache
the merged files and upload them for manual fallback refreshes.

Linux race shard -v output feeds these measurements. Keep the
partition unit-test job and the Windows verbose-output dependency.

Rerun artifacts are not bound to plan inputs, so stale records can drop
timing entries for one run without changing Go-checked test coverage.
@peterkc
peterkc marked this pull request as ready for review October 21, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: full suite on every OS, -race and coverage on Linux only, time-balanced shards

2 participants