fix(pro): synchronize Go ArrayCache polling and clear by pcriadoperez · Pull Request #30314 · ccxt/ccxt · GitHub
Skip to content

fix(pro): synchronize Go ArrayCache polling and clear - #30314

Draft
pcriadoperez wants to merge 2 commits into
ccxt:masterfrom
pcriadoperez:fix/go-arraycache-locking
Draft

pcriadoperez wants to merge 2 commits into
ccxt:masterfrom
pcriadoperez:fix/go-arraycache-locking

Conversation

@pcriadoperez

@pcriadoperez pcriadoperez commented Sep 7, 2026 •

Copy link
Copy Markdown
Member

Summary

Fix Go ArrayCache bookkeeping races discovered while verifying the independent performance PR #30300. This branch contains only the hand-written Go cache locking fix and native tests, not the performance changes.

  • Protect ArrayCache.GetLimit and ArrayCacheByTimestamp.GetLimit with the same mutex used by append. Polling writes deferred-reset flags as well as reading counters; it is not read-only.
  • Clear storage and metadata in one critical section. Previously a concurrent append could run after storage was cleared but before its index/counters were reset.
  • Document that callers must not hold the cache mutex when calling methods that acquire it, and that returned rows remain shared mutable values.

Regression tests

Native tests cover plain, symbol/id, outcome/id, symbol/side and timestamp caches:

  • Concurrent append, global/symbol/unknown-symbol polling and shallow snapshots.
  • Concurrent clear/append/poll with storage/index consistency checks performed under the cache mutex.
  • Serialized polling retains deferred-reset semantics before/after clear.

The new timestamp polling test reports DATA RACE on the original implementation. The existing keyed-cache race was separately reproduced during #30300. A mutation restoring only the old two-lock Clear makes the atomicity regression fail for all five variants, even without -race (five repeats). This checks the logical interleaving bug, not just data races.

The native cache race suite passed five repeats before rebasing onto current upstream, then 20 repeats on the final upstream base (Go 1.26.1, darwin/arm64). The production REST/Pro build and full WebSocket base suite also pass.

Scope and remaining limitations

This fixes cache-method synchronization, not all WebSocket concurrency:

  • GetLimit followed by ToArray is still two separately locked operations. A message can arrive between them. Exact per-watch count/snapshot consistency requires a broader change to the watch/filter interface; this PR does not introduce a new snapshot API or modify generated exchange files.
  • ToArray copies the container, not the row maps/slices. External reads or writes of those rows during an update still need synchronization. Direct access to public Data/Hashmap is not made thread-safe.
  • Internal callers were inspected: nested cache GetLimit methods delegate once, and no cache method holding Mu calls GetLimit. Clear no longer calls another mutex-acquiring clear method while holding the lock.
  • No exchange API calls or live trades are required to reproduce these races. Other language implementations are unchanged.

Verification checklist

  • npm run lint — not relevant to this Go-only source change; no TS files modified. The separate performance branch's lint hook currently hits the TypeScript/ESLint compatibility error.
  • npm run tsBuild — not rerun; no TS changes.
  • npm run transpile (Python + PHP) — not applicable; hand-written Go cache only.
  • npm run transpileCS + npm run buildCS — unchanged language, not rerun.
  • npm run transpileGO — not applicable to this hand-written cache.
  • Go target build — go -C go build ./v4 ./v4/pro passes.
  • npm run transpileRust + npm run buildRust — unchanged language, not rerun.
  • npm run check-python-syntax + npm run check-php-syntax — no Python/PHP code changes in this PR.
  • Offline tests — go -C go test -race ./v4 -run TestArrayCache -count=20 -timeout=120s passes; go -C go run ./tests --baseTests --ws reports Base WS tests passed!.
  • Live smoke — not run for this native concurrency fix; deterministic offline goroutine tests are the regression gate.
  • Diff contains only the hand-written Go cache and native test file; git diff --check passes. No generated edits.

@carlotestor carlotestor left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Reviewed at d661efb. I reproduced every claim in the description locally against this head rather than taking the body at its word; all of them hold.

Independent verification (Go 1.25.5, go/v4 resolved to the PR worktree via go list -m, confirming the local source was under test):

Check Result
go vet ./v4 clean
go test -race ./v4 (whole package) ok — 15/15 new subtests pass
go test -count=5 ./v4 (CI-equivalent, no -race) ok
Base WS suite go run . --baseTests --ws Base WS tests passed!
gofmt -l clean

The tests are genuine regression gates, not tautologies. Reverting exchange_cache.go to origin/master while keeping the new test file reproduces real failures:

  • The GetLimit race is exactly where you say it is — ArrayCacheByTimestamp.GetLimit writing clearUpdates against Append reading it. Polling is provably not read-only:
WARNING: DATA RACE
Write at 0x00c0002b6318 by goroutine 12:
  (*ArrayCacheByTimestamp).GetLimit()  exchange_cache.go:503
Previous read at 0x00c0002b6318 by goroutine 11:
  (*ArrayCacheByTimestamp).Append()    exchange_cache.go:435
  • The Clear atomicity claim also holds. Restoring only the old two-lock Clear (leaving the new GetLimit locking in place) fails all five variants without -race, five repeats — e.g. clear exposed mismatched rows/index: 1/5, 2/1, 1/0. That is the logical interleaving bug, not a detector artifact, so the point in your description is correct.

Correctness of the locking itself. sync.Mutex is not reentrant, so the risk with this change is a self-deadlock from taking Mu in GetLimit. I checked the reachable set and found none:

  • The nested caches (ArrayCacheBySymbolById, ArrayCacheByOutcomeById, ArrayCacheBySymbolBySide) delegate to ArrayCache.GetLimit exactly once, with no lock of their own.
  • AppendInternal, trackAppendLocked, resetUpdateTrackersLocked and mergeRow are lock-free helpers, so the append paths still take Mu once.
  • MathMin / ToString / ParseInt, now called under the lock, are pure and touch no cache state.
  • No code outside exchange_cache.go locks Mu, and the generated pro/*.go callers reach GetLimit only through ToGetsLimit(...) without holding it, so no caller can enter already-locked.

Inlining Data = Data[:0] is faithful to the old BaseCache.Clear() — on master that method was only Data[:0] under the lock, so no reset was dropped in the merge.

Findings

One durability issue with the regression gate, detailed inline: CI runs go test -C go -count=1 ./v4 without -race, and the timestamp half of this fix is invisible to that command. Not merge-blocking — the fix is correct either way — but worth knowing that half of what you just fixed would not be caught if it regressed.

The scope/limitations section is accurate and I would not expand it here: GetLimit + ToArray remaining two separately locked calls is a real property of the current watch/filter interface, and fixing it belongs in a broader change, not this one.

Merge gate: 🟢
Merge probability: 85%

Comment thread go/v4/exchange_cache.go
// GetLimit for timestamp cache ignores symbol because entries are not
// symbol-segmented. It mirrors the same precedence order as ArrayCache.
func (c *ArrayCacheByTimestamp) GetLimit(symbol any, limit any) any {
c.Mu.Lock()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This lock is the one CI will not protect. .github/workflows/go-app.yml runs go test -C go -count=1 ./v4 without -race, and I confirmed the difference matters: removing only these two lines (keeping the new atomic Clear) still passes go test -count=5 5/5 silently, while go test -race reports 12 data races. By contrast the ArrayCache.GetLimit mutation fails even without -race — it dies on fatal error: concurrent map read and map write — and the Clear mutation fails on the assertions. So the timestamp cache is the only part of this fix with no CI gate. Consider adding -race to the "Run Go Unit Tests" step (these tests run in ~4s under -race, so the cost is negligible), otherwise a future refactor can silently drop this lock.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added an explicit ArrayCache race-detector CI gate in 3e04957: go test -C go -race -count=1 -run '^TestArrayCache' ./v4. This runs for important/base changes alongside the ordinary Go unit tests, so timestamp counter races cannot pass on functional assertions alone. Locally the ArrayCache race suite passed with -count=20.

Comment thread go/v4/exchange_cache.go
Comment on lines +26 to +29

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good invariant to write down. Worth noting that after this change BaseCache.Clear() has zero callers in the tree — both overrides now inline Data = Data[:0]. The "clear rows and metadata in one critical section" rule is therefore enforced only by convention in the two overrides, and a future cache type embedding BaseCache would reintroduce exactly the split you just removed by calling it. Consider stating that constraint on BaseCache.Clear itself so the next implementer sees it at the call site.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the requested warning immediately above BaseCache.Clear in 3e04957: it clears base storage only; subclasses must clear Data and their own indexes/counters together under a single Mu critical section rather than calling BaseCache.Clear and reacquiring Mu.

@carlotestor carlotestor left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up on 3e04957.

Delta is two files and no behaviour change to the cache logic reviewed at d661efb:

  • .github/workflows/go-app.yml adds a go test -C go -race -count=1 -run '^TestArrayCache' ./v4 step, gated on the same important_modified == 'true' condition as the existing ./v4 test step, and placed after it / before the Ws build. Anchoring the filter with ^TestArrayCache picks up the new suite (including the ByTimestamp polling case that only fails under -race), and scoping the race detector to that one regex keeps CI time bounded instead of racing the whole package. Note the gate is conditional: pushes that do not touch the watched paths will not run it, so the race check protects go/v4 changes rather than acting as an unconditional guard.
  • go/v4/exchange_cache.go adds a doc comment on BaseCache.Clear telling subclasses with indexes or counters to reset storage and metadata in a single Mu critical section rather than calling this method and re-locking. That is exactly the invariant the earlier commit enforced in ArrayCacheBySymbolById/ByTimestamp, so the comment matches the shipped code.

Nothing new to flag; my earlier findings stand as addressed and this commit does not reopen any of them.

Merge gate: 🟢

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug go ws WebSocket (aka ccxt.pro)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants