fix(autobalancer): let recovered partitions settle before balancing by thiskian · Pull Request #3673 · AutoMQ/automq · GitHub
Skip to content

fix(autobalancer): let recovered partitions settle before balancing - #3673

Open
thiskian wants to merge 1 commit into
AutoMQ:mainfrom
thiskian:kabdolahi/3669-autobalancer-recovery-cooldown
Open

thiskian wants to merge 1 commit into
AutoMQ:mainfrom
thiskian:kabdolahi/3669-autobalancer-recovery-cooldown

Conversation

@thiskian

@thiskian thiskian commented Oct 20, 2026 •

Copy link
Copy Markdown

Why

Optional balancing can move recently reassigned partitions again before their new load reports have settled. Initial zero-valued rate reports can make a recovering destination appear idle, while a precomputed balancing plan can become stale after broker lifecycle or controller leadership changes.

What

Apply a configurable partition reassignment cooldown, require representative post-handoff load reports for optional optimization, and discard stale balancing plans.

How

Track accepted reassignment batches and observed handoffs in controller-local state. Add autobalancer.controller.partition.reassignment.cooldown.ms, defaulting to five minutes; zero disables the cooldown. Bound the whole-cluster healthy-source balancing hold to 30 seconds after broker recovery events.

Require distinct, complete post-handoff load reports spanning the rate meter's five-second tick before optional optimization treats a recovered replica as ready. Keep raw load freshness and mandatory recovery placement unchanged. Excluding a partition from optimization retains its broker's modeled load.

Initial metadata replay and duplicate lifecycle records do not fabricate or extend recovery. Detected slow-source evacuation remains eligible, and broker/controller transitions invalidate queued optional plans.

Testing

Coverage includes configuration validation and dynamic disabling, accepted versus rejected reassignment batches, expiry, duplicate lifecycle events, controller handoffs, slow-source evacuation, stale plans, and incomplete/duplicate/early load reports.

Local verification passed: the AutoBalancer tagged suites, Checkstyle and Java formatting. The repository's rat task was invoked and skipped by upstream configuration.

./gradlew --build-cache core:S3UnitTest --tests 'kafka.autobalancer.*'
./gradlew --build-cache rat checkstyleMain checkstyleTest spotlessJavaCheck

Reviewer notes

This change is independent of the router/WAL fixes and optional cross-rack placement. Read PartitionReassignmentCooldown and BrokerRecoveryCooldown, then their integration in AnomalyDetectorImpl, followed by the model warm-up rules and tests.

The cooldown bounds optional scheduling; it does not certify storage or application recovery. The issue's measurements do not isolate this PR's individual latency benefit.

Related to #3669.

AI assistance: OpenAI Codex (GPT-6) was used to prepare the contribution split, adapt the changes to main, and update the tests and descriptions.

Attribution

The initial partition reassignment cooldown was implemented by @deniszh.

@CLAassistant

CLAassistant commented Oct 20, 2026 •

Copy link
Copy Markdown

## Why

Optional balancing can move recently reassigned partitions again before their new load reports have settled. Initial zero-valued rate reports can make a recovering destination appear idle, while a precomputed balancing plan can become stale after broker lifecycle or controller leadership changes.

## What

Apply a configurable partition reassignment cooldown, require representative post-handoff load reports for optional optimization, and discard stale balancing plans.

## How

Track accepted reassignment batches and observed handoffs in controller-local state. Add `autobalancer.controller.partition.reassignment.cooldown.ms`, defaulting to five minutes; zero disables the cooldown. Bound the whole-cluster healthy-source balancing hold to 30 seconds after broker recovery events.

Require distinct, complete post-handoff load reports spanning the rate meter's five-second tick before optional optimization treats a recovered replica as ready. Keep raw load freshness and mandatory recovery placement unchanged. Excluding a partition from optimization retains its broker's modeled load.

Initial metadata replay and duplicate lifecycle records do not fabricate or extend recovery. Detected slow-source evacuation remains eligible, and broker/controller transitions invalidate queued optional plans.

## Testing

Coverage includes configuration validation and dynamic disabling, accepted versus rejected reassignment batches, expiry, duplicate lifecycle events, controller handoffs, slow-source evacuation, stale plans, and incomplete/duplicate/early load reports.

Local verification passed: the AutoBalancer tagged suites, Checkstyle and Java formatting. The repository's `rat` task was invoked and skipped by upstream configuration.

```bash
./gradlew --build-cache core:S3UnitTest --tests 'kafka.autobalancer.*'
./gradlew --build-cache rat checkstyleMain checkstyleTest spotlessJavaCheck
```

## Reviewer notes

This change is independent of the router/WAL fixes and optional cross-rack placement. Read `PartitionReassignmentCooldown` and `BrokerRecoveryCooldown`, then their integration in `AnomalyDetectorImpl`, followed by the model warm-up rules and tests.

The cooldown bounds optional scheduling; it does not certify storage or application recovery. The issue's measurements do not isolate this PR's individual latency benefit.

Related to AutoMQ#3669.

AI assistance: OpenAI Codex (GPT-6) was used to prepare the contribution split, adapt the changes to main, and update the tests and descriptions.

## Attribution

The initial partition reassignment cooldown was implemented by [@deniszh](https://github.com/deniszh).
@thiskian
thiskian force-pushed the kabdolahi/3669-autobalancer-recovery-cooldown branch from 1853921 to 98d273f Compare October 21, 2026 10:18
@thiskian
thiskian changed the base branch from 1.7 to main October 21, 2026 10:18

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants