Checkpoint only fully applied binlog transactions by coding-chimp · Pull Request #1790 · github/gh-ost · GitHub
Skip to content

Checkpoint only fully applied binlog transactions - #1790

Open
coding-chimp wants to merge 1 commit into
github:masterfrom
Shopify:bb/applied-transaction-checkpoint
Open

coding-chimp wants to merge 1 commit into
github:masterfrom
Shopify:bb/applied-transaction-checkpoint

Conversation

@coding-chimp

Copy link
Copy Markdown
Contributor

Description

Fix a risk of checkpointing a partially applied binlog transaction. Multiple row changes can share the same binlog coordinates, so applying one does not prove that all have reached the ghost table. Resuming from such a checkpoint can skip unapplied changes.

  • Send ordered transaction-completion markers through the existing reader, dispatcher, and apply queues.
  • Advance a separate applied-transaction watermark only after preceding destination SQL commits. Persist that completed boundary rather than sampled reader or heartbeat progress.
  • Preserve bounded queues and DML batching across markers. Failed SQL batches cannot acknowledge collected markers.
  • Stop synchronous dispatch after listener errors and reject marker acknowledgement after abort. Make reader operations, checkpoint waits, and checkpoint SQL observe migration cancellation.
  • Clone MySQL and MariaDB GTID coordinates so later mutations cannot change recorded transaction boundaries.

In case this PR introduced Go code changes:

  • contributed code is using same conventions as original code
  • script/cibuild returns with no formatting errors, build errors or unit test errors.

Route ordered completion markers through the existing DML queues and persist a separate applied-transaction watermark only after preceding SQL commits. Preserve batching and reject acknowledgements after failed delivery or migration cancellation.

Make reader operations and checkpoint SQL cancellation-aware. Cover multi-row events, MySQL/MariaDB GTID cloning, partial-apply replay, failed batches, and cancellation during both checkpoint polling and an in-flight INSERT.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant