fix(commit): align TableCommit validation and retries with Java by JingsongLi · Pull Request #913 · apache/paimon-rust · GitHub
Skip to content

fix(commit): align TableCommit validation and retries with Java - #913

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/table-commit-java-parity
Sep 22, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/table-commit-java-parity

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Align Rust TableCommit with Java for the currently supported commit paths. Reject conflicting deletion vectors and indexed-column updates, use authoritative catalog metadata, and recover safely when a snapshot publication response is lost.

Brief change log

  • Reject duplicate deletion vectors and stale deletion-vector replacements, including mixed data/index commits. Classify deletion-vector commits as OVERWRITE consistently with retry deduplication.
  • Apply indexed-column update protection even when the same batch deletes an unrelated index.
  • Read the latest snapshot and schema from REST catalogs; use the latest persisted schema for filesystem tables while preserving each data file's writer schema.
  • Inherit watermark and statistics only when the statistics file's schema is compatible.
  • Clean newly prepared metadata on preparation failure. Retry publication errors and retain files when publication may already have succeeded, including guarded index commits.
  • Remove the unused manifest.merge-min-count option and accessor. Verify that historical manifests remain unchanged across 35 consecutive commits.

Filesystem snapshot publication mechanics are outside this change.

Tests

Added 14 regression tests covering conflict validation, snapshot metadata, REST commits, publication response loss, preparation cleanup, and preservation of small manifests.

  • cargo test --offline --locked -p paimon --all-targets --features fulltext,vortex: 3,394 passed, 6 ignored.
  • cargo clippy --offline --locked --all-targets --workspace --features fulltext,vortex -- -D warnings: passed.
  • cargo fmt --all -- --check and git diff --check: passed.

API and Format

Remove CoreOptions::manifest_merge_min_count() and its configuration parsing; the option did not affect Rust commit behavior. Snapshot and manifest serialization formats are unchanged. Deletion-vector commits now use the existing OVERWRITE kind.

Documentation

Removed the obsolete option's API documentation. No other documentation references remain.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed ff0e4bc. No blocking findings.

Checked deletion-vector conflict validation and OVERWRITE retry identity, indexed-column protection with unrelated index deletions, authoritative REST snapshot/schema lookup, schema-compatible statistics and watermark inheritance, and ownership of metadata during preparation failures versus uncertain publication results.

Local validation:

  • 87 TableCommit tests passed, including the 14 new parity regressions.
  • Core library suite with default features: 3,044 passed, 6 ignored.
  • cargo fmt --all -- --check passed.
  • An additional temporary regression passed for a lost publication response followed by another writer committing: the retry finds its earlier committed snapshot before the expected-snapshot guard, publishes no duplicate, and preserves both writers data.

The full CI matrix is still running; no failed checks at review time. Filesystem snapshot publication mechanics remain outside this PR, as documented.

@JingsongLi
JingsongLi merged commit ef540b1 into apache:main Sep 22, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants