{{ message }}
DeltaLake: replace cached snapshot on update() so kernel sees fresh creds#107480
Merged
Conversation
…reds The fresh snapshot rebuilt inside `update()` already captures the C++ S3 client's current creds via `createBuilder` keep it instead of discarding it when the version matches, so the next kernel op runs on a freshly-credentialed engine and doesn't outlive the STS TTL.
Contributor
clickhouse-gh
Bot
added
the
pr-bugfix
![clickhouse-gh[bot]](https://avatars.githubusercontent.com/in/928874?s=60&v=4){kind=small}
kssenii
reviewed
Jun 16, 2026
Compare a SipHash128 of the C++ S3 client's current credentials against the one captured when kernel_snapshot_state was built, and rebuild only the kernel engine + snapshot + scan when they differ. Schema, stats, and the snapshot LRU cache are preserved across rebuilds.
kssenii
approved these changes
Jun 16, 2026
ahmadov
added
v26.2-must-backport
v26.3-must-backport
v26.4-must-backport
v26.5-must-backport
pr-must-backport
The previous wording change broke test_database_delta::test_complex_table_schema's contains_in_log assertion. Split the init and rebuild branches so the init path emits the original message verbatim.
1 task
kssenii
approved these changes
Jun 18, 2026
Vended catalog credentials (Glue / Unity / REST) are static in the C++
S3 client until `credentials_refresh_callback` fires, but delta-kernel's
Rust object_store bypasses the ClickHouse error handlers that normally
invoke it. So an expired vended session surfaces as `DELTA_KERNEL_ERROR`
with `ExpiredToken` and never recovers.
Plumb the callback through `IObjectStorage::tryRefreshCredentialsViaCallback`
to `IKernelHelper::refreshCredentials`, and add a one-shot retry on
stale-token errors at both kernel touchpoints:
- `TableSnapshot::initOrUpdateSnapshot` (engine build)
- `Iterator::scanDataFunc` (per-batch listing, guarded by
`total_data_files == 0` so we don't replay already-emitted paths)
This was referenced Jun 22, 2026
Contributor
robot-clickhouse-ci-1
added
the
pr-must-backport-synced
ahmadov
removed
pr-synced-to-cloud
robot-ch-test-poll2
added
the
pr-must-backport-synced
fm4v
removed
pr-must-backport
robot-clickhouse-ci-1
added
the
pr-must-backport-synced
robot-ch-test-poll
added a commit
that referenced
this pull request
Jun 25, 2026
Cherry pick #107480 to 26.3: DeltaLake: replace cached snapshot on update() so kernel sees fresh creds
robot-clickhouse
added a commit
that referenced
this pull request
Jun 25, 2026
…e() so kernel sees fresh creds
robot-ch-test-poll
added a commit
that referenced
this pull request
Jun 25, 2026
Cherry pick #107480 to 26.4: DeltaLake: replace cached snapshot on update() so kernel sees fresh creds
robot-clickhouse
added a commit
that referenced
this pull request
Jun 25, 2026
…e() so kernel sees fresh creds
robot-ch-test-poll
added a commit
that referenced
this pull request
Jun 25, 2026
Cherry pick #107480 to 26.5: DeltaLake: replace cached snapshot on update() so kernel sees fresh creds
robot-clickhouse
added a commit
that referenced
this pull request
Jun 25, 2026
…e() so kernel sees fresh creds
robot-clickhouse-ci-2
added
the
pr-backports-created
alexey-milovidov
added a commit
that referenced
this pull request
Jun 27, 2026
Backport #107480 to 26.5: DeltaLake: replace cached snapshot on update() so kernel sees fresh creds
alexey-milovidov
added a commit
that referenced
this pull request
Jun 27, 2026
Backport #107480 to 26.4: DeltaLake: replace cached snapshot on update() so kernel sees fresh creds
alexey-milovidov
added a commit
that referenced
this pull request
Jun 27, 2026
Backport #107480 to 26.3: DeltaLake: replace cached snapshot on update() so kernel sees fresh creds
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes https://github.com/ClickHouse/support-escalation/issues/7739.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
The fresh snapshot rebuilt inside
update()already captures the C++ S3 client's current creds viacreateBuilderkeep it instead of discarding it when the version matches, so the next kernel op runs on a freshly-credentialed engine and doesn't outlive the STS TTL.Note that besides DeltaLake, this also changes
AwsAuthSTSAssumeRoleCredentialsProvider::GetAWSCredentials/Reloadto honor and clearSetNeedRefresh()(matching the Web-Identity and SSO providers), which affects any S3 access using STS assume-role on auth retries. Describe the newIObjectStorage::tryRefreshCredentialsViaCallbackvirtual and the credentials-fingerprint rebuild path inTableSnapshot.Version info
26.6.1.110126.5.4.19,26.4.5.74,26.3.16.15