Repository navigation
Handle 409s from Rekor by fetching the entry - #647
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
@steiza @cmurphy @kommendorkapten Wanted to get some input on options here.
With Rekor v1, when we fetch an entry from one of the log's read APIs, the log returns a complete TLE structure. For Rekor v2, the client must reconstruct the TLE, since there is no endpoint. This is almost doable:
- The index is returned in both a custom header and the error message when responding with a 409
- The inclusion proof is constructed by fetching the relevant tiles computed with the index
- The canonicalized body and kind/version come from the entry bundle computed with the index
- The checkpoint comes from the log
Unfortunately, the log ID cannot be recomputed with anything from a Rekor v2 log. We've defined the log ID as the encoded public key hash, which comes from the trusted root (or in Rekor v1's case, it's either returned by the log or can be fetched from /api/v1/log). We aren't guaranteed to have the trusted root here.
The way we represent a log according to C2SP is a combination of the log's origin (e.g. log2025-1.rekor.sigstore.dev) and the first 4 bytes of its key ID as per c2sp.org/signed-note. Those values are included in the checkpoint.
So this will all be resolved going forward if we standardize on c2sp.org/tlog-proof and get rid of these redundant fields in the TLE. But what do we now?
Options:
- Rekor v2 uploads simply don't handle 409s
- Change the key lookup logic in pkg/verify/tlog to use the origin/key ID if there is no log ID present in the bundle. We also have to relax some of the constraints when parsing the bundle that mandate a log ID. I'd add a conformance test to verify if other clients can handle no log ID as well - some might, IIRC some just try to verify with all log keys.
- Inject the trusted root into the context higher up the call stack. If a trusted root is available, then fetch the entry and populate the log ID when the response is a 409.
Thoughts? Other options?
There was a problem hiding this comment.
Ignoring 409 will break certain flows, as the same entry can't be uploaded again, but the expected failure rate is low. How bad would it be if we resign then try again from a client side? For ECDSA this would work as the signatures are non-determinisitc, and getting a new signature would so create a new TLE, and so not hit by the deduplication. How bad of an idea is that?
If we think that is a terrible idea (to leave stray entries in the log) I would vote on injecting the trusted root, as verifying the received proof is before constructing the bundle is a good thing anyways.
There was a problem hiding this comment.
I don't think that's a significant ask. Unfortunately, I don't have an example of a pipeline where this is coming up, I assume it's from bespoke pipelines. Curious if anyone has hit this with artifact attentions or a trusted publishing workflow. Also, this hasn't come up for v2 yet since there is no real world usage.
My biggest concern is this ends up being a bigger effort because we produce bundles that aren't compatible across clients. I think it's most important to resolve in cosign,
I can fix this just for V1, since we know these errors are a server side flake occasionally. I can just leave a comment about we could go about resolving this for v2 if it becomes an issue.
There was a problem hiding this comment.
IIRC this has come up when the npm registry submits the publish attestation, due to the async nature of how they are processed/ack'ed it became an issue. But this is only due to internal failure, the processing pipeline could also be updated to "retry from the start" on failure.
As Rekor V2 will be a breaking change, I'm leaning into injecting the trust root for client side reconstruction, or a resign (with a small risk of stale entries).
There was a problem hiding this comment.
Just coming back to this - given the amount of code needed to rebuild the TLE, I'd lean towards just getting this change in as-is for Rekor v1 only if that's alright, since that flake can happen due to no fault of the signer.
e85bb06 to
c443e1e
Compare
When a CI pipeline fails midway, a client may try to upload the same signed entry again. This will result in Rekor returning a 409. Additionally, occasionally Rekor v1 fails with a 409 - the root cause isn't known, it's a flake in Trillian. This change gracefully handles a 409 Conflict by retrieving the transparency log entry. For Rekor v1, this is simple, just use the UUID in the response to lookup the entry by UUID. Rekor v2 is not handled because we aren't able to fully construct the TLE without the log's public key, which is the TLE's LogID. A comment has been added with details on how someone could implement it in the future. Signed-off-by: Hayden <8418760+Hayden-IO@users.noreply.github.com>
Cosign 3.1.3 aborts when Rekor v1 reports that the signed entry already exists, before uploading the signature to the registry. Build the same pinned Cosign source with released `sigstore-go` 1.3.0 ([upstream fix](sigstore/sigstore-go#647)), which retrieves the existing log entry and completes the bundle. Both image workflows verify source checksums and the built dependency versions, then retain mandatory signature and SBOM attestation verification. The Grype database producer and restore actions now share a run-ID-only cache key. Failed-only reruns reuse the successful producer's database; cache misses, stale databases and critical vulnerabilities with an available fix still fail. No official fixed Cosign release exists as of 2026-10-13, so this uses the approved isolated source-build fallback. It adds compilation time (104 seconds for the first local build, 16 seconds cached); the overall image-job budget accommodates this while signing deadlines stay unchanged. The engine's dependency module is unchanged. Validation: - Real signing-dependency conflict owner fails with 1.2.2 and passes with 1.3.0; upload/fetch errors and malformed evidence remain failures. - Cache owner fails on the original keys for attempts 2/3 and passes with stable keys and separate-run isolation. - Shell owner covers required signing/verification operations, signer constraints and timeout exit propagation. - Installer rejects a tampered checksum before building; installed binary contains the pinned versions. - Repository guards and Python check group pass (314 script tests). Fixes THE-1524. Independent review: `review-code-dev` v2 / Alibaba delegation passed, all ten files inspected, zero findings. Latest-head CI: **35 passed, 2 skipped** in the PR check display, including all thirteen real OCI image sign/attest/verify and vulnerability checks. [Verification run](https://github.com/The-Vibe-Company/quivr/actions/runs/38345125999). Engine-job logs confirm pinned artifact versions and a 150-second installer. Head: `a168639c913ac75c2108d81718d30455192869ca`. GitHub reports clean and mergeable.
When a CI pipeline fails midway, a client may try to upload the same
signed entry again. This will result in Rekor returning a 409.
Additionally, occasionally Rekor v1 fails with a 409 - the root cause
isn't known, it's a flake in Trillian.
This change gracefully handles a 409 Conflict by retrieving the
transparency log entry. For Rekor v1, this is simple, just use the UUID
in the response to lookup the entry by UUID.
Rekor v2 is not handled because we aren't able to fully construct the
TLE without the log's public key, which is the TLE's LogID. A comment
has been added with details on how someone could implement it in the
future.