Distrust the compromised eMudhra intermediates by borisschlosser · Pull Request #13055 · pulumi/registry · GitHub
Skip to content

Distrust the compromised eMudhra intermediates - #13055

Merged
jkodroff merged 2 commits into
masterfrom
borisschlosser/distrust-emudhra-intermediates
Oct 21, 2026
Merged

Distrust the compromised eMudhra intermediates#13055
jkodroff merged 2 commits into
masterfrom
borisschlosser/distrust-emudhra-intermediates

Conversation

@borisschlosser

@borisschlosser borisschlosser commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

The private keys of eight eMudhra intermediate CAs (six TLS, two code-signing) were generated with too little entropy and can be recovered from public certificate data. Browsers blocked them on 2026-10-17 and eMudhra revoked them. No misuse has been confirmed. This moves every toolchain the registry builds and publishes with to a release that distrusts them:

  • tools/resourcedocsgen/go.mod: add toolchain go1.26.10, the Go release the disclosure lists as fixed for 1.26. Every workflow that builds resourcedocsgen uses go-version-file, so CI moves with it. This morning's master build (run 39559241101) still set up Go 1.26.6, and resourcedocsgen fetches provider schemas over HTTPS with that binary.
  • bucket-cleanup.yml, test-ci-scripts.yml: read the toolchain from the same go.mod instead of a floating 1.26 spec, which setup-go resolves to whatever patch the runner has cached.
  • Node: pin 24.22.1, yesterday's security release that ships the NSS 3.131.1 root store without the three emSign roots, everywhere 24.x was used (this morning's run used cached 24.21.0) and on community-package-check.yml, which was still on Node 20.
  • mise.toml: pin both tools so mise install picks up the fixed releases locally.
  • registry-mirror-discover / registry-mirror-publish caches: the five actions/cache keys held only the mirror-tools commit, and the install step skips go install when a cached binary exists, so the toolchain bump would never have reached them. registry-mirror-publish is the primary HTTPS client to api.pulumi.com in publish.yml. Each setup-go step now has an id and its resolved go-version is part of the key, so the binaries rebuild on every toolchain change.
  • BUILD-AND-DEPLOY.md: version tables, plus a note that the Go and Node pins are exact and need bumping by hand on security releases, since Dependabot does not cover them.

Not changed, for the record:

  • Python scripts under scripts/ci/ run via uv run --with requests or pip install requests with no lock file, so they resolve the current certifi (2026.10.19, which drops the emSign roots) on every run.
  • Hugo 0.157.0 is a Go binary built before the fix and the pulumi-command shortcode fetches from api.github.com with it. No Hugo release built with a fixed Go exists yet (0.168.0 is from 2026-10-13). That one call is the residual exposure until Hugo ships one, tracked in Upgrade Hugo to a release built with a Go that distrusts the compromised eMudhra intermediates #13056.

References:

Test plan

  1. Automated checks
    • go mod tidy with Go 1.26.10: no diff beyond the toolchain line
    • go build ./..., golangci-lint run (0 issues) and go test ./... (all pass) in tools/resourcedocsgen
    • make lint-markdown: 0 errors
    • CI
  2. Manual checks
    • Ran a Go HTTP client against the disclosure's test host https://dv-g3b-emudhra.we-have-a-problem.com/ with the pre-fix Mozilla bundle (cacert-2026-09-25.pem) as the root pool, so Go's own verifier runs rather than the macOS one: go1.26.6 accepts the chain with HTTP 200, go1.26.10 rejects it with x509: no valid chains built
    • mise install resolves to go1.26.10 and v24.22.1

🤖 Generated with Claude Code

@borisschlosser @claude
Distrust the compromised eMudhra intermediates
2eb7f58 
The private keys of eight eMudhra intermediate CAs (six TLS, two code-signing) were generated with too little entropy and can be recovered from public certificate data. Browsers blocked them on 2026-10-17 and eMudhra revoked them. Go 1.26.10 and Node 24.22.1 are the releases that distrust them; this moves every toolchain the registry builds and publishes with to those releases.

- tools/resourcedocsgen/go.mod: add `toolchain go1.26.10`. Every workflow that builds resourcedocsgen reads `go-version-file`, so CI moves with it. This morning's master build still set up Go 1.26.6, and resourcedocsgen fetches provider schemas over HTTPS with that binary.
- bucket-cleanup.yml, test-ci-scripts.yml: read the toolchain from the same go.mod instead of a floating `1.26` spec that resolved to whatever patch the runner had cached.
- Node: pin 24.22.1 everywhere `24.x` was used (this morning's run used cached 24.21.0) and on community-package-check, which was still on Node 20.
- mise.toml: pin both tools so `mise install` picks up the fixed releases locally.

Verified with Go's own verifier against the disclosure's test host (https://dv-g3b-emudhra.we-have-a-problem.com/), using the pre-fix Mozilla bundle as the root pool: go1.26.6 accepts the chain with HTTP 200, go1.26.10 rejects it with "x509: no valid chains built".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisschlosser borisschlosser added the impact/no-changelog-required This issue doesn't require a CHANGELOG update label Oct 21, 2026
@claude

claude Bot commented Oct 21, 2026

Copy link
Copy Markdown

@borisschlosser borisschlosser mentioned this pull request Oct 21, 2026
Open
@borisschlosser
borisschlosser requested a review from a team October 21, 2026 12:32
@borisschlosser @claude
Key the registry-mirror binary caches on the Go version
b538808 
The registry-mirror-discover and registry-mirror-publish binaries are built with `go install` and cached under a key that held only REGISTRY_MIRROR_TOOLS_COMMIT. The install step skips the build whenever a cached binary exists, so a toolchain bump never reached them: after the Go 1.26.10 pin, CI would have kept restoring binaries built with 1.26.6, including registry-mirror-publish, the primary HTTPS client to api.pulumi.com in publish.yml.

Give each setup-go step an id and append its resolved go-version output to the five cache keys, so the binaries rebuild on every toolchain change. Using the resolved version rather than hashFiles(go.mod) avoids a rebuild on every dependency bump.

Also note in BUILD-AND-DEPLOY.md that the Go and Node pins are now exact and need bumping by hand on security releases, since Dependabot does not cover them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisschlosser

borisschlosser commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, the cache gap was real: every key held only the mirror-tools commit, and the install step skips go install when a cached binary exists, so the toolchain bump would never have reached registry-mirror-publish. Fixed in b538808.

I keyed on the resolved toolchain rather than hashFiles('tools/resourcedocsgen/go.mod'): each setup-go step now has id: setup-go and the five keys end in -go${{ steps.setup-go.outputs.go-version }}. That rebuilds on every Go bump without also rebuilding on every dependency bump in that go.mod. None of the five cache steps has restore-keys, so the old entries can't be restored through a prefix. The PR description now lists the binaries.

On the minor points:

  • Node pin staleness: added a note under the tool table in BUILD-AND-DEPLOY.md that the Go and Node pins are exact and need bumping by hand on security releases, since Dependabot doesn't cover node-version or the toolchain line.
  • community-package-check.yml on Node 24: that workflow is workflow_dispatch only, so this PR's CI doesn't exercise it, but its steps only run scripts/ci/community-package/cli.py under Python. Nothing in that script tree invokes Node, so the major bump is inert there.
  • Hugo: tracking issue is Upgrade Hugo to a release built with a Go that distrusts the compromised eMudhra intermediates #13056, labelled p1.

@github-actions

github-actions Bot commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Your site preview for commit b538808 is ready! 🎉

http://registry--origin-pr-13055-b5388087.s3-website.us-west-2.amazonaws.com/registry/

@jkodroff
jkodroff merged commit baaf692 into master Oct 21, 2026
17 checks passed
@jkodroff
jkodroff deleted the borisschlosser/distrust-emudhra-intermediates branch October 21, 2026 21:16
@github-actions

github-actions Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

This branch was successfully deployed

1 active deployment
testing — b5388087 Deployed Oct 21, 2026 by borisschlosser via Build and deploy preview #14740
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fnune fnune fnune approved these changes

Assignees

No one assigned

Labels

impact/no-changelog-required This issue doesn't require a CHANGELOG update

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@borisschlosser @fnune @jkodroff