[Nuclio] Fix missing S3 credentials for source archive with secret_env auto-mount by daanvdk-mck · Pull Request #10164 · mlrun/mlrun · GitHub
Skip to content

[Nuclio] Fix missing S3 credentials for source archive with secret_env auto-mount - #10164

Open
daanvdk-mck wants to merge 1 commit into
mlrun:developmentfrom
daanvdk-mck:bugfix/ML-13129-nuclio-s3-creds-env-from
Open

daanvdk-mck wants to merge 1 commit into
mlrun:developmentfrom
daanvdk-mck:bugfix/ML-13129-nuclio-s3-creds-env-from

Conversation

@daanvdk-mck

@daanvdk-mck daanvdk-mck commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

📝 Description

Deploying a with_repo=True serving function from an S3/MinIO project source still fails on MLRun CE with static credentials are empty, after the ML-11339 fixes (#9884, #10150). Those fixes resolve Nuclio's codeEntryAttributes S3 credentials from the storage auto-mount env vars that auto_mount_type=s3 produces. The CE chart default is auto_mount_type=secret_env with secret_name=storage-credentials and no keys, which mounts the whole secret through envFrom. Neither fix covers that case, so the access key, secret key and endpoint reach Nuclio empty.


🛠️ Changes Made

  • server/py/services/api/crud/runtimes/nuclio/helpers.py, _get_auto_mount_env_as_dict: when the scratch runtime's spec.env_from has a secretRef to the auto-mount secret_name, the secret's non-empty keys are now merged into the result. Values from spec.env (e.g. cleartext_env) take precedence, matching pod semantics. envFrom refs to other secrets are ignored, the same way [Nuclio] Fix missing S3 credentials for source archive when using auto-mount secret_name #10150 treats secretKeyRef.

✅ Checklist

  • I updated the documentation (if applicable)
  • I have tested the changes in this PR
  • I confirmed whether my changes are covered by system tests
    • If yes, I ran all relevant system tests and ensured they passed before submitting this PR
    • I updated existing system tests and/or added new ones if needed to cover my changes
  • If I introduced a deprecation:
  • Please run Smoke-tests workflow, providing it the PR number as input (upon success, the workflow run will add label "Smoke tests: Pass" to the PR)

🧪 Testing

  • New secret_env unit tests in test_nuclio.py:
    • CE default (secret_name, no keys): access key, secret key and endpoint are read from the secret.
    • cleartext_env overrides the same key from the secret.
    • builder_env still overrides secret values.
    • Empty secret: attributes stay empty.
  • The CE-default and precedence tests fail without the fix.
  • test_nuclio.py passes in full (300 tests) on the server test flavor (MLRUN_IS_API_SERVER=true).
  • Verified end to end on a CE install (chart 0.12.0, rc12 images, default storage auto-mount, no project secrets), using the ticket's repro (with_repo=True serving function, source s3://mlrun/project.tar.gz):
    • Unpatched rc12 API: fn.deploy() fails with static credentials are empty.
    • rc12 API with this fix: fn.deploy() succeeds and the function serves requests.

🔗 References


🚨 Breaking Changes?

  • Yes (explain below)
  • No

🔍️ Additional Notes

The same envFrom gap in the builder's fetch-source init container (Kaniko/Buildah) is fixed separately in #10167 (ML-13132).

…v auto-mount

On CE the chart default is auto_mount_type=secret_env with
secret_name=storage-credentials and no keys, which mounts the secret via
envFrom. _get_auto_mount_env_as_dict only harvested spec.env, so Nuclio's
codeEntryAttributes got empty S3 credentials and endpoint ("static
credentials are empty"). Also dereference envFrom refs to the auto-mount
secret_name, with env entries taking precedence as in a pod.

ML-13129

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

📊 Diff Coverage Report

📂 Click to view full coverage details

Diff Coverage

Diff: origin/development...HEAD, staged and unstaged changes

  • server/py/services/api/crud/runtimes/nuclio/helpers.py (100%)

Summary

  • Total: 4 lines
  • Missing: 0 lines
  • Coverage: 100%

@daanvdk-mck
daanvdk-mck marked this pull request as ready for review October 19, 2026 11:06
@daanvdk-mck
daanvdk-mck requested review from a team and Yacouby as code owners October 19, 2026 11:06
@daanvdk-mck
daanvdk-mck requested a review from eytand October 19, 2026 11:07

@Yacouby Yacouby left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The builder side has the same gap so a Kaniko/buildah source fetch under the CE default would hit the same thing. Is there a follow-up ticket for it?

@daanvdk-mck

Copy link
Copy Markdown
Contributor Author

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants