[Bug] Forward namespace to KFP create_experiment in multi-user mode by moutazmuhammad · Pull Request #10113 · mlrun/mlrun · GitHub
Skip to content

[Bug] Forward namespace to KFP create_experiment in multi-user mode - #10113

Open
moutazmuhammad wants to merge 9 commits into
mlrun:developmentfrom
moutazmuhammad:fix/kfp-namespace-multiuser
Open

moutazmuhammad wants to merge 9 commits into
mlrun:developmentfrom
moutazmuhammad:fix/kfp-namespace-multiuser

Conversation

@moutazmuhammad

@moutazmuhammad moutazmuhammad commented Sep 17, 2026 •

Copy link
Copy Markdown

📝 Description

When running MLRun in multi-user mode (with Kubeflow Pipelines + Dex/OIDC authentication), calling create_pipeline results in a 403 Forbidden error from KFP. This happens because the namespace parameter is never passed through to KFP's create_experiment() call, so KFP defaults to the kubeflow namespace where the user has no permissions.

Fixes #9960.


🛠️ Changes Made

  • server/py/services/api/api/endpoints/pipelines.py: added an optional namespace query parameter to the create_pipeline endpoint and forwarded it to the internal _create_pipeline() helper.
  • server/py/services/api/crud/pipelines.py: Pipelines.create_pipeline() now accepts namespace and uses it both to initialize the KFP client (_initialize_kfp_client(namespace)) and when calling kfp_client.create_experiment(name=experiment_name, namespace=namespace), so the experiment is created in the caller's profile namespace instead of the default one.

✅ Checklist

  • I updated the documentation (if applicable)
  • I have tested the changes in this PR
  • I confirmed whether my changes are covered by system tests
    • If yes, I ran all relevant system tests and ensured they passed before submitting this PR
    • I updated existing system tests and/or added new ones if needed to cover my changes
  • If I introduced a deprecation:
  • Please run Smoke-tests workflow, providing it the PR number as input (upon success, the workflow run will add label "Smoke tests: Pass" to the PR)

🧪 Testing

  • Verified namespace is correctly threaded end-to-end: endpoint → _create_pipeline → Pipelines.create_pipeline → _initialize_kfp_client(namespace) and kfp_client.create_experiment(namespace=namespace).
  • ruff check and ruff format --check pass on both changed files.
  • Validated in production on GKE and EKS clusters running MLRun 1.10.0 in multi-user mode with KFP 2.x and Dex authentication: before the fix, project.run(workflow_path=...) from a Jupyter notebook in a user profile namespace returned 403 Forbidden from KFP; after the fix, the pipeline is created successfully in the user's profile namespace.

🔗 References


🚨 Breaking Changes?

  • Yes (explain below)
  • No

The namespace parameter is optional and defaults to None, preserving existing behavior for single-user deployments.

…mode

Fixes mlrun#9960. create_pipeline now threads an optional namespace query
param through _create_pipeline and Pipelines.create_pipeline to
kfp_client.create_experiment and client initialization, so KFP creates
the experiment in the caller's profile namespace instead of defaulting
to the wrong one.
@moutazmuhammad
moutazmuhammad requested a review from a team as a code owner September 17, 2026 11:01
@moutazmuhammad moutazmuhammad changed the title [Pipelines] Forward namespace to KFP create_experiment in multi-user … [Fix] Forward namespace to KFP create_experiment in multi-user … Sep 17, 2026
@moutazmuhammad moutazmuhammad changed the title [Fix] Forward namespace to KFP create_experiment in multi-user … [Pipelines] Forward namespace to KFP create_experiment in multi-user mode Sep 17, 2026
@moutazmuhammad moutazmuhammad changed the title [Pipelines] Forward namespace to KFP create_experiment in multi-user mode [Bug] Forward namespace to KFP create_experiment in multi-user mode Sep 17, 2026
@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Smoke Tests with the PR synced to development HEAD 3d38e5b9: Pass ✅

@github-actions

Copy link
Copy Markdown
Contributor

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: create_pipeline does not forward namespace to KFP create_experiment in multi-user mode — causes 403 Forbidden

2 participants