### 📝 Description
Adds the dedicated, SA-gated `/api/v1/follower/projects/*` surface
(ML-12901) that lets Orca drive MLRun as a follower in the enterprise
project-sync epic: a two-phase-commit create/delete, a
fire-and-forget-with-CAS update, and a paginated list-states read for
reconciliation, all filtered/paginated at the DB layer. Builds on the
`op_id`/`phase`/`updated_at` columns already added to the projects table
(data-model PR #9639) — this PR is the first to actually write and read
them. Legacy user-facing `/api/v1/projects` endpoints and the CE
(`leader=mlrun`) path are untouched — this surface only matters once an
Orca leader is configured.

---

### 🛠️ Changes Made
- New `follower_contract.py`: a dependency-free
CAS/UUIDv7-ordering/state-machine module implementing the Follower
Contract's per-operation validation order and request grid — kept
storage/transport-agnostic so it can move into a shared cross-follower
library later.
- New `follower_schemas.py`: the wire request/response pydantic models
for this surface. Not in `mlrun.common.schemas` — that package is the
public SDK↔server contract, and the SDK never talks to this surface.
- New `projects_follower.py` router: `prepare-create`, `commit-create`,
`update` (PUT), `prepare-delete`, `commit-delete` (DELETE), and
`list-states` (GET, keyset-paginated at the DB layer).
- New `authenticate_leader_request` auth dependency
(`framework/api/deps.py`) — builds on the existing igz
session-verification flow (`AuthInfo.is_service_account()`), rejects
anything that isn't the configured leader's service account.
- The 5 2PC hooks (`prepare_create_project`, `commit_create_project`,
`prepare_delete_project`, `commit_delete_project`,
`update_project_follower`) are implemented directly on
`services.api.crud.projects.Projects`, not declared on the abstract
`follower.Member` interface — the endpoint always calls the concrete
class, never polymorphically, so the interface (and
`nop_follower`/`nuclio`/`iguazio/v4`) carries no stubs for them.
- Each hook takes a row lock (`for_update`) on its CAS pre-check read,
and persists `op_id`/`phase`/`updated_at` onto the project row alongside
`state` (`SQLDB.create_project`/`store_project`/`patch_project`).
- `updated_at` is a required field on every op (confirmed always sent by
Orca per the Follower Contract HLD):
`FollowerCommitCreateRequest`/`FollowerPrepareDeleteRequest` declare it
directly; `prepare-create`/`update`'s `project` payload is validated via
a new `_project_updated_at` helper, mirroring the existing
`_project_op_id`.
- New `httpdb.projects.follower.leader_identity` config key; new
`commit_delete_follower_project` on the chief client. `commit-delete`
blocks synchronously until the purge actually finishes (confirmed
requirement from the Orca feature owner) — no background task, no early
202 — so it's the one route needing chief re-routing (schedule deletion
is chief-only) and a timeout sized to the full `delete_project`
operation rather than the legacy chief-proxy default.

---

### ✅ Checklist
- [ ] I updated the documentation (if applicable)
- [x] I have tested the changes in this PR
- [ ] I confirmed whether my changes are covered by system tests
- [ ] If yes, I ran all relevant system tests and ensured they passed
before submitting this PR
- [ ] I updated existing system tests and/or added new ones if needed to
cover my changes
- [ ] If I introduced a deprecation:
  - [ ] I followed the [Deprecation Guidelines](./DEPRECATION.md)
  - [ ] I updated the relevant Jira ticket for documentation
- [ ] Please run
[Smoke-tests](https://github.com/mlrun/mlrun/actions/workflows/smoke-tests.yml)
workflow, providing it the PR number as input (upon success, the
workflow run will add label "Smoke tests: Pass" to the PR)

---

### 🧪 Testing
- 50 unit tests added and passing under pytest: full
CAS/ordering/state-machine coverage (`test_follower_contract.py`, 47
tests) + SA-gating coverage (`test_deps.py`, 3 tests).
- 21 more unit tests for the 5 crud-level 2PC hooks,
`get_follower_project_snapshot`, and `list_project_states`
(`test_projects.py`), verified passing via a manual harness — they can't
run under plain `pytest` in my dev venv because
`services/api/tests/unit/conftest.py` imports the full app, which hits a
pre-existing, unrelated pydantic v1-vs-v2 gap in that venv (confirmed
pre-existing: it breaks untouched tests the same way). Should run
normally in CI.
- No system tests added — this surface has no caller yet (Orca's MLRun
follower client, ORIS-3535, is a separate, not-yet-merged change in the
`orca` repo).

---

### 🔗 References
- Ticket link: https://ecliptos.atlassian.net/browse/ML-12901
- Design docs links:
- https://ecliptos.atlassian.net/wiki/spaces/IRG/pages/654737417
(Follower Contract HLD)
- https://ecliptos.atlassian.net/wiki/spaces/IRG/pages/611745844
(Project Sync Mechanism — Backend HLD)
- https://ecliptos.atlassian.net/wiki/spaces/IRG/pages/661979181
(Followers sub-page — MLRun section)
- External links:

---

### 🚨 Breaking Changes?

- [ ] Yes (explain below)
- [x] No

---

### 🔍️ Additional Notes
- Orca's MLRun follower client (ORIS-3535) depends on this branch
merging first — it's the actual caller of this surface.
- ML-12902 (mode-aware backward-compatibility proxy for the legacy
`/api/v1/projects` endpoints) is a separate, not-yet-started ticket;
deliberately out of scope here.
- `commit-delete`'s blocking behavior was confirmed directly with the
Orca feature owner, rather than assumed from the (still-open) Backend
HLD question on sync-vs-background cleanup.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: moranbental <107995850+moranbental@users.noreply.github.com>

### 📝 Description
Adds a single-record counterpart to the follower `list-states` endpoint:
`GET /follower/projects/states/{name}` returns the sync state of one
named project, so a caller that only needs one project's state doesn't
have to page through `list-states` to find it.

---

### 🛠️ Changes Made
- New `GET /follower/projects/states/{name}` route in
`projects_follower.py`, returning a `FollowerProjectState` for a single
project.
- Reuses the existing `Projects().get_follower_project_snapshot()`
(name/op_id/state) projection — the same one the 2PC hooks use for their
CAS pre-check reads — rather than adding a new crud method.
- Returns 404 (`MLRunNotFoundError`) when the project doesn't exist.
- Inherits the router's existing SA-gated `authenticate_leader_request`
auth automatically (applied at `include_router`, not per-route).

---

### ✅ Checklist
- [ ] I updated the documentation (if applicable)
- [ ] I have tested the changes in this PR
- [ ] I confirmed whether my changes are covered by system tests
- [ ] If yes, I ran all relevant system tests and ensured they passed
before submitting this PR
- [ ] I updated existing system tests and/or added new ones if needed to
cover my changes
- [ ] If I introduced a deprecation:
  - [ ] I followed the [Deprecation Guidelines](./DEPRECATION.md)
  - [ ] I updated the relevant Jira ticket for documentation
- [ ] Please run
[Smoke-tests](https://github.com/mlrun/mlrun/actions/workflows/smoke-tests.yml)
workflow, providing it the PR number as input (upon success, the
workflow run will add label "Smoke tests: Pass" to the PR)

---

### 🧪 Testing
- `ruff check` / `ruff format --check` pass on the changed file.
- Ran the existing crud-layer suite
(`services/api/tests/unit/crud/test_projects.py`, 29/29 passing)
covering `get_follower_project_snapshot()`'s found/not-found behavior,
which this endpoint calls directly.
- No new automated test exercises the route itself over HTTP — none of
the sibling follower routes (`prepare-create`, `update`, `list-states`,
etc.) have router-level tests either, so this follows the existing
coverage pattern rather than introducing a new one.

---

### 🔗 References
- Ticket link: https://ecliptos.atlassian.net/browse/ML-13069
- Design docs links:
- External links:

---

### 🚨 Breaking Changes?

- [ ] Yes (explain below)
- [x] No

---

### 🔍️ Additional Notes
- Stacked on `feature/project-sync-follower` (not yet merged) — base
this PR against that branch, not `development`.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

…10087)

### 📝 Description
`commit_delete_project` — the `/follower/projects/{name}` DELETE hook
Orca calls to purge a project's resources (ML-12901) — blocked the HTTP
request until the full resource cascade finished. This converts it to
schedule (or reuse) MLRun's existing project-deletion background task
instead, so the request returns quickly. The project row stays present
(`state=deleting`) until the task actually finishes — only *when* the
HTTP response returns changes, not the deletion semantics.

---

### 🛠️ Changes Made
- `crud/projects.py`: `commit_delete_project` now validates (unchanged
CAS/ordering/state logic) then schedules/reuses a background task via
`InternalBackgroundTasksHandler` instead of calling `delete_project`
inline. New `_purge_follower_project_resources` is the task body, run on
its own fresh DB session.
- Deliberately does **not** reuse
`framework.api.utils.get_or_create_project_deletion_background_task`:
its deletion step resolves through MLRun's own leader/follower selection
(`project_member.get_project_member()`), which would forward the delete
request back to the leader instead of purging locally — wrong here,
since the leader has already decided by the time it calls this hook.
Reuses the same background-task *kind* as that helper for cross-path
dedup, but always deletes locally.
- Handles the create-vs-active-check TOCTOU race: a concurrent loser now
folds into the reuse path instead of surfacing an uncaught
`MLRunConflictError`.
- `api/endpoints/projects_follower.py`: the endpoint returns once the
task is scheduled/reused (200 if the project was already gone, 202
otherwise), scheduling the returned callable via FastAPI's
`BackgroundTasks`.
- `framework/utils/projects/follower_schemas.py`: new
`FollowerCommitDeleteResponse(FollowerProjectState)` adding
`background_task_name: str | None`, scoped to this one response (not the
shared `FollowerProjectState` every other op returns) — an additive
field, not required by any existing caller.
- `framework/utils/clients/chief.py`: docstring fix on the chief-proxy
passthrough (no longer claims to block for the full purge), and drops
the special-cased 900s timeout -
`background_tasks.default_timeouts.operations.delete_project`, the
background task's own run budget, not the right knob for how long the
now-fast synchronous scheduling call should take - in favor of the
ordinary chief-proxy default, matching `build_function`'s call.

---

### ✅ Checklist
- [ ] I updated the documentation (if applicable)
- [x] I have tested the changes in this PR
- [x] I confirmed whether my changes are covered by system tests
- [x] If yes, I ran all relevant system tests and ensured they passed
before submitting this PR
- [x] I updated existing system tests and/or added new ones if needed to
cover my changes
- [ ] If I introduced a deprecation:
  - [ ] I followed the [Deprecation Guidelines](./DEPRECATION.md)
  - [ ] I updated the relevant Jira ticket for documentation
- [ ] Please run
[Smoke-tests](https://github.com/mlrun/mlrun/actions/workflows/smoke-tests.yml)
workflow, providing it the PR number as input (upon success, the
workflow run will add label "Smoke tests: Pass" to the PR)

---

### 🧪 Testing
- `server/py/services/api/tests/unit/crud/test_projects.py`: 6 unit
tests covering the new scheduling/dedup/no-op/create-race/purge-strategy
behavior, including a rewritten test asserting a same-`op_id` retry now
*reuses* the active task instead of re-running the purge. All 6 passing.
- No router-level (FastAPI endpoint) test added — the crud-level tests
already cover the new scheduling/dedup/no-op/purge-strategy logic; the
endpoint itself only adds thin status-code/scheduling glue on top.
- `ruff check`/`ruff format --check` and `lint-imports` clean on all
touched files.
- **Live end-to-end verification** on a lab cluster with Orca configured
as leader (`httpdb.projects.leader=orca`): drove `prepare-create` →
`commit-create` → `prepare-delete` → `commit-delete` directly against
`/api/v1/follower/projects/*`, exercising all three response paths this
diff changes:
1. `commit-delete` schedules a new task → `202` with a fresh
`background_task_name`.
2. An immediate second `commit-delete` on the same `op_id`, fired with
no delay while the first task is still running → `202` with the
**identical** `background_task_name`; server logs confirm only one
correlation-id ran the actual delete cascade (no double purge).
3. A `commit-delete` after the project has already fully disappeared →
`200` with `background_task_name: None`.

Also confirmed the project stays listed as `deleting` immediately after
a `202` response and disappears once the background task's cascade (DB
rows, downstream project cleanup, secrets, configmaps) completes — logs
clean, no errors, no unhealthy pods.

---

### 🔗 References
- Ticket link: https://ecliptos.atlassian.net/browse/ML-12904
- Design docs links:
- https://ecliptos.atlassian.net/wiki/spaces/IRG/pages/654737417
(Follower Contract HLD)
- https://ecliptos.atlassian.net/wiki/spaces/IRG/pages/611745844
(Project Sync Mechanism — Backend HLD)
- https://ecliptos.atlassian.net/wiki/spaces/IRG/pages/661979181
(Followers sub-page — MLRun section)
- External links:

---

### 🚨 Breaking Changes?

- [ ] Yes (explain below)
- [x] No

The `/follower/projects/{name}` DELETE response gains a field
(additive); existing callers are unaffected.

---

### 🔍️ Additional Notes
- Verified the shorter chief-proxy timeout can't cause a double purge:
`commit_delete_project` always re-checks for an active task before
scheduling (reuse-or-create-with-race-handling, covered above), and
since the background task runs in-process on chief with no separate
worker/queue, a chief crash mid-purge kills the task and its tracking
together - a retry after restart just re-runs the same idempotent
(query-then-delete-what's-found) cascade sequentially, never
concurrently.
- Two review findings from this PR's quality gate were left as-is (not
fixed) since they're pre-existing characteristics of the reused
background-task machinery, not introduced by this diff: (1) a task whose
internal timeout just flipped it to `failed` is still reported as
"active" by `get_active_background_task_by_kind`, so a retry can
silently skip re-purging until a later attempt; (2) the project row's
`deletion_background_task_name` DB column isn't populated on this path
(it has no read path anywhere in the codebase today, so purely
cosmetic).

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

### 📝 Description
Follower Contract HLD checklist item 10 (ML-12905): on boot (enterprise
mode, `leader=orca`), MLRun triggers Orca's leader-side reconciliation
sweep for itself (`POST /api/v1/projects/followers/sync`, MLRun's own SA
— not a user-identity relay) and polls the resulting Trackable Action to
a terminal state. MLRun never reads a states list or computes its own
diff — Orca pushes truth back onto MLRun's existing
`/follower/projects/*` surface (ML-12901) while the sweep runs.

**Only the chief triggers the sweep.** Workers converge on the same
readiness signal by asking the chief (new `GET
/api/v1/followers-sync-status`, mirroring the existing
`clusterization-spec` chief-proxy pattern) instead of each independently
triggering their own, redundant sweep against Orca.

**Scope note:** this PR is the trigger/poll machinery only. It tracks
readiness (a `threading.Event`, `is_followers_sync_ready()`) but nothing
consumes it for resource-creation blocking yet — the gate that reads
this signal (checklist item 9, plus the blocking half of item 10) is a
separate, follow-up PR, stacked on this branch, to be opened once this
merges.

**Verified live end-to-end** against a real implementation of orca's
trigger endpoint (`feature/ORIS-3532-trigger`) on a real lab: the full
chain — chief trigger → real Orca sweep → real callbacks into MLRun's
own `/follower/projects/*` endpoints → poll → chief readiness →
worker-proxied readiness — all confirmed working. Two wire-format bugs
(explicitly flagged as unverified guesses in the original code) were
found and fixed this way: the trigger response field is `correlationId`
(top-level), not `opId`/`status.opId`; the real action type is
`"follower-sync"`, not `"sync-followers"`.

Note: the ticket text describes startup-sync as a paginated `GET
/api/v1/project-states` pull-and-diff. That endpoint doesn't exist and
contradicts the current, decided HLD (same class of stale-ticket-text
issue ML-12903 hit) — this PR implements the HLD's actual, current
design instead.

---

### 🛠️ Changes Made
- `framework/utils/clients/iguazio/v4.py`: new
`Client.trigger_followers_sync()` (SA-authenticated POST +
poll-to-terminal, mirrors the existing project-CUD trigger/poll pattern)
and its supporting
`_send_self_authenticated_request`/`_wait_for_followers_sync`/`_verify_followers_sync_terminal`.
- `framework/utils/projects/follower.py`: chief-only
`_start_periodic_followers_sync_trigger()` and worker-only
`_start_periodic_followers_sync_wait_for_chief()`, both registered from
`start()` via the existing
`framework.utils.periodic.run_function_periodically` infra (the same
mechanism `_start_periodic_sync` and `framework/service/__init__.py`'s
`_start_chief_clusterization_spec_sync_loop` already use for analogous
chief/worker background polling) instead of a hand-rolled
`threading.Thread` + retry loop. Backoff-with-logging on failure comes
for free from that infra; each function cancels its own periodic slot
once it succeeds. New `is_followers_sync_ready()` accessor.
- `framework/utils/projects/member.py`: `is_followers_sync_ready()`
no-op-true default on the abstract base, so CE/leader mode (and any
polymorphic caller that doesn't know which mode is running) is
unaffected.
- `framework/utils/clients/chief.py`: new `is_followers_sync_ready()` —
asks the chief via the new endpoint.
- `services/api/api/endpoints/followers_sync_status.py` (new): `GET
/followers-sync-status`, chief answers locally, worker re-routes to
chief. Registered in `api.py`; added to the `verify_api_state`
allowlists in `deps.py` alongside `clusterization-spec` (same
"clusterization purposes" category).

---

### ✅ Checklist
- [ ] I updated the documentation (if applicable)
- [x] I have tested the changes in this PR
- [ ] I confirmed whether my changes are covered by system tests
- [ ] If yes, I ran all relevant system tests and ensured they passed
before submitting this PR
- [ ] I updated existing system tests and/or added new ones if needed to
cover my changes
- [ ] If I introduced a deprecation:
  - [ ] I followed the [Deprecation Guidelines](./DEPRECATION.md)
  - [ ] I updated the relevant Jira ticket for documentation
- [ ] Please run
[Smoke-tests](https://github.com/mlrun/mlrun/actions/workflows/smoke-tests.yml)
workflow, providing it the PR number as input (upon success, the
workflow run will add label "Smoke tests: Pass" to the PR)

---

### 🧪 Testing
- New unit tests: `test_iguazio_v4.py` (+4: trigger success/SA-auth,
missing-correlationId, poll-timeout, action-failed),
`test_follower_member.py` (+8: non-orca-leader-ready-immediately,
chief/worker `initialize()`+`start()` integration, trigger
sets-ready-and-cancels-on-success, trigger leaves-pending-on-failure,
wait-for-chief sets-ready-and-cancels-when-ready, wait-for-chief
stays-pending-when-not-ready, `is_followers_sync_ready` reflects the
event), `test_chief.py` (+1: `is_followers_sync_ready` round-trip).
- Live end-to-end verification on a real lab against orca's
`feature/ORIS-3532-trigger`: full trigger → sweep → callback → poll →
readiness chain confirmed working, including the chief/worker
readiness-consistency fix (both report identical readiness, worker via
the new chief-proxy endpoint, not an independent check).

---

### 🔗 References
- Ticket link: https://ecliptos.atlassian.net/browse/ML-12905
- Design docs links: [Follower Contract
HLD](https://ecliptos.atlassian.net/wiki/spaces/IRG/pages/654737417)
(checklist item 10) · [Followers
page](https://ecliptos.atlassian.net/wiki/spaces/IRG/pages/661979181)
(MLRun §1)

---

### 🚨 Breaking Changes?

- [ ] Yes (explain below)
- [x] No

---

### 🔍️ Additional Notes
- **Cross-repo dependency, now resolved for testing purposes:** Orca's
`/api/v1/projects/followers/sync` endpoint didn't exist on
`feature/project-sync` when this PR was originally opened; it now exists
on orca's `feature/ORIS-3532-trigger` branch and was used for the live
verification above. Still not merged to orca's mainline.
- **Found, root-caused, and deliberately deferred a separate bug**
(already-merged ML-12901/ML-12904 code, out of scope for this PR):
`commit_delete_project`'s underlying `delete_project` call
unconditionally blocks up to 300s waiting for Nuclio to confirm a
project's deletion whenever `nuclio_dashboard_url` is configured,
regardless of leader mode. In `leader=orca`, Nuclio is a separate peer
follower Orca drives independently — MLRun's own follower-delete hook
shouldn't wait on it. To be fixed in a separate session/PR.
- **Follow-up PR:** the resource-creation-blocking gate (checklist item
9 + the blocking half of item 10) is stacked on this branch as
`feature/ML-12905-resource-creation-gate`; its `mlrun/mlrun` PR will be
opened once this one merges (GitHub won't let a cross-repo PR base on a
fork-only branch).
- Base branch is `feature/project-sync-follower` (this epic's
integration branch), not `development`.

---

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

…10103)

### 📝 Description
Follower Contract HLD checklist item 9 (ML-12905): while this MLRun
follower's project state isn't trustworthy, block creation of new
logical resources (functions, API gateways) against it. Two conditions
gate creation: the follower's own startup sync with the leader hasn't
reached a terminal state yet (checklist item 10's readiness signal,
`is_followers_sync_ready()`, built in #10089), or the specific project's
own `status.state` is `creating`/`deleting`.

---

### 🛠️ Changes Made
- `framework/utils/projects/follower.py`: new
`Member.ensure_project_open_for_resource_creation()`. Raises
`MLRunPreconditionFailedError` if `is_followers_sync_ready()` is false,
or if the project's `status.state` is `creating`/`deleting`. A project
with no recorded state yet (pre-dates the 2PC follower interface) is
treated as online, not blocked.
- `framework/utils/projects/member.py`: no-op default on the abstract
base — MLRun-as-leader (CE) drives project state directly and
synchronously, so there's no window where a project sits non-online
while a caller races to create resources against it.
- `services/api/api/endpoints/functions.py` / `nuclio.py`: wired the
gate into `store_function`, `build_function`, `start_function`,
`store_api_gateway` (added a `db_session` dependency), and
`deploy_function`.

---

### ✅ Checklist
- [ ] I updated the documentation (if applicable)
- [x] I have tested the changes in this PR
- [ ] I confirmed whether my changes are covered by system tests
- [ ] If yes, I ran all relevant system tests and ensured they passed
before submitting this PR
- [ ] I updated existing system tests and/or added new ones if needed to
cover my changes
- [ ] If I introduced a deprecation:
  - [ ] I followed the [Deprecation Guidelines](./DEPRECATION.md)
  - [ ] I updated the relevant Jira ticket for documentation
- [ ] Please run
[Smoke-tests](https://github.com/mlrun/mlrun/actions/workflows/smoke-tests.yml)
workflow, providing it the PR number as input (upon success, the
workflow run will add label "Smoke tests: Pass" to the PR)

---

### 🧪 Testing
- New unit tests: `test_follower_member.py` (+4:
blocks-while-startup-sync-pending, blocks-non-online-project,
allows-online-project, allows-unset-state), `test_leader_member.py` (+1:
is-noop-for-leader).
- Live end-to-end verification on a lab patched with this branch (at the
time, stacked on #10089's pre-merge branch) against a real Orca
implementation: confirmed the gate blocks resource creation while a
project is mid-sync and clears once state settles.

---

### 🔗 References
- Ticket link: https://ecliptos.atlassian.net/browse/ML-12905
- Design docs links: [Follower Contract
HLD](https://ecliptos.atlassian.net/wiki/spaces/IRG/pages/654737417)
(checklist item 9) · [Followers
page](https://ecliptos.atlassian.net/wiki/spaces/IRG/pages/661979181)
(MLRun §1)

---

### 🚨 Breaking Changes?

- [ ] Yes (explain below)
- [x] No

---

### 🔍️ Additional Notes
- Base branch is `feature/project-sync-follower` (this epic's
integration branch), not `development`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

### 📝 Description
New SDK path for project create/update/patch/delete: in enterprise mode
(Orca as leader), the SDK now calls Orca's user-facing project endpoints
directly with the caller's own credentials, bypassing the MLRun API
entirely — instead of going through MLRun's existing
backward-compatibility proxy (#10043). This is "role 3" per the
Followers HLD; #10043 was role 2 (the backward-compat proxy for pre-Orca
SDK clients and the UI).

The actual request orchestration (what to call, in what order, how to
poll to completion) is shared between this SDK client and #10043's
server-side leader-proxy — see Changes Made.

---

### 🛠️ Changes Made
- `mlrun/utils/orca_projects.py` (new) — Orca's project wire protocol
(create/update wire-shape translation, response parsing, the
trackable-action completion-check logic, the relevant endpoint/query
constants, and `ProjectLike`/`ProjectMetadataLike`/`ProjectSpecLike` —
`typing.Protocol`s describing the structural shape the wire functions
need, deliberately duck-typed rather than coerced to a real pydantic
`Project` — see the docstring for why), extracted out of the server-side
leader-proxy (`v4.py`, #10043) so both implementations share one
derivation of Orca's actual contract instead of two
independently-maintained ones.
- `mlrun/utils/orca_client.py` (new) — `OrcaProjectsOrchestrator`: the
actual sequence of Orca requests behind create/update/patch/delete/get,
and how each polls to completion. Shared between this PR's SDK client
and `v4.py`'s server-side leader-proxy, which had nearly-identical "what
request to make when" logic duplicated — the only genuine difference
between the two is how each sends an authenticated request (the SDK uses
its own held credentials; the leader-proxy relays the acting user's
identity), so that's the one thing each supplies via an injected
`RequestSender` Protocol. The orchestrator returns raw pieces (response,
op_id, schema objects) rather than either caller's own public return
contract, since the server's `project_leader.Member` interface
(`bool`/`None`/`Project`) and the SDK's own (`MlrunProject`/`op_id`) are
genuinely different and interface-mandated.
- `mlrun/db/orca.py` (new) — `OrcaProjectsClient`, the SDK-direct
client. Builds one `OrcaProjectsOrchestrator` (bound to its own
credentials) and its public methods (create/update/patch/delete/get) are
thin adapters translating orchestrator results into the SDK's own return
contract. `@typing.overload` signatures make `wait_for_completion`'s
effect on the return type (`MlrunProject` vs. `op_id`) explicit to type
checkers.
- `v4.py` —
`Client.create_project`/`update_project`/`delete_project`/`get_project`
now build an orchestrator per call (identity, unlike the SDK's fixed
credentials, changes per request) and delegate to it, preserving their
exact `project_leader.Member` signatures/return semantics unchanged.
`_wait_for_op`/`_verify_op_terminal` deleted (now in the shared
orchestrator). Two small, intentional behavior improvements fall out of
this: `update_project` now skips polling when Orca settles synchronously
(previously it always polled, even with nothing left to wait for), and
resolving the CAS witness now tolerates a project that doesn't exist yet
(an upsert-create call would previously have raised instead of
proceeding with `prevOpId=null`).
- `mlrun/db/httpdb.py` —
`create_project`/`store_project`/`patch_project`/`delete_project` gate
on a new `_orca_direct_mode()` check and delegate to
`OrcaProjectsClient` when true (always blocking — no SDK caller today
needs async project creation, so this doesn't add new public API surface
for it); unchanged fallback to the existing MLRun-API path otherwise.
These four signatures are byte-for-byte unchanged. `get_project` is
deliberately left on the legacy path — out of scope for this ticket.
- `_orca_direct_mode()` requires `is_iguazio_v4_mode()`,
`iguazio_api_url` being configured client-side, **and** the API server's
project-sync leader actually being `"orca"` — the first two alone don't
guarantee the third (a v4-auth deployment can have `iguazio_api_url`
configured for unrelated reasons, e.g. oauth token endpoints, while
still self-leading projects). Since the SDK had no way to observe the
server's leader, added a `projects_leader` field to
`mlrun.common.schemas.ClientSpec` (both pydantic faces), populated
server-side from `httpdb.projects.leader`, and synced into the SDK's
config during `connect()` - mirroring the existing `authentication_mode`
field/sync exactly.
- Extracted `HTTPRunDB._auth_request_kwargs()` out of `api_call` (same
auth-header/cookie logic, unchanged) so the new Orca client reuses it
instead of duplicating it.
- `patch_project`'s Orca-direct path merges the caller's partial patch
into the project's current state (`mergedeep`, mirroring the exact
strategy MLRun's own server-side patch merge uses) before sending it to
Orca's PATCH — Orca's PATCH is full-replace, not merge, so a naive
pass-through would silently drop any field the caller didn't include.

---

### ✅ Checklist
- [ ] I updated the documentation (if applicable)
- [x] I have tested the changes in this PR
- [x] I confirmed whether my changes are covered by system tests
- [ ] If yes, I ran all relevant system tests and ensured they passed
before submitting this PR
- [x] I updated existing system tests and/or added new ones if needed to
cover my changes
- [ ] If I introduced a deprecation:
  - [ ] I followed the [Deprecation Guidelines](./DEPRECATION.md)
  - [ ] I updated the relevant Jira ticket for documentation
- [ ] Please run
[Smoke-tests](https://github.com/mlrun/mlrun/actions/workflows/smoke-tests.yml)
workflow, providing it the PR number as input (upon success, the
workflow run will add label "Smoke tests: Pass" to the PR)

---

### 🧪 Testing
- New unit tests: `tests/utils/test_orca_projects.py` (wire-shape
translation, response parsing, all four trackable-action completion
outcomes, MLRun-only spec fields never leaking onto Orca's wire) and
`tests/rundb/test_orca.py` (`OrcaProjectsClient` end-to-end —
create/update/patch/delete/get, `wait_for_completion` true/false,
action-failed vs. poll-timeout, the synchronous-200-skips-polling case,
the patch-merge preserving untouched fields — plus `HTTPRunDB`'s
Orca-direct gating). 28 test cases, all passing.
- Ran the full `tests/rundb/` + `tests/utils/` suite: 961 passed, 51
failed, 2 skipped. All 51 failures are pre-existing/environmental (need
a live local server or Docker, unavailable in this environment) or
already-failing-and-unrelated tests in files this PR doesn't touch —
none are in the changed files.
- `v4.py`'s refactor to use the shared orchestrator was verified with a
standalone script driving `Client` end-to-end against `requests_mock`
(create/update/delete/get, identity-relay header check, the
deletion-strategy `check` short-circuit, CAS prev_op_id resolution
including the new upsert-create tolerance, synchronous-vs-async delete)
— server/py's own pytest suite can't run in this dev environment
(unrelated local pydantic1-vs-2/FastAPI version mismatch).
- Live end-to-end verification against a real Orca-backed system
(`leader=orca`): create, get, update (via `project.save()`), and delete
all confirmed working through the SDK-direct path — including confirming
the deleted project actually 404s afterward. Also confirmed the
`projects_leader` client-spec gate against the real server (correctly
reports `"orca"`, correctly activates `_orca_direct_mode()`).

---

### 🔗 References
- Ticket link: https://ecliptos.atlassian.net/browse/ML-12903
- Design docs links: [Project Sync Mechanism – Backend
HLD](https://ecliptos.atlassian.net/wiki/spaces/IRG/pages/611745844),
[Orca as Leader
HLD](https://ecliptos.atlassian.net/wiki/spaces/IRG/pages/654737409),
[Followers](https://ecliptos.atlassian.net/wiki/spaces/IRG/pages/661979181)
("New MLRun SDK → Orca directly")
- External links: #10043 (the server-side backward-compat proxy this PR
shares its request orchestration with); orca#1059 (unmerged, prototypes
the same contract in orca's own Python SDK)

---

### 🚨 Breaking Changes?

- [ ] Yes (explain below)
- [x] No

---

### 🔍️ Additional Notes
- Orca's side of this contract lives on orca's `feature/project-sync`
branch. Live-tested end-to-end against a real deployment running that
branch (see Testing) — the create/read/update/delete round trip via this
PR's SDK-direct path works as designed.
- `HTTPRunDB`'s project CUD methods don't expose async
(`wait_for_completion=False`) at the SDK-public level yet — no known
caller needs it today. If one shows up, it's a small follow-up: thread
the kwarg through from `HTTPRunDB` to `OrcaProjectsClient`, which
already supports it.
- Follow-up idea, not done here: once orca's own Python SDK package
(`iguazio`, orca#1059) is published, revisit whether this client should
depend on it instead of maintaining a parallel implementation.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>