feat: add GitLab and Bitbucket to `--source` by l2ysho · Pull Request #1356 · apify/apify-cli · GitHub
Skip to content

feat: add GitLab and Bitbucket to --source - #1356

Merged
l2ysho merged 7 commits into
masterfrom
feat/git-source-gitlab-bitbucket
Sep 2, 2026
Merged

l2ysho merged 7 commits into
masterfrom
feat/git-source-gitlab-bitbucket

Conversation

@l2ysho

@l2ysho l2ysho commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

Note

TL;DR — adds --source gitlab and --source bitbucket alongside github. Both tested with personal and repo token.

Follows #1348. Part of #1153.

No longer blocked

PR What it gives us State
apify-core#30208 GitLab/Bitbucket branches for /integrations/git, create-repo and register-deployment-key merged, live
apify-core#30279 the Console page that starts the OAuth connect for the CLI merged, live

The shape change

GitHub is one connection addressed by the fixed id github-app. GitLab and Bitbucket are a UserIntegration per connected account, each addressed by its own id and reporting its own workspaces — one GitLab account can carry several namespaces, including subgroups. So:

  • integrations are matched by provider, not by a hardcoded id
  • every workspace from every matching integration is merged into one list, each keeping the providerId that create-repo and register-deployment-key address it by

Without that, a second connected account would have been silently ignored.

--git-repo matches labels

GitLab identifies a namespace by a number, so --git-repo 4711/my-scraper is what the API wants and nobody would type it. --git-repo my-group/sub resolves through the label. Since a GitLab label is a full_path and contains slashes of its own, the flag splits at the last slash — my-group/sub/my-scraper is workspace my-group/sub, repository my-scraper. The workspaces array in --json reports labels for the same reason.

Connecting

GitHub's authorize URL is two public query params, so the CLI builds it. GitLab and Bitbucket mint a CSRF state server-side and need a UserIntegration row to exist first — neither is something the CLI can produce. So connectViaConsole opens the Console integrations page with the localCli* deep-link params and waits on a loopback server, exactly as apify login does.

The callback is not treated as the only signal. The CLI polls GET /integrations/git alongside it and takes whichever lands first, because a user who connects on the Console page by hand never triggers the callback — and that is a real path, not a hypothetical: it is what happens when the hand-off does not fire. An AbortSignal releases whichever wait loses, so the loopback server never holds the command open.

Which Apify account the connection lands on

#1371 fixed this for GitHub. These two providers reach Console a different way — the CLI navigates to the integrations page rather than sending a routePrefix through an OAuth callback — so the bug survived.

Console creates the UserIntegration for the account its own route names: useEffectiveUserSession resolves the session token from viewedUserId, which comes only from the path. Where the path says nothing, Console redirects to whichever account the browser was last on — /settings/* is ROUTE_KEYS.ACCOUNT, which is in REMEMBER_LAST_ACCOUNT_ROUTE_NAMES. GET /integrations/git is scoped to the token owner, so an integration created elsewhere is invisible to the CLI, which then waits out its full timeout and reports the wrong cause.

It fails in both directions, unlike GitHub:

CLI token Browser last used Integration created on Result
organization personal personal CLI never sees it
personal organization that organization CLI never sees it

The account now rides in the path. An explicit /organization/<id> route is never redirected away (if (!isUserRoute) return in useRedirectToLastAccount), so it pins both cases.

getConsoleRoutePrefix moved to console-url.ts beside getConsoleIntegrationsUrl, which now takes the account — GitHub's query param and these providers' path prefix share one source. The account reaches the page the CLI opens, the connect URL printed for agents and CI, and the noWorkspace next steps.

Waiting for the connection

Two defects in the race, both found in review:

  • A failed listing read ended the wait. pollForWorkspaces had no guard around load(), so one 5xx or a dropped connection rejected out of the race and skipped controller.abort() — the only thing that closes the loopback server and clears its timer. apify create printed its next steps and then sat there for the remaining five minutes. The poll now survives a failed read, and the abort runs in a finally.
  • A successful callback got one read, not a poll. Console reports the connection as soon as the integration row exists, which can beat its workspaces into the listing. That single empty read reported a connected account as a missing one. It now waits the way the authorize branch already does.

A failed clone no longer costs the Actor

The clone is the only step needing the user's own provider credentials, and the only purely local one. It used to abort the run, leaving a repository with no Actor and a name that the next attempt could not reuse.

Now the error is held, the Actor and the deploy key are still created, and the stop reports the missing local copy with a git clone to run — over HTTPS or SSH, whichever the user has credentials for. Bitbucket also returns its clone URL with the account name embedded, which makes git ask only for a password and defeats every token; the CLI strips the userinfo before cloning or printing it.

Reviewer notes

This branch depends on Console's routing, and none of it is a contract. Three assumptions, all in apify-core, none covered by a test that would fail if someone changed them:

  1. every USER route gets an /organization/:viewedUserId twin (route_definitions.ts)
  2. an explicit organization route is never redirected to the last-used account (if (!isUserRoute) return)
  3. the path is what selects the effective account (useEffectiveUserSession)

A regression in any of them is silent — no error, no 404, the connection just lands on the wrong account and the user is told Apify has no GitLab account. Cheapest mitigation is a test on the apify-core side; the real one is the first follow-up below.

GitHub's provider value is confirmed, not assumed. src/api/src/routes/integrations/git_providers.ts builds the GitHub entry as { id: GITHUB_APP_PROVIDER_ID, provider: 'github' } with GITHUB_APP_PROVIDER_ID = 'github-app', so matching by provider is correct for all three.

Verification

Verified live against production, on a personal login:

  • GitLab — full chain: connect, create-repo, clone, Actor, deploy key.
  • Bitbucket — connect through the Console hand-off, create-repo, Actor, deploy key. The clone was confirmed separately with a working credential; the 403 first seen there was a workspace the account had no create rights in, not a CLI or scope fault.

The account routing and the two waiting fixes landed after those runs and are covered by unit tests only — an organization login has not been exercised end to end.

64 tests across gitSource.test.ts and connectViaConsole.test.ts. Full suite: 527 passed, 4 skipped. lint, format, build clean, docs/reference.md regenerated.

Follow-ups, not in this PR

  • Drop the loopback server. Print the account-scoped Console URL, let the user connect in Console's normal UI, and rely on the poll the CLI already runs. Nothing changes in Console — the deep link just stops being sent, and the hook it triggers is never entered. Removes ~110 lines plus its test, and both defects above were in code this deletes. The cost is the /exit signal, which tells the CLI the user closed the tab; GitHub has never had it, so this is parity rather than a regression. Losing it means a user who gives up waits out the timeout.
  • Have the CLI create the integration. A public POST /v2/integrations/git returning { integrationId, loginUrl } would let the CLI start the flow under its own token. The account question then never gets asked, and all three Console-routing assumptions above go away.
  • Clone over SSH when a key is available. The recovery steps offer it, but the run itself always tries HTTPS, which fails for anyone who only uses SSH.
  • Core-side, worth reporting: a 403 from create-repo maps to "Reconnect your account", which cannot help when the cause is missing permission; and shouldValidateScopes: false lets a Bitbucket token without repository:admin connect cleanly and fail three steps later.
  • The Console hand-off scaffolding is duplicated between connectViaConsole.ts and auth/login.ts. Worth extracting next to local-api-server.ts, which also touches edit-input-schema.ts. Note the double encodeURIComponent on localCliComputerName in both — searchParams.set already encodes, so whoever extracts this has to keep the quirk or fix both sides at once.
  • Cloning a private repository still needs the user's own provider credentials. The deploy key is Apify's and only lets the build worker read it.

🤖 Generated with Claude Code

Extends `apify create --source github` to the other two providers the API is
gaining in apify-core#30208.

The API addresses GitHub by a fixed `github-app` id, but GitLab and Bitbucket
by the id of each connected account — a user can connect several, and each
carries its own workspaces. So the CLI now matches integrations by `provider`,
merges the workspaces of every account into one list, and keeps the owning
account on each entry as the id later calls address it by.

`--git-repo` matches a workspace label as well as its id, because GitLab
identifies a namespace by a number nobody would type.

Connecting is the part the CLI cannot do alone: GitLab and Bitbucket mint their
CSRF state server-side, so `connectViaConsole` hands off to the Console page
from apify-core#30279 and waits on a loopback server, the same way `apify
login` does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added this to the 148th sprint - Tooling team milestone Aug 27, 2026
@github-actions github-actions Bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Aug 27, 2026
l2ysho and others added 6 commits August 27, 2026 13:19
- parse `--git-repo` at the last slash so GitLab subgroup workspaces work
- use provider labels ("GitLab") instead of raw ids in user-facing messages
- generalize `--source` help and `--git-repo` error text to all providers
- reuse `getConsoleIntegrationsUrl()` in `apify login`
- type `connectViaConsole` provider and `stopReason`
- add tests for the Console hand-off and `readProviderState`

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found by running the GitLab and Bitbucket flows against the live API.

- poll the integrations listing alongside the Console callback, so a
  connection made on the page by hand is picked up instead of hanging
  until the timeout; `AbortSignal` releases whichever wait loses
- keep the Actor and its deploy key when the clone fails, and report the
  missing local copy with a recoverable next step
- strip the account name Bitbucket embeds in its clone URL, which made
  git ask only for a password and no token could satisfy
- report git's own stderr instead of execa's wrapper, offer SSH beside
  HTTPS when the clone has to be redone, and stop leading with `cd` into
  a directory the clone never filled

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Master's #1371 and #1373 rewrote the same functions this branch generalizes.
Resolved by keeping this branch's multi-provider shape and folding both of
master's behaviours into it:

- `getGitConnectUrl` keeps the optional `authorize` block, so GitLab and
  Bitbucket still fall back to the Console page, and now also sends
  `routePrefix` for an organization login
- `ensureUsableIntegration` keeps the Console hand-off race for providers
  the CLI cannot authorize, and takes master's per-poll re-read of which
  grant is missing, its never-open-the-same-URL-twice rule, and the
  account-aware wait message — with provider labels rather than ids
- `runGitSourceFlow` keeps the non-fatal clone and per-account `providerId`,
  and takes master's re-prompt for a rejected repository name
- `notAuthorized` and `noWorkspace` name the Apify account and the provider
  label; next steps and `getGitStopUrl` carry the account
- both test suites kept in full

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…aces

Both found by reviewing the Console hand-off race.

- a failing integrations read no longer ends the wait: `pollForWorkspaces`
  keeps polling, so one 5xx or a dropped connection cannot turn a grant
  that is still coming into a lookup failure
- the race aborts in a `finally`. The abort is what closes the loopback
  server and clears its timer, so skipping it held `apify create` open for
  the full five minutes after the command had already printed its next steps
- a `connected: true` callback no longer stops on one empty read. Console
  reports the connection as soon as the integration exists, which can beat
  its workspaces into the listing, so the flow reported a connected account
  as a missing one. It now waits the way the authorize branch does
- both cases covered by tests, and the suite's real 2 s sleep replaced with
  an abort-faithful `node:timers/promises` mock: 2.19 s to 37 ms

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#1371 fixed this for GitHub. These two providers reach Console a different
way — the CLI navigates to the integrations page rather than sending a
`routePrefix` through an OAuth callback — so the same bug is still there.

Console creates the `UserIntegration` for the account its own route names
(`useEffectiveUserSession` reads `viewedUserId` from the path), and where
the path says nothing it redirects to whichever account the browser was
last on: `/settings/*` is `ROUTE_KEYS.ACCOUNT`, which is in Console's
`REMEMBER_LAST_ACCOUNT_ROUTE_NAMES`. `GET /integrations/git` is scoped to
the token owner, so an integration that lands elsewhere is invisible to
the CLI, which then waits out the full five minutes.

That makes it fail in both directions: an organization login connecting
to the personal account, and a personal login connecting to an
organization the browser was last on. GitHub only ever had the first.

The account now rides in the path. An explicit `/organization/<id>` route
is never redirected away, so it pins both cases.

- `getConsoleRoutePrefix` moves to `console-url.ts` beside
  `getConsoleIntegrationsUrl`, which takes the account. GitHub's query
  param and these providers' path prefix now share one source
- the account reaches the page the CLI opens, the connect URL printed for
  agents and CI, and the `noWorkspace` next steps
- `connectViaConsole` takes `{ abortSignal, account }` rather than two
  positional optionals

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is idea from https://github.com/apify/apify-core/pull/30279 but when I see it now, just poll GET integration/git seems to be enough and /exit is not so big deal (GitHub does not have it anyway) 🤔

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Relaying on the loopback server request from Console seems a bit cleaner solution to me than polling API, honestly.

The exit for the Gitlab and Bitbucket is also beneficial. Once we convert Github to UserIntegration, we will have to implement deeplink for the GH as well.

@l2ysho
l2ysho marked this pull request as ready for review September 1, 2026 16:54
@l2ysho
l2ysho merged commit fec0ea8 into master Sep 2, 2026
24 checks passed
@l2ysho
l2ysho deleted the feat/git-source-gitlab-bitbucket branch September 2, 2026 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants