Allow opt-in shallow cloning by zware · Pull Request #744 · python/buildmaster-config · GitHub
Skip to content

Allow opt-in shallow cloning#744

Open
zware wants to merge 1 commit into
python:mainfrom
zware:opt-in_shallow_cloning
Open

Allow opt-in shallow cloning#744
zware wants to merge 1 commit into
python:mainfrom
zware:opt-in_shallow_cloning

Conversation

@zware

@zware zware commented Jun 24, 2026

Copy link
Copy Markdown
Member

Opt-in on ware-win11-arm64; something seems to be going wrong
with the blobless clones on that worker.

Opt-in on ware-win11-arm64; something seems to be going wrong
with the blobless clones on that worker.
@zware zware requested a review from encukou June 24, 2026 15:35
Comment thread master/master.cfg

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to either only use worker Git options, or only use default Git options, but not mix the two. I suggest something like:

        git_options = worker.git_options
        if git_options is None:
            git_options = dict(
                # "git clean -fdx": remove all files which are not tracked
                # by Git, ignoring the .gitignore rules (ex: remove also
                # ".o" files).
                mode="full",
                method="fresh",
                # Partial (blobless) clone: only fetch commit and tree
                # objects, blobs are fetched on demand during checkout.
                filters=["blob:none"],
            )

        source = GitHub(
            repourl=str(settings.git_url),
            # If fetch fails, wipe and re-clone instead of failing the
            # build.
            clobberOnFailure=True,
            timeout=3600,
            **branch_args,
            **git_options,
        )

That way, you can remove filters=None, from ware-win11-arm64 configuration (git options).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's the way I was going initially, but I'd like to be able to set a default timeout here and override it per-worker (for example, timeout really ought to be closer to 5 minutes than 60 on ware-win11-arm64). I can also imagine allowing per-builder overrides on top of the per-worker ones.

Looking at some of my other workers, I'm wondering if the blobless clone is living up to expectations, or if the filters option should be an opt-in as well. @stratakis, how is blobless working out for you in practice?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't analyzed yet, I'll see if I can provide some stats one of these days. Bear in mind though that there is a bug in buildbot that the blobless fetches will grow blobfull overtime with subsequent fetches which I fixed here: buildbot/buildbot#9074

But it requires a new release of buildbot to pull the fix.

@encukou encukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to get this in as is & possibly clean up later :)

FWIW, here's how I'd do it -- assuming a worker admin doesn't generally care about Buildbot Git step arguments: https://github.com/python/buildmaster-config/compare/main...encukou:buildmaster-config:opt-in_shallow_cloning?expand=1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants