Allow opt-in shallow cloning#744
Conversation
Opt-in on ware-win11-arm64; something seems to be going wrong with the blobless clones on that worker.
There was a problem hiding this comment.
I would prefer to either only use worker Git options, or only use default Git options, but not mix the two. I suggest something like:
git_options = worker.git_options
if git_options is None:
git_options = dict(
# "git clean -fdx": remove all files which are not tracked
# by Git, ignoring the .gitignore rules (ex: remove also
# ".o" files).
mode="full",
method="fresh",
# Partial (blobless) clone: only fetch commit and tree
# objects, blobs are fetched on demand during checkout.
filters=["blob:none"],
)
source = GitHub(
repourl=str(settings.git_url),
# If fetch fails, wipe and re-clone instead of failing the
# build.
clobberOnFailure=True,
timeout=3600,
**branch_args,
**git_options,
)That way, you can remove filters=None, from ware-win11-arm64 configuration (git options).
There was a problem hiding this comment.
Hmm, that's the way I was going initially, but I'd like to be able to set a default timeout here and override it per-worker (for example, timeout really ought to be closer to 5 minutes than 60 on ware-win11-arm64). I can also imagine allowing per-builder overrides on top of the per-worker ones.
Looking at some of my other workers, I'm wondering if the blobless clone is living up to expectations, or if the filters option should be an opt-in as well. @stratakis, how is blobless working out for you in practice?
There was a problem hiding this comment.
Haven't analyzed yet, I'll see if I can provide some stats one of these days. Bear in mind though that there is a bug in buildbot that the blobless fetches will grow blobfull overtime with subsequent fetches which I fixed here: buildbot/buildbot#9074
But it requires a new release of buildbot to pull the fix.
encukou
left a comment
There was a problem hiding this comment.
Happy to get this in as is & possibly clean up later :)
FWIW, here's how I'd do it -- assuming a worker admin doesn't generally care about Buildbot Git step arguments: https://github.com/python/buildmaster-config/compare/main...encukou:buildmaster-config:opt-in_shallow_cloning?expand=1

Opt-in on ware-win11-arm64; something seems to be going wrong
with the blobless clones on that worker.