Enhance LFS transfer retries with maximum retry time limit by TheRootDaemon · Pull Request #6304 · git-lfs/git-lfs · GitHub
Skip to content

Enhance LFS transfer retries with maximum retry time limit - #6304

Merged
chrisd8088 merged 16 commits into
git-lfs:mainfrom
TheRootDaemon:maxretryafter
Aug 7, 2026
Merged

chrisd8088 merged 16 commits into
git-lfs:mainfrom
TheRootDaemon:maxretryafter

Conversation

@TheRootDaemon

Copy link
Copy Markdown
Contributor

Adds lfs.transfer.maxretryafter config option with a default value of 300 seconds.
Also treats 509 as 429 so that Retry-After headers on 509 responses are recognized.

Addresses #5475

@TheRootDaemon
TheRootDaemon requested a review from a team as a code owner July 15, 2026 07:21
@chrisd8088 chrisd8088 changed the title Enhance LFS transfer retries, handle 509 Bandwidth Limit Errros Enhance LFS transfer retries, handle 509 Bandwidth Limit Errors Jul 21, 2026

@chrisd8088 chrisd8088 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, thanks very much for picking up an issue from our project backlog and proposing a potential implementation!

I think we should name the new configuration option lfs.transfer.maxRetryTime in order to align with Git's http.maxRetryTime option, which looks to me to have the same effect.

(Arguably, we should also not have implemented our own lfs.transfer.* versions of Git's http.maxRetries and http.retryAfter options, but since we have our own settings, going forward we can at least try to keep the nomenclature we use as similar to Git's as possible.)

We'll also need to add a section to our git-lfs-config(5) manual page to describe the new lfs.transfer.maxRetryTime option. That section should follow the sections about the lfs.transfer.maxRetries and lfs.transfer.maxRetryDelay options.

The Go test functions in this PR generally look good, thank you for including those! We should also expand the integration tests in our shell test suite so they exercise the new configuration option.

We already have a range of integration tests which exercise the Git LFS client when 429 status code responses are received while making requests to the Batch API endpoint and to upload or download individual objects. The t/t-batch-retries-ratelimit.sh script checks requests made to the Batch API endpoint, while the t/t-batch-storage-retries-ratelimit.sh script checks requests made to upload and download individual objects. These tests generally operate by either creating repositories whose name includes a special string like batch-retry-later or by creating objects whose content includes a special string like storage-upload-retry-later. Our t/cmd/lfstest-gitserver.go test server then detects these repository names or object contents and responds by generating a certain number of 429 status code responses. The server's checkRateLimit() function in should cause a 429 status code to be returned for each request until 10 seconds have passed since the first request of the same type, as set by the secondsToRefillTokens constant.

Ideally, we'd include several new integration tests, one for the Batch API request and one each for object upload and download requests, which would set the lfs.transfer.maxRetryTime option to something like 5 and then confirm that the Git LFS client abandons the request when it receives a Retry-After header with a larger value.

Finally, with respect to our handling of 509 Bandwidth Limit Exceeded HTTP status code responses, I'm inclined to say that we should defer any changes to another PR. This comment on issue #5475 suggests to me that some users are explicitly converting 429 status code responses into 509 status code responses in order to force the Git LFS client to abandon its requests, so I wouldn't want to alter how we handle those without deeper consideration.

For one thing, our current behaviour differs depending on whether a 509 response is returned for a Batch API request or for an individual object transfer request. If a POST request to the Batch API endpoint receives a 509 status code response, the Git LFS client treats it as an error which should not be retried, so the call to the TransferQueue structure's canRetryObject() method results in a false return value and the objects listed in the request are not re-enqueued.

However, when making an individual GET or PUT object transfer request, our upload and download HTTP transfer adapters both treat a 509 status code response as an error which should be retried, so the call to the TransferQueue structure's canRetryObject() method results in a true return value and the request is repeated, up to the maximum number of times specified by the lfs.transfer.maxRetries option.

If we're going to address how we handle 509 status code responses, we should ideally make these behaviours consistent with each other, while also respecting the possibility that servers already return the 509 status code for Batch API requests with the expectation that the client will not retry them. So for various reasons, I'd suggest we don't try to change anything related to 509 status codes in this PR, and instead just focus on implementing the lfs.transfer.maxRetryTime option for 429 status code responses with Retry-After headers.

We really appreciate contributions like this, and I hope you'll enjoy working on our project! Thanks so much for taking the time to draft this PR!

Comment thread lfsapi/response_test.go Outdated
Comment thread lfsapi/response_test.go Outdated
Comment thread tq/transfer_queue.go Outdated
Comment thread lfshttp/errors.go Outdated
Rename test case variables, Remove redundant string prefix assertions
in TestAuthErrWithoutBody and TestFatalWithoutBody where the error type is already validated.
@TheRootDaemon
TheRootDaemon requested a review from chrisd8088 July 27, 2026 06:29
@TheRootDaemon

Copy link
Copy Markdown
Contributor Author

In a prior commit in this PR we removed a pair of assertions from
the TestAuthErrWithoutBody() and TestFatalWithoutBody() Go test
functions in our "lfsapi/response_test.go" source file, with the
understanding that these assertions were redundant.

However, these assertions explicitly verify that the NewAuthError()
and NewFatalError() functions from our "errors" package wrap their
error parameters with the strings "Authentication required" and
"Fatal error", respectively.

As well, the assertions in the TestAuthErrWithoutBody() and
TestFatalWithoutBody() functions also checked some of the error
messages defined by the defaultError() function in our "lfshttp"
package, specifically the "Authorization error" and "Server error"
messages for the HTTP status codes 401 and 500, respectively.

We therefore restore these two assertions, as they have some value
in checking that our error messages for specific HTTP conditions
match our expectations.
In prior commits in this PR we introduced several new Go test
functions to our "lfsapi" and "tq" packages, including the
TestRateLimitedWithBody() and TestRateLimitedWithoutBody() functions
in the former package and the TestCanRetryObjectLater() function in
the latter package.  These functions each define and verify multiple
test cases, using anonymous structures with a number of fields,
not all of which are used in every test case.

We can simplify the definitions of these test cases slightly by
relying on Go to initialize certain string and Boolean fields with
the appropriate default values, namely the empty string and the
"false" value, respectively.
In prior commits in this PR we introduced several new Go test
functions to our "lfsapi" and "tq" packages, including the
TestRateLimitedWithBody() and TestRateLimitedWithoutBody() functions
in the former package and the TestCanRetryObjectLater() function in
the latter package.  These functions each define and verify multiple
test cases, using anonymous structures with a number of fields,
some of which specify the expected state after the condition
defined by the test case is simulated.

In a previous commit, we renamed these types of fields in the
test case structures of the TestRateLimitedWithBody() and
TestRateLimitedWithoutBody() functions so they begin with the
prefix "expected" rather than "want".  We made this change because
we already have a number of other test case structures which
follow this naming scheme, such as the "dirWalkerWalkTestCase"
structure in our "tools" package.

We now make the same change for the test case structure in our
new TestCanRetryObjectLater() function, for the same reason.

@chrisd8088 chrisd8088 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for addressing our review comments and adding some new shell tests and documentation! This PR is looking quite good now.

I've made another round of suggestions and did some debugging of one shell test, and in the interest of trying to move this PR forward I just wrote up a small set of commits which implement my proposed changes.

Since they are all minor test and documentation changes, I thought I would try pushing my commits up to your branch and seeing if our CI jobs pass. I hope you don't mind my taking that approach!

Thank you again for helping to improve the Git LFS client; we really approach the assistance.

Comment thread lfsapi/response_test.go
Comment thread lfsapi/response_test.go Outdated
Comment thread lfsapi/response_test.go Outdated
Comment thread tq/transfer_queue_test.go Outdated
Comment thread tq/transfer_queue_test.go Outdated
Comment thread t/t-batch-storage-retries-ratelimit.sh Outdated
Comment thread t/t-batch-storage-retries-ratelimit.sh Outdated
Comment thread t/t-batch-storage-retries-ratelimit.sh Outdated
Comment thread t/t-batch-storage-retries-ratelimit.sh Outdated
Comment thread docs/man/git-lfs-config.adoc Outdated
@chrisd8088 chrisd8088 changed the title Enhance LFS transfer retries, handle 509 Bandwidth Limit Errors Enhance LFS transfer retries with maximum retry time limit Aug 6, 2026
In a prior commit in this PR we added a new shell integration test
named "batch API HTTP upload abandons when Retry-After exceeds
maxRetryTime" to our "t/t-batch-retries-ratelimit.sh" shell script.

The intent of this test is to verify that when the Git LFS client
makes a request to the Batch API and receives a 429 status code
response with a Retry-After header, if the value of that header is
greater than that specified by our new "lfs.transfer.maxRetryTime"
configuration option, then the client will report an error and not
retry the Batch API request.

However, this test does not succeed at present because our
"lfstest-gitserver" test server utility simply responds to the
client's Batch API request with a 200 status code, and so the client
is able to successfully proceed and upload an object to the server.

To resolve this issue, we revise the test to use the specific
repository name "upload-batch-retry-later-exceeds-max", and then
adjust the "lfstest-gitserver" server utility so that it recognizes
this repository name in Batch API requests and responds with a 429
status code and a Retry-After header, until at least 10 seconds
have elapsed since the first Batch API request.  This behaviour
is already supported by our server utility for several of our
other tests in the same "t/t-batch-retries-ratelimit.sh" script,
so we simply need to ensure that the server is aware of the
specific repository name used by our new test.

We then take the opportunity to make a number of other minor
adjustments to our new test, including adding a call to our
refute_server_object() test assertion function at the end of
the test.

We also add a comment to explain that the repository name used by
the test is significant, as it signals to the test server utility
that it should behave in a specific manner.

Finally, we rename the test slightly and make a few other revisions
to bring its format into alignment with our current idiom for new
shell integration tests.
In a prior commit in this PR we added two new shell integration
tests named "batch storage HTTP upload abandons when Retry-After
exceeds maxRetryTime" and "batch storage HTTP download abandons
when Retry-After exceeds maxRetryTime" to our
"t/t-batch-storage-retries-ratelimit.sh" shell script.

The intent of these tests is to verify that when the Git LFS client
makes a request to upload or download an individual object and
receives a 429 status code response with a Retry-After header, if
the value of that header is greater than that specified by our new
"lfs.transfer.maxRetryTime" configuration option, then the client
will report an error and not retry the object transfer request.

We can expand these tests slightly by adding a call to our
refute_server_object() assertion function at the end of the test
which expects an object upload to fail, and by adding a call to
our refute_local_object() assertion function at the end of the
test which expects an object download to fail.

As well, we indent the lines in the latter test which appear between
"pushd" and "popd" shell built-in commands, in order to match the
formatting of our other shell tests which use these commands.

We also add comments to explain that the contents of the objects
created by the tests are significant, as they signal to the test
server utility that it should return 429 status code responses
with Retry-After headers to the initial object transfer requests
it receives for an object.

Finally, we rename the tests slightly and make a few other revisions
to bring their format into alignment with our current idiom for new
shell integration tests.
In prior commits in this PR we implemented a new configuration
option for the Git LFS client named "lfs.transfer.maxRetryTime",
and then added documentation of the new option to our
git-lfs-config(5) manual page.

We now adjust that documentation slightly to correct one typo,
and also take the opportunity to simplify the description of
the "lfs.transfer.maxRetryTime" option's default value by using
a numeral value of 300.

Although we describe several of our other similar options' default
values using words (e.g., "eight" and "ten"), these are single words
rather than multiple words, and in any case we should probably revise
them to numerals in a future PR in order to align them with the
prevailing style of both our documentation and that of the Git project.

As well, we format the new option's name using the camel-case style
found in the Git project's own documentation, and which we are
slowly trying to adopt in our own documentation.
@chrisd8088
chrisd8088 merged commit af13aa8 into git-lfs:main Aug 7, 2026
10 checks passed
@TheRootDaemon

Copy link
Copy Markdown
Contributor Author

@TheRootDaemon
TheRootDaemon deleted the maxretryafter branch August 13, 2026 06:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants