Repository navigation
Enhance LFS transfer retries with maximum retry time limit - #6304
Conversation
chrisd8088
left a comment
There was a problem hiding this comment.
Hey, thanks very much for picking up an issue from our project backlog and proposing a potential implementation!
I think we should name the new configuration option lfs.transfer.maxRetryTime in order to align with Git's http.maxRetryTime option, which looks to me to have the same effect.
(Arguably, we should also not have implemented our own lfs.transfer.* versions of Git's http.maxRetries and http.retryAfter options, but since we have our own settings, going forward we can at least try to keep the nomenclature we use as similar to Git's as possible.)
We'll also need to add a section to our git-lfs-config(5) manual page to describe the new lfs.transfer.maxRetryTime option. That section should follow the sections about the lfs.transfer.maxRetries and lfs.transfer.maxRetryDelay options.
The Go test functions in this PR generally look good, thank you for including those! We should also expand the integration tests in our shell test suite so they exercise the new configuration option.
We already have a range of integration tests which exercise the Git LFS client when 429 status code responses are received while making requests to the Batch API endpoint and to upload or download individual objects. The t/t-batch-retries-ratelimit.sh script checks requests made to the Batch API endpoint, while the t/t-batch-storage-retries-ratelimit.sh script checks requests made to upload and download individual objects. These tests generally operate by either creating repositories whose name includes a special string like batch-retry-later or by creating objects whose content includes a special string like storage-upload-retry-later. Our t/cmd/lfstest-gitserver.go test server then detects these repository names or object contents and responds by generating a certain number of 429 status code responses. The server's checkRateLimit() function in should cause a 429 status code to be returned for each request until 10 seconds have passed since the first request of the same type, as set by the secondsToRefillTokens constant.
Ideally, we'd include several new integration tests, one for the Batch API request and one each for object upload and download requests, which would set the lfs.transfer.maxRetryTime option to something like 5 and then confirm that the Git LFS client abandons the request when it receives a Retry-After header with a larger value.
Finally, with respect to our handling of 509 Bandwidth Limit Exceeded HTTP status code responses, I'm inclined to say that we should defer any changes to another PR. This comment on issue #5475 suggests to me that some users are explicitly converting 429 status code responses into 509 status code responses in order to force the Git LFS client to abandon its requests, so I wouldn't want to alter how we handle those without deeper consideration.
For one thing, our current behaviour differs depending on whether a 509 response is returned for a Batch API request or for an individual object transfer request. If a POST request to the Batch API endpoint receives a 509 status code response, the Git LFS client treats it as an error which should not be retried, so the call to the TransferQueue structure's canRetryObject() method results in a false return value and the objects listed in the request are not re-enqueued.
However, when making an individual GET or PUT object transfer request, our upload and download HTTP transfer adapters both treat a 509 status code response as an error which should be retried, so the call to the TransferQueue structure's canRetryObject() method results in a true return value and the request is repeated, up to the maximum number of times specified by the lfs.transfer.maxRetries option.
If we're going to address how we handle 509 status code responses, we should ideally make these behaviours consistent with each other, while also respecting the possibility that servers already return the 509 status code for Batch API requests with the expectation that the client will not retry them. So for various reasons, I'd suggest we don't try to change anything related to 509 status codes in this PR, and instead just focus on implementing the lfs.transfer.maxRetryTime option for 429 status code responses with Retry-After headers.
We really appreciate contributions like this, and I hope you'll enjoy working on our project! Thanks so much for taking the time to draft this PR!
Rename test case variables, Remove redundant string prefix assertions in TestAuthErrWithoutBody and TestFatalWithoutBody where the error type is already validated.
c39c9aa to
2a928a7
Compare
In a prior commit in this PR we removed a pair of assertions from the TestAuthErrWithoutBody() and TestFatalWithoutBody() Go test functions in our "lfsapi/response_test.go" source file, with the understanding that these assertions were redundant. However, these assertions explicitly verify that the NewAuthError() and NewFatalError() functions from our "errors" package wrap their error parameters with the strings "Authentication required" and "Fatal error", respectively. As well, the assertions in the TestAuthErrWithoutBody() and TestFatalWithoutBody() functions also checked some of the error messages defined by the defaultError() function in our "lfshttp" package, specifically the "Authorization error" and "Server error" messages for the HTTP status codes 401 and 500, respectively. We therefore restore these two assertions, as they have some value in checking that our error messages for specific HTTP conditions match our expectations.
In prior commits in this PR we introduced several new Go test functions to our "lfsapi" and "tq" packages, including the TestRateLimitedWithBody() and TestRateLimitedWithoutBody() functions in the former package and the TestCanRetryObjectLater() function in the latter package. These functions each define and verify multiple test cases, using anonymous structures with a number of fields, not all of which are used in every test case. We can simplify the definitions of these test cases slightly by relying on Go to initialize certain string and Boolean fields with the appropriate default values, namely the empty string and the "false" value, respectively.
In prior commits in this PR we introduced several new Go test functions to our "lfsapi" and "tq" packages, including the TestRateLimitedWithBody() and TestRateLimitedWithoutBody() functions in the former package and the TestCanRetryObjectLater() function in the latter package. These functions each define and verify multiple test cases, using anonymous structures with a number of fields, some of which specify the expected state after the condition defined by the test case is simulated. In a previous commit, we renamed these types of fields in the test case structures of the TestRateLimitedWithBody() and TestRateLimitedWithoutBody() functions so they begin with the prefix "expected" rather than "want". We made this change because we already have a number of other test case structures which follow this naming scheme, such as the "dirWalkerWalkTestCase" structure in our "tools" package. We now make the same change for the test case structure in our new TestCanRetryObjectLater() function, for the same reason.
chrisd8088
left a comment
There was a problem hiding this comment.
Thank you very much for addressing our review comments and adding some new shell tests and documentation! This PR is looking quite good now.
I've made another round of suggestions and did some debugging of one shell test, and in the interest of trying to move this PR forward I just wrote up a small set of commits which implement my proposed changes.
Since they are all minor test and documentation changes, I thought I would try pushing my commits up to your branch and seeing if our CI jobs pass. I hope you don't mind my taking that approach!
Thank you again for helping to improve the Git LFS client; we really approach the assistance.
In a prior commit in this PR we added a new shell integration test named "batch API HTTP upload abandons when Retry-After exceeds maxRetryTime" to our "t/t-batch-retries-ratelimit.sh" shell script. The intent of this test is to verify that when the Git LFS client makes a request to the Batch API and receives a 429 status code response with a Retry-After header, if the value of that header is greater than that specified by our new "lfs.transfer.maxRetryTime" configuration option, then the client will report an error and not retry the Batch API request. However, this test does not succeed at present because our "lfstest-gitserver" test server utility simply responds to the client's Batch API request with a 200 status code, and so the client is able to successfully proceed and upload an object to the server. To resolve this issue, we revise the test to use the specific repository name "upload-batch-retry-later-exceeds-max", and then adjust the "lfstest-gitserver" server utility so that it recognizes this repository name in Batch API requests and responds with a 429 status code and a Retry-After header, until at least 10 seconds have elapsed since the first Batch API request. This behaviour is already supported by our server utility for several of our other tests in the same "t/t-batch-retries-ratelimit.sh" script, so we simply need to ensure that the server is aware of the specific repository name used by our new test. We then take the opportunity to make a number of other minor adjustments to our new test, including adding a call to our refute_server_object() test assertion function at the end of the test. We also add a comment to explain that the repository name used by the test is significant, as it signals to the test server utility that it should behave in a specific manner. Finally, we rename the test slightly and make a few other revisions to bring its format into alignment with our current idiom for new shell integration tests.
In a prior commit in this PR we added two new shell integration tests named "batch storage HTTP upload abandons when Retry-After exceeds maxRetryTime" and "batch storage HTTP download abandons when Retry-After exceeds maxRetryTime" to our "t/t-batch-storage-retries-ratelimit.sh" shell script. The intent of these tests is to verify that when the Git LFS client makes a request to upload or download an individual object and receives a 429 status code response with a Retry-After header, if the value of that header is greater than that specified by our new "lfs.transfer.maxRetryTime" configuration option, then the client will report an error and not retry the object transfer request. We can expand these tests slightly by adding a call to our refute_server_object() assertion function at the end of the test which expects an object upload to fail, and by adding a call to our refute_local_object() assertion function at the end of the test which expects an object download to fail. As well, we indent the lines in the latter test which appear between "pushd" and "popd" shell built-in commands, in order to match the formatting of our other shell tests which use these commands. We also add comments to explain that the contents of the objects created by the tests are significant, as they signal to the test server utility that it should return 429 status code responses with Retry-After headers to the initial object transfer requests it receives for an object. Finally, we rename the tests slightly and make a few other revisions to bring their format into alignment with our current idiom for new shell integration tests.
In prior commits in this PR we implemented a new configuration option for the Git LFS client named "lfs.transfer.maxRetryTime", and then added documentation of the new option to our git-lfs-config(5) manual page. We now adjust that documentation slightly to correct one typo, and also take the opportunity to simplify the description of the "lfs.transfer.maxRetryTime" option's default value by using a numeral value of 300. Although we describe several of our other similar options' default values using words (e.g., "eight" and "ten"), these are single words rather than multiple words, and in any case we should probably revise them to numerals in a future PR in order to align them with the prevailing style of both our documentation and that of the Git project. As well, we format the new option's name using the camel-case style found in the Git project's own documentation, and which we are slowly trying to adopt in our own documentation.
86331e9 to
6300aa8
Compare

Adds
lfs.transfer.maxretryafterconfig option with a default value of300seconds.Also treats
509as429so thatRetry-Afterheaders on 509 responses are recognized.Addresses #5475