Repository navigation
Configurable HTTP transfer encoding #6193
Description
Feature proposal
Instead of defaulting HTTP transfer encoding to gzip, have it configurable and allow other types of compression.
Context
Right now, golang's http framework by default and silently (not visible in GIT_CURL_VERBOSE!) adds Accept-Encoding: gzip to requests. This can make an LFS server send gzip-compressed data.
I stumbled upon a case where the LFS servers were spending more than 60% of their time doing CPU-bound gzip compression. LFS servers should have high egress, and generally be bounded on network egress.
I experimented with other compression algorithms using -c http.extraheader="accept-encoding: zstd" and it generally worked, but it perhaps would be better if it was just an lfs config, not git-wide?
Details
Have lfs.acceptEncoding which will accept a comma-separated list, resulting in Accept-Encoding header being set in the HTTP requests git-lfs is making for download and upload urls. Batch endpoints will remain unaffected.
E.g. lfs.acceptEncoding=gzip,zstd will result in Accept-Encoding: gzip,zstd being added.
golang's http lib supports zstd, so not having it shouldn't be a problem.
Potential extension of this could include preferences using weights like Accept-Encoding: gzip;q=1.0, identity;q=0.5. It's pretty much a standard
Questions
Does it make sense to do it just for git-lfs, or is it better (and safe) to use git-wide http.extraheader="accept-encoding: zstd" config?
My thinking is it's better to target the requests that would benefit from this the most - download requests. If so, this has to be done via config in git-lfs.
Activity
mzr commentedon Jan 19, 2026
I implemented something quick here which I tested against 2 different implementations of "LFS servers", and it worked.
LFS servers in quotes, because these speak both git-lfs protocol and serve the objects behind the urls from download action. This separation actually makes me think it's good to have this as an explicit config just for the "download" urls - the provider of the objects can support or not given encoding.
Anyway, if you also think it's a good idea, LMK and I can put up a PR and get it over the line.
lfshttp: log more request headers when tracing
lfshttp: log more request headers when tracing
chrisd8088 commentedon Jan 21, 2026
Hey, thanks very much for the report and for the proposed enhancement!
First of all, great catch that the
Accept-Encodingrequest header is missing from our trace logs. I think that oversight should be corrected with the changes from PR #6195.Second, I like your suggested changes a great deal, so please do open a PR so we can review them in detail!
I think you've got a good point that we don't necessarily want to alter the
Accept-Encodingheader on all types of requests. Requests to the Git LFS Batch API, in particular, should only return small JSON bodies and so would likely not benefit much from the use of other encoding algorithms.So if your PR only allows a user to customize the encoding used for object download requests, my one immediate suggestion would be that we name any new configuration option accordingly. For example, we might introduce a
lfs.transfer.httpDownloadEncoding/lfs.transfer.<url>.httpDownloadEncodingoption which controlled theAccept-Encodingheader for HTTP object downloads only. (After all, some users now perform their object transfers entirely over SSH, so we can't use a generic configuration option name which doesn't specify that this applies only to HTTP-based transfers.)Thanks again for taking a look into this issue and providing so much great detail in your report, and for drafting a potential PR as well!
mzr commentedon Jan 24, 2026
Here's the PR: #6196. Let me close this issue in favor of that.
mzr commentedon Apr 29, 2026
@chrisd8088 , just FYI on our git-lfs server implementation, I ran some quick initial tests, and I see significant improvement compared to gzip. Around 2.5x perf win. It's yet to be tested at scale - I tried this only on one client machine for one repo. Because of that this result should be treated as preliminary, as the measured sample obviously lacks analytical rigour. I will followup later with more data.
For one of our large aosp repositories (44451 LFS objects) containing a lot of build artifacts, with gzip I get:
and with zstd:
I expect CPU savings server-side too, once the change gets applied at scale. Possibly further improvement with this change that skips compression serving already compressed blobs.
chrisd8088 commentedon Apr 30, 2026
Hey, that's fantastic news! Thank you so much for the work on this new feature!