GitHub Community
Unauthenticated clone of a public repo intermittently returns 401 (prompts for Username) from CI workers #206581
Replies: 37 comments 61 replies
|
💬 Your Product Feedback Has Been Submitted 🎉 Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users. Here's what you can expect moving forward ⏩
Where to look to see what's shipping 👀
What you can do in the meantime 💻
As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities. Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐ |
|
+1 Sometimes it work, sometimes not, so I guess it depends on worker which handled the request. 
|
|
We are experiencing the same issue beginning around 09:26 UTC today. Adding the following line resolved the issue for us: No credentials or other pipeline changes were needed. |
|
Same here, same fix works. By the way we get this using git 2.34.1 When we use git 2.53.0 there is no such problem and no workaround needed. |
I got the issue with git 2.55.0, on a fetch from my local machine. |
|
Same issue here |
|
Another possible fix seems to be to use "actions/checkout" instead of "git clone" |
|
Oh, I guess then I was just lucky that my workflows worked after the change. I'll watch this issue to see what other solutions pop up. |
|
Also same, for us configuring git to use http/1.1 works around it |
|
We're also observing some |
|
@nebuk89 Do you have any known active issues with your European servers? |
|
Hey everyone! 👋 Thank you for raising this. We have shared this with our internal teams to investigate - if we have any updates we will post here and in appropriate discussions ✨ |
|
why is this not seen as an outage this is heavily affecting us |
|
@samus-aran so this answer confirms that the change we see now is a feature of rolling rate limiting on open repos, not a bug, right? |
|
@samus-aran Could you please clarify whether this is a bug you’re fixing rn or a feature you’re rolling out? It seems like it might be the latter, which would mean many of us need to rethink and potentially rewrite parts of our setups. |
|
Our CI/CD templates are centrally managed, but we have no way to quickly provide a central PAT for everyone. Apart from potentially leaking the PAT, as we do not have centrally managed service connections currently, I guess a central technical user may sooner or later also cause rate limit issues if reused by everyone? (if this is not a bug indeed a heads-up before introducing rate limits without auth would have been cool) |
|
@samus-aran authenticated checkouts were failing at our 3-hour job timeout with transfer speeds between 15 and 1 KB/s. If that doesn't count against the SLO, the SLO isn't measuring what customers actually experience, and the status page didn't reflect reality. Worse, there was no communication at all, so we — like many other customers, presumably — wasted hours troubleshooting a problem that was on GitHub's side. Incidents are multiplying and clients are looking for alternatives, example (btw probably commented because of the checkout issue): Please consider reflecting this incident in the status history, or at least communicate to all clients when there's ongoing issues. |
|
In case if you need more traces. Here is one from ancient With HTTP/1.1 |
|
I have made an interesting observation - this exact problem occurs with gnutls, but not with openssl (for example on arch linux) |
|
It's probably just random, I got the issue on Fedora (git 2.53.0, curl 8.15.0, OpenSSL 3.5.8, nghttp2 1.66.0). |
|
I probably didn't express myself clearly - I meant git built against gnutls vs git built against openssl. Basically if you run |
|
For me, it fails on these versions (when using HTTP/2):
And it works on these versions:
|
|
Seeing the same intermittent failures on EU-based CI runners. Switching to HTTP/1.1 seems to reduce the issue, but it doesn’t feel like a reliable long-term workaround. |
|
FTR, I stumbled on this when trying to build zephyr which clones modules as part of the build. Chatting with AI gave the hint to try HTTP/1.1 (which works), further investigation then led me to this discussion. |
|
I just encountered this issue when trying to update Homebrew on Ubuntu 24.04 in EU |
|
TrueNAS users are affected too (both community and enterprise I guess), see https://forums.truenas.com/t/truenas-apps-catalog-fails-to-sync-git-http-2-returns-401-http-1-1-works/67742 |
|
Why is this not reported on https://www.githubstatus.com/ ? |
|
It just seems to affect Europeans, so it doesn't have priority. (Sarcasm) |
|
+1 |
|
There is an alternative / inofficial status page which shows more details. Maybe you want to check: https://mrshu.github.io/github-statuses/ |
|
Just to confirm, we are experiencing the same issue in our CI when fetching public GitHub repositories. I saw two similar error messages so far: It happened on both our self-hosted GitLab instance and ghe.com. |
|
We confirm, we are experiencing the same issue in our CI when fetching private GitHub repositories |
|
Thanks to anyone reporting that downgrading the connection to For anyone interested |
|
The incident seems to be over now? Or at least we can't reproduce it anymore: |
|
It's quite unpredictable - we thought that yesterday and then it re-appeared this morning. Would still take counter-measures! |
|
Problem seems to be resolved on our end (Switzerland). |
|
It's possible to define the HTTP/1.1 only for the step that needs it (like CMake dependencies in our case) |
|
+1 We're seeing the same thing in our CI/CD. repo sync, which uses git under the hood, is failing while fetching multiple manifests. |
|
same issue here on our Azure self-hosted (private, no public IP) Virtual Machine Scale Sets (Azure DevOps Runners). |
|
+1 Seems to have some dependency on the host - trying to clone from my Windows laptop works consistently. Cloning from some of our Ubuntu machines fails 99% of the time. Update 03/09/26 13:16 CET: first time CI (which makes use of cloning a public repo via HTTPS) successfully completed on Ubuntu 20.04, 22.04 and 24.04 machines after consistent failures during the last ~36 hours. |
|
We've seen the same thing. On my macbook with git 2.55.0, works every time. With Ubuntu Images we also had a failure rate of 99%. Quick edit: |
|
Wonder if these problems arise because of too many traffic through bots. |
|
Hi everyone! We wanted to thank everyone for their patience and apologise for the confusion we see many have had when being prompted to authenticate to clone public repositories. Our aim is to make public repositories accessible without authentication as much as possible. However, like much of the Internet, we continue to see significant increases in the volume of robot traffic recently which has increased the need for verification, for example CAPTCHAs. Striking the right balance means accommodating our current users whilst protecting the broader community we support ensuring GitHub remains fast and reliable for all of our users. For Git commands, we achieve this by applying higher limits to your account if you're authenticated. There is a challenge, however – in most cases Git will attempt all requests without authentication first, and only authenticate if it receives a
The key element here is that "the user agent MAY repeat the request with a new or replaced Authorization header field" – which is what Git then does in this case. Previously, we have limited traffic with a Git has two ways to force authentication – you can configure http.proactiveAuth, or you can use Git over SSH, which is always authenticated. However, either of these would require you to change your Git configuration to use authentication and receive higher limits than we are able to offer currently to unauthenticated traffic. To address the recent increase in abusive traffic, we have expanded our traffic protections for unauthenticated requests. When these protections are applied, we now return We understand this isn't ideal. We’re continuously improving our systems to identifying and responding to excessive abusive traffic, reduce false positives while also keeping GitHub available, and fast, and reliable for all our developers. As automated abusive traffic continues to increase and those traffic patterns frequently change, some requests may still require additional verification. By returning a Therefore, if possible, we recommend configuring your Git client with authentication credentials for GitHub. If you need assistance with this, we have documented options in "Authenticating with the command line." Thank you again for your patience and understanding as we work to maintain GitHub's availability and performance for every developer! |
|
Why you insist it was planned and where were no problems on your side? |
|
GitHub may have legitimate reasons for limiting the traffic due to bots, but I do believe you failed to adequately understand the consequences of this. As users before have pointed out, thousands of tools rely on these connections and we developers may have little ability to do any changes on them. In my case, I found my way into this topic because when the cert renewal process failed because of this (Pulls https://github.com/dehydrated-io/dehydrated). I have zero control for the server management tool cert renewal pipeline, luckily in this case this attempt was 2 weeks before renewal. But what if the expiry was the next day? Our entire product would have dropped offline. |
|
Do you understand exactly how many widely-deployed workflows and processes you've broken by doing this? We're talking hundreds of millions here. This is pants-on-head stupid. All you're going to accomplish is to push the OSS community sharply away from GitHub in the long term, and pissing a HUGE amount of users and sysadmins off in the short term. |
Because it never was about traffic. |
|
This change really needs to be reverted immediately. Then we need a proper announcement with planning time. I'm one of the devs for AMP by CubeCoders and it's broken the installs of a majority of our users. We rely on GitHub to host the configurations of the game servers anonymously. Requiring a login means we have to change our approach, which means an emergency patch. This isn't an easy ask and something that will require us to change how our program works in major ways. |
|
Previously public Github repos are not public anymore and are basicly limited to registered users? Also - if you are a paying customer your public repos are now limited? |
|
+1 to @stavros-k !! Literally first request from fresh new VM which was idle for 2 days got 401 |
I suspect that unauthenticated traffic from a previously unknown cloud host is an abusive traffic pattern in itself. That's how attackers scale or ingest all changes on Github. |
New vms can't clone. Old vms can't clone. Then who can? |
VMs using an access token, GitHub app token, GitHub actions token. A simple PAT with only public repo access would suffice. |
Then it's not public if you need an identity. Is it? |
|
@github Community Admin: Whatever changes GitHub have made, the situation is now better for our build servers: builds are now succeeding, without the 'use HTTP/1.1' workaround: no authentication-related failures so far 🤞 |
|
same for us. everything seems working like before |
|
I think the execution here could have been better. The small heads up like: "Hey, in the near term future, we will make a change, which is going to affect fetching public repositories in github actions. In order to stay not affected, please generate PAT token with empty privileges, add it to you Github Actions as secret and then use this token to fetch public repositories. We expect this change to land in one week from now" That would have been a completely other look then what happened here ;/ |
and others
Hi everyone! We wanted to thank everyone for their patience and apologise for the confusion we see many have had when being prompted to authenticate to clone public repositories.
Our aim is to make public repositories accessible without authentication as much as possible. However, like much of the Internet, we continue to see significant increases in the volume of robot traffic recently which has increased the need for verification, for example CAPTCHAs.
Striking the right balance means accommodating our current users whilst protecting the broader community we support ensuring GitHub remains fast and reliable for all of our users.
For Git commands, we achieve this by applying higher limit…