Add pure SSH LFS support by n0routine · Pull Request #31516 · go-gitea/gitea · GitHub
Skip to content

Add pure SSH LFS support - #31516

Merged
techknowlogick merged 9 commits into
go-gitea:mainfrom
n0routine:lfs-ssh
Sep 27, 2024
Merged

techknowlogick merged 9 commits into
go-gitea:mainfrom
n0routine:lfs-ssh

Conversation

@n0routine

@n0routine n0routine commented Jun 28, 2024 •

Copy link
Copy Markdown
Contributor

Fixes #17554
/claim #17554

Docs PR https://gitea.com/gitea/docs/pulls/49

To test, run pushes like: GIT_TRACE=1 git push. The trace output should mention "pure SSH connection".

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jun 28, 2024
@lunny lunny added this to the 1.23.0 milestone Jun 28, 2024
Comment thread modules/lfstransfer/backend/backend.go
Comment thread modules/lfstransfer/logger.go
Comment thread modules/lfstransfer/main.go
Comment thread modules/lfstransfer/transfer/README.md Outdated
@n0routine

Copy link
Copy Markdown
Contributor Author

@n0routine
n0routine force-pushed the lfs-ssh branch 2 times, most recently from efe3893 to d506260 Compare June 28, 2024 12:37
@n0routine

Copy link
Copy Markdown
Contributor Author

Improved error handling.

Comment thread modules/lfstransfer/main.go Outdated
@n0routine

Copy link
Copy Markdown
Contributor Author

Accidentally resolved the conversation about DB connections above, but reiterating:

Pushes and clones work... but there's sometimes (by which I mean decently reproducible) bug where after transferring and verification and everything... the connection just hangs for almost exactly 120 seconds. After which it just ends gracefully and git proceeds with its push.

It's particularly nasty, I have no leads on it yet. The best I can figure out is that it looks like the git-lfs client seems to not send the quit command that it should at the end of the protocol, or that it isn't coming through.

@n0routine

Copy link
Copy Markdown
Contributor Author

Added locking API support.

Well at least I figured out what's causing the hang. Setting lfs.ssh.automultiplex to false on the client side:
https://github.com/git-lfs/git-lfs/blob/8e36a03d85bf05bbca004dd7e8b55b147809b3e0/docs/man/git-lfs-config.adoc?plain=1#L90
gets rid of the hang. The bad news is that it's true by default.

I think this is feature interacting badly with the SSH server?

I traced the traffic, and indeed, something weird is happening on the client side. Git LFS spawns several workers each making its own connection. In a multiplexed connection, only the "first" spawn of the our commands gets a quit message. The rest of them have to wait 120 seconds for an EOF. I'm almost tempted to believe this is a bug on the git-lfs client side.

Can you check and see if this is reproducible in your environments?

@n0routine

Copy link
Copy Markdown
Contributor Author

Yeah, I'm not going crazy. This commit:
git-lfs/git-lfs@44b8801
breaks pure SSH LFS sessions.

And this has history, it isn't even the first time lol:
git-lfs/git-lfs#5537

@n0routine

Copy link
Copy Markdown
Contributor Author

just made: git-lfs/git-lfs#5816

@n0routine
n0routine force-pushed the lfs-ssh branch 2 times, most recently from 5fadb5d to de9a3cf Compare July 1, 2024 20:15

@lafriks lafriks left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As pointed in quite some places already but all should be fixed

Comment thread modules/lfstransfer/backend/backend.go Outdated
Comment thread modules/lfstransfer/backend/backend.go Outdated
Comment thread modules/lfstransfer/backend/lock.go Outdated
Comment thread modules/lfstransfer/backend/lock.go Outdated
@n0routine

Copy link
Copy Markdown
Contributor Author

@lafriks adressed the url joins.

@lunny

lunny commented Sep 24, 2024 •

Copy link
Copy Markdown
Member

It looks like LFS_ALLOW_PURE_SSH hasn't been read from [lfs] but [server]. Otherwise I can test it successfully. And you need to add an example in app.example.ini file.

@n0routine

Copy link
Copy Markdown
Contributor Author

It looks like LFS_ALLOW_PURE_SSH hasn't been read from [lfs] but [server]. Otherwise I can test it successfully. And you need to add an example in app.example.ini file.

I added it under server because LFS_START_SERVER was there already.

@lunny

lunny commented Sep 26, 2024

Copy link
Copy Markdown
Member

It looks like LFS_ALLOW_PURE_SSH hasn't been read from [lfs] but [server]. Otherwise I can test it successfully. And you need to add an example in app.example.ini file.

I added it under server because LFS_START_SERVER was there already.

OK. It's a legacy problem. So could you add example in app.example.ini?

@github-actions github-actions Bot added the docs-update-needed The document needs to be updated synchronously label Sep 26, 2024
@n0routine

Copy link
Copy Markdown
Contributor Author

@lunny added it to the example config.

@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Sep 26, 2024
@lunny lunny added the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Sep 26, 2024
@techknowlogick
techknowlogick merged commit 8a9fd7f into go-gitea:main Sep 27, 2024
@GiteaBot GiteaBot removed the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Sep 27, 2024
@techknowlogick

Copy link
Copy Markdown
Member

Thanks so much @ConcurrentCrab!!! <3

@techknowlogick

Copy link
Copy Markdown
Member

I'll sort out Algora stuff right now. Related, if you have other issues you'd like to suggest for bounties, please get in touch with me, as I'm working on a list right now.

@n0routine

n0routine commented Sep 27, 2024 •

Copy link
Copy Markdown
Contributor Author

Thanks! and I appreciate the tip.

Related, if you have other issues you'd like to suggest for bounties, please get in touch with me, as I'm working on a list right now.

Was this directed at me? If so, I'd be happy to take a shot at more bounties. I quite enjoyed the work part of this.

My only sore point here was the turnaround time on this one. I get that y'all have a business to run. One feature PR isn't exactly at the top of the priority list, I wasn't expecting instant replies or anything. But to me it felt like this was basically completely ignored for quite a while, and as a result took far longer than it should have.

@6543

6543 commented Sep 27, 2024 •

Copy link
Copy Markdown
Member

-> mind the documentation ...
... i'm fine with the pull itselve but next to the config flag we should link to the upstream issue

@6543

6543 commented Sep 27, 2024 •

Copy link
Copy Markdown
Member

But to me it felt like this was basically completely ignored for quite a while, and as a result took far longer than it should have.

Sorry for that i personnally was just waiting for some references in the docs and you would also have got my lgtm

(#31516 (comment))

@n0routine

Copy link
Copy Markdown
Contributor Author

-> mind the documentation ... ... i'm fine with the pull itselve but next to the config flag we should link to the upstream issue

@6543 does this cover what you mean: https://gitea.com/gitea/docs/pulls/49/files

I thought this was linked in reply to you above somewhere, but the discussion might've been collapsed before you saw it.

zjjhot added a commit to zjjhot/gitea that referenced this pull request Sep 30, 2024
* giteaofficial/main:
  Change the code search to sort results by relevance (go-gitea#32134)
  [skip ci] Updated translations via Crowdin
  Add pure SSH LFS support (go-gitea#31516)
@wxiaoguang

wxiaoguang commented Nov 11, 2024 •

Copy link
Copy Markdown
Contributor

Could you take a look at " Refactor LFS SSH and internal routers #32473 " ?

If I understand correctly, when using LFS SSH transfer, there should be some "PUT" requests, but I can't see them in the test.

Could you help to improve the test? Otherwise the feature would be broken some time later when code changes. Thank you.


Update: resolved.

@wxiaoguang wxiaoguang added the type/feature Completely new functionality. Can only be merged if feature freeze is not active. label Nov 13, 2024
@go-gitea go-gitea locked as resolved and limited conversation to collaborators Dec 26, 2024
@wxiaoguang

wxiaoguang commented Oct 14, 2025 •

Copy link
Copy Markdown
Contributor

@ConcurrentCrab it seems that there are resource leaking: During git clone, gitea fails with accept4: too many open files; retrying in 1s #35649

I proposed " LFS SSH: Fix missing Close when error occurs and add more tests #35658 ", but not 100% sure.

Do you have some ideas? Thank you.

@wxiaoguang

Copy link
Copy Markdown
Contributor

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

docs-update-needed The document needs to be updated synchronously lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/feature Completely new functionality. Can only be merged if feature freeze is not active.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support LFS purely over SSH protocol

9 participants