ssh_exec transport never reaps its ssh child: one zombie process per connection · Issue #7437 · libgit2/libgit2 · GitHub
Skip to content

ssh_exec transport never reaps its ssh child: one zombie process per connection #7437

Open

Description

@odrobnik

The ssh_exec transport never waits for its ssh child. ssh_exec_subtransport_close calls git_process_close() and git_process_free(). Neither one waits: git_process_free only closes the pipes, and waitpid runs only in git_process_wait(), which nothing on this path calls. So every fetch or push over ssh_exec leaves its ssh process as a zombie until the host process exits.

A short-lived CLI never notices. A long-running process accumulates one zombie per connection. A daemon that fetched and pushed over SSH through libgit2 had 71 zombie ssh children after two hours. They count against the per-user process limit.

Reproduction steps

A C program that fetches an ssh:// remote a few times, then lists its zombie children and reaps them:

zombierepro.c
/*
 * Fetch "origin" (an ssh:// remote, so libgit2 uses the ssh_exec
 * transport) a few times, then list this process's zombie children and
 * reap them.
 *
 * usage: zombierepro <repository> [<fetches>]
 */
#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <unistd.h>
#include <git2.h>

int main(int argc, char **argv)
{
	git_repository *repo = NULL;
	char ps[160];
	int fetches = argc > 2 ? atoi(argv[2]) : 3, i, status;
	pid_t pid;

	if (argc < 2) {
		fprintf(stderr, "usage: %s <repository> [<fetches>]\n", argv[0]);
		return 2;
	}
	git_libgit2_init();
	if (git_repository_open(&repo, argv[1]) < 0)
		return 1;

	for (i = 0; i < fetches; i++) {
		git_remote *remote = NULL;
		git_fetch_options opts = GIT_FETCH_OPTIONS_INIT;

		if (git_remote_lookup(&remote, repo, "origin") < 0 ||
		    git_remote_fetch(remote, NULL, &opts, NULL) < 0) {
			const git_error *e = git_error_last();
			fprintf(stderr, "fetch failed: %s\n", e ? e->message : "unknown error");
			return 1;
		}
		git_remote_free(remote);   /* also disconnects */
	}
	git_repository_free(repo);

	sleep(1);  /* let the ssh children finish exiting */
	printf("zombie children after %d fetches:\n", fetches);
	fflush(stdout);
	snprintf(ps, sizeof(ps),
		"ps -o pid=,ppid=,stat=,command= -ax | awk '$2 == %d && $3 ~ /Z/'", (int)getpid());
	system(ps);

	while ((pid = waitpid(-1, &status, WNOHANG)) > 0)
		printf("reaped %d, which libgit2 never waited for\n", (int)pid);

	git_libgit2_shutdown();
	return 0;
}

No SSH server is needed: GIT_SSH_COMMAND points at a stand-in for ssh that runs the remote command locally.

#!/bin/sh
# libgit2's ssh_exec transport never reaps its ssh child. GIT_SSH_COMMAND
# points at a stand-in for ssh that runs the remote command locally, so
# no ssh server is needed:
#     fake-ssh [-p port] [user@]host '<git-upload-pack ...>'
set -e
FETCH=${FETCH:-./zombierepro}
export GIT_CONFIG_GLOBAL=/dev/null GIT_AUTHOR_NAME=a GIT_AUTHOR_EMAIL=a@b \
       GIT_COMMITTER_NAME=a GIT_COMMITTER_EMAIL=a@b
rm -rf src.git work fake-ssh
git init -q --bare -b main src.git
git init -q -b main work
git -C work commit -q --allow-empty -m one
git -C work push -q "$PWD/src.git" main
git -C work remote add origin "ssh://localhost$PWD/src.git"

printf '#!/bin/sh\nfor last; do :; done\neval "$last"\n' > fake-ssh
chmod +x fake-ssh
GIT_SSH_COMMAND="$PWD/fake-ssh" $FETCH work 3

Expected behavior

No zombie children once the fetches are done: libgit2 waits for the ssh process it started.

Actual behavior

One zombie per fetch:

zombie children after 3 fetches:
40203 40202 Z    <defunct>
40205 40202 Z    <defunct>
40207 40202 Z    <defunct>
reaped 40207, which libgit2 never waited for
reaped 40205, which libgit2 never waited for
reaped 40203, which libgit2 never waited for

Version of libgit2 (release number or SHA1)

v1.9.4 (built with GIT_SSH_EXEC). The close path is unchanged in v1.9.7 and on main (42a89d7).

Operating system(s) tested

macOS 27 (arm64). The code involved (src/util/unix/process.c) is the same on Linux.


A fix could be to reap in ssh_exec_subtransport_close() after closing the pipes: ssh exits as soon as its stdin reaches EOF, so git_process_wait() returns right away. git_process_free() could also reap as a backstop, with a bounded wait in case ssh is stuck. That's what GitKit does (Cocoanetics/GitKit#39) in its shims over the pristine sources. I can send a PR if that sounds right.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions