hashsig: carry partial runs across file read blocks by m0g3r · Pull Request #7349 · libgit2/libgit2 · GitHub
Skip to content

hashsig: carry partial runs across file read blocks - #7349

Open
m0g3r wants to merge 1 commit into
libgit2:mainfrom
m0g3r:hashsig-runs-across-read-blocks
Open

m0g3r wants to merge 1 commit into
libgit2:mainfrom
m0g3r:hashsig-runs-across-read-blocks

Conversation

@m0g3r

@m0g3r m0g3r commented Aug 17, 2026

Copy link
Copy Markdown

What

git_hashsig_create_fromfile reads the file in 4096-byte blocks and calls
hashsig_add_hashes once per block. That function started a fresh run at the
beginning of every call, and recorded whatever run was in progress when it ran
out of data. A run of characters spanning a block boundary was therefore hashed
as two shorter runs rather than one.

For content whose lines are shorter than a block this never shows, because every
run ends at a newline anyway. For a file with lines longer than 4096 characters
the block boundaries land in the middle of runs, so the signature no longer
matches the one git_hashsig_create computes over the same bytes in memory.

Rename detection compares an in-memory blob against an on-disk file, so
identical content scored as dissimilar and renames of such files were missed.
That is the behaviour reported in #7196.

The fix

Keep the in-progress run in hashsig_in_progress so it continues across calls,
record a run only once it has actually ended, and flush the trailing run before
finalizing. The in-memory path is unaffected: it makes a single call, so the
trailing run that used to be recorded at the end of the buffer is now recorded
by the flush instead.

Reproduction and evidence

Using the reporter's recipe (six files of random characters on a single line,
sizes 1024 to 32768, all renamed, GIT_DIFF_FIND_RENAMES | GIT_DIFF_FIND_FOR_UNTRACKED,
threshold 50):

Before — only the files at or under one block are matched, exactly as reported:

[R] random.1024.txt -> random.1024.txt.moved
[D] random.16384.txt
[?] random.16384.txt.moved
[R] random.2048.txt -> random.2048.txt.moved
[D] random.32768.txt
[?] random.32768.txt.moved
[R] random.4096.txt -> random.4096.txt.moved
[D] random.8192.txt
[?] random.8192.txt.moved
renames detected: 3

After:

[R] random.1024.txt -> random.1024.txt.moved
[R] random.16384.txt -> random.16384.txt.moved
[R] random.2048.txt -> random.2048.txt.moved
[R] random.32768.txt -> random.32768.txt.moved
[R] random.4096.txt -> random.4096.txt.moved
[R] random.8192.txt -> random.8192.txt.moved
renames detected: 6

Test

core::hashsig::similarity_metric_long_lines extends the invariant the existing
similarity_metric test already checks for short data — that a signature built
from a file equals one built from the same bytes in memory — to content that
spans several read blocks with no newlines. It uses deterministic content rather
than random data.

Without the change it fails with 100 != 68; with the change it passes.

Testing performed

Built with CMake 4.3.3 / Ninja 1.13.2 / Apple clang 21, Debug, macOS arm64.

  • libgit2_tests — 3031 passed, 22 skipped, 0 failed
  • util_tests — 441 passed, 10 skipped, 0 failed
  • Targeted before and after: core::hashsig, diff (219), status (84),
    merge (181), rebase (53), checkout (168) — all green

Note this changes the signature computed for on-disk files with lines longer
than 4096 characters, which is the point of the fix; similarity scores for such
files move toward the in-memory result. No existing test changed behaviour.

Only tested on macOS arm64; I have not run the Windows or Linux CI matrix
locally.


AI assistance disclosure: this change — the diagnosis, the patch, the test, and
this description — was produced with AI assistance. The reproduction, the
red/green verification, and the test runs reported above were actually executed
and the numbers are real. No human has separately reviewed the change.

`git_hashsig_create_fromfile` reads the file in 4096 byte blocks and
calls `hashsig_add_hashes` once per block. That function started a fresh
run at the beginning of every call and recorded whatever run was in
progress when it ran out of data, so a run of characters that spanned a
block boundary was hashed as two shorter runs instead of one.

For content whose lines are shorter than a block this is invisible,
because every run ends at a newline. For a file with lines longer than
4096 characters the block boundaries fall in the middle of runs, and the
resulting signature no longer matches the one `git_hashsig_create`
computes over the same bytes in memory. Rename detection compares an
in-memory blob against an on-disk file, so identical content was scored
as dissimilar and renames of such files were missed.

Keep the in-progress run in `hashsig_in_progress` so it continues across
calls, only record a run once it has actually ended, and flush the final
run before finalizing.

Fixes libgit2#7196
@MarkRydholm

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants