{{ message }}
Repository navigation
Define lockable files, make read-only in working copy - #1870
Merged
Merged
Conversation
Also add functions to fix writeable flag based on these patterns
Also requires dropping recursive option which is fine since always recursive
…laces `locking` still depends on `config` but will be refactored out later
Still using strings.Replace, this doesn’t justify using text/template
Define lockable files, make read-only in working copy when not locked
# Conflicts: # locking/locks.go
ttaylorr
reviewed
Jan 18, 2017
ttaylorr
left a comment
Contributor
There was a problem hiding this comment.
Thanks for tackling those merge conflicts 👍 . I'm on board with this, but I'll defer to @technoweenie's judgement before approving this into master.
technoweenie
approved these changes
Jan 18, 2017
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Feb 28, 2025
As of Go 1.24, if the Go version number in the "go.mod" file is set to 1.24 or higher, the "go vet" command now reports misuses of non-constant strings as format strings. In previous commits in this PR we have now resolved all but one of the instances where we provided a non-constant string as a format string. The remaining instance is our use of the Errorf() function of the "fmt" package from the Go standard library at the end of the Combine() function in our "errors" package. The Combine() function was added to our custom "errors" package in commit 08e3e5b of PR git-lfs#1870, originally for use in the "locking" package. This function merges multiple errors into a single error by concatenating their error messages with a newline separator character between each original message. In Go 1.20 the Join() function was added to the "errors" package of the Go standard library, and it performs the same concatenation of error messages as our Combine() function, including the use of a newline character as a separator between the original messages, except that it delays the concatenation until the Error() method is called. To resolve the remaining case where we pass a non-constant string as a format string, we remove the Combine() function from our "errors" package and replace it with a Join() function that simply invokes the Join() function of the standard library's "errors" package, which we can expect to be defined as we currently require the use of at least Go 1.21 (per the Go version specified in our "go.mod" file). To make this change, we alias the standard library's "errors" package as "goerrors", following the pattern established in our "lfshttp" package where we use that alias as well. We then revise the two callers of our Combine() function to make use of the new Join() function instead. One of these callers is the startConnection() function in our "ssh" package, which simply joins two errors, and the other caller is the FixLockableFileWriteFlags() method of the Client structure in our "locking" package, which collects zero or more errors while iterating over a list of files and repeatedly calling another method. Because the Join() functions accept variadic arguments rather than an array of errors, we revise these callers to pass the errors they intend to concatenate as direct arguments rather than in an array. For the FixLockableFileWriteFlags() method this means that as it iterates over its list of files, if any errors occur we immediately call the Join() function to add them to the error value we return at the end of the function, whereas previously we appended the errors to an array, and then called the Combine() function after the loop exited to concatenate any errors in the array into the function's final error return value.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Mar 5, 2025
As of Go 1.24, if the Go version number in the "go.mod" file is set to 1.24 or higher, the "go vet" command now reports misuses of non-constant strings as format strings. In previous commits in this PR we have now resolved all but one of the instances where we provided a non-constant string as a format string. The remaining instance is our use of the Errorf() function of the "fmt" package from the Go standard library at the end of the Combine() function in our "errors" package. The Combine() function was added to our custom "errors" package in commit 08e3e5b of PR git-lfs#1870, originally for use in the "locking" package. This function merges multiple errors into a single error by concatenating their error messages with a newline separator character between each original message. In Go 1.20 the Join() function was added to the "errors" package of the Go standard library, and it performs the same concatenation of error messages as our Combine() function, including the use of a newline character as a separator between the original messages, except that it delays the concatenation until the Error() method is called. To resolve the remaining case where we pass a non-constant string as a format string, we remove the Combine() function from our "errors" package and replace it with a Join() function that simply invokes the Join() function of the standard library's "errors" package, which we can expect to be defined as we currently require the use of at least Go 1.21 (per the Go version specified in our "go.mod" file). To make this change, we alias the standard library's "errors" package as "goerrors", following the pattern established in our "lfshttp" package where we use that alias as well. We then revise the two callers of our Combine() function to make use of the new Join() function instead. One of these callers is the startConnection() function in our "ssh" package, which simply joins two errors, and the other caller is the FixLockableFileWriteFlags() method of the Client structure in our "locking" package, which collects zero or more errors while iterating over a list of files and repeatedly calling another method. Because the Join() functions accept variadic arguments rather than an array of errors, we revise these callers to pass the errors they intend to concatenate as direct arguments rather than in an array. For the FixLockableFileWriteFlags() method this means that as it iterates over its list of files, if any errors occur we immediately call the Join() function to add them to the error value we return at the end of the function, whereas previously we appended the errors to an array, and then called the Combine() function after the loop exited to concatenate any errors in the array into the function's final error return value.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
May 15, 2025
In PR git-lfs#1870, which was based on the work in PR git-lfs#1822, we introduced support for "lockable" files whose permissions would be managed by several new Git LFS commands such as "git lfs post-checkout" and "git lfs post-merge", both of which we register as Git hooks. In commit 585befa of PRs git-lfs#1822 and git-lfs#1870 we added manual pages for these two commands. (Although these PRs also introduced the "git lfs post-commit" command, they did not include a manual page for it, so we added one belatedly in commit ae5d5b0 of PR git-lfs#4052.) From the time it was written, our git-lfs-post-checkout(1) manual page has indicated that the command accepts three arguments. This accurately reflects the fact that Git will invoke our hook command with parameters specifying the SHAs from before and after the checkout operation, plus a status flag indicating whether a branch checkout was performed or not. However, our manual page does not explain what values these arguments are expected to contain, other than by dint of the names given to the arguments in the page's synopsis section. We therefore expand our description of the "git lfs post-checkout" command to detail the values Git will provide in each of the three arguments. We also take the opportunity to adjust the name of the second argument to clarify that we expect this parameter to contain a Git revision SHA and not the name of a Git reference such as a branch or tag. As well, we replace the underscores in the argument names with hyphens to match the style we use in all our other manual pages for the names of command parameters.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
May 15, 2025
In PR git-lfs#1870, which was based on the work in PR git-lfs#1822, we introduced support for "lockable" files whose permissions would be managed by several new Git LFS commands such as "git lfs post-checkout" and "git lfs post-merge", both of which we register as Git hooks. In commit 585befa of PRs git-lfs#1822 and git-lfs#1870 we added manual pages for these two commands. (Although these PRs also introduced the "git lfs post-commit" command, they did not include a manual page for it, so we added one belatedly in commit ae5d5b0 of PR git-lfs#4052.) From the time it was written, our git-lfs-post-merge(1) manual page has indicated that the command accepts a single "<is_squash>" argument. This accurately reflects the fact that Git will invoke our hook command with a single flag parameter that indicates whether or not a squash merge was performed. However, our "git lfs post-merge" command disregards this parameter entirely, so we adjust our manual page now to clarify that this command requires no arguments and ignores the one provided by Git.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Nov 26, 2025
Since commit 3652681 of PR git-lfs#1870 the assert_attributes_count() assertion function in our shell test library has used the grep(1) and wc(1) commands to count the number of lines in the ".gitattributes" file which match a given pattern. We now simplify this operation by using the -c option of the "grep" command to count the number of matching lines instead of a separate "wc" command. We also replace the "!=" comparison operator in the subsequent shell test with the -eq operator, since we expect both of its operands to be integers. These changes slightly simplify the steps in the assert_attributes_count() function, and also bring them into alignment with the many similar commands throughout our test suite that we have revised or corrected in prior commits in this PR. Note that unlike those other instances where we make use of the "grep" command's -c option, in this case the command runs in a subshell within a simple assignment rather than within a "[" shell builtin, so we need to append a "true" command following an "||" OR list operator. This is required to handle the case where the "grep" command finds no lines in its input which match the given pattern. The command will output a line count of "0" to its standard error file descriptor, but will also return an exit code of 1, indicating that the command did not find any matches. Our tests always enable the "errexit" shell attribute using the "set -e" command, and so this would cause the test which called the assert_attributes_count() to fail, because the "grep" command's exit status would be treated as the status of the simple variable assignment. We therefore need to append an "||" operator and a "true" command so that the subshell will always return a zero exit status. By contrast, all our other uses of the "grep" command with the -c option occur in subshells within "[" shell builtins, and so the commands' exit statuses do not supplant the statuses of the "[" builtins themselves.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Nov 26, 2025
Since commit 3652681 of PR git-lfs#1870 the assert_attributes_count() assertion function in our shell test library has used the grep(1) and wc(1) commands to count the number of lines in the ".gitattributes" file which match a given pattern. We now simplify this operation by using the -c option of the "grep" command to count the number of matching lines instead of a separate "wc" command. We also replace the "!=" comparison operator in the subsequent shell test with the -eq operator, since we expect both of its operands to be integers. These changes slightly simplify the steps in the assert_attributes_count() function, and also bring them into alignment with the many similar commands throughout our test suite that we have revised or corrected in prior commits in this PR. Note that unlike those other instances where we make use of the "grep" command's -c option, in this case the command runs in a subshell within a simple assignment rather than within a "[" shell builtin, so we need to append a "true" command following an "||" OR list operator. This is required to handle the case where the "grep" command finds no lines in its input which match the given pattern. The command will output a line count of "0" to its standard error file descriptor, but will also return an exit code of 1, indicating that the command did not find any matches. Our tests always enable the "errexit" shell attribute using the "set -e" command, and so this would cause the test which called the assert_attributes_count() to fail, because the "grep" command's exit status would be treated as the status of the simple variable assignment. We therefore need to append an "||" operator and a "true" command so that the subshell will always return a zero exit status. By contrast, all our other uses of the "grep" command with the -c option occur in subshells within "[" shell builtins, and so the commands' exit statuses do not supplant the statuses of the "[" builtins themselves.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Mar 16, 2026
In commit 13a8af6 of PR git-lfs#1616 we added a FastWalkGitRepo() function to our "tools" package for the purpose of improving the performance of the "git lfs track" command. At the time, this command used the Walk() function of the "filepath" package from the Go standard library to traverse the contents of the current Git working tree and locate all ".gitattributes" files. Then in commit f1fdc85 of the same PR git-lfs#1616 we updated the "git lfs track" command's findAttributeFiles() function to use our new FastWalkGitRepo() function instead of the "filepath" package's Walk() function. This change made searches for ".gitattributes" files in large repositories faster for several reasons. Unlike the Walk() function from the "filepath" package, the functions called by the FastWalkGitRepo() function to traverse a directory hierarchy did not sort the entries in each directory, and also ignored all ".git" directories and all entries which matched any pattern found in a ".gitignore" file. Later, in PRs git-lfs#1870 and git-lfs#2689, we expanded the number of callers of the FastWalkGitRepo() function. In particular, in PR git-lfs#2689 we began to make use of the function during the final phase of all Git LFS commands where we find and delete any stale temporary files stored in our ".lfs/tmp" directory. This PR introduced our "fs" package whose cleanupTmp() method calls the FastWalkGitRepo() function, passing the path to the ".lfs/tmp" directory and an anonymous callback function which removes any temporary object data files that are more than an hour old. We then added a FastWalkGitRepoAll() function to our "fs" package in PR git-lfs#3190, which operated in a similar fashion as the FastWalkGitRepo() function but did not read ".gitignore" files and so also did not skip directory entries matching any patterns found in those files. Next, in PR git-lfs#3686, we updated the internal implementation of the FastWalkGitRepo() and FastWalkGitRepoAll() functions to avoid entering submodules when traversing a Git working tree. To make this change, we added a check to the Walk() method of the "fastWalker" structure so it would return immediately when processing a directory if the directory contained an entry named ".git", unless the directory was the root of the working tree. Note that the Walk() method already ignored any directory entries with the name ".git", but this only meant it would traverse through the contents of a submodule checkout in a working tree while skipping the submodule's ".git" directory. Then in commit 83d7f76 of PR git-lfs#3823 we first implemented the NewLsFiles() function in our "git" package, which executes a "git ls-files" command and returns list of files it outputs. As well, we updated the findAttributeFiles() function of our "git" package and the fixFileWriteFlags() function in our "locking" package to both call the NewLsFiles() function instead of calling either the FastWalkGitRepo() or FastWalkGitRepoAll() functions. Since these were the only instances where we actually used the FastWalkGitRepo() or FastWalkGitRepoAll() functions to traverse a Git working tree, in the same commit of PR git-lfs#3823 we removed the FastWalkGitRepo() function and renamed the FastWalkGitRepoAll() function to FastWalkDir(). We also simplified some of the internal functions called by the FastWalkDir() function because they no longer needed to detect or parse ".gitignore" files, or skip directory entries matching the patterns from those files. Although the two remaining use cases for the FastWalkDir() function also did not need the function to detect and skip submodules or directories named ".git", the logic to do so was retained in the internal functions of the "tools" package. Specifically, the fastWalkWithExcludeFiles() function, which is called by the FastWalkDir() function, establishes two file path filters with the patterns ".git" and "**/.git", which are then passed down to the Walk() method of the "fastWalker" structure. That method checks whether the current directory entry's name matches either of the filter patterns, and if it does returns immediately without processing the entry any further. In addition, the Walk() method still also performs the check added in PR git-lfs#3686 to try to avoid traversing into Git submodules. As noted above, however, neither of the two remaining callers of the FastWalkDir() function require these checks, because they only need the function to traverse directory hierarchies within the ".git/lfs" directory. The cleanupTmp() method of the Filesystem structure in our "fs" package uses the FastWalkDir() function to find stale temporary files within the ".git/lfs/tmp" directory. The EachObject() method of the same structure, meanwhile, uses the FastWalkDir() function to invoke a callback function for each object file under the ".git/lfs/objects" directory. In both cases, the directory hierarchy traversed by the FastWalkDir() function is entirely within the ".git/lfs" directory, so there is no value in trying to exclude ".git" directories while performing the traversal. We therefore now simplify the FastWalkDir() function and the internal functions it invokes by removing the unnecessary checks for submodules and for directory entries named ".git". First, we update the fastWalkWithExcludeFiles() function so that does not initialize any file path filters, and we rename the function to fastWalkDir(). Next, we remove the Walk() method's "excludePaths" parameter, and alter the method so that it no longer skips directory entries based on whether their names match the patterns in that parameter's file path filters. We also eliminate the check in the Walk() method for directories containing ".git" directory entries, as this check's only purpose was to skip Git submodules within a working tree. As well, we revise the code comments relating to all of these functions and methods to reflect their new names and their simplified behaviours. To minimize the changes in this commit, however, we leave the names of the functions' parameters and internal variables intact, even though some of them still reflect the original design and the expectation that the functions would be used with Git working trees. In a subsequent commit in this PR we will then rename these variables and parameters, along with the "rootDir" field of the "fastWalker" structure, so that they more accurately represent the functions' current purpose and implementation. Finally, note that the "cleans only temp files and directories older than an hour" test in our "t/t-tempfile.sh" shell test script verifies the behaviour of cleanupTmp() function, which employs the FastWalkDir() function, while the TestFastWalkBasic() test function in our Go test suite directly exercises the fastWalkDir() internal function. Both tests thus provide some assurance that our changes in this commit have not introduced any unexpected regressions.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Mar 16, 2026
In commit 13a8af6 of PR git-lfs#1616 we added a FastWalkGitRepo() function to our "tools" package for the purpose of improving the performance of the "git lfs track" command. At the time, this command used the Walk() function of the "filepath" package from the Go standard library to traverse the contents of the current Git working tree and locate all ".gitattributes" files. Then in commit f1fdc85 of the same PR git-lfs#1616 we updated the "git lfs track" command's findAttributeFiles() function to use our new FastWalkGitRepo() function instead of the "filepath" package's Walk() function. This change made searches for ".gitattributes" files in large repositories faster for several reasons. Unlike the Walk() function from the "filepath" package, the functions called by the FastWalkGitRepo() function to traverse a directory hierarchy did not sort the entries in each directory, and also ignored all ".git" directories and all entries which matched any pattern found in a ".gitignore" file. Later, in PRs git-lfs#1870 and git-lfs#2689, we expanded the number of callers of the FastWalkGitRepo() function. In particular, in PR git-lfs#2689 we began to make use of the function during the final phase of all Git LFS commands where we find and delete any stale temporary files stored in our ".lfs/tmp" directory. This PR introduced our "fs" package whose cleanupTmp() method calls the FastWalkGitRepo() function, passing the path to the ".lfs/tmp" directory and an anonymous callback function which removes any temporary object data files that are more than an hour old. We then added a FastWalkGitRepoAll() function to our "fs" package in PR git-lfs#3190, which operated in a similar fashion as the FastWalkGitRepo() function but did not read ".gitignore" files and so also did not skip directory entries matching any patterns found in those files. Next, in PR git-lfs#3686, we updated the internal implementation of the FastWalkGitRepo() and FastWalkGitRepoAll() functions to avoid entering submodules when traversing a Git working tree. To make this change, we added a check to the Walk() method of the "fastWalker" structure so it would return immediately when processing a directory if the directory contained an entry named ".git", unless the directory was the root of the working tree. Note that the Walk() method already ignored any directory entries with the name ".git", but this only meant it would traverse through the contents of a submodule checkout in a working tree while skipping the submodule's ".git" directory. Then in commit 83d7f76 of PR git-lfs#3823 we first implemented the NewLsFiles() function in our "git" package, which executes a "git ls-files" command and returns list of files it outputs. As well, we updated the findAttributeFiles() function of our "git" package and the fixFileWriteFlags() function in our "locking" package to both call the NewLsFiles() function instead of calling either the FastWalkGitRepo() or FastWalkGitRepoAll() functions. Since these were the only instances where we actually used the FastWalkGitRepo() or FastWalkGitRepoAll() functions to traverse a Git working tree, in the same commit of PR git-lfs#3823 we removed the FastWalkGitRepo() function and renamed the FastWalkGitRepoAll() function to FastWalkDir(). We also simplified some of the internal functions called by the FastWalkDir() function because they no longer needed to detect or parse ".gitignore" files, or skip directory entries matching the patterns from those files. Although the two remaining use cases for the FastWalkDir() function also did not need the function to detect and skip submodules or directories named ".git", the logic to do so was retained in the internal functions of the "tools" package. Specifically, the fastWalkWithExcludeFiles() function, which is called by the FastWalkDir() function, establishes two file path filters with the patterns ".git" and "**/.git", which are then passed down to the Walk() method of the "fastWalker" structure. That method checks whether the current directory entry's name matches either of the filter patterns, and if it does returns immediately without processing the entry any further. In addition, the Walk() method still also performs the check added in PR git-lfs#3686 to try to avoid traversing into Git submodules. As noted above, however, neither of the two remaining callers of the FastWalkDir() function require these checks, because they only need the function to traverse directory hierarchies within the ".git/lfs" directory. The cleanupTmp() method of the Filesystem structure in our "fs" package uses the FastWalkDir() function to find stale temporary files within the ".git/lfs/tmp" directory. The EachObject() method of the same structure, meanwhile, uses the FastWalkDir() function to invoke a callback function for each object file under the ".git/lfs/objects" directory. In both cases, the directory hierarchy traversed by the FastWalkDir() function is entirely within the ".git/lfs" directory, so there is no value in trying to exclude ".git" directories while performing the traversal. We therefore now simplify the FastWalkDir() function and the internal functions it invokes by removing the unnecessary checks for submodules and for directory entries named ".git". First, we update the fastWalkWithExcludeFiles() function so that does not initialize any file path filters, and we rename the function to fastWalkDir(). Next, we remove the Walk() method's "excludePaths" parameter, and alter the method so that it no longer skips directory entries based on whether their names match the patterns in that parameter's file path filters. We also eliminate the check in the Walk() method for directories containing ".git" directory entries, as this check's only purpose was to skip Git submodules within a working tree. As well, we revise the code comments relating to all of these functions and methods to reflect their new names and their simplified behaviours. To minimize the changes in this commit, however, we leave the names of the functions' parameters and internal variables intact, even though some of them still reflect the original design and the expectation that the functions would be used with Git working trees. In a subsequent commit in this PR we will then rename these variables and parameters, along with the "rootDir" field of the "fastWalker" structure, so that they more accurately represent the functions' current purpose and implementation. Finally, note that the "cleans only temp files and directories older than an hour" test in our "t/t-tempfile.sh" shell test script verifies the behaviour of cleanupTmp() function, which employs the FastWalkDir() function, while the TestFastWalkBasic() test function in our Go test suite directly exercises the fastWalkDir() internal function. Both tests thus provide some assurance that our changes in this commit have not introduced any unexpected regressions.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Mar 17, 2026
In commit 13a8af6 of PR git-lfs#1616 we added a FastWalkGitRepo() function to our "tools" package for the purpose of improving the performance of the "git lfs track" command. At the time, this command used the Walk() function of the "filepath" package from the Go standard library to traverse the contents of the current Git working tree and locate all ".gitattributes" files. Then in commit f1fdc85 of the same PR git-lfs#1616 we updated the "git lfs track" command's findAttributeFiles() function to use our new FastWalkGitRepo() function instead of the "filepath" package's Walk() function. This change made searches for ".gitattributes" files in large repositories faster for several reasons. Unlike the Walk() function from the "filepath" package, the functions called by the FastWalkGitRepo() function to traverse a directory hierarchy did not sort the entries in each directory, and also ignored all ".git" directories and all entries which matched any pattern found in a ".gitignore" file. Later, in PRs git-lfs#1870 and git-lfs#2689, we expanded the number of callers of the FastWalkGitRepo() function. In particular, in PR git-lfs#2689 we began to make use of the function during the final phase of all Git LFS commands where we find and delete any stale temporary files stored in our ".lfs/tmp" directory. This PR introduced our "fs" package whose cleanupTmp() method calls the FastWalkGitRepo() function, passing the path to the ".lfs/tmp" directory and an anonymous callback function which removes any temporary object data files that are more than an hour old. We then added a FastWalkGitRepoAll() function to our "fs" package in PR git-lfs#3190, which operated in a similar fashion as the FastWalkGitRepo() function but did not read ".gitignore" files and so also did not skip directory entries matching any patterns found in those files. Next, in PR git-lfs#3686, we updated the internal implementation of the FastWalkGitRepo() and FastWalkGitRepoAll() functions to avoid entering submodules when traversing a Git working tree. To make this change, we added a check to the Walk() method of the "fastWalker" structure so it would return immediately when processing a directory if the directory contained an entry named ".git", unless the directory was the root of the working tree. Note that the Walk() method already ignored any directory entries with the name ".git", but this only meant it would traverse through the contents of a submodule checkout in a working tree while skipping the submodule's ".git" directory. Then in commit 83d7f76 of PR git-lfs#3823 we first implemented the NewLsFiles() function in our "git" package, which executes a "git ls-files" command and returns list of files it outputs. As well, we updated the findAttributeFiles() function of our "git" package and the fixFileWriteFlags() function in our "locking" package to both call the NewLsFiles() function instead of calling either the FastWalkGitRepo() or FastWalkGitRepoAll() functions. Since these were the only instances where we actually used the FastWalkGitRepo() or FastWalkGitRepoAll() functions to traverse a Git working tree, in the same commit of PR git-lfs#3823 we removed the FastWalkGitRepo() function and renamed the FastWalkGitRepoAll() function to FastWalkDir(). We also simplified some of the internal functions called by the FastWalkDir() function because they no longer needed to detect or parse ".gitignore" files, or skip directory entries matching the patterns from those files. Although the two remaining use cases for the FastWalkDir() function also did not need the function to detect and skip submodules or directories named ".git", the logic to do so was retained in the internal functions of the "tools" package. Specifically, the fastWalkWithExcludeFiles() function, which is called by the FastWalkDir() function, establishes two file path filters with the patterns ".git" and "**/.git", which are then passed down to the Walk() method of the "fastWalker" structure. That method checks whether the current directory entry's name matches either of the filter patterns, and if it does returns immediately without processing the entry any further. In addition, the Walk() method still also performs the check added in PR git-lfs#3686 to try to avoid traversing into Git submodules. As noted above, however, neither of the two remaining callers of the FastWalkDir() function require these checks, because they only need the function to traverse directory hierarchies within the ".git/lfs" directory. The cleanupTmp() method of the Filesystem structure in our "fs" package uses the FastWalkDir() function to find stale temporary files within the ".git/lfs/tmp" directory. The EachObject() method of the same structure, meanwhile, uses the FastWalkDir() function to invoke a callback function for each object file under the ".git/lfs/objects" directory. In both cases, the directory hierarchy traversed by the FastWalkDir() function is entirely within the ".git/lfs" directory, so there is no value in trying to exclude ".git" directories while performing the traversal. We therefore now simplify the FastWalkDir() function and the internal functions it invokes by removing the unnecessary checks for submodules and for directory entries named ".git". First, we update the fastWalkWithExcludeFiles() function so that does not initialize any file path filters, and we rename the function to fastWalkDir(). Next, we remove the Walk() method's "excludePaths" parameter, and alter the method so that it no longer skips directory entries based on whether their names match the patterns in that parameter's file path filters. We also eliminate the check in the Walk() method for directories containing ".git" directory entries, as this check's only purpose was to skip Git submodules within a working tree. As well, we revise the code comments relating to all of these functions and methods to reflect their new names and their simplified behaviours. To minimize the changes in this commit, however, we leave the names of the functions' parameters and internal variables intact, even though some of them still reflect the original design and the expectation that the functions would be used with Git working trees. In a subsequent commit in this PR we will then rename these variables and parameters, along with the "rootDir" field of the "fastWalker" structure, so that they more accurately represent the functions' current purpose and implementation. Finally, note that the "cleans only temp files and directories older than an hour" test in our "t/t-tempfile.sh" shell test script verifies the behaviour of cleanupTmp() function, which employs the FastWalkDir() function, while the TestFastWalkBasic() test function in our Go test suite directly exercises the fastWalkDir() internal function. Both tests thus provide some assurance that our changes in this commit have not introduced any unexpected regressions.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Mar 17, 2026
In commit 13a8af6 of PR git-lfs#1616 we added a FastWalkGitRepo() function to our "tools" package for the purpose of improving the performance of the "git lfs track" command. At the time, this command used the Walk() function of the "filepath" package from the Go standard library to traverse the contents of the current Git working tree and locate all ".gitattributes" files. Then in commit f1fdc85 of the same PR git-lfs#1616 we updated the "git lfs track" command's findAttributeFiles() function to use our new FastWalkGitRepo() function instead of the "filepath" package's Walk() function. This change made searches for ".gitattributes" files in large repositories faster for several reasons. Unlike the Walk() function from the "filepath" package, the functions called by the FastWalkGitRepo() function to traverse a directory hierarchy did not sort the entries in each directory, and also ignored all ".git" directories and all entries which matched any pattern found in a ".gitignore" file. Later, in PRs git-lfs#1870 and git-lfs#2689, we expanded the number of callers of the FastWalkGitRepo() function. In particular, in PR git-lfs#2689 we began to make use of the function in our "git lfs prune" command to locate all the object files in our local storage directories, which by default are located within the ".git/lfs/objects" directory. As well, with this PR we began to use the FastWalkGitRepo() function during the final phase of all Git LFS commands where we find and delete any stale temporary files from the directory where we store such files, which by default is the ".git/lfs/tmp" directory. PR git-lfs#2689 introduced our "fs" package, whose Filesystem structure's EachObject() and cleanupTmp() methods both called the FastWalkGitRepo() function, passing the paths to the local object storage directory and the local temporary file storage directory, respectively. We then added a FastWalkGitRepoAll() function to our "fs" package in PR git-lfs#3190. This function operated in a similar fashion as the FastWalkGitRepo() function but did not read ".gitignore" files and so also did not skip directory entries matching any patterns found in those files. Next, in PR git-lfs#3686, we updated the internal implementation of the FastWalkGitRepo() and FastWalkGitRepoAll() functions to avoid entering submodules when traversing a Git working tree. To make this change, we added a check to the "fastWalker" structure's Walk() method so it would return immediately when processing a directory if the directory contained an entry named ".git", unless the directory was the root of the working tree. Note that the Walk() method already ignored any directory entries with the name ".git", but this only meant it would traverse through the contents of a submodule checkout in a working tree while skipping the submodule's ".git" directory. Then in commit 83d7f76 of PR git-lfs#3823 we first implemented the NewLsFiles() function in our "git" package, which executes a "git ls-files" command and returns the list of files it outputs. As well, we updated the findAttributeFiles() function of our "git" package and the fixFileWriteFlags() function in our "locking" package to both call the NewLsFiles() function instead of calling either the FastWalkGitRepo() or FastWalkGitRepoAll() functions. Since these were the only instances where we actually used the FastWalkGitRepo() or FastWalkGitRepoAll() functions to traverse a Git working tree, in the same commit of PR git-lfs#3823 we removed the FastWalkGitRepo() function and renamed the FastWalkGitRepoAll() function to FastWalkDir(). We also simplified some of the internal functions called by the FastWalkDir() function because they no longer needed to detect or parse ".gitignore" files, or skip directory entries matching the patterns from those files. Although the two remaining use cases for the FastWalkDir() function also did not need the function to detect and skip submodules or directories named ".git", the logic to do so was retained in the internal functions of the "tools" package. Specifically, the fastWalkWithExcludeFiles() function, which is called by the FastWalkDir() function, establishes two file path filters with the patterns ".git" and "**/.git", which are then passed down to the Walk() method of the "fastWalker" structure. That method checks whether the current directory entry's name matches either of the filter patterns, and if it does returns immediately without processing the entry any further. In addition, the Walk() method still also performs the check added in PR git-lfs#3686 to try to avoid traversing into Git submodules. As noted above, however, neither of the two remaining callers of the FastWalkDir() function require these checks, because they only need the function to traverse directory hierarchies which the Git LFS client has created, and which are specifically not Git working trees. The cleanupTmp() method of the Filesystem structure in our "fs" package uses the FastWalkDir() function to find stale files within the local temporary file storage directory. The EachObject() method of the same structure, meanwhile, uses the FastWalkDir() function to invoke a callback function for each object file in the local object storage directory. By default, these local storage directories are the ".git/lfs/tmp" and ".git/lfs/objects" directories. If the "lfs.storage" configuration option is set to a relative path, then these directories will be located somewhere within the ".git" directory, while if the option is set to an absolute path, our local storage directories will be located under that arbitrary location. In all these cases the Git LFS client creates and manages these local storage directories, so we can expect them not to contain ".git" directories or submodules. This is true even when a user has configured the "lfs.storage" option with an absolute path, since the client is still responsible for creating and managing "tmp" and "objects" directories within that arbitrary location. We therefore now simplify the FastWalkDir() function and the internal functions it invokes by removing the unnecessary checks for submodules and for directory entries named ".git". First, we update the fastWalkWithExcludeFiles() function so that does not initialize any file path filters, and we rename the function to fastWalkDir(). Next, we remove the Walk() method's "excludePaths" parameter, and alter the method so that it no longer skips directory entries based on whether their names match the patterns in that parameter's file path filters. We also eliminate the check in the Walk() method for directories containing ".git" directory entries, as this check's only purpose was to skip Git submodules within a working tree. As well, we revise the code comments relating to all of these functions and methods to reflect their new names and their simplified behaviours. To minimize the changes in this commit, however, we leave the names of the functions' parameters and internal variables intact, even though some of them still reflect the original design and the expectation that the functions would be used with Git working trees. In a subsequent commit in this PR we will then rename these variables and parameters, along with the "rootDir" field of the "fastWalker" structure, so that they more accurately represent the functions' current purpose and implementation. Finally, note that the "cleans only temp files and directories older than an hour" test in our "t/t-tempfile.sh" shell test script verifies the behaviour of cleanupTmp() function, which employs the FastWalkDir() function, while the TestFastWalkBasic() test function in our Go test suite directly exercises the fastWalkDir() internal function. Both tests thus provide some assurance that our changes in this commit have not introduced any unexpected regressions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

This is really just #1822 brought up to date with master. I had quite a hassle resolving all the conflicts since quite a lot of structural changes are being made in master that overlapped with this and I'd rather not have to keep doing that, so would like to merge it.