In prior commits in this PR we introduced a new configuration option
named "lfs.transfer.httpDownloadEncoding", which may be set to either
of the values "gzip" or "zstd", with a default value of "gzip".

When our new option is not defined or is set to "gzip", the Git LFS
client simply follows its legacy behaviour when making object transfer
download requests over HTTP.  Specifically, the client should allow the
"net/http" package in the Go standard library to insert an
"Accept-Encoding: gzip" header into the requests, and if the server
chooses to respond with gzip-encoded data, the client should expect
the "net/http" package to automatically decode the body of the response.

By contrast, when our new option is set to the value "zstd", the
Git LFS client will explicitly add an "Accept-Encoding: zstd" header
when it makes an object transfer download HTTP request.  If the server
chooses to encode the body of its response, it should include a
"Content-Encoding: zstd" header which the client will recognize and
then attempt to decode the response body using the zstd algorithm.

In either case, the server may also send a Content-Length header, in
which case the value of that header will be the length of the encoded
content and not the length of the original, uncompressed data.

For this reason, when the "net/http" package handles gzip-encoded
response data, it returns a Response structure whose ContentLength
field is set to -1.  This informs any reader of the data stream in
the structure's Body field that the length of the stream is unknown:

  https://github.com/golang/go/blob/go1.25.7/src/net/http/response.go#L90-L92
  https://github.com/golang/go/blob/go1.25.7/src/net/http/transport.go#L2385

The same issue pertains when we handle a zstd-encoded response to an
object download request, except that the "net/http" package is not
responsible for decoding the data in the response body.  Instead, we
decode this data in the download() method of the basicDownloadAdapter
structure in our "tq" package.

Later in its operation, the download() method passes the Response
structure's ContentLength field to the CopyWithCallback() function
from our "tools" package.  Thus when handling a response with a
Content-Length header and a zstd-encoded body, we currently pass an
inaccurate value to the CopyWithCallback() function, one which
represents the length of the encoded object data and not its
uncompressed actual size.

We therefore update the download() to set the ContentLength field
to -1 after decoding a zstd-encoded response body, to match the
behaviour of the "net/http" package when it decodes a gzip-encoded
response body.

One consequence of this change will be visible if a user has set the
GIT_LFS_PROGRESS environment variable to an absolute file path.
In this case, the progress log messages we write to that path will
contain a misleading "-1" value as the denominator of the fraction
which indicates how much object data has been received so far.

However, this is a known issue which already occurs when a server
returns a gzip-encoded response body, and one which we expect to
address comprehensively in a future PR.  See, for reference:

  git-lfs#6011 (comment)

In prior commits in this PR we introduced a new configuration option
named "lfs.transfer.httpDownloadEncoding", which may be set to either
of the values "gzip" or "zstd", with a default value of "gzip".

When our new option is not defined or is set to "gzip", the Git LFS
client simply follows its legacy behaviour when making object transfer
download requests over HTTP.  Specifically, the client should allow the
"net/http" package in the Go standard library to insert an
"Accept-Encoding: gzip" header into the requests, and if the server
chooses to respond with gzip-encoded data, the client should expect
the "net/http" package to automatically decode the body of the response.

By contrast, when our new option is set to the value "zstd", the
Git LFS client will explicitly add an "Accept-Encoding: zstd" header
when it makes an object transfer download HTTP request.  If the server
chooses to encode the body of its response, it should include a
"Content-Encoding: zstd" header which the client will recognize and
then attempt to decode the response body using the zstd algorithm.

In either case, the server may also send a Content-Length header, in
which case the value of that header will be the length of the encoded
content and not the length of the original, uncompressed data.

For this reason, when the "net/http" package handles gzip-encoded
response data, it returns a Response structure whose ContentLength
field is set to -1.  This informs any reader of the data stream in
the structure's Body field that the length of the stream is unknown:

  https://github.com/golang/go/blob/go1.25.7/src/net/http/response.go#L90-L92
  https://github.com/golang/go/blob/go1.25.7/src/net/http/transport.go#L2385

The same issue pertains when we handle a zstd-encoded response to an
object download request, except that the "net/http" package is not
responsible for decoding the data in the response body.  Instead, we
decode this data in the download() method of the basicDownloadAdapter
structure in our "tq" package.

Later in its operation, the download() method passes the Response
structure's ContentLength field to the CopyWithCallback() function
from our "tools" package.  Thus when handling a response with a
Content-Length header and a zstd-encoded body, we currently pass an
inaccurate value to the CopyWithCallback() function, one which
represents the length of the encoded object data and not its
uncompressed actual size.

We therefore update the download() to set the ContentLength field
to -1 after decoding a zstd-encoded response body, to match the
behaviour of the "net/http" package when it decodes a gzip-encoded
response body.

One consequence of this change will be visible if a user has set the
GIT_LFS_PROGRESS environment variable to an absolute file path.
In this case, the progress log messages we write to that path will
contain a misleading "-1" value as the denominator of the fraction
which indicates how much object data has been received so far.

However, this is a known issue which already occurs when a server
returns a gzip-encoded response body, and one which we expect to
address comprehensively in a future PR.  See, for reference:

  git-lfs#6011 (comment)

In PR #6011 we will add a test to our "t/t-filter-process.sh" shell
script with the intent that it should validate the revisions made in
the PR to our "clean" Git filter commands.  Specifically, the new
test creates the conditions described in this comment:

  #4974 (comment)

To reproduce these conditions, the test first needs to create a file
larger than the size of the buffer we use to detect and decode
Git LFS pointers.  The test then adds the file as a Git LFS object,
causing its contents to be streamed to our "clean" filter process.
However, the test also ensures that a file smaller than the size of
our pointer detection buffer is written to the same location as the
original file before our filter process actually executes.

Given these conditions, our "clean" filter process would at present
report an error, but the new test will demonstrate that the changes
made in PR #6011 will allow the filter to succeed.

However, to ensure that we do not accidentally introduce some version
of the same problem addressed by PR #6011 again in the future, the
new test must always create files that are both larger and smaller
than the size of the buffer the Git LFS client uses to read and
decode pointers.

If we change this buffer size value in the future, we will need to
remember to adjust our new test, or else it may silently become
ineffective.  The test might continue to pass, but it would no longer
accurately reproduce the conditions necessary to verify that our
filter program does not report an error if the size of a file appears
to drop below the size of our initial read buffer.

In fact, we already have a number of other tests in our shell suite
which depend on knowledge of the size of the read buffer, which at
present is 1024 bytes and is set by the "blobSizeCutoff" constant
in our "lfs" package.

Moreover, at least one of these existing tests, the "clean pseudo
pointer with extra data" test in our "t/t-clean.sh" script, no
longer creates test data of the appropriate size because its initial
version was written at a time when the size of the read buffer used
to decode Git LFS pointers was only 512 bytes, and the test has never
been appropriately revised.  (We will document the history of this
test's development in a subsequent commit in this PR, at which time
we will also update the test.)

In order to help avoid these types of issues with our shell tests,
we start by revising our "lfs" package so that the constant which
sets the size of our pointer read buffer is now exported with the
name BlobSizeCutoff.

In a subsequent commit in this PR we will then be able to add a
Go helper program to our test suite which outputs the value of this
constant.  This will allow us to also adjust all the tests which
implicitly depend on this value so they no longer create files
with sizes set by hard-coded values but instead scale the size of
the files to the current value of the BlobSizeCutoff constant.

In PR #6011 we will add a test to our "t/t-filter-process.sh" shell
script with the intent that it should validate the revisions made in
the PR to our "clean" Git filter commands.  Specifically, the new
test creates the conditions described in this comment:

  #4974 (comment)

To reproduce these conditions, the test first needs to create a file
larger than the size of the buffer we use to detect and decode
Git LFS pointers.

We also have a number of other tests in our shell suite which depend
on knowledge of the size of the read buffer, which at present is 1024
bytes and is set by the "blobSizeCutoff" constant in our "lfs" package.

At least one of these existing tests, the "clean pseudo pointer with
extra data" test in our "t/t-clean.sh" script, no longer creates test
data of the appropriate size because its initial version was written
at a time when the size of the read buffer used to decode Git LFS
pointers was only 512 bytes, and the test has never been appropriately
revised.

In order to help avoid these types of issues with our shell tests,
in a previous commit in this PR we revised our "lfs" package so that
the constant which sets the size of our pointer read buffer is now
exported with the name BlobSizeCutoff.

We can now introduce a new Go helper program named "lfstest-getlimit"
which when run with the command-line option --max-pointer-size will
output the value of the BlobSizeCutoff constant.

Our helper utility also supports three other possible command-line
options.  When the --max-bufio-scan-token-size option is provided,
the program will return the value of the MaxScanTokenSize constant
from the "bufio" package in the Go standard library.  We will be
able to make use of this value in the "prune doesn't hang on long
lines in diff" and "prune doesn't hang on long lines in stash diff"
tests in our "t/t-prune.sh" script, as they are intended to exercise
our "git lfs prune" command with files larger than the maximum token
size supported by the "bufio" package's Scanner interface.

Alternatively, when our new helper utility is run with the option
--max-pktline-len, it will output the maximum length of a Git packet
line, as specified by the MaxPacketLength constant from our
"git-lfs/pktline" Go package.  We will be able to make use of this
value in the "filter process: hash-object --stdin --path does not
hang" test in our "t/t-filter-process.sh" script, as it is intended
to exercise our "clean" filter with a file larger than the maximum
length of a single Git packet line.

Lastly, when the --max-spool-mem-buffer-size option is provided to
our new helper program, it will output the value of the
MemoryBufferLimit constant from our "tools" package.  We will be
able to make use of this value in the "smudge with invalid pointer"
test in our "t/t-smudge.sh" script, as it is designed to exercise
our "smudge" filter with input data larger than the size of the
internal memory buffer used to spool data by the Spool() function
in our "tools" package.

In PR #5504 we modified the CopyCallbackFile() method of the
GitFilter structure in our "lfs" package so that the callback
function it creates only outputs a progress log message when
at least 200 milliseconds have elapsed since the last message
was written, unless the number of bytes read has now met or
exceeded the total size of the given Git LFS object.  (The
minimum time allowed between messages is specified by the
DefaultLoggingThrottle constant from our "tasklog" package.)

In commit 00f1e95 of PR #5504 we then
also introduced the TestCopyCallbackFileThrottle() test function to
verify that the callback function created by the CopyCallbackFile()
method obeyed the new limits on how often it should write to the
progress log.

At present, the callback function checks whether the number of
bytes read has met or exceeded the total size of the given object,
and if it has, unconditionally writes a message to the progress
log, on the assumption that no further callbacks will occur because
the final bytes of the object have now been read.

However, this assumes the total size of the object is known and
correct when the callback is utilized by the CopyWithCallback()
function of our "tools" package.

While this remains true at present for most of the situations in
which we pass a callback function to the CopyWithCallback() function,
these will be more limited in number after we make the change to our
"clean" filter commands proposed in PR #6011.

PR #6011 resolves a bug in our "git lfs filter-progress" and
"git lfs clean" commands whereby they can pass an incorrect
"totalSize" parameter to the CopyWithCallback() function.  The only
solution for this problem is for the commands to simply not attempt
to determine the total size of an object.

Absent other changes, the result of this bug fix would be that
even if a progress log file is specified by the GIT_LFS_PROGRESS
environment variable, no messages would be written to the log file
during "clean" filter operations.  This would occur because the
copyToTemp() method of the GitFilter structure in our "lfs" package
would find a negative value in its "fileSize" parameter and then
set its "cb" callback function parameter to nil.

We intend to address this issue in PR #6011, but before we do so,
in subsequent commits in this PR we will revise the callback
function created by the CopyCallbackFile() method to handle the
case where it is passed a negative value in its "total" parameter,
which indicates that the total size of the object is unknown.

Before we make those revisions to the CopyCallbackFile() method,
though, we first expand our test suite to exercise the callback
function created by the method under a wider range of conditions.

Specifically, we now refactor the TestCopyCallbackFileThrottle()
function so we will be able to easily expand it to simulate a
number of different test cases.  We define these test cases using
a new copyCallbackFileThrottleTestCase structure and an associated
copyCallbackFileThrottleTestMode enumerated value.

At present, the only test case we specify is one where the initial
total size of an object is both known and correct, but in future
commits in this PR we will add other test cases, including one
where the total size of an object is unknown and is therefore
represented by a negative value.

We alter the TestCopyCallbackFileThrottle() function itself to
iterate over one or more test cases and check each one using both
the EagerEOFByteReader and the DeferredEOFByteReader structures from
our "tools" package.  We use these structures' Read() methods to
simulate conditions where the method returns either an EOF error or
a nil error, respectively, at the same time as the final byte of an
input source is read.

For each test case and underlying implementation of the Reader
interface from the Go standard library's "io" package, the
TestCopyCallbackFileThrottle() function now calls the setup()
method of the copyCallbackFileThrottleTestCase structure, which
initializes the progress log file, mock time clock, callback
function, and input data source for a CallbackReader structure
from our "tools" package.

The test function then runs the Assert() method of the
copyCallbackFileThrottleTestCase structure to perform a series of
calls to the Read() method of the CallbackReader structure after
simulated periods of time have elapsed, and checks that the final
contents of the progress log file match the expected results for
the given test case.

In later commits in this PR we will expand the set of test cases run
by the TestCopyCallbackFileThrottle() function as we revise the
callback function created by the CopyCallbackFile() method and the
Read() methods of the BodyWithCallback and CallbackReader structures
in our "tools" package.

In subsequent commits in this PR we expect to make several revisions
to the anonymous callback function created by the CopyCallbackFile()
method of the GitFilter structure in our "lfs" package.  In
particular, we intend to alter the function to more appropriately
handle the case where its "total" parameter is negative, which
indicates that the total size of the current object is unknown.

At present, when the callback function is passed a negative value in
its "total" parameter, it simply writes that value to the progress
log as the denominator of the fraction representing the amount of
the object's data processed so far.

As well, if the callback function is passed a negative value in
its "total" parameter, it will always write to the progress log
regardless of how much time has elapsed since the last message was
written.

In PR #5504 we modified the CopyCallbackFile() method of the
GitFilter structure in our "lfs" package so that the callback
function it creates only outputs a progress log message when
at least 200 milliseconds have elapsed since the last message
was written, unless the number of bytes read has now met or
exceeded the total size of the given Git LFS object.  (The
minimum time allowed between messages is specified by the
DefaultLoggingThrottle constant from our "tasklog" package.)

The callback function checks whether the number of bytes read has
met or exceeded the total size of the given object, and if it has,
unconditionally writes a message to the progress log, on the
assumption that no further callbacks will occur because the final
bytes of the object have now been read.

However, this assumes the total size of the object is known and
correct when the callback is utilized by the CopyWithCallback()
function of our "tools" package.  When the "total" parameter is
negative, though, the number of bytes read so far is always larger
than the total size, so a message is always written.

While it is currently rare for the value in the callback function's
"total" parameter to be negative, we expect it to become more common
in the future as a result of the bug fix to our "clean" filter
commands implemented by PR #6011.

Before we address this issue, we first refactor the callback function
created by the CopyCallbackFile() method to make the logic in the
function easier to modify and understand.  In particular, we separate
the multiple parts of the function's single conditional expression
into distinct steps.  We also revise the function so that it returns
immediately when the condition checked by one of these individual
steps is indicates that no log message should be written.

In PR #5504 we modified the CopyCallbackFile() method of the
GitFilter structure in our "lfs" package so that the callback
function it creates only outputs a progress log message when
at least 200 milliseconds have elapsed since the last message
was written, unless the number of bytes read has now met or
exceeded the total size of the given Git LFS object.  (The
minimum time allowed between messages is specified by the
DefaultLoggingThrottle constant from our "tasklog" package.)

The callback function checks whether the number of bytes read has
met or exceeded the total size of the given object, and if it has,
unconditionally writes a message to the progress log, on the
assumption that no further callbacks will occur because the final
bytes of the object have now been read.

However, this assumes the total size of the object is known and
correct when the callback is utilized by the CopyWithCallback()
function of our "tools" package.  When the "total" parameter is
negative, though, the number of bytes read so far is always larger
than the total size, so a message is always written.

While it is currently rare for the value in the callback function's
"total" parameter to be negative, we expect it to become more common
in the future as a result of the bug fix to our "clean" filter
commands implemented by PR #6011.

PR #6011 resolves a bug in our "git lfs filter-progress" and
"git lfs clean" commands whereby they can pass an incorrect
"totalSize" parameter to the CopyWithCallback() function.  The only
solution for this problem is for the commands to simply not attempt
to determine the total size of an object.

Absent other changes, the result of this bug fix would be that
even if a progress log file is specified by the GIT_LFS_PROGRESS
environment variable, no messages would be written to the log file
during "clean" filter operations.  This would occur because the
copyToTemp() method of the GitFilter structure in our "lfs" package
would find a negative value in its "fileSize" parameter and then
set its "cb" callback function parameter to nil.

We intend to address this issue in PR #6011, but the consequence
when we do so will be that the callback function created by the
CopyCallbackFile() method will receive a negative value in its
"total" parameter during "clean" filter operations if a progress log
is configured with the GIT_LFS_PROGRESS environment variable.

At present, the callback function may already receive a negative
value in its "total" parameter, but only under a relatively unusual
combination of circumstances.  First, of course, a progress log must
be enabled with the GIT_LFS_PROGRESS environment variable.  Second,
the legacy "git lfs smudge" command must be used during a "smudge"
filter operation rather than the "git lfs filter-process" command;
this may occur if a version of Git older than v2.11.0 is in use.
Third, the "git lfs smudge" command must download an object that is
not available locally from a server which compresses the content
for transit and therefore sends a Content-Encoding header but no
Content-Length header.

Under these conditions, the lack of a Content-Length header means
the size of the HTTP response body will be represented with a value
of -1.  When our HTTP download adapter passes this negative value to
the CopyWithCallback() function, it will be used to initialize a new
CallbackReader structure from our "tools" package, and ultimately
passed to the callback function created by the CopyCallbackFile()
method.  Note that the callback function itself is passed from the
Smudge() method of the GitFilter structure to the same structure's
downloadFile() method, which creates a new transfer queue using
the WithProgressCallback() option function.

By contrast, when the "git lfs filter-process" command is in use,
it creates a transfer queue without a defined callback function.
As a result, objects that not are available locally are downloaded
and written to local storage without generating any progress log
messages.  When the Smudge() method is later invoked (because Git
re-sends a "smudge" command to our long-running filter process),
it runs the GitFilter structure's readLocalFile() method instead of
the downloadFile() method.  While the readLocalFile() method does
invoke the CopyWithCallback() function and passes a callback
function to it, at this point the size of the object can be
determined from the file in local storage, so the callback function
receives a non-negative value in its "total" parameter.

However, as described above, when the "git lfs smudge" command is
in use and a content-encoded download occurs, the callback function
created by the CopyCallbackFile() method will receive a negative
value in its "total" parameter, which means the function will write
to the progress log regardless of the amount of time that has
elapsed since the previous message was written.

Further, we expect the changes in PR #6011 to result in a wider
range of conditions under which the callback function will receive
a negative value in its "total" parameter.

We therefore now revise the callback function created by the
CopyCallbackFile() method so that it checks whether the value of
the "total" parameter is negative, and if so will not write a
progress log message unless a sufficient period of time has
elapsed since the previous log message was written.

To verify that our change is effective, we define a second test
case for the TestCopyCallbackFileThrottle() test function that
we refactored in a prior commit in this PR.  Our new test case
is named "initial total unknown" and simulates multiple invocations
of the callback function where the "total" parameter has a negative
value each time.  In this test case, we expect that the modifications
from this commit should ensure that the callback function only writes
one log message, after the test advances the mock time clock by the
full duration specified in the DefaultLoggingThrottle constant from
our "tasklog" package.

Note that the log message expected by our new "initial total unknown"
test case contains a "-1" in the denominator of the fraction
representing the amount of source data read so far.  We will refine
this field in a subsequent commit in this PR so that negative
"total" values are no longer reported in a literal manner.

In PR #5504 we modified the CopyCallbackFile() method of the
GitFilter structure in our "lfs" package so that the callback
function it created only output a progress log message when
at least 200 milliseconds had elapsed since the last message
was written, unless the number of bytes read had met or exceeded
the total size of the given Git LFS object.  (The minimum time
allowed between messages was specified by the DefaultLoggingThrottle
constant from our "tasklog" package.)

This design depended on the assumption that the total size of the
given object was known and correct when the callback was utilized
by the CopyWithCallback() function of our "tools" package.  When
the "total" parameter was negative, though, a message would always
be written because the number of bytes read so far would always be
larger than the total size.

In a previous commit in this PR we adjusted the callback function
so that it now checks whether the value of the "total" parameter
is negative, and if so will not write a progress log message unless
a sufficient period of time has elapsed since the previous log
message was written.

However, one consequence of this adjustment is that when the "total"
parameter is negative, it is not guaranteed that a log message will
be written when the last callback is made after the final bytes of
the given object's data are read.  Further, if an object's data is
fully processed in less than 200 milliseconds, no progress log
message will be written for the object at all.

While it is currently rare for the value in the callback function's
"total" parameter to be negative, we expect it to become more common
in the future as a result of the bug fix to our "clean" filter
commands implemented by PR #6011.

PR #6011 resolves a bug in our "git lfs filter-progress" and
"git lfs clean" commands whereby they can pass an incorrect
"totalSize" parameter to the CopyWithCallback() function.  The only
solution for this problem is for the commands to simply not attempt
to determine the total size of an object.

Absent other changes, the result of this bug fix would be that
even if a progress log file is specified by the GIT_LFS_PROGRESS
environment variable, no messages would be written to the log file
during "clean" filter operations.  This would occur because the
copyToTemp() method of the GitFilter structure in our "lfs" package
would find a negative value in its "fileSize" parameter and then
set its "cb" callback function parameter to nil.

We intend to address this issue in PR #6011, but the consequence
when we do so will be that the callback function created by the
CopyCallbackFile() method will receive a negative value in its
"total" parameter during "clean" filter operations if a progress log
is configured with the GIT_LFS_PROGRESS environment variable.

At present, the callback function may already receive a negative
value in its "total" parameter, but only under a relatively unusual
combination of circumstances.  First, of course, a progress log must
be enabled with the GIT_LFS_PROGRESS environment variable.  Second,
the legacy "git lfs smudge" command must be used during a "smudge"
filter operation rather than the "git lfs filter-process" command;
this may occur if a version of Git older than v2.11.0 is in use.
Third, the "git lfs smudge" command must download an object that is
not available locally from a server which compresses the content
for transit and therefore sends a Content-Encoding header but no
Content-Length header.

Under these conditions, the lack of a Content-Length header means
the size of the HTTP response body will be represented with a value
of -1.  When our HTTP download adapter passes this negative value to
the CopyWithCallback() function, it will be used to initialize a new
CallbackReader structure from our "tools" package, and ultimately
passed to the callback function created by the CopyCallbackFile()
method.  Note that the callback function itself is passed from the
Smudge() method of the GitFilter structure to the same structure's
downloadFile() method, which creates a new transfer queue using
the WithProgressCallback() option function.

By contrast, when the "git lfs filter-process" command is in use,
it creates a transfer queue without a defined callback function.
As a result, objects that are not available locally are downloaded
and written to local storage without generating any progress log
messages.  When the Smudge() method is later invoked (because Git
re-sends a "smudge" command to our long-running filter process),
it runs the GitFilter structure's readLocalFile() method instead of
the downloadFile() method.  While the readLocalFile() method does
invoke the CopyWithCallback() function and passes a callback
function to it, at this point the size of the object can be
determined from the file in local storage, so the callback function
receives a non-negative value in its "total" parameter.

However, as described above, when the "git lfs smudge" command is
in use and a content-encoded download occurs, the callback function
created by the CopyCallbackFile() method will receive a negative
value in its "total" parameter, which means the function will write
to the progress log regardless of the amount of time that has
elapsed since the previous message was written.

Further, we expect the changes in PR #6011 to result in a wider
range of conditions under which the callback function will receive
a negative value in its "total" parameter.

We therefore now enhance both the callback function and the internal
implementation of the Read() method of the BodyWithCallback and
CallbackReader structures in our "tools" package so that a progress
log message will always be written when the final bytes of a given
object are read.

First, we update the doCallbackIfRequired() method of the
"callbackState" structure in our "tools" package so that it checks
if an EOF condition has been encountered, and if so, checks whether
the amount of data read so far exceeds the total size that was
provided initially for the given object.  Should both of these
conditions apply, the method resets the "totalSize" field of the
"callbackState" structure to the final value of the "readSize"
field, and then runs the doCallback() method to invoke the callback
function.

As a result of this change, a callback will be performed the first
time the EOF condition is encountered while processing an object
whose total size was initially unknown, and when this callback is
made, the callback function's "total" parameter will not have a
negative value but will have the same value as the function's
"written" parameter.

Next, we update the CopyCallbackFile() method of the GitFilter
structure in our "lfs" package so that the callback function the
method creates now checks whether the "total" parameter passed to
the function has a different value than it did the previous time
the function was invoked.  If this is the case, and if the current
value of the "total" is also equal to the non-negative value of the
"written" parameter, then the function writes a message to the
progress log regardless of the amount of time that has elapsed since
the previous message was written.

This change has the key consequence that if the callback function
is invoked one or more times with a negative value for its "total"
parameter, and is invoked a final time with matching non-negative
values for its "written" and "total" parameters, it will always
write a progress log message at that time.

When this change is combined with the revisions we have made to the
"callbackState" structure's doCallbackIfRequired() method in this
commit, the result is that if an object's total size is initially
unknown, when an EOF condition is encountered while reading the
object's data, then a final progress log message is guaranteed to
be written with the total size of the object represented as equal
to the actual number of bytes that were read.

To verify this new behaviour, we add or update a number of our tests,
starting with the TestCopyCallbackFileThrottle() test function in
our "lfs" package.  Most significantly, we update this function's
"initial total unknown" test case so that it now expects a final
log message in which the total size of the object is not negative
but matches the total number of bytes read.

Note that the first log message expected by our new "initial total
unknown" test case still contains a "-1" in the denominator of the
fraction representing the amount of source data read so far.  We will
refine this field in a subsequent commit in this PR so that negative
"total" parameter values are no longer reported in a literal manner.

We also add two new test cases for the TestCopyCallbackFileThrottle()
function, in order to simulate situations where the total size of
an object is initially either overestimated or underestimated.  In
both cases, a final callback should be made in which the "total"
parameter value has been adjusted to match the actual number of
bytes that have been read when the EOF condition is reached.

To simulate these additional test cases, we expand the setup() and
Assert() methods of the "copyCallbackFileThrottleTestCase" structure
we introduced to our "lfs/util_test.go" source file in a previous
commit in this PR.

For the particular test case where the total size of an object is
underestimated, the Assert() method now performs three extra calls to
the Read() method of the CallbackReader structure, with a simulated
delay between the first two of these calls.  The first of these calls
should never result in a log message being written, while the second
should produce a log message in which the number of bytes read so far
exceeds the total size of the object.  For both new test cases, when
the EOF condition is reached, a callback should be made which produces
a log message with the object's total size adjusted in the final log
message to match the actual number of bytes read.

To independently verify the new behaviour of the "callbackState"
structure's doCallbackIfRequired() method, we add a test function to
our "tools" package which also simulates situations in which the
total size of the object is either unknown, overestimated, or
underestimated.

The TestBothCallbackReadersInvokeCallbackOnEOFWhenTotalSizeIncorrect()
function confirms that the Read() methods of both the BodyWithCallback
and CallbackReader structures perform a final callback when the EOF
condition is reached and pass the callback function an updated value
for the total size of the object which matches the actual number of
bytes read.  Note that in a prior commit in this PR we refactored the
Read() methods of both of the BodyWithCallback and CallbackReader
structures so that these methods now just invoke the read() method of
a "callbackState" structure, which in turn executes the
doCallbackIfRequired() method, so we do not need to provide a test
of the doCallbackIfRequired() method itself.

In addition to these changed or new Go test functions, we also expand
our "t/t-progress.sh" shell script with two new tests, one of which
exercises the "git lfs smudge" command under the specific set of
conditions we described above.

Our new "progress log: unknown initial size (smudge)" shell test
forces the use of the "git lfs smudge" command by a "git clone"
command and specified a progress log file using the GIT_LFS_PROGRESS
environment variable.  As well, in the repository cloned by the
"git clone" command, the test establishes a Git LFS object with
the special key string "storage-download-encoding-gzip" as its
contents, which causes our "lfstest-gitserver" test utility program
to return the object's contents in an HTTP response with a
gzip-encoded body and without a Content-Length header.

The test then checks that the progress log file contains an entry
with the full size of the object reported in both the numerator and
denominator of the field which represents the amount of data
downloaded for the object.  Previously, all progress log entries
generated by the "git lfs smudge" command while downloading a
content-encoded object would have a "-1" in the field's denominator.

Note that any intermediate entries preceding the final log entry
will still have a negative value in this field, but we will address
that issue in a subsequent commit in this PR.

(Note also that we use the same "storage-download-encoding-gzip"
special object content key for a more complete test of the Git LFS
client's handling of gzip-encoded responses in the "batch storage
HTTP download with gzip encoding" test in our
"t/t-batch-storage-encoding.sh" shell script.)

The other new shell test in our "t/t-progress.sh" script is named
"progress log: unknown initial size (fetch/pull)", and operates in
a similar manner to the "progress log: unknown initial size (smudge)"
test except that it exercises the "git lfs fetch" and "git lfs pull"
commands rather than the "git lfs smudge" command.

As noted in the following comment on PR #6011, our "git lfs fetch"
and "git lfs pull" commands may write progress log entries with a "-1"
value in the denominator of the field which represents the amount of
data downloaded for the object, if the HTTP response to an object
download request is content-encoded and lacks a Content-Length header.

  #6011 (comment)

In these commands, a transfer queue is created using the
WithProgress() option function rather than the WithProgressCallback()
option function, so the "cb" field of the TransferQueue structure
from our "tq" package is nil.  However, both commands pass a Meter
structure from our "tq" package to the WithProgress() function,
so the TransferQueue structure's "m" field is non-nil.

As a result, when the TransferQueue structure's ensureAdapterBegun()
method is executed, the callback function it creates will execute
only the TransferBytes() method of the Meter structure, which in
turn does not immediately return without taking any action because
its "m" receiver parameter is non-nil.  By comparison, when the
WithProgressCallback() option function is used instead of the
WithProgress() option function, this same callback function runs
the TransferBytes() method, which returns immediately, and then
runs the callback function specified by the non-nil "cb" field.

The callback function created by the ensureAdapterBegun() method
is then passed to the Begin() method of the transfer adapter, and
ultimately passed to the CopyWithCallback() function of our "tools"
package by the "basicDownloadAdapter" structure's download() method.
(Note that this is also true for our "git lfs filter-process" and
"git lfs smudge" commands, not just for the "git lfs fetch"
and "git lfs pull" commands.)

As described above regarding the "git lfs smudge" command, the
CopyWithCallback() function initializes a new CallbackReader structure
and so the callback function passed to the CopyWithCallback() function
will ultimately be invoked as object data is read from the HTTP
response body.  With the "git lfs fetch" and "git lfs pull" commands,
though, this callback function only runs the TransferBytes() method
of the Meter structure, and that method does not return immediately
but runs the logBytes() method of the same structure.

For historical reasons which date back to the introduction of the
logBytes() method in commit 710862b
of PR #476, this method writes a line to the progress log file with
the identical format as the lines generated by the callback function
created by the CopyCallbackFile() method of the GitFilter structure.

However, unlike that anonymous callback function, the logBytes()
method does not limit its output based on the amount of time that
has elapsed since it was previously invoked.  When we revised the
CopyCallbackFile() method in PR #5504 to implement a mechanism
which throttled the frequency of its output to the progress log,
we did not also update the logBytes() method to do the same.

While we likely should refactor the logBytes() method and the
CopyCallbackFile() method's callback function to share a common
implementation, that change falls outside of the scope of this PR,
so we defer it to a future PR.

What this PR does accomplish, though, is to ensure that regardless
of their source, when progress log messages are written for an object
whose total size is initially unknown, a final callback will be made
after the EOF condition is encountered, and this will generate a
progress log message in which the total size has been adjusted to
match the actual number of bytes read.

For this reason, our "progress log: unknown initial size (fetch/pull)"
shell test exercises the "git lfs fetch" and "git lfs pull" commands
with a progress log file specified in the GIT_LFS_PROGRESS environment
variable and an HTTP object download response that lacks a
Content-Length header.  Like our other new shell test, this test
also uses the "storage-download-encoding-gzip" special object
content key to cause the "lfstest-gitserver" utility to gzip-encode
its response to an object download request.

The test then confirms that the progress logs generated by the
"git lfs fetch" and "git lfs pull" commands contain an entry with
the full size of the object reported in both the numerator and
denominator of the field which represents the amount of data
downloaded for the object.  Previously, all progress log entries
generated by these commands while downloading a content-encoded
object would have a "-1" in the field's denominator.

Note that any intermediate entries preceding the final log entry
will still have a negative value in this field, but we will address
that issue in a subsequent commit in this PR.

As described in a previous commit in this PR, several of our Git LFS
commands may write progress log messages with a negative value in
the denominator of the fraction representing the amount of data
downloaded so far for an individual object.  Specifically, the
"git lfs fetch", "git lfs pull", and "git lfs smudge" commands
will write such log messages when a progress log is enabled with
the GIT_LFS_PROGRESS environment variable and the response to an
HTTP object download request is content-encoded and therefore lacks
a Content-Length header.

As described in a previous commit in this PR, for historical reasons
progress log messages are written either by the anonymous callback
function created by the CopyCallbackFile() method of the GitFilter
structure in our "lfs" package, or by the logBytes() method of the
Meter structure in our "tq" package.  However, the format of the
messages is identical regardless of which function or method
generates them.

When the logBytes() method was first introduced in commit
710862b of PR #476, it duplicated
the format of the progress log messages generated by the callback
function created by the CopyCallbackFile() method.  The current
progress log message format in that method dates from commit
36fa23d of PR #100, but the original
version of the method and its callback function themselves were added
in commit fd41906 of PR #97.

While we likely should refactor the logBytes() method and the
CopyCallbackFile() method's callback function to share a common
implementation, that change falls outside of the scope of this PR,
so we defer it to a future PR.

However, one of the changes we made in a prior commit in this PR
now ensures that if an object's total size is initially unknown,
when an EOF condition is encountered while reading the object's
data, then a final progress log message is guaranteed to be written
with the total size of the object represented as equal to the
actual number of bytes that were read.  Further, this final
progress log entry will be output regardless of whether the
CopyCallbackFile() method's callback function or the logBytes()
method is being used to write messages to the log file.

While this change affects the content of the last progress log
entry written for a given object, until the EOF condition is
reached, intermediate log messages may be generated which still
have a negative value in the denominator of the fraction
representing the amount of the object's data processed so far.

These negative values occur because the length of the data to be
read is initially unknown.  At present, the only circumstance in
which this is possible is when the HTTP response to an object
download request lacks a Content-Length header, as will be the
case when the response body is gzip-encoded, as we noted in the
following comment on PR #6011:

  #6011 (comment)

PR #6011 resolves a bug in our "git lfs filter-progress" and
"git lfs clean" commands whereby they can pass an incorrect
"totalSize" parameter to the CopyWithCallback() function.  The only
solution for this problem is for the commands to simply not attempt
to determine the total size of an object.  As a consequence, we
expect that our "git lfs filter-progress" and "git lfs clean" commands
will never know in advance the length of the object data they are
reading from Git, and it will become a more common occurrence for
the CopyCallbackFile() method's callback function and the logBytes()
method to receive a "total" parameter with a negative value.

We therefore now refine the format of the progress log when the
total size of an object is unknown by writing a series of question
mark characters ("?") in place of the denominator of the fraction
which represents the amount of the object's data processed so far.
We adjust the number of these question mark characters to match
the length of the numerator, e.g., "123/???" and "12345/?????".

To avoid implementing the necessary string formatting logic in
both the CopyCallbackFile() method's callback function and the
logBytes() method, we introduce a new FormatFraction() function
to our "tools" package and then substitute its output in place
of the appropriate fraction field in the log messages.

We then add a TestFormatFraction() Go test function which verifies
that the FormatFraction() function generates the expected output
under a range of conditions, including when the number of bytes
read so far exceeds the value provided as the total size of an
object, and when the total size of an object is unknown and should
be represented using various numbers of question mark characters.

Finally, we update the comments in several of the tests we added
to the "t/t-progress.sh" shell script in a prior commit of this PR,
and we revise the "initial total unknown" test case for our
TestCopyCallbackFileThrottle() test function.  In that test case,
we update the expected contents of the progress log because the
intermediate entry which precedes the final entry should now have
a series of question mark characters rather than a "-1" value in
the denominator of the fraction which represents the amount of the
object's data processed so far.