We first introduced support for the use of HTTP/1.1 chunked
transfer encoding during Git LFS object uploads in commit
962be37 of PR git-lfs#386, and in that
commit we added the original version of what is now the "chunked
transfer encoding" test in our "t/t-chunked-transfer-encoding.sh"
shell test script.

The initial version of that script was located in a directory named
"test" and was named "test-chunked-transfer-encoding.sh".
Then in PR git-lfs#3125 we renamed the "test" directory to "t", and in
commit 219b7e0 of that PR we also
renamed all the shell test scripts within the directory to have
the prefix "t-" instead of "test-".

One unintended consequence of this change, though, was that it
would have silently rendered the "chunked transfer encoding" test
ineffective, except that for several unrelated reasons the test
was already ineffective and had been so since its initial version.

We explain the chain of events which led to this state of affairs
in more detail below, but the key reason why we have not detected
any of these problems until now is that the test is simply a near-
duplicate of the generic, example "happy path" test from our
"t/t-happy-path.sh" script.  As such it relies entirely on our
"lfstest-gitserver" utility program to modify its behaviour after
recognizing the name of the test's repository, and to cause the
test to fail if the Git LFS client does not behave as the server
expects.

However, the server has never correctly validated that the client
will send a Transfer-Encoding header with the value "chunked" when
instructed to do so.  Because the shell test does not verify this
itself, the test has always trivially passed, even as subsequent
changes further degraded the implementation of the related code in
our server utility.

To help resolve these issues and prevent a similar recurrence in
the future, we start by replacing the test with two new tests
in the "t/t-batch-storage-encoding.sh" script that we added recently
in PR git-lfs#6207.

The first of these tests, named "batch storage HTTP upload with
chunked transfer encoding", creates a single Git LFS object and
uploads it to the test server utility, as did the original "chunked
transfer encoding" test.  Unlike the original test, though, our new
test captures the trace log output from the Git LFS client and
verifies using the grep(1) command that the headers of the PUT request
used to upload the object include a Transfer-Encoding header with the
value "chunked".  Further, the test also checks that the PUT request's
headers did not include a Content-Length header.

Because this test independently verifies the presence in the trace
log of a Transfer-Encoding header from the object upload PUT request,
we are now guaranteed that the test itself will catch any future
regression of the Git LFS client's support for chunked transfer
encoding, even if our "lfstest-gitserver" program fails to do so.

We want to avoid the possibility that our test's "grep" commands could
match against the HTTP headers captured in the trace log from the
client's Git LFS Batch and Locking API requests, which precede the
final object upload PUT request.  To exclude these prior requests from
the log, we use the awk(1) command to match against the PUT request
line, and then only output that line and all subsequent lines into
a new "put.log" file.

Unlike the legacy "chunked transfer encoding" test, our new test
simply checks that its Git LFS object is successfully uploaded, as
there is no particular value in then performing a download operation
as well.  The legacy test only included this extra step because the
whole test was copied from the sample "happy path" test without
modification, and that test illustrates both upload and download
operations.

Our second new test, named "batch storage HTTP upload without transfer
encoding", performs a similar set of steps as our first new test, but
assumes that the client has not been instructed to use chunked
transfer encoding.  The test therefore checks that the headers of the
object upload PUT request do not include a Transfer-Encoding header,
but do include a Content-Length header whose value is the size of
the object.

While this test does not require the "lfstest-gitserver" utility to
act differently than normal, our first new test obviously does, as
we need the server to instruct the Git LFS client to include a
Transfer-Encoding header when uploading objects.

As with the original "chunked transfer encoding" test from PR git-lfs#386,
we expect the test server utility to change its behaviour when it
recognizes the name of the test's repository.  Both the utility's
lfsBatchHandler() and storageHandler() functions were revised in that
PR to call the testingChunkedTransferEncoding() function, which is
intended to return "true" if the name of the repository begins with
a specific prefix.

Since the initial implementation of the "chunked transfer encoding"
test, the name of the repository the test creates has been derived
from the name of the enclosing shell script.  Because the script was
named "test/test-chunked-transfer-encoding.sh", the server utility's
testingChunkedTransferEncoding() function was written to compare a
request's URL path with the prefix "/test-chunked-transfer-encoding",
as our shell test suite makes Git LFS Batch API requests whose first
URL path component is the name of a repository.

This design worked as intended for Batch API requests (at least until
the changes from PR git-lfs#3125).  The Git LFS client would make a Batch API
request to upload a single object, and the server utility would return
a JSON response whose "header" element contained a "Transfer-Encoding"
key with the value "chunked".  The client then uploaded the object
using a PUT request with a "Transfer-Encoding: chunked" header.

Our "lfstest-gitserver" utility handles PUT requests with its
storageHandler() function, and the intent of the changes from
962be37 of PR git-lfs#386 was that this
function would call the testingChunkedTransferEncoding() function
to determine if the PUT request should include a Transfer-Encoding
header or not.  If the testingChunkedTransferEncoding() function
returned "true", then the storageHandler() function would check
for a Transfer-Encoding header with the value "chunked", and if one
was not provided by the client, log an error and halt the server.
If this occurred, the client would have then reported an error and
exited, causing the test to fail.

However, even if the Git LFS client did not send a Transfer-Encoding
header when instructed to by the Batch API response, the test server
utility's storageHandler() function could never have detected this
because its call to the testingChunkedTransferEncoding() function
always received a return value of "false".

The reason the testingChunkedTransferEncoding() function would always
return "false" when called by the storageHandler() function is that
the URLs generated by the "lfstest-gitserver" utility for individual
Git LFS objects always begin with the path prefix "/storage/" rather
than a repository name.  We do now include the repository name in
these requests as the value of a query parameter named "r", but
this enhancement was only made subsequent to PR git-lfs#386, specifically
in commit 27cf051 of PR git-lfs#463.
Even so, the testingChunkedTransferEncoding() function still only
checks the initial component of a request's URL path, and so can
never return "true" when passed an object upload request.

To resolve this long-standing problem, we now revise the
testingChunkedTransferEncoding() function so that it requires a
repository name as its sole parameter, and then returns "true" if
that name matches a specific string (without a leading URL path
separator, since repository names are simply directory names).
Since the function is called from both the lfsBatchHandler() and
storageHandler() functions, we update each of these to pass the
appropriate "repo" variable to the testingChunkedTransferEncoding()
function.  In the case of the lfsBatchHandler() function, the
"repo" variable is populated from the URL path of the current
request, whereas in the storageHandler() function the variable
is set with the value of the "r" query parameter.

We also set the repository name created by our new "batch
storage HTTP upload with chunked transfer encoding" test to
"batch-storage-upload-encoding-chunked", and update the
testingChunkedTransferEncoding() function to compare its input
parameter against that name.  This repository name better reflects
both the name of the test and the name of the test's shell script,
and as such aligns with the repository names used by the other
tests in the same script.

We next address another problem with the storageHandler() function
and the way it responds if the testingChunkedTransferEncoding()
function returns "true".  In the original implementation from commit
962be37 of PR git-lfs#386, if the
storageHandler() function were to receive a return value of "true"
from the testingChunkedTransferEncoding() function, the
storageHandler() function would then check that the current request
included a Transfer-Encoding header with the value of "chunked".

If this value was not found, then the Fatal() function from the Go
standard library's "log" package would be called, causing the server
to halt, which in turn would cause the client to exit with a
non-zero status code and the test to fail.  Note, though, that as
we described above, this sequence of events could never actually
occur because the testingChunkedTransferEncoding() function would
always return "false" to the storageHandler() function.

When we introduced the debug() function to the "lfstest-gitserver"
utility in commit e2046fd of PR git-lfs#1375,
we replaced calls to the Fatal() function of the "log" package with
calls to the debug() function.  Our debug() function does not cause
the calling program to halt immediately, unlike the "log" package's
Fatal() function.

In the other two instances where we replaced the Fatal() function
with our own debug() function, this change did not alter the
effectiveness of our tests.  In these instances, while the server
would no longer halt if an unexpected condition occurred, it still
proceeded to return a 4xx status code to the Git LFS client, which
would in turn cause the currently-executing test to fail.

However, the storageHandler() function depended on the "log" package's
Fatal() function to cause the server to exit immediately, and did
not implement any other indication to the client that because a
Transfer-Encoding header was expected but not received, an error
condition had occurred.

To resolve this problem, we change the storageHandler() function
so that if it finds that the Git LFS client has not sent a
"Transfer-Encoding: chunked" header when it was instructed to do so,
the function now returns a 400 Bad Request HTTP status code to the
client.  This will cause the client to retry the request, and then
after repeatedly receiving 400 status code responses, the client
will report an error and exit with a non-zero status, causing the
test to fail.

We also take the opportunity to simplify the code used by the
storageHandler() function to check for a "Transfer-Encoding: chunked"
header.  Previously, the function had to iterate through the list of
values in the TransferEncoding field of the Request structure from
the Go standard library's "http" package, checking each value against
the string "chunked".  We can now replace this explicit loop with
a call to the Contains() function of the Go standard library's
"slices" package, since that package was added in Go v1.21.

Finally, because our new "batch storage HTTP upload with chunked
transfer encoding" test creates a repository with the fixed name of
"batch-storage-upload-encoding-chunked", and because we have modified
the testingChunkedTransferEncoding() function to check for that
specific repository name, we have implicitly addressed the third
problem with our existing "chunked transfer encoding" test.

As noted above, in commit 219b7e0
of PR git-lfs#3125 we renamed all our shell test scripts to begin with "t-"
rather than "test-", and an unintended consequence of this change
was that the testingChunkedTransferEncoding() function would no longer
recognize the repository name from the "chunked transfer encoding"
test, even when handling Batch API requests made by the test.

As the test was originally written, it passed the name of the script
containing the test to the basename(1) command and then used the
result as the name of the test's primary Git repository.  Since the
test's script was initially named "test-chunked-transfer-encoding.sh",
the testingChunkedTransferEncoding() function checked for a URL path
which began with the prefix "/test-chunked-transfer-encoding" in
order to match the name of the test's repository.

When the script was renamed to "t-chunked-transfer-encoding.sh"
in PR git-lfs#3125, though, the test's repository name automatically
changed to "t-chunked-transfer-encoding".  As a result, the
testingChunkedTransferEncoding() function could no longer recognize
the repository name, and so the lfsBatchHandler() function stopped
sending Batch API responses with "Transfer-Encoding" keys in the
"header" element.  Since the test made no independent check that
the Git LFS client sent a Transfer-Encoding header, the test still
continued to succeed.

Fortunately, the legacy "chunked transfer encoding" test was the
only test affected in this way by the shell script name changes from
PR git-lfs#3125.  While the "lfstest-gitserver" utility at that point had
several functions similar to the testingChunkedTransferEncoding()
function, each of which checked for a specific repository name prefix
in the URL path of a request, the tests which relied on these
functions to recognize their repositories all used fixed names for
their repositories rather than deriving them from the names of the
enclosing scripts.