{{ message }}
Repository navigation
fix: language files are never parsed where file(1) is unavailable - #5872
Merged
Merged
Conversation
Fixes ossf#5066. The checker selection in scan_file read: valid_file = valid_file | (file in output) if output else (file == basename) The conditional binds the whole right-hand side, so the two branches are not "accumulate one way or the other" — they are accumulate, or ASSIGN. With `output` empty the loop overwrote `valid_file` on every iteration and only the last checker name in the dict could ever match. `output` is filled by is_linux_kernel() only when `file` is on PATH, and `file` is a Unix utility. So on a Windows host without it, 17 of the 18 language file types are silently never parsed — `Package.resolved` survives alone, by sorting last. Measured against the real `valid_files` dict: conan.lock pubspec.lock .env go.mod pom.xml package-lock.json yarn.lock .control .pyd cpanfile composer.lock requirements.txt PKG-INFO: METADATA: renv.lock Gemfile.lock Cargo.lock -> all dead Package.resolved -> the only survivor The reported symptom, a jar producing an SBOM with no Java component and a scan exiting 0, is one instance. Java, Go, npm, yarn, Python, Rust, Ruby, PHP, Dart, Perl, R, conan and opkg are all affected the same way, and a vulnerability scanner reporting clean because it never looked is the expensive part. WHY THE WINDOWS CI JOB DID NOT CATCH IT. `windows-latest` carries Git for Windows, which ships `file.exe` in `usr\bin`. `inpath("file")` is therefore True on the runner, CI takes the same branch Linux does, and the defect is invisible there. Confirmed on this machine: `where file` resolves to `C:\Program Files\Git\usr\bin\file.exe`, and the bug only appears once that directory is removed from PATH. Verified end to end on Windows 10, driving the real VersionScanner against test/language_data/pom.xml: file on PATH parser reached before, CI-like env True True (masked) before, reporter's env False False <- the bug after, CI-like env True True after, reporter's env False True The loop only accumulated, so it is now the `any()` it always was, which also removes the precedence trap rather than re-parenthesising around it. The minimal alternative is one pair of brackets; happy to switch if preferred. Two notes on the issue thread, since they cost fifteen months: - The missing "Done scanning file" is a red herring. That line is logged at the end of run_checkers, which scan_file only calls for executables and kernel images, so it never appears for a language file on any platform. - Line endings are not involved. The pom.xml inside log4j-core-2.24.3.jar is pure ASCII with LF, and validate_pom() returns True for it on a cp950 host. The added test patches `inpath` to reproduce a host without `file`, so it exercises the defect on Linux CI too. I could not execute it locally: this test class's setup_class calls get_cvelist_if_stale(), which wants the full CVE database, so the whole class errors at setup here. The behaviour it asserts is the one measured in the table above. Signed-off-by: ppcvote <risky9763@gmail.com>
ppcvote
force-pushed
the
fix/windows-language-file-detection
branch
from
August 10, 2026 07:07
64340d3 to
8dce233
Compare
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fixes #5066.
The jar is a symptom. The defect is in checker selection in
scan_file:The conditional binds the whole right-hand side, so the branches are not "accumulate one way or the other" — they are accumulate, or assign. With
outputempty the loop overwrotevalid_fileevery iteration, and only the last checker name in the dict could ever match.outputis filled byis_linux_kernel()only whenfileis on PATH, andfileis a Unix utility.Scale
Measured against the real
valid_filesdict rather than reasoned about — on a host withoutfile, 17 of 18 language file types are never parsed:So this is not only jars: Java, Go, npm, yarn, Python, Rust, Ruby, PHP, Dart, Perl, R, conan and opkg all produce empty SBOM components on such a host, and the scan exits 0. For a vulnerability scanner, reporting clean because it never looked is the expensive part of the bug.
Why the Windows CI job never caught this
windows-latestcarries Git for Windows, which shipsfile.exeinusr\bin.inpath("file")is True on the runner, CI takes the same branch Linux does, and the defect cannot appear there.Confirmed on this machine —
where fileresolves toC:\Program Files\Git\usr\bin\file.exe, and the bug only shows once that directory is off PATH. That is also why @smirgel saw it on a plain Windows Server box and @terriko could not reproduce it.Verification
Windows 10, driving the real
VersionScanneragainsttest/language_data/pom.xml:fileon PATHTwo dead ends in the thread, since they cost fifteen months
The missing
Done scanning fileis a red herring. That line is logged at the end ofrun_checkers, whichscan_fileonly calls for executables and kernel images. It never appears for a language file, on any platform. @smirgel's observation was accurate but does not indicate the failure.Line endings are not involved. The
pom.xmlinsidelog4j-core-2.24.3.jaris pure ASCII with LF endings, andvalidate_pom()returnsTruefor it on a cp950 Windows host. I checked this first, since it was the standing hypothesis.The change
The loop only ever accumulated, so it is now the
any()it always was. That removes the precedence trap rather than re-parenthesising around it. The minimal alternative is one pair of brackets:Happy to switch if you prefer the smaller diff.
On the test
It patches
inpathto reproduce a host withoutfile, so it exercises the defect on Linux CI rather than needing a Windows runner without Git.I could not execute it locally, and would rather say so than imply otherwise:
TestLanguageScanner.setup_classcallsget_cvelist_if_stale(), so the entire class errors at setup without the full CVE database. The behaviour it asserts is exactly what the table above measures directly.Worth considering separately:
_validate_xmland this selection both fail closed and silently, and a scan that parses nothing is indistinguishable from a scan that found nothing. A single warning when a file matchedPARSER_MATCH_FILENAMESbut no parser ran would have surfaced this in a day rather than fifteen months. Happy to send that as its own PR if it is wanted.