{{ message }}
Repository navigation
fix(meta_analyzer): parse stringified findings array from LLM - #91
Merged
rng1995 merged 1 commit intoJun 22, 2026
Merged
Conversation
MetaAnalyzerResult had a field_validator to repair `overall_assessment`
when an LLM returns it as a JSON string, but no equivalent for `findings`.
Some models/endpoints (e.g. Claude Sonnet via an OpenAI-compatible
gateway) return the top-level `findings` array as a JSON string, which
fails Pydantic list validation and crashes the scan with exit code 2
before any report is written.
Add a mirrored `field_validator("findings", mode="before")` that
json.loads a string and falls back to an empty list on parse failure or
non-list payloads, matching the existing `overall_assessment` handling.
Covered by new unit tests in TestMetaAnalyzerResultFindingsValidator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rng1995
approved these changes
Jun 21, 2026
rng1995
left a comment
rng1995
left a comment
Collaborator
There was a problem hiding this comment.
Clean, well-justified fix. Approving.
This mirrors the existing overall_assessment mode="before" validator for the top-level findings array, so models/gateways that return findings as a stringified JSON array no longer crash the scan with a list_type error after all per-file analyzers ran. The fallback to [] on parse failure or non-list payloads matches the established pattern, and the tests cover the stringified-array, native-list, invalid-string, and non-list-JSON cases. Sensible and low-risk.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
MetaAnalyzerResulthas afield_validatorthat repairsoverall_assessmentwhen an LLM returns it as a JSON string (mode="before"→json.loads), but there is no equivalent validator for the top-levelfindingsarray.Some models/endpoints — reproducibly Claude Sonnet via an OpenAI-compatible gateway (
SKILLSPECTOR_PROVIDER=openaipointed at a router) — returnfindingsas a stringified JSON array rather than a real list. This fails Pydantic'slist_typevalidation and crashes the scan:The process exits with code 2 and no report is written, even though all per-file analyzers ran successfully. I hit this on multiple real scans (a skill and an MCP-server repo).
Fix
Add a mirrored
field_validator("findings", mode="before")thatjson.loadsa string payload, falling back to an empty list on parse failure or a non-list result — exactly matching the existing_parse_stringified_assessmentbehavior foroverall_assessment.Tests
New
TestMetaAnalyzerResultFindingsValidatorintests/nodes/test_llm_analyzer_base.pycovering:MetaAnalyzerFindingobjects[][]Full file suite passes (
83 passed);ruff check/ruff formatclean.🤖 Generated with Claude Code