perf(interpreter): near-linear variable resolver (fix hang on large stacks) by swirle13 · Pull Request #212 · patrickchugh/terravision · GitHub
Skip to content

perf(interpreter): near-linear variable resolver (fix hang on large stacks) - #212

Open
swirle13 wants to merge 1 commit into
patrickchugh:mainfrom
swirle13:perf/near-linear-variable-resolver
Open

perf(interpreter): near-linear variable resolver (fix hang on large stacks)#212
swirle13 wants to merge 1 commit into
patrickchugh:mainfrom
swirle13:perf/near-linear-variable-resolver

Conversation

@swirle13

@swirle13 swirle13 commented Sep 4, 2026

Copy link
Copy Markdown

Summary

modules.interpreter.find_replace_values can hang for many minutes (and grow to
multiple GB of RSS) on larger stacks. This PR makes the variable resolver
near-linear so those stacks render in seconds, while leaving the resolved
output of normal stacks unchanged.

This is unrelated to #209 (that fixed an interactive terraform stdin prompt
hang). This one is an algorithmic blow-up inside the resolver itself.

Root cause

find_replace_values resolves a resource attribute string by regex-scanning it
for var. / local. / data. / module. references and substituting each.
The problem is in replace_module_vars: when a referenced module output's
value itself contains further unresolved references, it recurses by calling
find_replace_values on the entire attribute string again (not just on the
sub-value), once per reference:

# replace_module_vars(...)
if ("module." in out_val and ...) or "var." in out_val or "local." in out_val:
    value = find_replace_values(value, mod, tfdata, recursion_depth + 1)
    continue

For an attribute with R references this fans out roughly O(R^depth) (capped
only by recursion_depth >= 50), and the working string can grow on each pass as
outputs expand. The pathological shape is common: e.g. a jsonencode({...})
secret assembled from several other modules' endpoints/ARNs, where those values
are themselves references.

The fix (within one handle_metadata_vars pass)

  1. Memoize (value, module) -> resolved result.
  2. Re-entrancy guard — if the same (value, module) is already being
    resolved on the call stack, return it unchanged. That recursion cannot make
    further progress on that exact string and is the source of the fan-out.
  3. Bounded-cost guard — a per-attribute call budget (_FRV_MAX_CALLS) and a
    working-string length cap (_FRV_MAX_LEN). This only fires on values the
    resolver could never fully resolve anyway (they would otherwise reach the
    existing recursion_depth >= 50"UNKNOWN" path) — so it fails in bounded
    time instead of after a long hang.

Caches are module-level and cleared at the start of each handle_metadata_vars
pass, so nothing leaks across CLI invocations.

Before / after (real-world)

Measured on real AWS Terraform stacks (planfile mode, so only the enrichment
pipeline is timed):

Stack shape Before After
~115 resources, one jsonencode() secret referencing ~8 module outputs did not complete — killed at ~18 min, >2 GB RSS ~33 s total
~650 resources did not complete resolver completes (~6.5 min); the remaining time is Graphviz layout, unrelated to this change

Profiling the 15 enrichment steps on the ~115-resource stack showed
resolve_all_variables was the sole hotspot (every other step < 0.1 s);
after the patch it is ~27 s and the whole run is ~33 s.

Correctness

  • Normal attributes resolve in a handful of small calls and never hit the
    caps, so their resolved values are unchanged.
  • Verified on a stack that already worked pre-patch (CloudFront/WAF/S3/etc.): the
    resulting diagram is structurally identical — same node set and same edges;
    the only diff is TerraVision's own random per-run node UUIDs.

Notes

  • _FRV_MAX_CALLS / _FRV_MAX_LEN are module constants and easy to tune; happy
    to make them configurable (env var / flag) or to split this into a
    memoization-only change plus a separate discussion on the cap if you'd prefer.
  • A deeper follow-up would be to change replace_module_vars to resolve the
    referenced output value in isolation and substitute it, rather than
    re-resolving the whole string — but that is a larger, behavior-sensitive change
    and I kept this PR minimal.
  • Developed with AI assistance.

…rge stacks)

find_replace_values re-resolves the entire attribute string once per nested
reference (via replace_module_vars), which is O(refs**depth) and can hang for
minutes / exhaust memory on large stacks whose attributes reference several
module outputs (e.g. a jsonencode() secret built from other modules' endpoints).

This adds, within a single handle_metadata_vars pass:
- Memoization of (value, module) -> resolved result.
- A re-entrancy guard that returns the string unchanged when the same
  (value, module) is already being resolved on the stack (that recursion cannot
  make progress and is the source of the exponential fan-out).
- A per-attribute bounded-cost guard (call budget + working-string length) so a
  value the resolver could never fully resolve anyway (it would otherwise reach
  the existing recursion_depth>=50 'UNKNOWN' path) bails out in bounded time
  instead of after a hang.

Normal attributes resolve in a handful of small calls and never hit the caps,
so their resolved output is unchanged.
@swirle13
swirle13 force-pushed the perf/near-linear-variable-resolver branch from a808873 to cd5c859 Compare September 4, 2026 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant