[dynamo]: [dynamo] Fix silent drop of closure mutations in methods of locally instantiated classes (#197896) by syerva-g · Pull Request #198055 · pytorch/pytorch · GitHub
Skip to content

[dynamo]: [dynamo] Fix silent drop of closure mutations in methods of locally instantiated classes (#197896) - #198055

Open
syerva-g wants to merge 5 commits into
pytorch:mainfrom
syerva-g:fix-197896-dynamo-closure-mutation
Open

syerva-g wants to merge 5 commits into
pytorch:mainfrom
syerva-g:fix-197896-dynamo-closure-mutation

Conversation

@syerva-g

@syerva-g syerva-g commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #197896

Summary

When a user-defined class or context manager is instantiated inside a @torch.compile region (self.source is None), UserDefinedObjectVariable still tracks the class provenance via self.cls_source. However:

  1. UserDefinedObjectVariable.resolve_type_attr required self.cls_source is not None and self.source is not None for can_use_mro_source, dropping source_fn when binding methods on newly constructed instances (and similarly in call_method and resolve_data_descriptor).
  2. GenericContextWrappingVariable.enter / exit did not pass source_fn via get_source_by_walking_mro when self.cls_source was present.
  3. NestedUserFunctionVariable.get_function did not register materialized closure cells in side_effects.track_cell_existing when allow_sourced_cells=True (__build_class__).

Because the unbound method lacked a Source, UserFunctionVariable.bind_args built the method's closure cells without a Source, causing SideEffects to treat closure-captured lists/dicts as MutationNew and silently drop mutations upon function exit.

Test Plan

  • pytest test/dynamo/test_functions.py -k test_locally_defined_class_closure_mutation

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @kadeng @chauhang @amjames @jataylo @azahed98

@pytorch-bot

pytorch-bot Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

@pytorch-bot pytorch-bot Bot added ciflow/inductor ciflow/torchtitan Run TorchTitan integration tests module: dynamo labels Sep 21, 2026
@linux-foundation-easycla

linux-foundation-easycla Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: syerva-g / name: syerva-g (a861c3d)

@pytorch-bot

pytorch-bot Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

The following ciflow label(s) have been added but CI has not been triggered yet because the workflows are awaiting approval:

  • ciflow/inductor
  • ciflow/torchtitan

Once a maintainer approves the workflows (scroll to the bottom of the PR page), the corresponding CI jobs will be triggered automatically. Please ping one of the reviewers if you do not have access to approve and run workflows.

1 similar comment
@pytorch-bot

pytorch-bot Bot commented Sep 21, 2026

Copy link
Copy Markdown

The following ciflow label(s) have been added but CI has not been triggered yet because the workflows are awaiting approval:

  • ciflow/inductor
  • ciflow/torchtitan

Once a maintainer approves the workflows (scroll to the bottom of the PR page), the corresponding CI jobs will be triggered automatically. Please ping one of the reviewers if you do not have access to approve and run workflows.

@syerva-g syerva-g closed this Sep 21, 2026
@syerva-g
syerva-g deleted the fix-197896-dynamo-closure-mutation branch September 21, 2026 23:26
@syerva-g syerva-g changed the title [dynamo] Fix silent drop of closure mutations in methods of locally instantiated classes (#197896) [Closed] Sep 21, 2026
@syerva-g
syerva-g restored the fix-197896-dynamo-closure-mutation branch September 22, 2026 00:02
@syerva-g syerva-g reopened this Sep 22, 2026
@syerva-g syerva-g changed the title [Closed] Fix silent drop of closure mutations in methods of locally instantiated classes (#197896) Sep 22, 2026
@syerva-g syerva-g changed the title Fix silent drop of closure mutations in methods of locally instantiated classes (#197896) [dynamo]: [dynamo] Fix silent drop of closure mutations in methods of locally instantiated classes (#197896) Sep 22, 2026

@williamwen42 williamwen42 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Address these concerns discovered.

Comment thread torch/_dynamo/variables/functions.py Outdated
Comment thread torch/_dynamo/variables/user_defined.py
@syerva-g

Copy link
Copy Markdown
Contributor Author

Thanks for the review @williamwen42! I've addressed both review comments in e988db5 and also updated the test_sourceless_* tests in test/dynamo/test_misc.py (which previously relied on A().method having cls_source=None when A was a sourced local class) so they construct a sourceless MappingProxy lookup instead. Could you please re-approve the CI workflow run when you get a chance?

@syerva-g

Copy link
Copy Markdown
Contributor Author

Address these concerns discovered.

Done

@syerva-g syerva-g closed this Sep 24, 2026
@syerva-g syerva-g reopened this Sep 24, 2026
@soulitzer soulitzer added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Sep 25, 2026
@guilhermeleobas

Copy link
Copy Markdown
Collaborator

@syerva-g can you rebase

@syerva-g
syerva-g force-pushed the fix-197896-dynamo-closure-mutation branch from e988db5 to 8004523 Compare September 30, 2026 18:21
@syerva-g

Copy link
Copy Markdown
Contributor Author

@syerva-g can you rebase

done

@guilhermeleobas guilhermeleobas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Just one nit

Comment thread torch/_dynamo/variables/ctx_manager.py Outdated
Comment thread torch/_dynamo/variables/ctx_manager.py Outdated
Comment thread torch/_dynamo/variables/user_defined.py Outdated
Comment thread test/dynamo/test_functions.py
@syerva-g

syerva-g commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the reviews @guilhermeleobas and @rtimpe! Addressed all comments in 7465dda.
@williamwen42 friendly ping to take another look when you get a chance so we can clear the changes-requested status and merge.

@williamwen42

Copy link
Copy Markdown
Member

Please address test failures.

@syerva-g
syerva-g force-pushed the fix-197896-dynamo-closure-mutation branch from 19781ab to 4c59c57 Compare October 2, 2026 23:13
@syerva-g

syerva-g commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

@williamwen42 Fixed the test_sourceless_bound_* failure (removed @staticmethod on _make_sourceless_cls_getter so make_test_cls_with_patches in test_dynamic_shapes / test_nested_graph_breaks_wrapped binds self properly) and rebased onto latest main in 4c59c57. Could you please approve the workflow run when you get a chance?

@syerva-g

syerva-g commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Please address test failures.

@williamwen42 tests pass now, the 3 tests failing seems unstable and not related to this change in particular

@syerva-g
syerva-g force-pushed the fix-197896-dynamo-closure-mutation branch from 4c59c57 to fc1f8d7 Compare October 6, 2026 18:36
@syerva-g

syerva-g commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto latest main (fc1f8d7) to pick up the upstream fixes for the unrelated Inductor CI flakes. All Dynamo tests (test_functions.py and test_misc.py) pass on the previous run.

@williamwen42 @guilhermeleobas @rtimpe — could you please approve the CI workflow run and take another look when you get a chance so we can clear the changes-requested status and merge? Thanks!

@williamwen42 williamwen42 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI found some more soundness concerns related to changing function attributes.

Comment thread torch/_dynamo/variables/ctx_manager.py Outdated

def enter(self, tx: "InstructionTranslatorBase") -> VariableTracker:
source = None if self.source is None else AttrSource(self.source, "__enter__")
fn_source = self.cls_source and self.get_source_by_walking_mro(tx, "__enter__")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI-generated review comment - Finding 1: Guard newly MRO-sourced functions

This new MRO-derived fn_source is passed to UserFunctionVariable directly. That source lets bind_args reconstruct closure cells, but it does not guard that the function at the source is still the one Dynamo traced. Rebinding Helper.value therefore reuses the cached graph: the old return value is retained and the old mutation is replayed into the replacement function's closure. The same unguarded construction occurs in UserDefinedObjectVariable.call_method (including __init__) and both context-manager entry/exit paths.

Existing-work / severity: PARTIALLY_TRACKED. #197860 and #198124 cover the ordinary resolve_type_attr bound-method case, so that subcase is existing and its severity is lowered. This finding is retained for the untracked PR-specific delta: #198055 newly makes MRO-derived sources reachable in call_method/__init__ and context-manager __enter__/__exit__ without installing the function guard.

Reproduction:

import torch
from torch._dynamo.testing import CompileCounterWithBackend
from torch.testing._internal.common_utils import run_tests, TestCase


class TestMethodRebind(TestCase):
    def test_locally_instantiated_method_rebind(self):
        old_log = []
        new_log = []

        class Helper:
            def value(self):
                old_log.append("old")
                return 1

        def new_value(self):
            new_log.append("new")
            return 2

        def fn(x):
            return x + Helper().value()

        backend = CompileCounterWithBackend("eager")
        compiled = torch.compile(fn, backend=backend, fullgraph=True)
        out1 = compiled(torch.ones(1))
        Helper.value = new_value
        out2 = compiled(torch.ones(1))

        self.assertEqual(out1, torch.tensor([2.0]))
        self.assertEqual(out2, torch.tensor([3.0]))
        self.assertEqual(old_log, ["old"])
        self.assertEqual(new_log, ["new"])
        self.assertEqual(backend.frame_count, 2)


if __name__ == "__main__":
    run_tests()

Observed on the exact current base and head:

base 4eab5fba: out1=[2] out2=[2] old_log=[]      new_log=[]      frames=1
head fc1f8d74: out1=[2] out2=[2] old_log=['old'] new_log=['old'] frames=1
guarded fix:  out1=[2] out2=[3] old_log=['old'] new_log=['new'] frames=2

The head result is particularly problematic: it silently applies the traced function's mutation to the replacement function's closure. I also reproduced this for __init__ and __enter__ rebinding.

Suggested fix:

Use UserFunctionVariable.create_with_source whenever the function source came from the class MRO, retaining the existing fallback when no MRO source is available:

# UserDefinedObjectVariable.call_method
if source_fn is not None:
    fn_vt = variables.UserFunctionVariable.create_with_source(method, source_fn)
else:
    fn_vt = variables.UserFunctionVariable(
        method, source=source and AttrSource(source, "__func__")
    )
return UserMethodVariable(fn_vt, self, source=source).call_function(
    tx, args, kwargs
)

# UserDefinedObjectVariable.resolve_type_attr
if var_source is not None:
    fn_vt = variables.UserFunctionVariable.create_with_source(
        type_attr, var_source
    )
else:
    fn_vt = variables.UserFunctionVariable(type_attr, source=fn_source)
return variables.UserMethodVariable(fn_vt, self, source=source)

# GenericContextWrappingVariable.enter/exit
if fn_source is not None:
    fn_vt = variables.UserFunctionVariable.create_with_source(fn, fn_source)
else:
    fn_vt = variables.UserFunctionVariable(fn, source=fn_source)

create_with_source installs the required CLOSURE_MATCH guard. PR #198124 covers the ordinary resolve_type_attr site when composed with this change, but not call_method or context-manager entry/exit, so those sites still need to be handled here or coordinated explicitly.

Verification used source-spliced worktrees at exact base 4eab5fba (then-current origin/main, gap 0) and head fc1f8d74; the only environment adaptation was a compatibility-only HealthCheckHook import shim. All five tests added by this PR still passed with the guarded patch.

type_attr = unpatched_nn_module_init

can_use_mro_source = self.cls_source is not None and self.source is not None
can_use_mro_source = self.cls_source is not None

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI-generated review comment - Finding 2: Recover sources for lru_cache and __getattr__ methods

Widening can_use_mro_source fixes the plain-function branch, but two adjacent method paths still ignore it. The lru_cache branch passes the incoming instance source, which is None for a locally instantiated object, and call_getattr_fallback derives __getattr__ only from self.source, which is also None. Their closure cells therefore remain sourceless and their mutations are silently dropped even though both functions compile with fullgraph=True.

Existing-work / severity: EXISTING-IN-SCOPE; lowered. Both reproductions fail identically on the base and head, so this is not a regression introduced by #198055. It is an incomplete-fix gap within #197896: that issue tracks dropped closure mutations broadly, but no live issue or PR separately covers the lru_cache and custom __getattr__ variants.

Reproduction (lru_cache(0) disables caching so this only tests closure handling):

import functools

import torch
from torch._dynamo.testing import CompileCounter
from torch.testing._internal.common_utils import run_tests, TestCase


class TestMissingMethodSources(TestCase):
    def test_lru_cache_method_closure_mutation(self):
        log = []

        class Helper:
            @functools.lru_cache(0)
            def touch(self):
                log.append("touch")

        def fn(x):
            Helper().touch()
            return x + 1

        counter = CompileCounter()
        compiled = torch.compile(fn, backend=counter, fullgraph=True)
        self.assertEqual(compiled(torch.ones(1)), torch.tensor([2.0]))
        self.assertEqual(compiled(torch.ones(1)), torch.tensor([2.0]))
        self.assertGreater(counter.frame_count, 0)
        self.assertEqual(log, ["touch", "touch"])

    def test_getattr_closure_mutation(self):
        log = []

        class Helper:
            def __getattr__(self, name):
                log.append(name)
                return 1

        def fn(x):
            return x + Helper().missing

        counter = CompileCounter()
        compiled = torch.compile(fn, backend=counter, fullgraph=True)
        self.assertEqual(compiled(torch.ones(1)), torch.tensor([2.0]))
        self.assertEqual(compiled(torch.ones(1)), torch.tensor([2.0]))
        self.assertGreater(counter.frame_count, 0)
        self.assertEqual(log, ["missing", "missing"])


if __name__ == "__main__":
    run_tests()

Exact base and head both produced:

GETATTR frames=1 log=[]
LRU     frames=1 log=[]
FAILED (failures=2)

Suggested fix:

Give the wrapper's underlying function the MRO source while preserving the bound-method source. For __getattr__, keep the bound-method and function sources separate and use create_with_source for the MRO-derived function so this does not introduce Finding 1's rebinding bug:

elif is_lru_cache_wrapped_function(type_attr):
    fn_source = source
    if can_use_mro_source:
        fn_source = self.get_source_by_walking_mro(tx, name)
    fn_vt = variables.WrapperUserFunctionVariable(
        type_attr, "__wrapped__", source=fn_source
    )
    return variables.WrapperUserMethodVariable(fn_vt, self, source=source)
method_source = None
if self.source:
    method_source = AttrSource(self.source, "__getattr__")

if self.cls_source is not None:
    fn_source = self.get_source_by_walking_mro(tx, "__getattr__")
    fn_vt = variables.UserFunctionVariable.create_with_source(
        getattr_fn, fn_source
    )
else:
    fn_source = method_source and AttrSource(method_source, "__func__")
    fn_vt = variables.UserFunctionVariable(getattr_fn, source=fn_source)

out = variables.UserMethodVariable(
    fn_vt, self, source=method_source
).call_function(tx, [variables.ConstantVariable.create(name)], {})

With this patch, the same test produced:

GETATTR frames=2 log=['missing', 'missing']
LRU     frames=2 log=['touch', 'touch']
OK

Verification used source-spliced worktrees at exact base 4eab5fba and head fc1f8d74; the only environment adaptation was the compatibility-only import shim noted in Finding 1. The PR's five new tests also passed with this patch.

Comment thread test/dynamo/test_functions.py Outdated
return x + 1

with torch._dynamo.config.patch(enable_trace_load_build_class=True):
g_nonlocal = torch.compile(f_nonlocal, backend="eager")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI-generated review comment - Finding 3: Separate redundant build-class coverage from missing supported-path coverage

There are two independent coverage concerns here:

  1. test_build_class_in_compiled_fn_nonlocal_rebind exercises behavior already covered more precisely by test_misc.py::MiscTests::test_build_class_closure_rebinding, which asserts that rebinding a materialized sourceless build-class cell remains unsupported under fullgraph=True. Because this new test omits fullgraph=True, it succeeds through graph-break/eager fallback and does not exercise this PR's production change. It can be dropped as redundant.
  2. Separately, the supported locally defined class path still lacks coverage for nonlocal cell reassignment. The existing tests and Findings 1/2 cover container mutation, function-identity guarding, and the lru_cache/__getattr__ branches; none covers STORE_DEREF through an ordinary method whose class is defined outside the compiled function.

The distinction is observable on the exact current base and head:

added build-class test:                 base PASS, head PASS
added build-class test + fullgraph=True: base Write to immutable cell,
                                          head Write to immutable cell
supported-path test below:              base Write to immutable cell,
                                          head PASS (n == 2)

Suggested fix:

Drop the newly added fallback-only build-class test because that unsupported path is already pinned elsewhere. Independently add the following supported-path regression test alongside the other new closure-mutation tests.

Suggested test:

def test_locally_defined_method_nonlocal_rebind(self):
    n = 0

    class Helper:
        def touch(self):
            nonlocal n
            n += 1

    def fn(x):
        Helper().touch()
        return x + 1

    compiled = torch.compile(fn, backend="eager", fullgraph=True)
    self.assertEqual(compiled(torch.ones(1)), torch.tensor([2.0]))
    self.assertEqual(compiled(torch.ones(1)), torch.tensor([2.0]))
    self.assertEqual(n, 2)

This is not a semantic replacement for the existing build-class test: one confirms the unsupported build-class boundary, while the new test fills missing coverage for the supported ordinary-method path.

@syerva-g

syerva-g commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the great catch @williamwen42! You're completely right on both points:

  1. Missing CLOSURE_MATCH guard when constructing UserFunctionVariable with an MRO source: Passing source=method_source directly to UserFunctionVariable(...) bypassed UserFunctionVariable.create_with_source, so install_guard(source.make_guard(GuardBuilder.CLOSURE_MATCH)) was not called when self.source is None. I audited all places where functions/callables are constructed from get_source_by_walking_mro and updated UserDefinedObjectVariable (call_method, resolve_type_attr, invoke_descriptor_get, and call_getattr_fallback) as well as GenericContextWrappingVariable (enter and exit) to use UserFunctionVariable.create_with_source (and propagated the MRO source to LruCacheVariable and __getattr__ as well).
  2. Test coverage (test_build_class_in_compiled_fn_nonlocal_rebind): Replaced with test_locally_defined_method_nonlocal_rebind and test_locally_instantiated_method_rebind in test/dynamo/test_functions.py, which mutate the outer nonlocal referenced by the method across calls to the compiled function (failing without create_with_source and passing now), plus added tests for @functools.lru_cache methods and __getattr__.

Ready for another look whenever you have a chance!

@williamwen42

Copy link
Copy Markdown
Member

Please address test failures.

@syerva-g
syerva-g force-pushed the fix-197896-dynamo-closure-mutation branch from ae4eca8 to 8c883e2 Compare October 14, 2026 18:28
@syerva-g

Copy link
Copy Markdown
Contributor Author

…cells (pytorch#197896)

Fixes pytorch#197896.

When a locally defined class is instantiated inside a compiled function
(e.g. `Helper().touch()`), the instance itself is sourceless (`self.source is None`),
but its class source (`self.cls_source`) is available. Use `self.cls_source` to
resolve the method/property/__enter__/__exit__ source via MRO walking so that
`UserFunctionVariable.bind_args` tracks closure cells with a valid `ClosureSource`
and replays in-place mutations to captured containers.
…k xfail

- call_getattr_fallback: inspect.getattr_static(type(value), '__getattr__') also finds a __getattr__ defined on the metaclass (EnumType.__getattr__ on Python < 3.12), which is not in type(value).__mro__.  Walking the MRO for it raised Unsupported, which _load_attr's python-constant fast path turned into a real AttributeError -> InternalTorchDynamoError (test_compilation_evnum_hasattr_fail on 3.11).  Only source/guard __getattr__ when a class in the MRO defines it.
- test_nested_graph_breaks_wrapped: drop the expectedFailure for test_udf_classes_reconstruction_nested_graph_breaks -- the MRO-sourced CLOSURE_MATCH guard now makes it recompile correctly when the class is rebound, so it was reported as 'Unexpected success'.
- test_functions: noqa B019 on the intentional lru_cache method and reflow an overlong assertEqual for PYFMT.
…recompile test

test_default_guard_filter_keeps_the_pytree_registry_keys_match counts the
guards that the default guard filter keeps over a control filter.  With this
PR, methods resolved through the class MRO of a locally constructed object are
built with UserFunctionVariable.create_with_source, which installs a
CLOSURE_MATCH guard (serialized as FUNCTION_CODE_MATCH).  pytree.tree_flatten
constructs a TreeSpec during tracing, so TreeSpec.__init__ and
TreeSpec.__post_init__ now get code checks -- exactly the guards that make a
rebinding of those methods recompile instead of silently reusing the graph.
Update the expected FUNCTION_CODE_MATCH count from 4 to 6 and document which
functions they cover.
The previous commit installed CLOSURE_MATCH on every MRO-sourced method,
including methods of objects that already have a source. Upstream sources
those through the class MRO too but does not guard them (pytorch#197860, which
pytorch#198124 addresses), and precompile now relies on that: since pytorch#200823,
test_package.py's unnamed-scope tests call an exec()-defined method of a
module-level object, and the new guard on it cannot be made portable, so
saving the package raised "CLOSURE_MATCH guard cannot be serialized".

Keep the guard for the sources this PR introduces - methods of objects
constructed during tracing, which are reachable only through cls_source -
and give objects that have a source back exactly the function source they
had before.
@syerva-g
syerva-g force-pushed the fix-197896-dynamo-closure-mutation branch 2 times, most recently from b8802ba to 4be1753 Compare October 21, 2026 13:26

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/inductor ciflow/torchtitan Run TorchTitan integration tests module: dynamo open source release notes: dynamo triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

6 participants