feat: add async_update_interfaces to rescan network interfaces at runtime by bdraco · Pull Request #1797 · python-zeroconf/python-zeroconf · GitHub
Skip to content

feat: add async_update_interfaces to rescan network interfaces at runtime#1797

Merged
bdraco merged 23 commits into
masterfrom
feat/async-update-interfaces
Jun 22, 2026
Merged

feat: add async_update_interfaces to rescan network interfaces at runtime#1797
bdraco merged 23 commits into
masterfrom
feat/async-update-interfaces

Conversation

@bdraco

@bdraco bdraco commented Jun 21, 2026

Copy link
Copy Markdown
Member

Closes #1777.

Summary

Adds a runtime API to reconcile the sockets zeroconf uses to the live interface set, so an interface that appears or disappears no longer requires recreating the whole Zeroconf. async_update_interfaces (with a sync update_interfaces wrapper) adds a responder socket for each interface that showed up, tears down the socket for each interface that went away, and re-announces existing registrations when a new sender appeared.

It accepts the same interfaces, ip_version, and apple_p2p arguments as the constructor, each defaulting to the construction-time value, so a caller reconfigures with the same vocabulary it started with. The listen socket is rebuilt when the new set needs a different address family; unicast mode is fixed at construction.

How this maps to #1777

  1. Works live, never tears down listeners for interfaces that haven't changed — an interface present in both the old and new set keeps its exact transport object; no teardown, no rebind, no packet loss on it.
  2. Accepts all the arguments we accept at startupinterfaces, ip_version, apple_p2p.
  3. Only tears down listeners for addresses that changed — the diff keys on (address, scope_id); only senders whose key left the desired set are closed, only keys that are new get a socket. An IP change is remove-old + add-new.

Behavior

  • No-op when nothing changed: if the reconciled set matches what is already bound, no socket is touched and nothing is re-announced. Re-announcement only fires when a new sender was actually added (a returning/new interface needs to advertise; removals need no re-announce on the survivors).
  • Listen socket adapts when it must: a rescan that keeps the same address family never touches the listen socket; a set that needs a new family rebuilds it (re-joining the interfaces that stay), and moving a Default single-family instance to an explicit list converts its dual-use listen/responder socket to a pure listener, then adds per-interface responders for the whole set. The conversion rebuilds the socket clean so the new joins do not collide with the memberships it already held, and demotes it first so nothing double-announces.

Details

  • Zeroconf.__init__ retains interfaces, ip_version, and apple_p2p; the rescan re-runs normalize_interface_choice against the live set and brings each interface up through the same net.add_interface primitive construction uses, so setup and rescan stay in lockstep.
  • (address, scope_id) keying lives on _WrappedTransport.interface_key (paired with _interface_key for the desired side), so link-local IPv6 addresses that repeat across interfaces don't collide; _WrappedTransport.multicast_interface gives the value membership calls take.
  • New drop_multicast_member in net.py is the inverse of add_multicast_member (IP_DROP_MEMBERSHIP / IPV6_LEAVE_GROUP), swallowing the same benign errnos (including WSAEINVAL on Windows).
  • Removal is proactive: the gone sender's protocol and reader/sender wrappers are dropped from the engine lists, then the transport is closed.
  • Re-announce piggybacks on the existing _async_broadcast_service; the listener answers on whichever transport received the query, so a new reader needs no extra wiring. RFC 6762 section 8.3.
  • AsyncZeroconf.async_update_interfaces delegates to Zeroconf, mirroring async_register_service.

How a consumer drives this (e.g. Home Assistant)

A consumer that constructs Zeroconf(interfaces=[...]) with an explicit address list calls async_update_interfaces(new_list) from its own adapter-change signal; the diff reconciles the new list against the bound sockets and leaves unchanged ones in place. Consumers with native adapter-change events use this directly; those without can opt into the polling monitor in the follow-up PR. This is the socket-side half of cases like home-assistant/core#59553 (it re-announces whatever ServiceInfo the consumer registered; correcting announced URL/TXT content after a network change is the consumer's job via async_update_service).

Test plan

  • tests/test_interface_update.py covers no-op (no sockets touched, no re-announce), default-to-stored-choice, ip_version/apple_p2p overrides, keep-unchanged-sender-untouched (same transport object), remove and re-add, re-announce on add, failed join, None responder rollback, the dual-use guard, the unicast no-listen-socket path, and the sync wrapper, plus _WrappedTransport.interface_key / multicast_interface / _interface_key unit tests.
  • test_update_interfaces_reconciles_mixed_set drives the engine diff over a 4-sender v4+v6 set in one pass: keeps unchanged, drops gone, adds new, and distinguishes two interfaces sharing fe80::1 by scope_id (scope-2 kept, scope-3 dropped) end to end, asserting each gone interface leaves its group with its own representation.
  • tests/utils/test_net.py::test_drop_multicast_member covers success, IPv6, benign errnos, reraise, no-IPv6, and the Windows WSAEINVAL path.
  • _engine and _transport diff fully covered (lines and branches); compiled (REQUIRE_CYTHON=1) and pure-Python suites green; pre-commit run clean.

@codecov

codecov Bot commented Jun 21, 2026

Copy link
Copy Markdown

@codspeed-hq

codspeed-hq Bot commented Jun 21, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 21 untouched benchmarks


Comparing feat/async-update-interfaces (c2df902) with master (797fdc0)

Open in CodSpeed

@bdraco bdraco force-pushed the fix/self-heal-dead-transport branch from 37e3ceb to 530b7ff Compare June 21, 2026 23:29
@bdraco bdraco force-pushed the feat/async-update-interfaces branch from cc6dd24 to f624cb2 Compare June 21, 2026 23:31
@bdraco bdraco force-pushed the feat/async-update-interfaces branch from f624cb2 to f7b6624 Compare June 21, 2026 23:50
@bdraco bdraco changed the base branch from fix/self-heal-dead-transport to master June 21, 2026 23:50
@bdraco bdraco requested a review from Copilot June 22, 2026 00:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a runtime API to rescan and reconcile Zeroconf’s active sockets with the current set of network interfaces, avoiding the need to recreate a Zeroconf instance when interfaces appear/disappear.

Changes:

  • Add async_update_interfaces() (and sync update_interfaces()) to reconcile per-interface responder sockets and re-announce existing registrations.
  • Introduce drop_multicast_member() as the inverse of add_multicast_member() for proactive multicast membership teardown.
  • Add targeted test coverage for interface updates and multicast leave behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/zeroconf/_core.py Stores construction parameters and adds sync/async interface-rescan APIs plus re-announce behavior.
src/zeroconf/_engine.py Implements sender/socket reconciliation logic and multicast membership add/drop integration.
src/zeroconf/_utils/net.py Adds drop_multicast_member() to leave multicast groups with benign-error handling.
src/zeroconf/asyncio.py Adds AsyncZeroconf.async_update_interfaces() wrapper delegating to Zeroconf.
tests/test_interface_update.py New tests for interface diffing helpers and runtime interface update scenarios.
tests/utils/test_net.py Adds unit tests for drop_multicast_member().

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/zeroconf/_engine.py Outdated
Comment thread src/zeroconf/_engine.py Outdated
Comment thread tests/test_interface_update.py Outdated
Comment thread tests/test_interface_update.py Outdated
Comment thread src/zeroconf/_utils/net.py
@bluetoothbot

bluetoothbot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Previous review — superseded by a newer review below.

@bluetoothbot bluetoothbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking issues found.

  • IPv6 link-local diff key collapses interfaces sharing an address
  • drop_multicast_member omits WSAEINVAL handled by its inverse

@bdraco bdraco force-pushed the feat/async-update-interfaces branch from f7b6624 to 14d468f Compare June 22, 2026 01:19
@bdraco

bdraco commented Jun 22, 2026

Copy link
Copy Markdown
Member Author

Thanks, both review points addressed in the latest push:

IPv6 link-local diff key collision (bluetoothbot + Copilot): _interface_bind_address and _wrapped_bind_address now return (address, scope_id) instead of the bare address string, so the same link-local address on two interfaces no longer collapses to one key. The desired side reads the scope_id from the normalized interface tuple and the current side reads it from getsockname()[3], which are the same value the socket binds with, so an unchanged interface still matches itself across a rescan. Helper tests updated to assert the tuple key and a non-collision case.

drop_multicast_member WSAEINVAL (bluetoothbot + Copilot): now mirrors add_multicast_member and adds WSAEINVAL to the benign set on win32. New tests/utils/test_net.py::test_drop_multicast_member_wsaeinval covers it.

Also added test_update_interfaces_keeps_unchanged_sender_untouched, which pins the core invariant: an interface present in both the old and new set keeps its exact transport object (no teardown, no rebind), while only the gone interface's transport is closed.

@bdraco bdraco force-pushed the feat/async-update-interfaces branch from 14d468f to 9c65842 Compare June 22, 2026 01:28
@bluetoothbot

Copy link
Copy Markdown
Contributor

@bdraco: Thanks, both review points addressed in the latest push:

Verified. Scope_id keying sound. Desired reads interface[0][2]; current reads getsockname()[3]. Same value socket binds with. Link-local collision resolved. Dual-use guard and rollback correct. Three minor suggestions left inline: missing log on IPv6 inet_pton failure, unsurfaced per-interface add failure, silent duplicate sender on Default-to-list switch.

@bluetoothbot

bluetoothbot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Previous review — superseded by a newer review below.

@bluetoothbot bluetoothbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found.

@bdraco bdraco force-pushed the feat/async-update-interfaces branch 2 times, most recently from 4029b73 to b2f6c89 Compare June 22, 2026 02:00
@bluetoothbot

bluetoothbot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Previous review — superseded by a newer review below.

@bluetoothbot bluetoothbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking issues found.

  • IPv6 multicast leave uses scope_id where the join used the interface index

@bdraco bdraco force-pushed the feat/async-update-interfaces branch from b2f6c89 to 320f800 Compare June 22, 2026 02:10
@bdraco

bdraco commented Jun 22, 2026

Copy link
Copy Markdown
Member Author

Good catch on the IPv6 membership asymmetry — fixed in the latest push.

Important #1 (IPv6 leave used scope_id where join used the index): the responder socket already holds the join index in IPV6_MULTICAST_IF (set by new_respond_socket), so I capture it at wrap time. _WrappedTransport now carries multicast_index, read via getsockopt(IPPROTO_IPV6, IPV6_MULTICAST_IF) in make_wrapped_transport, and multicast_interface uses it for the v6 leave instead of the bound scope_id. Leave and join are now symmetric for global IPv6 (scope_id 0, index N) as well as link-local, and it works for both startup-created and rescan-added senders since both go through make_wrapped_transport. The diff key (interface_key) stays scope_id-based since both sides derive from the same bind, as you noted.

New tests:

  • test_wrapped_multicast_interface now pins that the leave value carries multicast_index (9), not the bound scope_id (5).
  • test_make_wrapped_transport_captures_v6_multicast_index confirms the index is read via getsockopt.
  • test_update_interfaces_reconciles_mixed_set drops a global-style v6 sender and asserts the leave uses its join index.

Suggestion #1: drop_multicast_member now logs the same IPv6-disabled info line its inverse emits before returning False.

Suggestion #3 / silent partial failure: _async_add_interface now logs (debug) naming the interface when a join fails or no responder socket is created. Surfacing the failed set to the caller via the return type would be a larger API change; left as-is to match construction semantics (the constructor also skips unusable interfaces), but the per-interface skip is now observable.

@bluetoothbot bluetoothbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking issues found.

  • getsockopt re-raise widens the startup failure surface for every IPv6 transport

make_wrapped_transport runs on the startup/connection path; re-raising a
failed IPV6_MULTICAST_IF read there could abort instance startup to protect
multicast_index, which only selects the interface for a benign group leave
that drop_multicast_member already tolerates. Fall back to the default index
with a debug log instead. Apply the same graceful fallback to
_listen_socket_supports (assume dual-stack) so a getsockopt failure can't
abort a rescan, and the two paths agree. Soften the _core config-commit
comment, which overstated the invariant on a partial reconcile failure.
@bdraco

bdraco commented Jun 22, 2026

Copy link
Copy Markdown
Member Author

Good catch on the startup blast radius. Fixed in 37f6b1a:

  • make_wrapped_transport no longer re-raises a failed IPV6_MULTICAST_IF read on non-Windows. It runs on the startup/connection path (connection_made for every transport), and the index it protects only selects the interface for a future group leave that drop_multicast_member already treats as benign, so a read failure now falls back to the default index 0 with a debug log instead of aborting setup.
  • _listen_socket_supports gets the same graceful fallback (assume dual-stack + debug log) so a getsockopt failure can't abort a rescan, and the two paths now agree. This supersedes the re-raise from the previous round; degrading is the right trade given make_wrapped_transport can't safely abort startup.
  • Softened the _core config-commit comment, which overstated the invariant: a mid-reconcile failure may have changed some sockets, so the comment now just says the stored config is left unchanged rather than claiming it matches the bound sockets.

On the two silent-failure notes (surface failed/degraded interfaces to the caller): keeping that best-effort by design for now; the per-interface warnings name the interface, and an aggregate failed-interface return is a deliberate API addition I'd do as a follow-up rather than fold into this PR.

@bluetoothbot

bluetoothbot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Previous review — superseded by a newer review below.

@bluetoothbot bluetoothbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found.

…rt bring-up

A failed IPV6_V6ONLY read assumes dual-stack (returning False could loop
rebuilds when the rebuilt socket's read also fails), but if the socket really
were v6-only that skips a needed rebuild and strands an added IPv4 family, so
log it at warning rather than debug. Document that interface bring-up is
best-effort, and note the benign group leave on the rebuilt listen socket.
@bdraco

bdraco commented Jun 22, 2026

Copy link
Copy Markdown
Member Author

Addressed in c93f537:

  • The IPV6_V6ONLY read-failure fallback now logs at warning, not debug. You're right that assume-dual-stack is the risky direction (skips a needed rebuild, strands a v4 family) and that interface updates may never recur; I kept assume-dual-stack rather than force-rebuild because returning False could loop rebuilds when the rebuilt dual-stack socket's read also fails, but warning makes a real failure visible. Comment updated to spell out that trade.
  • Documented best-effort bring-up in the async_update_interfaces docstring: a requested interface that fails to bind or re-join is logged, not raised (alongside the existing best-effort re-announce note).
  • Added the clarifying comment for the benign group leave on the rebuilt listen socket in the close loop.

The programmatic partial-reconcile signal (return failed interfaces) stays a deliberate follow-up, as you noted; no API change in this PR.

@bluetoothbot

bluetoothbot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Previous review — superseded by a newer review below.

@bluetoothbot bluetoothbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking issues found.

  • normalize_interface_choice can raise RuntimeError on an empty interface set, bypassing best-effort contract

… bind failure

normalize_interface_choice raises RuntimeError for an All/Default instance
that transiently resolves to zero addresses during adapter churn; catch it as
a logged no-op so a momentary down state doesn't crash a caller's
adapter-change handler. Make a per-interface endpoint-creation failure roll
back and skip (log_warning_once) rather than abort the whole reconcile, so
other interfaces still come up and get re-announced, matching the documented
best-effort contract. Log a non-Windows IPV6_MULTICAST_IF read failure at
warning (Windows WSAEINVAL stays debug) since a wrong-index leave leaks the
membership.
@bdraco

bdraco commented Jun 22, 2026

Copy link
Copy Markdown
Member Author

All three addressed in 423f875:

  • Important Fix ServiceInfo __repr__ #1 (transient empty set): an All/Default instance that momentarily resolves to zero addresses during adapter churn made normalize_interface_choice raise straight into the caller's handler. Now caught as a logged no-op (current sockets left intact), so a transient down state doesn't crash the adapter-change handler. Test added.
  • Suggestion Fix ServiceInfo __repr__ #1 (endpoint-creation failure aborting the reconcile): _async_add_interface now rolls back and skips with log_warning_once instead of re-raising, so other interfaces still come up and get re-announced. This matches the documented best-effort contract and the add_interface-is-None path. The rebuild's listen-socket wrap failure still raises (can't continue without a listen socket). Tests updated.
  • Silent-failure (make_wrapped_transport debug log): a non-Windows IPV6_MULTICAST_IF read failure now logs at warning (Windows WSAEINVAL stays debug via a platform-branched level), since a wrong-index leave silently leaks the membership.

@bluetoothbot

bluetoothbot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Previous review — superseded by a newer review below.

@bluetoothbot bluetoothbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found.

Best-effort bring-up should downgrade an expected socket-level failure, not a
real bug. Roll back on any wrap failure, but re-raise anything that is not an
OSError so a TypeError/AttributeError surfaces instead of being deduped into a
one-time 'interface not added' warning.
@bdraco

bdraco commented Jun 22, 2026

Copy link
Copy Markdown
Member Author

On the two notes:

MEDIUM #1 (bare Exception) - fixed in 55ca5ea. Good catch; that was a regression from making the wrap failure best-effort. _async_add_interface now rolls back on any failure but only swallows OSError; a non-OSError (TypeError/AttributeError etc.) propagates so a real bug isn't downgraded to a deduped 'interface not added' warning. Added a test that a TypeError propagates after rollback.

MEDIUM #2 (_listen_socket_supports assume-dual-stack) - holding deliberately. This exact line has now been flagged in four directions across reviews (silent True, re-raise on non-win32, degrade to True, warning, now force-rebuild). force-rebuild is not strictly better: returning False on a getsockopt failure rebuilds to a dual-stack AF_INET6 socket whose IPV6_V6ONLY read fails the same way, so the next rescan rebuilds again - churning the listen socket on every reconcile. getsockopt(IPV6_V6ONLY) on a valid AF_INET6 socket effectively never fails on any platform, so this branch is defensive against the near-impossible; assume-dual-stack is the non-looping choice and the warning makes the rare event visible. A real fix (cache V6ONLY at socket creation, or a caller-visible degraded-family signal) is the deferred API follow-up, not a log-level or fallback-direction tweak. Leaving as-is unless a maintainer wants the churn tradeoff.

@bluetoothbot

Copy link
Copy Markdown
Contributor

@bdraco: On the two notes:

Agree on holding MEDIUM #2. force-rebuild loops: rebuilt AF_INET6 socket fails same getsockopt, churns every rescan. assume-dual-stack non-looping; warning surfaces the rare failure. Correct call. Real fix is caching V6ONLY at socket creation, a follow-up.

@bluetoothbot

Copy link
Copy Markdown
Contributor

@bdraco: On the cross-family finding (Important #1): rather than reject family changes, this now supports...

Verified rebuild rollback and staying-interface re-join — sound. One gap: rebuild failure raises through the best-effort API after demoting the dual-use socket from senders. Leaves degraded partial state, crashes adapter-change handler. See inline comment. Demote after rebuild succeeds, or log-and-return-False like the transient empty-set branch.

@bluetoothbot

bluetoothbot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Previous review — superseded by a newer review below.

@bluetoothbot bluetoothbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking issues found.

  • Rebuild failure raises through a best-effort API after mutating sender state

The Default dual-use conversion demoted the socket from senders before the
fallible rebuild, so a rebuild failure left it pulled from senders but not
replaced (instance stops responding) and propagated into the caller's handler.
The explicit demote was redundant; the rebuild removes the old transport from
senders itself, but only after the new socket succeeds. Drop the pre-demote
(keep only the diff-view pop) so a failed rebuild leaves senders intact, and
catch the rebuild failure as a logged no-op to honor the best-effort contract.
@bdraco

bdraco commented Jun 22, 2026

Copy link
Copy Markdown
Member Author

Important #1 - fixed in f0090e4. Genuinely good catch. The Default dual-use conversion demoted the socket from senders before the fallible rebuild, so a rebuild failure left it pulled-but-not-replaced (instance goes silent) and the exception crashed the caller's handler.

The fix: that explicit demote was redundant. _async_rebuild_listen_socket already removes the old transport from senders via _async_remove_transport, but only after the new socket is successfully created and wrapped. So I dropped the pre-demote (keeping only the local diff-view current.pop) and wrapped the rebuild in try/except -> log + return False. Now a rebuild failure (transient bind/endpoint failure during churn) leaves senders untouched (no degraded state) and is a logged no-op rather than a crash, consistent with the transient-empty-set branch. Added tests for both the family-rebuild failure and the dual-use-conversion failure leaving the dual-use sender intact.

Thanks for confirming the MEDIUM #2 hold (assume-dual-stack) - agreed, force-rebuild would churn. The remaining silent-failure note (send-only after a failed re-join) and the partial-failure return value stay the documented best-effort follow-up.

@bluetoothbot

bluetoothbot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Previous review — superseded by a newer review below.

@bluetoothbot bluetoothbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found.

gather(return_exceptions=True) also captures BaseExceptions like
CancelledError, which the isinstance(result, Exception) check skipped, so a
cancelled re-announce vanished silently; re-raise a captured BaseException
instead of swallowing it. Document on the public update_interfaces methods
that apple_p2p on a non-Apple platform raises RuntimeError.
@bdraco

bdraco commented Jun 22, 2026

Copy link
Copy Markdown
Member Author

Addressed the two actionable items in c2df902:

  • Silent-failure Redesign listener system #3 (new): gather(return_exceptions=True) also captures BaseExceptions like CancelledError, which the isinstance(result, Exception) check skipped, so a cancelled re-announce vanished. Now a captured BaseException is re-raised rather than swallowed (and a real Exception is still logged per-service). Test added that a CancelledError propagates.
  • Suggestion Redesign listener system #3: documented on update_interfaces / async_update_interfaces / the AsyncZeroconf delegate that apple_p2p on a non-Apple platform raises RuntimeError (input validation, matching the constructor).

MEDIUM #1 (assume-dual-stack) and #2 (send-only after failed re-join) remain held as the documented best-effort tradeoffs we agreed on; the aggregate partial-failure return is the post-PR follow-up.

@bluetoothbot

Copy link
Copy Markdown
Contributor

@bluetoothbot bluetoothbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found.

@bdraco bdraco merged commit 471feb4 into master Jun 22, 2026
38 checks passed
@bdraco bdraco deleted the feat/async-update-interfaces branch June 22, 2026 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Need a way to reconfigure interfaces at run time

3 participants