gpufault: trust a pod's fault event only from its own node by samhita-alla · Pull Request #8153 · flyteorg/flyte · GitHub
Skip to content

gpufault: trust a pod's fault event only from its own node - #8153

Open
samhita-alla wants to merge 4 commits into
gpu-forward/child-pod-gpu-faultfrom
gpu-forward/gpufault-trusted-pod-events
Open

samhita-alla wants to merge 4 commits into
gpu-forward/child-pod-gpu-faultfrom
gpu-forward/gpufault-trusted-pod-events

Conversation

@samhita-alla

@samhita-alla samhita-alla commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Why

A GPU fault event is only worth believing when the fault emitter on the pod's own node wrote it. The emitter's credentials only let it record events for its own node, so an event reported from any other node did not come from the emitter watching the pod's GPUs. The executor's failure classifier checked the event reason and the regarding UID, but not the reporting node, and its event watcher did not record the reporting node at all. A consumer that wanted a typed fault for an attached pod event had to parse the free-text message, with none of these checks.

What

  • gpufault gains trust.go. FromPodEvent(PodEvent, Pod) returns the typed fault only for a trusted event: a GPUXidError or GPUSXidError reason, recorded against this very pod, and reported from the pod's node. The trade the classifier already made stays and is documented: when the pod is gone and only its name is left, the name match is accepted. A pod with a UID but no node was never scheduled and has no trusted faults. IsFaultReason is exported for callers that filter before logging.
  • pluginsCore.ReasonInfo gains GpuFault, set only for an event that passed FromPodEvent.
  • The executor's event watcher records the reporting node of each event: reportingInstance, else the deprecated source host.
  • classifyGpuFailure uses FromPodEvent for its per-event checks, so a fault reported from another node no longer reclassifies a failure. RelevantToFailure still bounds the search in time.
  • The executor sets ReasonInfo.GpuFault on the pod events it attaches to a phase, and toClusterEvents copies it onto ClusterEvent.gpu_fault. The message is never parsed for a fault there.

Tests

  • New table in gpufault/trust_test.go covering each rule and each trade, a wrong reason with a valid-looking message, and an SXid.
  • plugin_manager_test: a fault reported from another node does not classify, and attached reasons carry a fault only when trusted. event_watcher_test: the reporting node is recorded and survives aggregation.
  • go test -race passes for gpufault, pluginmachinery/core and executor/pkg/plugin/k8s, and the lint diff against main is clean. The new toClusterEvents spec is in the controller's Ginkgo suite, which needs envtest binaries and was not run locally.

A downstream consumer that tags faults on pod events switches to FromPodEvent once this lands.

🤖 Generated with Claude Code

A GPU fault event is now believed only when its reason is one the fault
emitter writes, it was recorded against this very pod, and it was
reported from the node the pod ran on. The emitter's credentials only
let it record events for its own node, so an event reported from any
other node did not come from the emitter watching the pod's GPUs. A pod
that has a UID but no node was never scheduled and has no faults. When
the pod is gone and its UID and node are unknown, the name match is
accepted, as before.

gpufault.FromPodEvent is the single place this rule lives. The event
watcher records the reporting node of each event, classifyGpuFailure
uses FromPodEvent for its per-event checks, and every reason attached
to a phase now carries its fault as data on ReasonInfo.GpuFault when
the event passes the rule. toClusterEvents copies that fault onto the
cluster event, so consumers no longer need to parse message text to get
a typed fault.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the flyte2 label Oct 9, 2026
samhita-alla and others added 3 commits October 12, 2026 16:05
I, Samhita Alla <aallasamhita@gmail.com>, hereby add my Signed-off-by to this commit: 51e881d

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@EngHabu
EngHabu changed the base branch from main to gpu-forward/child-pod-gpu-fault October 14, 2026 16:03

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants