{{ message }}
Repository navigation
gpufault: trust a pod's fault event only from its own node - #8153
Open
samhita-alla wants to merge 4 commits into
Open
samhita-alla wants to merge 4 commits into
samhita-alla wants to merge 4 commits into
Conversation
A GPU fault event is now believed only when its reason is one the fault emitter writes, it was recorded against this very pod, and it was reported from the node the pod ran on. The emitter's credentials only let it record events for its own node, so an event reported from any other node did not come from the emitter watching the pod's GPUs. A pod that has a UID but no node was never scheduled and has no faults. When the pod is gone and its UID and node are unknown, the name match is accepted, as before. gpufault.FromPodEvent is the single place this rule lives. The event watcher records the reporting node of each event, classifyGpuFailure uses FromPodEvent for its per-event checks, and every reason attached to a phase now carries its fault as data on ReasonInfo.GpuFault when the event passes the rule. toClusterEvents copies that fault onto the cluster event, so consumers no longer need to parse message text to get a typed fault. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
I, Samhita Alla <aallasamhita@gmail.com>, hereby add my Signed-off-by to this commit: 51e881d Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Why
A GPU fault event is only worth believing when the fault emitter on the pod's own node wrote it. The emitter's credentials only let it record events for its own node, so an event reported from any other node did not come from the emitter watching the pod's GPUs. The executor's failure classifier checked the event reason and the regarding UID, but not the reporting node, and its event watcher did not record the reporting node at all. A consumer that wanted a typed fault for an attached pod event had to parse the free-text message, with none of these checks.
What
Tests
A downstream consumer that tags faults on pod events switches to FromPodEvent once this lands.
🤖 Generated with Claude Code
main