Description
When a workflow run is re-run on the same commit SHA, GitHub attaches check runs from both the old and new runs to the commit. If a job in the old (superseded) run fails for infrastructure reasons (e.g. expired build artifacts), allcheckspassed picks up that stale failure and reports the overall status as failed — even though the exact same job passes in the newer run.
This is a variant of #102 (which covered CANCELLED checks from superseded runs). In this case the stale check has conclusion FAILURE, not CANCELLED.
Reproduction
Real-world example: kubeflow/pipelines#13325
- Workflow run
25758917901 was created on May 12 for commit 8fee435.
- On May 14, attempt 2 (re-run) of that workflow was triggered. The job
End to End E2EEssential Tests - K8s v1.34.0 cacheEnabled=false argoVersion=v3.5.14 failed because the image artifacts from the original May 12 build had expired:
ERROR: stat images_25758917901/apiserver/apiserver.tar: no such file or directory
- Meanwhile, a fresh workflow run
25859935988 was also triggered on May 14 for the same commit. The exact same job passed in this run.
allcheckspassed evaluated all check runs on the commit, saw the FAILURE from the old re-run, and reported passed: false — blocking CI.
Root Cause
Same as #102: the deduplication logic keeps the check with the highest job ID per name, without considering which workflow run it belongs to. A failed job from a stale re-run can have a higher job ID than the passing job from the current run, causing the failure to win.
Even when the passing job has a higher ID, the action doesn't distinguish between runs — it sees both a FAILURE and a SUCCESS for the same job name and the outcome depends on which one the dedup happens to keep.
Expected Behavior
When multiple workflow runs exist for the same commit, allcheckspassed should only evaluate check runs from the most recent run of each workflow. Stale runs (whether cancelled, re-run, or superseded) should not influence the result.
Suggested Fix
Same as proposed in #102: group check runs by their workflow run, identify the most recent run per workflow name, and only evaluate checks from that run. This would prevent stale failures, cancellations, and expired-artifact errors from blocking CI.
Description
When a workflow run is re-run on the same commit SHA, GitHub attaches check runs from both the old and new runs to the commit. If a job in the old (superseded) run fails for infrastructure reasons (e.g. expired build artifacts),
allcheckspassedpicks up that stale failure and reports the overall status as failed — even though the exact same job passes in the newer run.This is a variant of #102 (which covered
CANCELLEDchecks from superseded runs). In this case the stale check has conclusionFAILURE, notCANCELLED.Reproduction
Real-world example: kubeflow/pipelines#13325
25758917901was created on May 12 for commit8fee435.End to End E2EEssential Tests - K8s v1.34.0 cacheEnabled=false argoVersion=v3.5.14failed because the image artifacts from the original May 12 build had expired:25859935988was also triggered on May 14 for the same commit. The exact same job passed in this run.allcheckspassedevaluated all check runs on the commit, saw theFAILUREfrom the old re-run, and reportedpassed: false— blocking CI.Root Cause
Same as #102: the deduplication logic keeps the check with the highest job ID per name, without considering which workflow run it belongs to. A failed job from a stale re-run can have a higher job ID than the passing job from the current run, causing the failure to win.
Even when the passing job has a higher ID, the action doesn't distinguish between runs — it sees both a
FAILUREand aSUCCESSfor the same job name and the outcome depends on which one the dedup happens to keep.Expected Behavior
When multiple workflow runs exist for the same commit,
allcheckspassedshould only evaluate check runs from the most recent run of each workflow. Stale runs (whether cancelled, re-run, or superseded) should not influence the result.Suggested Fix
Same as proposed in #102: group check runs by their workflow run, identify the most recent run per workflow name, and only evaluate checks from that run. This would prevent stale failures, cancellations, and expired-artifact errors from blocking CI.