Report what the launcher logged when a signal test fails by danielhanchen · Pull Request #9079 · unslothai/unsloth · GitHub
Skip to content

Report what the launcher logged when a signal test fails - #9079

Merged
danielhanchen merged 3 commits into
mainfrom
kaggle-signal-diagnostics
Aug 17, 2026
Merged

Report what the launcher logged when a signal test fails#9079
danielhanchen merged 3 commits into
mainfrom
kaggle-signal-diagnostics

Conversation

@danielhanchen

Copy link
Copy Markdown
Member

tests/kaggle/test_launch_cleanup.py drives a real subprocess, signals it, and then discards its output. When a signal test fails, the message says only what the exit status was.

The discarded output contains the one line that decides where to look. _install_release_handlers logs received signal N; deleting kernels before exiting the moment its handler runs, so its presence separates two different faults:

  • the signal never reached the handler, or
  • the handler ran and the process still exited 0.

Why now

CI produced the second symptom on a runner whose checkout carries the handler fix from #9072 (verified against the branch, not assumed). It does not reproduce locally at four cores under -n 4, across repeated runs. The only evidence that could distinguish those two cases was being thrown away.

The change

Both failure paths report the tail:

  • the assertions on exit status and on deletion include it;
  • the wait is routed through a helper, so a launcher that never dies names the handler too rather than raising a bare TimeoutExpired. Killing first is what lets the pipe reach EOF so the tail can be read at all.

The read happens only after the process has exited, so it cannot block.

Verification

Against two mutations of the handler, one exiting 0 and one returning without dying, each failure now carries:

Launcher said: [launch] received signal 15; deleting kernels before exiting

No behaviour changes and no bound moves. This is diagnostics only.

These tests drive a real subprocess and then throw its output away, so a failure
says only what the exit status was. The one line that decides where to look is
discarded: _install_release_handlers logs 'received signal N' the moment its
handler runs, which separates 'the signal never reached the handler' from 'the
handler ran and the process still exited 0'.

That matters right now. CI produced the second symptom on a runner carrying the
handler fix, and it does not reproduce locally at four cores under -n 4, so the
only evidence available is what the child said.

Both failure paths report it. The wait is routed through a helper so a launcher
that never dies names the handler too, rather than a bare TimeoutExpired; killing
first is what lets the pipe reach EOF so the tail can be read. The read happens
only after the process has exited, so it cannot block.

Verified against two mutations of the handler, one exiting 0 and one returning
without dying: each failure now carries '[launch] received signal 15; deleting
kernels before exiting'.
@danielhanchen

Copy link
Copy Markdown
Member Author

@chatgpt-codex-connector

Copy link
Copy Markdown

@danielhanchen
danielhanchen merged commit 9156c36 into main Aug 17, 2026
16 of 17 checks passed
@danielhanchen
danielhanchen deleted the kaggle-signal-diagnostics branch August 17, 2026 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant