Repository navigation
DAGs with external python task are not killed when marked as Failed. #74298
Replies: 2 comments 2 replies
|
This is a known bug in the 3.2 line, and it's fixed in Airflow 3.3.1. When you mark the task as failed, the next heartbeat from the task gets a 409 and the supervisor kills the task process. That's the "Server indicated the task shouldn't be running anymore" line in your log. In 3.2.2 the signal only goes to the task-runner process itself. Your It was reported in #65505 and fixed by #65738. The fix puts the task-runner in its own process group and sends the signal to the whole group with If you can't upgrade yet, one workaround is to have the long-running function notice that its parent is gone and exit on its own: import os
import sys
import time
parent = os.getppid()
while True:
if os.getppid() != parent:
sys.exit(1)
print("alive", flush=True)
time.sleep(1)That only helps for code that loops like this, but it stops the orphaned process from running forever. |
This is a known bug in the 3.2 line, and it's fixed in Airflow 3.3.1.
When you mark the task as failed, the next heartbeat from the task gets a 409 and the supervisor kills the task process. That's the "Server indicated the task shouldn't be running anymore" line in your log. In 3.2.2 the signal only goes to the task-runner process itself. Your
@task.external_pythonfunction runs in a separate Python process that the task-runner starts withsubprocess.Popen, so it never receives the signal. Once the task-runner is killed, that child gets reparented and keeps running, which is why your log keeps printingaliveafter the termination message.It was reported in #65505 and fixed by #65738. Th…