Fix airflow db clean never purging the callback table by steveahnahn · Pull Request #70923 · apache/airflow · GitHub
Skip to content

Fix airflow db clean never purging the callback table - #70923

Merged
vatsrahul1001 merged 3 commits into
apache:mainfrom
steveahnahn:fix-db-clean-callback-table
Sep 10, 2026
Merged

vatsrahul1001 merged 3 commits into
apache:mainfrom
steveahnahn:fix-db-clean-callback-table

Conversation

@steveahnahn

@steveahnahn steveahnahn commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

airflow db clean has not purged the callback table since Airflow 3.2.0.

Migration b87d2135fa50 renamed callback_request to callback, but the cleanup configuration in db_cleanup.py kept the old name. A configured table that does not exist is skipped with only a warning, so the entry became a no-op and nothing purges callback any more.

The table does accumulate in practice. Executor callbacks are set to success/failed by the scheduler and never deleted (only Dag-processor callbacks are removed, as they are dispatched). And because deadline.callback_id references callback.id with ON DELETE CASCADE, callback is the parent, so purging deadline leaves its callback rows orphaned permanently.

There was also no workaround available: --tables callback was rejected as an invalid choice, so operators could not purge the table through airflow db clean at all.

Why this was not caught

test_no_models_missing exists to catch exactly this: every model table must either be cleaned or documented as deliberately excluded. It has been passing vacuously. It located the models package by walking up from the test file (Path(__file__).parents[2] / "airflow/models"), a path that stopped existing when the sources moved under airflow-core/src, so it discovered zero models and asserted over an empty set. Confirmed against the pre-fix code: the check passed while callback was in neither the cleanup config nor the exclusion list.

With the check repaired, it reports one more table with no way to be purged: partitioned_asset_key_log. It carries no foreign key, so its rows are left behind when the partition Dag run they describe is cascade-deleted with its dag_run. The only existing delete (in SchedulerJobRunner) covers partition runs invalidated by a rollup-definition change, which is not a retention mechanism.

Only those orphans are purged. Rows whose partition Dag run still exists are the evidence the scheduler evaluates to decide when that pending run fires (SchedulerJobRunner reads them for every pending AssetPartitionDagRun), so a purely age-based delete could silently stall a rollup still accumulating keys past the retention window. They are kept regardless of age via a NOT IN (SELECT id FROM asset_partition_dag_run) filter.

What this changes

First commit — the callback fix:

  • Register the table under its current name, with created_at as the recency column.
  • Purge only callbacks that can no longer run. A callback still awaiting execution owns its deadline row through the ON DELETE CASCADE foreign key, so deleting one would silently drop a deadline that has not fired yet. Dag-processor callbacks carry no state and reference no deadline, so rows outliving the retention window are purged too. The existing TERMINAL_STATES constant is reused rather than re-hardcoding the state literals, since that kind of drift is what caused this bug.
  • Add dependent_tables=["deadline"] so deadline rows are cleaned and archived ahead of their parent, matching how dag and dag_run already handle the same cascade.
  • Drop the sla_miss entry, whose table no longer exists in Airflow 3.

Second commit — the coverage check and the table it found:

  • Walk airflow.models.__path__ instead of reconstructing the path from the test's location, so the check keeps working wherever the sources live (it now discovers 52 models, against 0 before), and assert that models were discovered at all so it can never pass on an empty set again.
  • Register partitioned_asset_key_log for cleanup, restricted to orphaned rows as above.
  • Record the remaining uncovered tables in the exclusion list with the reason each is safe — delete rules read from the live schema after airflow db migrate, not from the ORM metadata: asset_partition_dag_run, asset_watcher, dag_favorite, hitl_detail, hitl_detail_history, task_inlet_asset_reference, deadline_alert, and asset_state_store are all removed by ON DELETE CASCADE from tables that are already cleaned ; asset_state_store is a per-asset key/value store whose rows are upserted in place (bounded, current state rather than history); team is configuration, not run data.

The coverage check cannot pass without the callback fix — with it working, callback is reported as uncovered until the first commit registers it — which is why both land in one PR.

Verification

Verified end to end against real metadata databases, seeding rows and running the real run_cleanup, with a log row as a control to prove the run did actual work:

Case Result
old + success / failed purged
old + NULL state (Dag-processor) purged
old + scheduled / queued / running kept
recent + success kept
unfired deadline owned by a live callback survives
old orphaned partitioned_asset_key_log row purged
old partitioned_asset_key_log row whose partition run still exists kept
recent orphaned partitioned_asset_key_log row kept
control log row purged

The archiving path (the default, skip_archive=False) was exercised on Postgres, MySQL and SQLite, confirming the _airflow_deleted__callback__* table is created and the purged row preserved; a committed test covers it. Removing callback from the config makes the repaired check flag it, so the check now catches the defect that motivated this PR. The full test_db_cleanup.py suite passes on all three backends, and the new tests fail without the source changes.

Compatibility

Removing the two stale entries changes what --tables accepts. A list that mixes them with live tables still works, warning about the unknown names and cleaning the rest:

The following table(s) are not valid choices and will be skipped: ['callback_request', 'sla_miss']
Data prior to ... would be purged from tables ['log', 'job']

A --tables list naming only removed tables now stops with "No tables selected for db cleanup" instead of silently reporting success while doing nothing. That seems preferable, since such an invocation has been a no-op since 3.2.0, but it is a visible change for anyone whose cron passes only those names.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5, Fable 5) following the guidelines

@steveahnahn
steveahnahn force-pushed the fix-db-clean-callback-table branch from 1c1d6a5 to d1872d9 Compare August 1, 2026 17:24
@steveahnahn
steveahnahn marked this pull request as ready for review August 1, 2026 18:18
@steveahnahn
steveahnahn force-pushed the fix-db-clean-callback-table branch 3 times, most recently from 1805a44 to 367f2ad Compare August 1, 2026 20:17
Comment thread airflow-core/tests/unit/utils/test_db_cleanup.py
@steveahnahn
steveahnahn force-pushed the fix-db-clean-callback-table branch 3 times, most recently from 21e2a39 to 31f5e16 Compare August 4, 2026 15:12
@steveahnahn
steveahnahn force-pushed the fix-db-clean-callback-table branch 2 times, most recently from 32064de to a4473b4 Compare August 14, 2026 22:52
@ferruzzi ferruzzi added this to the Airflow 3.3.2 milestone Aug 28, 2026
The table was renamed from callback_request in Airflow 3.2.0 and the
cleanup configuration was not updated with it. A configured table that
does not exist is skipped with only a warning, so these rows were never
deleted and the table grew without bound, with no way for an operator to
purge it.

Only callbacks that can no longer run are removed. A callback still
awaiting execution owns its deadline row through an ON DELETE CASCADE
foreign key, so deleting one would silently drop a deadline that has not
fired yet.

The sla_miss entry is dropped as well, since that table no longer exists
in Airflow 3.
…n coverage check

The check that is meant to catch a metadata table being left out of
airflow db clean looked for the models package at a path relative to the
test file. That path stopped resolving when the sources moved under
airflow-core, so the check examined no models at all and its assertions
held trivially for an empty set. It has not been able to report a missing
table since, which is how the callback table went unpurged for several
releases.

Walking the package's own search path instead keeps the check working
wherever the sources live, and asserting that models were found stops it
from passing on an empty set again.

With the check restored, partitioned_asset_key_log is the one table it
reports that genuinely has no way to be purged: it carries no foreign
key, so its rows are left behind when the partition Dag run they describe
is cascade-deleted with its dag_run. Only those orphans are deleted --
rows whose partition Dag run still exists are the evidence the scheduler
evaluates to decide when that pending run fires, so they are kept
regardless of age. The other uncovered tables are recorded in the
exclusion list with the reason each is safe.
@steveahnahn
steveahnahn force-pushed the fix-db-clean-callback-table branch from f962705 to 523299a Compare September 8, 2026 16:16
Comment thread airflow-core/src/airflow/utils/db_cleanup.py
Comment thread airflow-core/src/airflow/utils/db_cleanup.py Outdated
… db clean

Every callback starts in SCHEDULED, a state outside both the active and the
terminal sets, and an unfired deadline's callback stays there until the deadline
is missed. Deny-listing active states would therefore purge those callbacks and,
through the ON DELETE CASCADE foreign key, silently drop deadlines that have not
fired. The finished states stay allow-listed, so an unknown state keeps its rows.
A SCHEDULED callback is purged only once no deadline references it: deleting a
Dag run cascades away its deadline at the database level, which leaves the
callback behind forever.

@ferruzzi ferruzzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround on that. Approved pending CI passing.

@vatsrahul1001 vatsrahul1001 added backport-to-v3-3-test Backport to v3-3-test type:bug-fix Changelog: Bug Fixes labels Sep 10, 2026
@vatsrahul1001
vatsrahul1001 merged commit 478745a into apache:main Sep 10, 2026
155 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 478745a v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

vatsrahul1001 pushed a commit that referenced this pull request Sep 10, 2026
* Fix airflow db clean never purging the callback table

The table was renamed from callback_request in Airflow 3.2.0 and the
cleanup configuration was not updated with it. A configured table that
does not exist is skipped with only a warning, so these rows were never
deleted and the table grew without bound, with no way for an operator to
purge it.

Only callbacks that can no longer run are removed. A callback still
awaiting execution owns its deadline row through an ON DELETE CASCADE
foreign key, so deleting one would silently drop a deadline that has not
fired yet.

The sla_miss entry is dropped as well, since that table no longer exists
in Airflow 3.

* Purge orphaned partitioned_asset_key_log rows and restore the db clean coverage check

The check that is meant to catch a metadata table being left out of
airflow db clean looked for the models package at a path relative to the
test file. That path stopped resolving when the sources moved under
airflow-core, so the check examined no models at all and its assertions
held trivially for an empty set. It has not been able to report a missing
table since, which is how the callback table went unpurged for several
releases.

Walking the package's own search path instead keeps the check working
wherever the sources live, and asserting that models were found stops it
from passing on an empty set again.

With the check restored, partitioned_asset_key_log is the one table it
reports that genuinely has no way to be purged: it carries no foreign
key, so its rows are left behind when the partition Dag run they describe
is cascade-deleted with its dag_run. Only those orphans are deleted --
rows whose partition Dag run still exists are the evidence the scheduler
evaluates to decide when that pending run fires, so they are kept
regardless of age. The other uncovered tables are recorded in the
exclusion list with the reason each is safe.

* Purge orphaned scheduled callbacks while keeping unfired deadlines in db clean

Every callback starts in SCHEDULED, a state outside both the active and the
terminal sets, and an unfired deadline's callback stays there until the deadline
is missed. Deny-listing active states would therefore purge those callbacks and,
through the ON DELETE CASCADE foreign key, silently drop deadlines that have not
fired. The finished states stay allow-listed, so an unknown state keeps its rows.
A SCHEDULED callback is purged only once no deadline references it: deleting a
Dag run cascades away its deadline at the database level, which leaves the
callback behind forever.

(cherry picked from commit 478745a)
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Sep 11, 2026
* Fix airflow db clean never purging the callback table

The table was renamed from callback_request in Airflow 3.2.0 and the
cleanup configuration was not updated with it. A configured table that
does not exist is skipped with only a warning, so these rows were never
deleted and the table grew without bound, with no way for an operator to
purge it.

Only callbacks that can no longer run are removed. A callback still
awaiting execution owns its deadline row through an ON DELETE CASCADE
foreign key, so deleting one would silently drop a deadline that has not
fired yet.

The sla_miss entry is dropped as well, since that table no longer exists
in Airflow 3.

* Purge orphaned partitioned_asset_key_log rows and restore the db clean coverage check

The check that is meant to catch a metadata table being left out of
airflow db clean looked for the models package at a path relative to the
test file. That path stopped resolving when the sources moved under
airflow-core, so the check examined no models at all and its assertions
held trivially for an empty set. It has not been able to report a missing
table since, which is how the callback table went unpurged for several
releases.

Walking the package's own search path instead keeps the check working
wherever the sources live, and asserting that models were found stops it
from passing on an empty set again.

With the check restored, partitioned_asset_key_log is the one table it
reports that genuinely has no way to be purged: it carries no foreign
key, so its rows are left behind when the partition Dag run they describe
is cascade-deleted with its dag_run. Only those orphans are deleted --
rows whose partition Dag run still exists are the evidence the scheduler
evaluates to decide when that pending run fires, so they are kept
regardless of age. The other uncovered tables are recorded in the
exclusion list with the reason each is safe.

* Purge orphaned scheduled callbacks while keeping unfired deadlines in db clean

Every callback starts in SCHEDULED, a state outside both the active and the
terminal sets, and an unfired deadline's callback stays there until the deadline
is missed. Deny-listing active states would therefore purge those callbacks and,
through the ON DELETE CASCADE foreign key, silently drop deadlines that have not
fired. The finished states stay allow-listed, so an unknown state keeps its rows.
A SCHEDULED callback is purged only once no deadline references it: deleting a
Dag run cascades away its deadline at the database level, which leaves the
callback behind forever.
vatsrahul1001 pushed a commit that referenced this pull request Sep 11, 2026
* Fix airflow db clean never purging the callback table

The table was renamed from callback_request in Airflow 3.2.0 and the
cleanup configuration was not updated with it. A configured table that
does not exist is skipped with only a warning, so these rows were never
deleted and the table grew without bound, with no way for an operator to
purge it.

Only callbacks that can no longer run are removed. A callback still
awaiting execution owns its deadline row through an ON DELETE CASCADE
foreign key, so deleting one would silently drop a deadline that has not
fired yet.

The sla_miss entry is dropped as well, since that table no longer exists
in Airflow 3.

* Purge orphaned partitioned_asset_key_log rows and restore the db clean coverage check

The check that is meant to catch a metadata table being left out of
airflow db clean looked for the models package at a path relative to the
test file. That path stopped resolving when the sources moved under
airflow-core, so the check examined no models at all and its assertions
held trivially for an empty set. It has not been able to report a missing
table since, which is how the callback table went unpurged for several
releases.

Walking the package's own search path instead keeps the check working
wherever the sources live, and asserting that models were found stops it
from passing on an empty set again.

With the check restored, partitioned_asset_key_log is the one table it
reports that genuinely has no way to be purged: it carries no foreign
key, so its rows are left behind when the partition Dag run they describe
is cascade-deleted with its dag_run. Only those orphans are deleted --
rows whose partition Dag run still exists are the evidence the scheduler
evaluates to decide when that pending run fires, so they are kept
regardless of age. The other uncovered tables are recorded in the
exclusion list with the reason each is safe.

* Purge orphaned scheduled callbacks while keeping unfired deadlines in db clean

Every callback starts in SCHEDULED, a state outside both the active and the
terminal sets, and an unfired deadline's callback stays there until the deadline
is missed. Deny-listing active states would therefore purge those callbacks and,
through the ON DELETE CASCADE foreign key, silently drop deadlines that have not
fired. The finished states stay allow-listed, so an unknown state keeps its rows.
A SCHEDULED callback is purged only once no deadline references it: deleting a
Dag run cascades away its deadline at the database level, which leaves the
callback behind forever.

(cherry picked from commit 478745a)
xvega pushed a commit to xvega/airflow that referenced this pull request Sep 13, 2026
* Fix airflow db clean never purging the callback table

The table was renamed from callback_request in Airflow 3.2.0 and the
cleanup configuration was not updated with it. A configured table that
does not exist is skipped with only a warning, so these rows were never
deleted and the table grew without bound, with no way for an operator to
purge it.

Only callbacks that can no longer run are removed. A callback still
awaiting execution owns its deadline row through an ON DELETE CASCADE
foreign key, so deleting one would silently drop a deadline that has not
fired yet.

The sla_miss entry is dropped as well, since that table no longer exists
in Airflow 3.

* Purge orphaned partitioned_asset_key_log rows and restore the db clean coverage check

The check that is meant to catch a metadata table being left out of
airflow db clean looked for the models package at a path relative to the
test file. That path stopped resolving when the sources moved under
airflow-core, so the check examined no models at all and its assertions
held trivially for an empty set. It has not been able to report a missing
table since, which is how the callback table went unpurged for several
releases.

Walking the package's own search path instead keeps the check working
wherever the sources live, and asserting that models were found stops it
from passing on an empty set again.

With the check restored, partitioned_asset_key_log is the one table it
reports that genuinely has no way to be purged: it carries no foreign
key, so its rows are left behind when the partition Dag run they describe
is cascade-deleted with its dag_run. Only those orphans are deleted --
rows whose partition Dag run still exists are the evidence the scheduler
evaluates to decide when that pending run fires, so they are kept
regardless of age. The other uncovered tables are recorded in the
exclusion list with the reason each is safe.

* Purge orphaned scheduled callbacks while keeping unfired deadlines in db clean

Every callback starts in SCHEDULED, a state outside both the active and the
terminal sets, and an unfired deadline's callback stays there until the deadline
is missed. Deny-listing active states would therefore purge those callbacks and,
through the ON DELETE CASCADE foreign key, silently drop deadlines that have not
fired. The finished states stay allow-listed, so an unknown state keeps its rows.
A SCHEDULED callback is purged only once no deadline references it: deleting a
Dag run cascades away its deadline at the database level, which leaves the
callback behind forever.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-to-v3-3-test Backport to v3-3-test type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants