{{ message }}
Fix airflow db clean never purging the callback table - #70923
Merged
vatsrahul1001 merged 3 commits intoSep 10, 2026
Merged
Conversation
steveahnahn
force-pushed
the
fix-db-clean-callback-table
branch
from
August 1, 2026 17:24
1c1d6a5 to
d1872d9
Compare
steveahnahn
marked this pull request as ready for review
August 1, 2026 18:18
1 task
steveahnahn
force-pushed
the
fix-db-clean-callback-table
branch
3 times, most recently
from
August 1, 2026 20:17
1805a44 to
367f2ad
Compare
steveahnahn
force-pushed
the
fix-db-clean-callback-table
branch
3 times, most recently
from
August 4, 2026 15:12
21e2a39 to
31f5e16
Compare
steveahnahn
force-pushed
the
fix-db-clean-callback-table
branch
2 times, most recently
from
August 14, 2026 22:52
32064de to
a4473b4
Compare
The table was renamed from callback_request in Airflow 3.2.0 and the cleanup configuration was not updated with it. A configured table that does not exist is skipped with only a warning, so these rows were never deleted and the table grew without bound, with no way for an operator to purge it. Only callbacks that can no longer run are removed. A callback still awaiting execution owns its deadline row through an ON DELETE CASCADE foreign key, so deleting one would silently drop a deadline that has not fired yet. The sla_miss entry is dropped as well, since that table no longer exists in Airflow 3.
…n coverage check The check that is meant to catch a metadata table being left out of airflow db clean looked for the models package at a path relative to the test file. That path stopped resolving when the sources moved under airflow-core, so the check examined no models at all and its assertions held trivially for an empty set. It has not been able to report a missing table since, which is how the callback table went unpurged for several releases. Walking the package's own search path instead keeps the check working wherever the sources live, and asserting that models were found stops it from passing on an empty set again. With the check restored, partitioned_asset_key_log is the one table it reports that genuinely has no way to be purged: it carries no foreign key, so its rows are left behind when the partition Dag run they describe is cascade-deleted with its dag_run. Only those orphans are deleted -- rows whose partition Dag run still exists are the evidence the scheduler evaluates to decide when that pending run fires, so they are kept regardless of age. The other uncovered tables are recorded in the exclusion list with the reason each is safe.
steveahnahn
force-pushed
the
fix-db-clean-callback-table
branch
from
September 8, 2026 16:16
f962705 to
523299a
Compare
ferruzzi
reviewed
Sep 9, 2026
… db clean Every callback starts in SCHEDULED, a state outside both the active and the terminal sets, and an unfired deadline's callback stays there until the deadline is missed. Deny-listing active states would therefore purge those callbacks and, through the ON DELETE CASCADE foreign key, silently drop deadlines that have not fired. The finished states stay allow-listed, so an unknown state keeps its rows. A SCHEDULED callback is purged only once no deadline references it: deleting a Dag run cascades away its deadline at the database level, which leaves the callback behind forever.
ferruzzi
approved these changes
Sep 9, 2026
Contributor
Backport failed to create: v3-3-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel. You can attempt to backport this manually by running: cherry_picker 478745a v3-3-testThis should apply the commit to the v3-3-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
1 task
vatsrahul1001
pushed a commit
that referenced
this pull request
Sep 10, 2026
* Fix airflow db clean never purging the callback table The table was renamed from callback_request in Airflow 3.2.0 and the cleanup configuration was not updated with it. A configured table that does not exist is skipped with only a warning, so these rows were never deleted and the table grew without bound, with no way for an operator to purge it. Only callbacks that can no longer run are removed. A callback still awaiting execution owns its deadline row through an ON DELETE CASCADE foreign key, so deleting one would silently drop a deadline that has not fired yet. The sla_miss entry is dropped as well, since that table no longer exists in Airflow 3. * Purge orphaned partitioned_asset_key_log rows and restore the db clean coverage check The check that is meant to catch a metadata table being left out of airflow db clean looked for the models package at a path relative to the test file. That path stopped resolving when the sources moved under airflow-core, so the check examined no models at all and its assertions held trivially for an empty set. It has not been able to report a missing table since, which is how the callback table went unpurged for several releases. Walking the package's own search path instead keeps the check working wherever the sources live, and asserting that models were found stops it from passing on an empty set again. With the check restored, partitioned_asset_key_log is the one table it reports that genuinely has no way to be purged: it carries no foreign key, so its rows are left behind when the partition Dag run they describe is cascade-deleted with its dag_run. Only those orphans are deleted -- rows whose partition Dag run still exists are the evidence the scheduler evaluates to decide when that pending run fires, so they are kept regardless of age. The other uncovered tables are recorded in the exclusion list with the reason each is safe. * Purge orphaned scheduled callbacks while keeping unfired deadlines in db clean Every callback starts in SCHEDULED, a state outside both the active and the terminal sets, and an unfired deadline's callback stays there until the deadline is missed. Deny-listing active states would therefore purge those callbacks and, through the ON DELETE CASCADE foreign key, silently drop deadlines that have not fired. The finished states stay allow-listed, so an unknown state keeps its rows. A SCHEDULED callback is purged only once no deadline references it: deleting a Dag run cascades away its deadline at the database level, which leaves the callback behind forever. (cherry picked from commit 478745a)
1 task
imrichardwu
pushed a commit
to imrichardwu/airflow
that referenced
this pull request
Sep 11, 2026
* Fix airflow db clean never purging the callback table The table was renamed from callback_request in Airflow 3.2.0 and the cleanup configuration was not updated with it. A configured table that does not exist is skipped with only a warning, so these rows were never deleted and the table grew without bound, with no way for an operator to purge it. Only callbacks that can no longer run are removed. A callback still awaiting execution owns its deadline row through an ON DELETE CASCADE foreign key, so deleting one would silently drop a deadline that has not fired yet. The sla_miss entry is dropped as well, since that table no longer exists in Airflow 3. * Purge orphaned partitioned_asset_key_log rows and restore the db clean coverage check The check that is meant to catch a metadata table being left out of airflow db clean looked for the models package at a path relative to the test file. That path stopped resolving when the sources moved under airflow-core, so the check examined no models at all and its assertions held trivially for an empty set. It has not been able to report a missing table since, which is how the callback table went unpurged for several releases. Walking the package's own search path instead keeps the check working wherever the sources live, and asserting that models were found stops it from passing on an empty set again. With the check restored, partitioned_asset_key_log is the one table it reports that genuinely has no way to be purged: it carries no foreign key, so its rows are left behind when the partition Dag run they describe is cascade-deleted with its dag_run. Only those orphans are deleted -- rows whose partition Dag run still exists are the evidence the scheduler evaluates to decide when that pending run fires, so they are kept regardless of age. The other uncovered tables are recorded in the exclusion list with the reason each is safe. * Purge orphaned scheduled callbacks while keeping unfired deadlines in db clean Every callback starts in SCHEDULED, a state outside both the active and the terminal sets, and an unfired deadline's callback stays there until the deadline is missed. Deny-listing active states would therefore purge those callbacks and, through the ON DELETE CASCADE foreign key, silently drop deadlines that have not fired. The finished states stay allow-listed, so an unknown state keeps its rows. A SCHEDULED callback is purged only once no deadline references it: deleting a Dag run cascades away its deadline at the database level, which leaves the callback behind forever.
vatsrahul1001
pushed a commit
that referenced
this pull request
Sep 11, 2026
* Fix airflow db clean never purging the callback table The table was renamed from callback_request in Airflow 3.2.0 and the cleanup configuration was not updated with it. A configured table that does not exist is skipped with only a warning, so these rows were never deleted and the table grew without bound, with no way for an operator to purge it. Only callbacks that can no longer run are removed. A callback still awaiting execution owns its deadline row through an ON DELETE CASCADE foreign key, so deleting one would silently drop a deadline that has not fired yet. The sla_miss entry is dropped as well, since that table no longer exists in Airflow 3. * Purge orphaned partitioned_asset_key_log rows and restore the db clean coverage check The check that is meant to catch a metadata table being left out of airflow db clean looked for the models package at a path relative to the test file. That path stopped resolving when the sources moved under airflow-core, so the check examined no models at all and its assertions held trivially for an empty set. It has not been able to report a missing table since, which is how the callback table went unpurged for several releases. Walking the package's own search path instead keeps the check working wherever the sources live, and asserting that models were found stops it from passing on an empty set again. With the check restored, partitioned_asset_key_log is the one table it reports that genuinely has no way to be purged: it carries no foreign key, so its rows are left behind when the partition Dag run they describe is cascade-deleted with its dag_run. Only those orphans are deleted -- rows whose partition Dag run still exists are the evidence the scheduler evaluates to decide when that pending run fires, so they are kept regardless of age. The other uncovered tables are recorded in the exclusion list with the reason each is safe. * Purge orphaned scheduled callbacks while keeping unfired deadlines in db clean Every callback starts in SCHEDULED, a state outside both the active and the terminal sets, and an unfired deadline's callback stays there until the deadline is missed. Deny-listing active states would therefore purge those callbacks and, through the ON DELETE CASCADE foreign key, silently drop deadlines that have not fired. The finished states stay allow-listed, so an unknown state keeps its rows. A SCHEDULED callback is purged only once no deadline references it: deleting a Dag run cascades away its deadline at the database level, which leaves the callback behind forever. (cherry picked from commit 478745a)
xvega
pushed a commit
to xvega/airflow
that referenced
this pull request
Sep 13, 2026
* Fix airflow db clean never purging the callback table The table was renamed from callback_request in Airflow 3.2.0 and the cleanup configuration was not updated with it. A configured table that does not exist is skipped with only a warning, so these rows were never deleted and the table grew without bound, with no way for an operator to purge it. Only callbacks that can no longer run are removed. A callback still awaiting execution owns its deadline row through an ON DELETE CASCADE foreign key, so deleting one would silently drop a deadline that has not fired yet. The sla_miss entry is dropped as well, since that table no longer exists in Airflow 3. * Purge orphaned partitioned_asset_key_log rows and restore the db clean coverage check The check that is meant to catch a metadata table being left out of airflow db clean looked for the models package at a path relative to the test file. That path stopped resolving when the sources moved under airflow-core, so the check examined no models at all and its assertions held trivially for an empty set. It has not been able to report a missing table since, which is how the callback table went unpurged for several releases. Walking the package's own search path instead keeps the check working wherever the sources live, and asserting that models were found stops it from passing on an empty set again. With the check restored, partitioned_asset_key_log is the one table it reports that genuinely has no way to be purged: it carries no foreign key, so its rows are left behind when the partition Dag run they describe is cascade-deleted with its dag_run. Only those orphans are deleted -- rows whose partition Dag run still exists are the evidence the scheduler evaluates to decide when that pending run fires, so they are kept regardless of age. The other uncovered tables are recorded in the exclusion list with the reason each is safe. * Purge orphaned scheduled callbacks while keeping unfired deadlines in db clean Every callback starts in SCHEDULED, a state outside both the active and the terminal sets, and an unfired deadline's callback stays there until the deadline is missed. Deny-listing active states would therefore purge those callbacks and, through the ON DELETE CASCADE foreign key, silently drop deadlines that have not fired. The finished states stay allow-listed, so an unknown state keeps its rows. A SCHEDULED callback is purged only once no deadline references it: deleting a Dag run cascades away its deadline at the database level, which leaves the callback behind forever.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

airflow db cleanhas not purged thecallbacktable since Airflow 3.2.0.Migration
b87d2135fa50renamedcallback_requesttocallback, but the cleanup configuration indb_cleanup.pykept the old name. A configured table that does not exist is skipped with only a warning, so the entry became a no-op and nothing purgescallbackany more.The table does accumulate in practice. Executor callbacks are set to
success/failedby the scheduler and never deleted (only Dag-processor callbacks are removed, as they are dispatched). And becausedeadline.callback_idreferencescallback.idwithON DELETE CASCADE,callbackis the parent, so purgingdeadlineleaves its callback rows orphaned permanently.There was also no workaround available:
--tables callbackwas rejected as an invalid choice, so operators could not purge the table throughairflow db cleanat all.Why this was not caught
test_no_models_missingexists to catch exactly this: every model table must either be cleaned or documented as deliberately excluded. It has been passing vacuously. It located the models package by walking up from the test file (Path(__file__).parents[2] / "airflow/models"), a path that stopped existing when the sources moved underairflow-core/src, so it discovered zero models and asserted over an empty set. Confirmed against the pre-fix code: the check passed whilecallbackwas in neither the cleanup config nor the exclusion list.With the check repaired, it reports one more table with no way to be purged:
partitioned_asset_key_log. It carries no foreign key, so its rows are left behind when the partition Dag run they describe is cascade-deleted with itsdag_run. The only existing delete (inSchedulerJobRunner) covers partition runs invalidated by a rollup-definition change, which is not a retention mechanism.Only those orphans are purged. Rows whose partition Dag run still exists are the evidence the scheduler evaluates to decide when that pending run fires (
SchedulerJobRunnerreads them for every pendingAssetPartitionDagRun), so a purely age-based delete could silently stall a rollup still accumulating keys past the retention window. They are kept regardless of age via aNOT IN (SELECT id FROM asset_partition_dag_run)filter.What this changes
First commit — the callback fix:
created_atas the recency column.deadlinerow through theON DELETE CASCADEforeign key, so deleting one would silently drop a deadline that has not fired yet. Dag-processor callbacks carry no state and reference no deadline, so rows outliving the retention window are purged too. The existingTERMINAL_STATESconstant is reused rather than re-hardcoding the state literals, since that kind of drift is what caused this bug.dependent_tables=["deadline"]so deadline rows are cleaned and archived ahead of their parent, matching howdaganddag_runalready handle the same cascade.sla_missentry, whose table no longer exists in Airflow 3.Second commit — the coverage check and the table it found:
airflow.models.__path__instead of reconstructing the path from the test's location, so the check keeps working wherever the sources live (it now discovers 52 models, against 0 before), and assert that models were discovered at all so it can never pass on an empty set again.partitioned_asset_key_logfor cleanup, restricted to orphaned rows as above.airflow db migrate, not from the ORM metadata:asset_partition_dag_run,asset_watcher,dag_favorite,hitl_detail,hitl_detail_history,task_inlet_asset_reference,deadline_alert, andasset_state_storeare all removed byON DELETE CASCADEfrom tables that are already cleaned ;asset_state_storeis a per-asset key/value store whose rows are upserted in place (bounded, current state rather than history);teamis configuration, not run data.The coverage check cannot pass without the callback fix — with it working,
callbackis reported as uncovered until the first commit registers it — which is why both land in one PR.Verification
Verified end to end against real metadata databases, seeding rows and running the real
run_cleanup, with alogrow as a control to prove the run did actual work:success/failedscheduled/queued/runningsuccesspartitioned_asset_key_logrowpartitioned_asset_key_logrow whose partition run still existspartitioned_asset_key_logrowlogrowThe archiving path (the default,
skip_archive=False) was exercised on Postgres, MySQL and SQLite, confirming the_airflow_deleted__callback__*table is created and the purged row preserved; a committed test covers it. Removingcallbackfrom the config makes the repaired check flag it, so the check now catches the defect that motivated this PR. The fulltest_db_cleanup.pysuite passes on all three backends, and the new tests fail without the source changes.Compatibility
Removing the two stale entries changes what
--tablesaccepts. A list that mixes them with live tables still works, warning about the unknown names and cleaning the rest:A
--tableslist naming only removed tables now stops with "No tables selected for db cleanup" instead of silently reporting success while doing nothing. That seems preferable, since such an invocation has been a no-op since 3.2.0, but it is a visible change for anyone whose cron passes only those names.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5, Fable 5) following the guidelines