{{ message }}
Repository navigation
Conversation
Streaming producers such as Spark Structured Streaming micro-batches or Iceberg commit jobs publish one partition every few minutes. The temporal rollup windows assumed a fixed key spacing (one minute for an hour, one hour for a day, one day for a week), so a five-minute producer could never satisfy WaitForAll and the rollup Dag run waited forever. The only workarounds were emitting fake per-minute events, counting events with MinimumCount and losing the ability to see which batch is missing, or maintaining a custom window plugin per deployment.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Streaming producers (Spark Structured Streaming micro-batches, Iceberg commit jobs) publish one partition every few minutes, but the temporal rollup windows expect a fixed key spacing:
HourWindowDayWindowWeekWindowA producer that emits one key every 5 minutes sends 12 keys per hour, so with
HourWindow()and the defaultWaitForAll()the hourly Dag run waits forever. The workarounds are emitting fake per-minute events, usingMinimumCount(12)(which no longer tells you which batch is missing), or a custom window plugin per deployment.This adds an optional
steptoHourWindow,DayWindowandWeekWindow:step=5 minstep=15 minHourWindowDayWindowWeekWindowBehaviour
to_upstreamwalks the period instepincrements and reuses_build_directional_steps, soWindow.Direction.BACKWARDkeeps working.stepmust be a positivetimedeltawith whole seconds that divides the period evenly (HourWindow(step=timedelta(minutes=7))raisesValueError).MonthWindow,QuarterWindowandYearWindowrejectstepwithTypeError, since their periods vary in length.FanOutMapperrequires an explicitdownstream_mapperwhen its window has a custom step, because the built-in defaults assume the window's default unit and would collapse several members onto one key.step=Nonekeeps the current behaviour. Serialization omitsstepwhen it isNone, so existing Dags keep their rollup fingerprint and pending partition Dag runs are not reset on upgrade. No DB migration;steplives in the serialized Dag JSON.to_upstream, so they show "12 / 12" without UI changes.Out of scope / open questions
10:03is logged but never matches. Should ingestion warn?WeekWindow(step=timedelta(minutes=1))expects 10,080 keys per pending partition Dag run. Should there be a limit similar tomax_downstream_keys?[start, end)) is a larger change to the AIP-76 model; it is proposed separately in Coverage-based completeness check for partitioned assets (streaming producers) #74595 rather than folded into this PR.Testing
airflow-core/tests/unit/partition_mappers(390 passed),task-sdk/tests/task_sdk/definitions/test_partition_mappers.py(49 passed), serialization tests for window/partition/rollup/fingerprint, and scheduler rollup/partition tests (-k "rollup or partition", 51 passed).prekpre-commit and manual stages, includingcheck-window-in-sync,mypy-airflow-coreandmypy-task-sdk.breeze testing task-sdk-tests,breeze testing core-tests(DB on SQLite and non-DB, test types API/Always/CLI/Core/Other/Serialization) andbreeze build-docs apache-airflow task-sdkpass locally. The provider suites selected bybreeze verifywere not run locally (memory limits); they are left to CI.related: #64571, #66848, #69456
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines