Apache Airflow version

3.3.1

If "Other Airflow 2/3 version" selected, which one?

No response

What happened?

With OTel metrics enabled, the scheduler exports two copies of its metrics: one from the main process, which behaves correctly, and one from a forked child that never advances. The child's copy is republished on every export interval, indefinitely, holding the counter values it inherited at the moment of the fork.

The scheduler container runs the main process plus four forked children:

401440 ppid=401414 threads=18 elapsed=2488  airflow scheduler   <- main
402459 ppid=401440 threads=17 elapsed=2473  airflow scheduler
402492 ppid=401440 threads=18 elapsed=2473  airflow scheduler
402524 ppid=401440 threads=15 elapsed=2473  airflow scheduler
407463 ppid=401440 threads=17 elapsed=2240  airflow scheduler   <- exports the frozen copy

The three children forked at startup are quiet. The one forked ~4 minutes later — after the parent had already been counting — inherited a non-zero accumulation and exports it forever:

airflow_scheduler_heartbeat{service_instance_id="airflow-scheduler-6545f66d48-tqgjv"}   277  (climbing)
airflow_scheduler_heartbeat{service_instance_id="c674b1e6-8df6-4fb1-ba05-161e0ce45725"}  44  (frozen, last sample 8s old)

Seven of the ten live airflow_scheduler_* series are duplicated this way, and each scheduler restart mints another frozen set.

This looks like the other side of the fix for #64690. get_otel_logger() now resets the SDK's Once() guard before set_meter_provider():

_metrics_internal._METER_PROVIDER_SET_ONCE._done = False
_metrics_internal._METER_PROVIDER = None

That gives a forked child a working provider where before it had a dead one. Children that never emit anything again still get a live PeriodicExportingMetricReader and keep flushing the instrument state they inherited.

Two consequences:

1. Duplicated series. Anything that sums or averages across instances double-counts while both copies are live.
2. Per-series alerting on a heartbeat is impossible. rate(airflow_scheduler_heartbeat[5m]) == 0 is the natural liveness check, and it fires forever on the frozen copy while the scheduler is perfectly healthy. The only workaround is to aggregate first — sum(rate(...)) == 0 — which silently gives up the ability to alert on one unhealthy instance among several.

Why the two copies have different identities

Worth noting for anyone reproducing on an older SDK. opentelemetry-sdk 1.44 added ServiceInstanceIdResourceDetector, whose docstring says:

because this detector is process dependent, providers refresh it automatically after a fork and merge the newly detected value on top of the existing resource. This means that if a user explicitly sets service.instance.id (e.g. via OTEL_RESOURCE_ATTRIBUTES or Resource.create(attributes=...)), that value will be overwritten with a newly generated UUID the next time the process forks.

So on 1.44 the child's resource carries a fresh UUID and the frozen copy lands in its own series, which is how this became visible at all. The child's environment is correct — Resource.create() run by hand inside the same container returns the pod name — the UUID comes from the post-fork refresh.

On earlier SDKs the child would carry the same service.instance.id as the parent, so the two copies collide on one series and the frozen values interleave with the real ones instead of sitting beside them. That is worse and harder to notice.

What you think should happen instead?

A forked child that does not emit metrics should not export. Stopping the inherited metric reader in the child — for example from an os.register_at_fork(after_in_child=...) hook, or by shutting the provider down before re-initialising — would leave one copy of each series and make a per-instance heartbeat check work again.

How to reproduce

1. Airflow 3.3.1, opentelemetry-sdk 1.44.0, KubernetesExecutor.
2. Enable OTel metrics: AIRFLOW__METRICS__OTEL_ON=True, plus OTEL_METRICS_EXPORTER=otlp, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf, OTEL_METRIC_EXPORT_INTERVAL=30000.
3. Let the scheduler run for ten minutes or so, long enough for it to fork a child after the heartbeat has been counting.
4. Query airflow_scheduler_heartbeat. Two series appear: one climbing, one frozen and still being re-sent.

ps -eo pid,ppid,etimes,args inside the scheduler container shows the child, and its elapsed time matches the moment the frozen series first appears.

Operating System

Ubuntu 24.04.4 LTS

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

KubernetesExecutor, one scheduler replica, k3s 1.36. Metrics go to an OpenTelemetry Collector on each node over OTLP HTTP and on to Prometheus by remote write.

Anything else?

Happens on every scheduler start. The number of frozen series grows by one set per restart.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct