Gateway metrics: make the OTLP aggregation temporality configurable (Delta is hard-coded) · Issue #788 · documentdb/documentdb · GitHub
Skip to content

Gateway metrics: make the OTLP aggregation temporality configurable (Delta is hard-coded) #788

Open

Description

@peterekhator

The gateway's metrics exporter sets the temporality to Delta in
pg_documentdb_gw/documentdb_gateway_otel/src/lib.rs (.with_temporality(Temporality::Delta)),
with no way to change it.

Many OpenTelemetry pipelines that end in Prometheus-compatible storage expect cumulative
sums and histograms. For example, Grafana Alloy's otelcol.exporter.prometheus drops
delta points, and the delta-to-cumulative processor is still marked experimental there.
With the current gateway, the metrics never reach the store unless an experimental
component is enabled.

Request: honour the standard OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE
environment variable (cumulative | delta | lowmemory), and/or a
TelemetryOptions.Metrics.Temporality setting in SetupConfiguration.json. The default can
stay Delta.

Two related details we noticed while wiring this up (happy to open separate issues):

  1. db.client.operation.duration is recorded in seconds but uses the SDK's default
    histogram boundaries (0, 5, 10, 25, … 10000), so nearly every request falls in the first
    bucket. Second-scale boundaries (e.g. 0.001 … 10) would make the percentiles usable.
  2. The unit suffix varies between series of the same metric (some are exported with the
    unit s/By, some without), which gives two Prometheus names for one metric
    (db_client_operation_duration_seconds_* and db_client_operation_duration_*).

Seen at commit cdf2ee2 (1.0 RC1).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions