Summary

Backend ClickHouse queries emitted no OpenTelemetry spans, so Tempo and the span-metrics RED panels only ever showed Postgres and Redis. This wires the existing instrument_clickhouse_client helper into the three places ClickHouse clients are created, so every query gets a db.system=clickhouse client span with a truncated statement, operation, row count and error status. The AgentCC gateway's Prometheus exposition did not escape label values, and the model label comes from the request body, so a quoted or multi-line model name could corrupt /-/metrics; label values are now escaped per the text exposition format.

Linked issues

Contributes to TH-8757
Linear: https://linear.app/future-agi/issue/TH-8757

AI use

AI use:

Type of change

• ✨ New feature (non-breaking change which adds functionality)
• 🐛 Bug fix (non-breaking change which fixes an issue)

E2E coverage

E2E: exempt (backend-internal and gateway-internal telemetry; no user-visible flow)

1) What changes were done

A. ClickHouse query spans (tfc/telemetry/instrumentation.py)

• instrument_clickhouse_client now wraps every query entry point present on the client (execute, execute_iter, query, command, query_row_block_stream), is a no-op when OTEL_ENABLED is false, and is idempotent.
• New trace_clickhouse_stream keeps the span open until a streaming result is consumed or closed, so execute_iter and query_row_block_stream report real durations and errors.
• Shared helpers build span name, SpanKind.CLIENT, attributes and error recording for the sync, async and stream wrappers; db.rows_returned unwraps (rows, column_types) tuples and is omitted for command() strings.

B. Wiring at the client factories

• tfc/utils/clickhouse.py singleton, tracer/services/clickhouse/client.py pooled native driver, and tracer/services/clickhouse/application_read_transport.py HTTP read client now return instrumented clients.

C. Gateway metrics label escaping (agentcc-gateway/internal/metrics/registry.go)

• formatLabels escapes backslash, double quote and newline in label values.

2) Why the changes were done

• Product/UX: Database latency and error dashboards in Grafana need ClickHouse alongside Postgres and Redis; the db.system dimension on Tempo span metrics is the only DB dimension those panels use.
• Technical: clickhouse-driver and clickhouse-connect ship no OpenTelemetry support and the urllib3 spans carry no DB attributes, so instrumentation has to happen at the client. Wrapping at the factories covers every caller with three small edits.
• Technical: The gateway metrics endpoint is about to be scraped by Alloy; an unescaped request-controlled label would break the whole scrape.

3) Tests written + scenarios each covers

tfc/tests/test_telemetry_clickhouse.py — span emission through the three real client factories with an in-memory exporter

• test_singleton_client_is_instrumented — the analytics singleton returns a wrapped client
• test_native_execute_emits_client_span — execute span has db.system, statement, operation, CLIENT kind and row count
• test_with_column_types_counts_rows_not_tuple_parts — with_column_types=True tuple results count rows, not tuple parts
• test_execute_iter_span_ends_when_rows_are_consumed — stream span stays open until the generator is exhausted
• test_execute_iter_error_while_consuming_marks_span_error — errors during iteration set ERROR status
• test_http_application_client_query_emits_span — clickhouse-connect query is traced
• test_http_application_client_command_records_no_row_count — string results record no row count
• test_http_application_client_block_stream_span_ends_on_exit — context-manager stream span ends on exit
• test_otel_disabled_returns_client_untouched — no wrapping when OTel is off
• test_instrumentation_is_idempotent — wrapping twice does not double-trace
• plus error-status and attribute-shape tests

agentcc-gateway/internal/metrics/registry_test.go

• TestFormatLabels_EscapesValues — quote, backslash and newline render escaped

tfc/tests/test_telemetry_clickhouse.py ............                      [100%]
============================== 12 passed in 0.69s ==============================

ok  	github.com/futureagi/agentcc-gateway/internal/metrics	0.392s
ok  	github.com/futureagi/agentcc-gateway/internal/plugins/prometheus	0.744s

4) How to run / test

cd futureagi
bin/test tfc/tests/test_telemetry_clickhouse.py -v

cd ../agentcc-gateway
go test ./internal/metrics/... ./internal/plugins/prometheus/...

5) Rollout notes

• Ships through the normal release flow (dev to main, release-please, release-images); no migration, no new env vars, no values change. ClickHouse spans start the moment the new backend image is live because OTEL_ENABLED and the Tempo endpoint are already set in prod.
• Expected new span volume: one client span per ClickHouse query; the busiest pattern alone is about 1,900 queries/h, well within current Tempo ingest.
• After the new agentcc-gateway image is deployed, flip prometheus.enabled to true in the deployment repo (templates/agentcc-gateway-cm.yaml, step 5 of deployment PR fix(annotations): consolidate TH-4812 annotation fixes #401) and redeploy the umbrella chart; the Alloy scrape for /-/metrics is already live and will go from HTTP 404 to up == 1.
• Rollback: redeploy the previous image tag; nothing persisted changes.

6) Edge cases & considerations

• OTEL_ENABLED=false: factories return the raw client; no OTel import happens on the hot path.
• Streaming reads: the span is started non-current so generator yields do not leak context; it ends on exhaustion, close, or exception.
• command() returns a string: no db.rows_returned, so dashboards do not see string lengths as row counts.
• Health-check SELECT 1 on pooled connections is traced like any query; volume is low and it is filterable by statement.

7) Pre-existing issues (NOT introduced by this PR)

• black --check flags a hunk in tracer/services/clickhouse/client.py and ruff reports five UP035/UP045 findings in tfc/telemetry/instrumentation.py; both reproduce on dev on lines this PR does not touch.

8) Architectural / important decisions

• Instrument at the factories, not per call site: three edits cover every ClickHouse read path and keep the helper as the single place span shape is defined.
• Spans are CLIENT kind: matches the psycopg and redis instrumentors so DB panels can filter by kind uniformly.
• Three-pair replacer for label escaping: the gateway has no Prometheus client dependency and strconv.Quote escapes more than the text format allows.

Checklist

• My code follows the style guide
• I've added tests that prove my fix is effective or that my feature works
• Targeted tests pass locally
• No hardcoded secrets, URLs, or PII
• PR title follows Conventional Commits