Add R2DBC instrumentation via r2dbc-proxy listener SPI by jordan-wong · Pull Request #12441 · DataDog/dd-trace-java · GitHub
Skip to content

Add R2DBC instrumentation via r2dbc-proxy listener SPI - #12441

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 38 commits into
masterfrom
feat/r2dbc-listener-spi-instrumentation-20260904
Oct 2, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 38 commits into
masterfrom
feat/r2dbc-listener-spi-instrumentation-20260904

Conversation

@jordan-wong

@jordan-wong jordan-wong commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

What

Adds R2DBC tracing (r2dbc-1.0, integration name r2dbc), on by default.

  • Tracing: hooks ConnectionFactories.find and wraps the returned factory with r2dbc-proxy. A listener creates one span per query. This is the same hook point and approach as the OpenTelemetry agent.
  • DBM: a separate module advises the driver's io.r2dbc.spi.Connection#createStatement(String), like DBMCompatibleConnectionInstrumentation does for JDBC. OpenTelemetry has no equivalent. The comment carries static service metadata only, with no traceparent, because the query span doesn't exist yet when the statement is created.
  • Span data: resource names use normalized SQL (DBQueryInfo). r2dbc:pool: factories are left unwrapped, because the pool resolves its delegate through a nested, instrumented find. That gives one span per query and keeps the ConnectionPool object intact.

For reviewers

  • New bundled third-party library: io.r2dbc:r2dbc-proxy:1.1.6.RELEASE`` (Apache-2.0). 77 of its classes are injected as helpers. The module only applies when Reactor 3.4+ is present (enforced by muzzle). Adds ~104 KiB
  • Factories, connections, statements and results are proxied. This is r2dbc-proxy's own design and the approach R2DBC recommends for observability, also used by the OpenTelemetry agent. Two consequences:
    • Each call goes through an extra proxy layer (connection → statement → result), which adds some overhead.
    • Code that casts a non-pooled factory or connection to a driver-specific type, or branches on instanceof VendorConnection, sees the proxy instead. Pooled factories keep their type.

Known gaps

  • No spans for factories created without ConnectionFactories.find, such as ones built directly or supplied as framework beans.
  • DBM does not cover Batch#add.

Release notes

R2DBC connections are traced by wrapping them with r2dbc-proxy. Applications that cast R2DBC connections to driver-specific types, or check instanceof on them, should verify behavior with the integration enabled (DD_TRACE_R2DBC_ENABLED=false turns it off).

Testing

  • Unit and instrumentation tests (JUnit) against in-memory H2, including DBM comment content, queries cancelled via take(1), next(), blockFirst() and Mono.from(...), and an r2dbc:pool: URL (one span per query; the pool keeps its type and disposes).
  • muzzle and latestDepTest (Java 11+).
  • Compared against the OpenTelemetry Java agent on a Spring WebFlux + PostgreSQL app. DB spans match on type, instance, user, host and SQL, with no class-loading errors.
  • DBM checked on real PostgreSQL: with DD_DBM_PROPAGATION_MODE=full, the server receives /*ddps=...,dddbs=...,ddh=...,dddb=...*/ on every statement, and resource names stay normalized.

🤖 Generated with Claude Code

jordan-wong and others added 3 commits September 1, 2026 06:03
…e hand-writing method advice

Adds a general heuristic to instrumenter-module.md: before instrumenting a client
library via manual method advice, check whether the library (or an official
companion, e.g. r2dbc-proxy for R2DBC, ClientInterceptor for gRPC,
Producer/ConsumerInterceptor for Kafka) exposes a first-class interception/listener
SPI, and prefer it.

Rationale, aimed at reactive/async clients: a listener SPI delivers one
well-defined start/end/error/cancel callback per operation with metadata already
assembled, whereas hand-written advice on a reactive method must re-implement the
lifecycle by wrapping the returned Publisher/Mono/Flux — a wrapper that finishes
the span only on onComplete/onError leaks every cancelled span (reactive pipelines
routinely cancel: take(1), timeouts, DiscardOnCancel), so spans are created but
never finished/exported, and it's easy to wrap at the wrong granularity.

Notes the no-forced-dependency approach: declare the interception library
compileOnly and inject the listener via helperClassNames() (or shade it), so it
does not land on the user's runtime classpath. Cross-links the existing R2DBC
connection-metadata guidance to this section.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Concrete R2DBC/r2dbc-proxy example showing the factory-hook +
AssignReturned + wrap-helper + listener shape, so the existing
"check for an interception SPI" guidance is operationalizable, not
just prose.
Adds tracing for R2DBC (reactive database access) by installing an
official r2dbc-proxy ProxyMethodExecutionListener instead of hand-writing
advice on Statement.execute()/Batch.execute(). The listener owns the
reactive lifecycle (complete/error/cancel) so spans can't leak on
cancellation, which a hand-rolled Publisher wrapper is prone to since
reactive pipelines cancel constantly (take(1), timeouts, driver-level
DiscardOnCancel).

- R2dbcInstrumentation hooks ConnectionFactories.find(ConnectionFactoryOptions)
  via @Advice.AssignReturned-style exit advice and wraps the returned
  ConnectionFactory with r2dbc-proxy.
- TraceProxyExecutionListener implements ProxyMethodExecutionListener,
  starting/finishing spans in beforeQuery/afterQuery.
- R2dbcConnectionCallbackInstrumentation + R2dbcSqlCommentInjector add
  DBM SQL comment injection, mirroring JDBC's SQLCommenter.
- r2dbc-proxy is declared implementation (not compileOnly) and its full
  class set is listed in helperClassNames() in dependency order: ordinary
  R2DBC apps don't depend on r2dbc-proxy, so its classes must be injected
  into the app's classloader (and added to the muzzle validation
  classpath via extraDependency) or the module is blocked/fails to load
  at runtime despite passing build-time checks.

Verified via a live trace comparison against the OpenTelemetry Java
agent on a Spring WebFlux + R2DBC/PostgreSQL sample app: matching
INSERT/CREATE TABLE/error-SELECT spans with correct DBM tags
(db.instance, db.type, db.user, peer.hostname, _dd.dbm_trace_injected).
@jordan-wong
jordan-wong requested review from a team as code owners September 9, 2026 13:37
@jordan-wong
jordan-wong requested review from sarahchen6 and vandonr and removed request for a team September 9, 2026 13:37
@dd-octo-sts

dd-octo-sts Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

@dd-octo-sts dd-octo-sts Bot added the tag: ai generated Largely based on code generated by an AI or LLM label Sep 9, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-09T13:56:50.080662Z 58cfdba PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 58cfdba670

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

The new module fails the required integration-list check. It also uses raw SQL resource names and puts the parent span context in full-mode DBM comments.

Open Bits AI session

🤖 Datadog Autotest · Commit 58cfdba · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Comment thread settings.gradle.kts
Comment thread .agents/skills/apm-integrations/references/instrumenter-module.md Outdated
The r2dbc module was missing from expected.integrations, so the
agent-jar structural check never verified it's bundled in the shaded
agent jar.
R2dbcConnectionCallbackInstrumentation listed ConnectionCallbackHandler
in its own helperClassNames() — that class is this module's
instrumentedType(), so ByteBuddy already defines it on the target
classloader. Listing it as a helper injects a second, redundant
definition of the class currently being transformed.
r2dbc-proxy is bundled as a real dependency, so its own
ConnectionFactoryProvider (DRIVER="proxy") is discoverable by
ConnectionFactories.find() alongside real drivers. If an application
requests that URL scheme directly, find() already returns an
r2dbc-proxy-wrapped factory; wrapping it again would double the query
listener callbacks and produce duplicate spans. Skip wrapping in that
case — this means apps using DRIVER="proxy" directly get no DD spans,
which is preferable to silently duplicating them.
extractDbType returned DRIVER unconditionally. For pooled URLs like
r2dbc:pool:postgresql://..., DRIVER is the wrapper's own name ("pool"),
losing the real database type. Prefer PROTOCOL only when DRIVER is a
known wrapper (currently "pool") — other drivers legitimately use the
driver:protocol URL shape for their own sub-mode designator (e.g.
r2dbc:h2:mem:///..., where DRIVER="h2" is already the real type and
PROTOCOL="mem" must not override it).
…atches

Several interlocking fixes to the DBM/resource-name path:

- Helper-injection bug: ConnectionCallbackHandler (and BatchCallbackHandler)
  were listed as helper classes in the other r2dbc modules. Helper-injected
  classes are loaded untransformed, so the createStatement/add advice that
  injects the DBM comment never actually ran — the _dd.dbm_trace_injected tag
  only appeared because it was set unconditionally. Removed both instrumented
  types from every helper list so their advice is applied.

- _dd.dbm_trace_injected is now set only when the executed query actually
  carries a DD comment (SharedDBCommenter.containsTraceComment), instead of
  unconditionally on every query.

- Statement-creation-time injection no longer embeds a per-execution
  traceparent (it referenced the ambient span, not the query span, which does
  not exist yet). It injects only static connection metadata, matching JDBC's
  prepareStatement path; the tag reflects that.

- Resource name goes through DBQueryInfo/SQLNormalizer (literals stripped),
  with any injected DBM comment stripped first so it doesn't leak into it.

- Batch#add(String) is now instrumented (R2dbcBatchCallbackInstrumentation),
  so batched statements get DBM comments too.

- Connection metadata is held in a WeakHashMap keyed by ConnectionInfo, so
  entries are evicted when the connection is unreachable rather than relying
  on a close() callback that abrupt disconnects never fire.

Tests updated for normalized resource names and comment-aware assertions.
The bundled r2dbc-proxy requires Reactor at runtime (e.g.
DelegatingContextView implements reactor.util.context.ContextView), but
the R2DBC SPI itself only requires Reactive Streams. On an R2DBC app
without reactor-core, injecting those proxy helper classes would fail and
leave the factory uninstrumented. Guard all three modules with a
classLoaderMatcher on reactor.core.publisher.Flux so instrumentation only
applies when Reactor is available.
The repo reserves Java/JUnit for unit tests and requires Groovy/Spock for
instrumentation tests (AGENTS.md). Port all three r2dbc instrumentation
test classes to Spock specs under src/test/groovy, preserving every test
and assertion (span shape, normalized resource names, DBM tag/comment
behavior, peer.service, error tags). No production changes.
The listener-SPI guidance told authors to declare the interception
library compileOnly and inject its classes via helperClassNames(). That
does not work: a compileOnly dependency is absent from the agent jar, so
there is nothing to inject and both build-time muzzle and runtime helper
definition fail. Correct it to use implementation scope (bundled into the
agent), and note that bundling pulls in transitive deps (guard on their
presence with classLoaderMatcher) and that the full transitive class set
must be listed in topological order.
…Spock"

This reverts commit 551ab66.

A review comment asked to move the tests to Groovy/Spock, but the repo's
"Enforce Groovy migration" CI check rejects any new .groovy file — the
project has migrated to JUnit 5 for instrumentation tests (see
docs/how_to_test_with_junit.md). Keep the tests as Java/JUnit, which is
the current convention. The resource-name normalization and DBM
assertion updates remain (they were made before the port).
@jordan-wong jordan-wong added type: feature Enhancements and improvements inst: others All other instrumentations labels Sep 23, 2026
@datadog-prod-us1-4

This comment has been minimized.

@mcculls
mcculls force-pushed the feat/r2dbc-listener-spi-instrumentation-20260904 branch from 8b4fe75 to bd3c373 Compare September 30, 2026 12:29
@vandonr

vandonr commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

@autotest review

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

DBM comment injection can silently disable PostgreSQL pg_hint_plan directives by moving them away from the required leading position.

Open Bits AI session

🤖 Bits Code Review · Commit f66349f · @DataDog review to ask questions

jordan-wong and others added 2 commits October 1, 2026 11:24
…irst

- Append the DBM comment (instead of prepending) for CALL statements, for
  PostgreSQL SQL carrying a pg_hint_plan hint (`/*+`), and when
  dd.dbm.always_append_sql_comment is set, keeping a closing semicolon last.
  Mirrors JDBC's SQLCommenter.
- Check for an existing DD comment before building the new one, in both
  prepend and append positions.
- Rename R2dbcDecorator#extractDbType to getDbType, matching getDbService.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
JDBC's DBMCompatibleConnectionInstrumentation always appends the comment
for SQL Server at prepareStatement; createStatement is R2DBC's equivalent.
r2dbc-mssql registers the drivers "sqlserver" and "mssql".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Move JDBC's SQLCommenter into agent-bootstrap (next to SharedDBCommenter)
and have R2DBC call it instead of keeping its own copy of the placement and
duplicate-comment logic. R2DBC now only decides the caller-side reasons to
append (dd.dbm.always_append_sql_comment, SQL Server), in one method, like
JDBC's call sites. getFirstWord becomes public @VisibleForTesting for the
JDBC test and benchmarks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jordan-wong
jordan-wong requested a review from a team as a code owner October 1, 2026 21:10

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The Reactor version guard in R2dbcModule.java (line 28) checks only for Flux, which is present in Reactor 3.3, but the bundled r2dbc-proxy requires reactor.util.context.ContextView (introduced in Reactor 3.4+). On Reactor 3.3, instrumentation is incorrectly enabled and throws NoClassDefFoundError during connection or query subscription instead of leaving the application uninstrumented.

Open Bits AI session

🤖 Bits Code Review · Commit 6e4675c · @DataDog review to ask questions

…cted

Drop the redundant classLoaderMatcher() on reactor.core.publisher.Flux:
the relocated r2dbc-proxy classes are claimed as helpers, so runtime muzzle
already blocks the module when the application lacks a compatible Reactor
(no Flux, or no ContextView on Reactor < 3.4). Reword the muzzle comment,
and add a muzzle fail directive on Reactor 3.3.22 so CI proves it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@vandonr vandonr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for addressing comments

@amarziali amarziali left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks OK from the LP standpoint. However I would verify few raised points before merging

Comment thread dd-java-agent/instrumentation/r2dbc/r2dbc-1.0/build.gradle
jordan-wong and others added 3 commits October 2, 2026 11:21
r2dbc-proxy runs on Reactor, so add the reactive-streams-1.0 and
reactor-core-3.1 instrumentations as testRuntimeOnly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Besides take(1), check spans still finish when a query is cancelled via
next(), blockFirst() and Mono.from(...) (the shape behind Spring
DatabaseClient's first()).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jordan-wong
jordan-wong enabled auto-merge October 2, 2026 16:59
@jordan-wong
jordan-wong added this pull request to the merge queue Oct 2, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 2, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit afff86a into master Oct 2, 2026
625 of 627 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the feat/r2dbc-listener-spi-instrumentation-20260904 branch October 2, 2026 19:01
@github-actions github-actions Bot added this to the 1.67.0 milestone Oct 2, 2026
@ValentinZakharov ValentinZakharov changed the title feat(r2dbc): add R2DBC instrumentation via r2dbc-proxy listener SPI Add R2DBC instrumentation via r2dbc-proxy listener SPI Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst: others All other instrumentations tag: ai generated Largely based on code generated by an AI or LLM tag: apm integration toolkit Changes generated by DataDog/apm-instrumentation-toolkit as part of IDM AIT experimentation type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants