R2DBC: SQLNormalizer runs on the I/O thread for every statement of a large Batch, blocking the event loop for ~55 s · Issue #12932 · DataDog/dd-trace-java · GitHub
Skip to content

R2DBC: SQLNormalizer runs on the I/O thread for every statement of a large Batch, blocking the event loop for ~55 s #12932

Open

Description

@AdrienCadoretDKT

Tracer Version(s)

1.67.0

Java Version(s)

21

JVM Vendor

Eclipse Adoptium / Temurin

Bug Report

Since the R2DBC integration added in #12441 (enabled by default), the agent normalizes the SQL of every statement in TraceProxyExecutionListener.beforeQuery, through DBQueryInfo and SQLNormalizer.normalize, on the thread that executes the query.

Our application (Spring Boot, r2dbc-postgresql 1.1.3, r2dbc-pool, jOOQ 3.21) saves thousands of rows in one transaction with an R2DBC Batch of INSERT statements. The values are inlined in the SQL, so every statement of the batch is distinct. Each one is normalized: the DBQueryInfo cache (DDCaches.newFixedSizeWeightedCache(512, ..., 2 * 1024 * 1024)) cannot help since no statement repeats.

Stack sampled on a Netty I/O thread (reactor-tcp-epoll):

SQLNormalizer.normalize
 <- DBQueryInfo.<init>
 <- TraceProxyExecutionListener.beforeQuery
 <- QueryInvocationSubscriber

Measured on a staging pod: about 6 ms of CPU per statement (statements of about 400 characters). With 8,760 statements in the batch, the I/O thread is busy for about 55 s. Consequences we observed:

  • the HTTP request reached our 60 s timeout;
  • the transaction stayed idle in transaction after the cancellation;
  • the connection pool was starved on the retry.

Setting DD_INTEGRATION_R2DBC_ENABLED=false restored the normal latency immediately.

Expected Behavior

Tracing a Batch should not keep I/O threads busy proportionally to the number of statements. For example: skip or truncate the normalization above a size or count threshold, normalize lazily when the span is finished or off the I/O thread, or create one span for the whole Batch instead of one per statement.

Reproduction Code

Not yet reduced to a standalone project. The scenario: with r2dbc-postgresql, create a Batch with connection.createBatch(), add about 10,000 distinct INSERT statements with literal values, execute it with the agent attached, and compare the time spent on the I/O thread with -Ddd.integration.r2dbc.enabled=false.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions