[Bug]: some observability capabilities loss on A2A · Issue #1132 · a2aproject/a2a-java · GitHub
Skip to content

[Bug]: some observability capabilities loss on A2A #1132

Open
1 of 7 issues completed
Open

Description

@khunj

What happened?

First, I would like to confirm whether the community has a roadmap for observability capabilities and whether the current capabilities are considered complete by the community.
Below are some issues I encountered when building observability capabilities based on a2a-java.


Issue 1: Server-side SSE per-event spans are missing

Summary

When a Server Agent sends SSE streaming events back to the Client, there are no per-event spans on the Server side. The Server span ends as soon as the stream publisher is created ("Stream publisher created"), making the actual event emission process a complete black box.

Current Behavior

Client Agent                          Server Agent
─────────                             ─────────
[CLIENT Span: sendMessage] ──HTTP──→ [SERVER Span: onMessageSendStream]
       │                                     │
       │  Span ends after stream init        │  Span ends after publisher returns
       ▼                                     ▼
  [CLIENT Span: event-1]              ❌ No Span
  [CLIENT Span: event-2]              ❌ No Span
  [CLIENT Span: event-3]              ❌ No Span
       │                                     │
       ▼                                     ▼
  Link to initial                     Event sending completely
  CLIENT Span (non-parent-child)      untraceable
  • The Server span closes before any SSE events are actually sent.
  • The Client side has per-event spans (linked via addLink), but the Server side has nothing.
  • Client and Server event spans cannot be correlated one-to-one.

Expected Behavior

Each SSE event sent by the Server should have an INTERNAL span as a child of the Server span (via context restoration), with attributes such as streaming.event.kind (status / artifact / completed / error). This would close the end-to-end trace loop.

Impact

  • Server-side event emission is a black box — no visibility into which events were sent, when, or how long each took.
  • End-to-end trace tree is broken: the Server span has no children, and Client event spans are only linked (not parent-child), so they don't appear in the trace tree view.
  • Cannot correlate Client-side event spans with Server-side event sending.

Issue 2: Push notification delivery is completely untraceable

Summary

Push notification delivery has no tracing at all. The active push callback from the Server Agent to the Client's webhook endpoint cannot be correlated to the original request.

Current Behavior

  • Push notification configuration management (set/get/delete webhook) has spans.
  • But the actual push notification delivery (the webhook callback) has zero tracing.
  • There is no span, no trace context propagation, and no way to link the push delivery back to the original task/request.

Expected Behavior

  • Push notification delivery should create a span with attributes such as push.notification.url, notification.topic, and streaming.event.kind.
  • The delivery span should be linkable to the original request span (via stored context or span link).

Impact

  • Cannot track whether push notifications were successfully delivered.
  • Cannot measure push notification delivery latency.
  • Cannot correlate push notifications back to the original task or subscription.

Issue 3: No business-level semantic attributes on spans

Summary

A2A protocol-layer spans carry basic attributes (operation name, task/message/context ID), but there are no business-level semantic attributes such as task type,.

Current Behavior

no genai attributes, no business attributes.

Expected Behavior

The SDK should provide a mechanism (e.g., callback, annotation, or API) for users to enrich spans with these business-level attributes.

Impact

  • Traces show "what happened at the protocol level" but not "what happened at the business level."

Issue 4: Task state transitions are not tracked

Summary

Task status attributes (e.g., submitted → working → completed) are reserved in the code but not activated. Task state transitions are invisible in traces.

Current Behavior

  • The span attribute for task status appears to be defined but never populated with actual values.
  • When a task transitions from submitted to working to completed, no span attribute records this transition.

Expected Behavior

Each SSE status event should carry a task.status attribute reflecting the current task state, enabling state-machine visualization and duration analysis per state.

Impact

  • Cannot track task progress through its lifecycle.
  • Cannot measure time spent in each task state.
  • Cannot build task state-machine visualizations in observability dashboards.

Issue 5: No Metrics implementation in core module

Summary

The a2a-java core module has zero Metrics implementation. There are no histograms, counters, or any other instruments measuring operation duration, throughput, or token usage.

Current Behavior

Expected Metric Type Status
gen_ai.client.operation.duration Histogram ❌ Missing
gen_ai.client.token.usage Histogram ❌ Missing
a2a.task.request.duration Histogram ❌ Missing
  • There is no gen_ai.client.operation.duration metric, which is the standard OTel GenAI operation duration histogram.

Expected Behavior

The core SDK module should provide built-in metrics instruments, at minimum:

  • gen_ai.client.operation.duration (OTel GenAI standard) — operation latency histogram
  • gen_ai.client.token.usage (OTel GenAI standard) — token usage histogram by gen_ai.token.type
  • Task request duration covering the full SSE streaming lifecycle (from stream initiation to last event)

Impact

  • Cannot measure operation latency or throughput at scale.
  • Cannot set up SLO/SLA alerts based on latency percentiles.
  • Cannot quantify agent interaction performance trends.
  • Token usage is not tracked, making cost analysis impossible.

Issue 6: Span attributes use custom gen_ai.agent.a2a.* prefix instead of official gen_ai.* SemConv

Summary

All A2A span attributes use a custom gen_ai.agent.a2a.* prefix (e.g., gen_ai.agent.a2a.operation.name, gen_ai.agent.a2a.context_id) instead of the official OpenTelemetry GenAI Semantic Conventions (gen_ai.*). This prevents automatic integration with OTel GenAI dashboards and alerting rules.

Current Behavior

Official GenAI SemConv Attribute a2a-java Attribute Status
gen_ai.provider.name (formerly gen_ai.system, deprecated) — ❌ Missing
gen_ai.operation.name gen_ai.agent.a2a.operation.name ⚠️ Prefixed
gen_ai.conversation.id gen_ai.agent.a2a.context_id ⚠️ Prefixed (code comment notes should use standard key)
gen_ai.request.model — ❌ Missing
gen_ai.response.model — ❌ Missing
gen_ai.usage.input_tokens — ❌ Missing
gen_ai.usage.output_tokens — ❌ Missing
gen_ai.client.operation.duration — ❌ Missing
gen_ai.client.token.usage — ❌ Missing
server.address / server.port — ❌ Missing

Notably, the sample/example code uses the correct gen_ai.* keys for metric attributes, creating an inconsistency between the core module and the examples.

Expected Behavior

  • Use official gen_ai.* attribute keys as defined by the OTel GenAI SemConv.
  • For A2A-specific extensions that don't have an official SemConv equivalent, use a clearly namespaced prefix (e.g., a2a.*) rather than embedding under gen_ai.agent.a2a.*.
  • Align the core module attributes with the example code (which already uses correct keys).

Impact

  • OTel GenAI-compatible dashboards (e.g., Grafana, Jaeger) cannot auto-discover and display A2A spans.
  • Alerting rules built for gen_ai.* attributes do not match A2A traces.
  • Users must build custom queries and dashboards instead of reusing existing GenAI observability tooling.
  • Inconsistency between core module and example code creates confusion.

Issue 7: SSE streaming operation duration is incomplete — span ends at stream initiation

Summary

The current span for SSE streaming requests ends immediately after the stream is initiated (when the publisher returns), not when the stream actually completes. This means gen_ai.client.operation.duration (if it existed) would only record the stream setup time — potentially 0ms — rather than the full end-to-end streaming duration.

Current Behavior

CLIENT Span: sendMessage
  │
  │  ← Span ends here (stream publisher created)
  │
  [CLIENT Span: event-1] (addLink)
  [CLIENT Span: event-2] (addLink)
  [CLIENT Span: event-3] (addLink)
  │
  │  ← Stream actually ends here (no span records this)
  ▼
  • The parent span closes before any events are sent.
  • There is no span or metric that captures the full duration from stream initiation to the last event.
  • The measured "operation duration" may be 0ms or near-0ms, completely misleading.

Expected Behavior

  • Either extend the parent span's lifetime to cover the full streaming duration, or
  • Record a dedicated metric (a2a.task.request.duration or similar) that measures the time from stream initiation to the final event.

Impact

  • Operation latency appears as ~0ms in dashboards, masking actual processing time.
  • Cannot measure true end-to-end streaming duration.
  • SLO/SLA monitoring based on operation duration would be incorrect.

Issue 8: No structured payload logging

Summary

There is no structured logging of A2A protocol message payloads. The only option is an opt-in toString() on span attributes, which is not a proper logging mechanism.

Current Behavior

  • a2a-java has gen_ai.agent.a2a.request / gen_ai.agent.a2a.response span attributes (enabled via system property), but these are span attributes, not structured logs.
  • There is no structured log event for task requests, state changes, result artifacts, negotiation payloads, authorization decisions, or notification deliveries.
  • No payload size truncation or sensitive data masking is provided.

Expected Behavior

  • Provide structured log events (at DEBUG level by default) for key protocol messages: task requests, state changes, result artifacts, negotiation payloads, authorization decisions, notification subscriptions and deliveries.
  • Include configurable payload size truncation (default 4KB) with [truncated] marker.
  • Provide sensitive data masking with customizable strategies.
  • Logs should be correlated to traces via trace_id / span_id.

Impact

  • Cannot inspect actual message payloads when debugging issues.
  • No audit trail of authorization decisions or negotiation content.
  • Cannot correlate log entries to traces for root-cause analysis.
  • The toString() approach is not suitable for production use (no truncation, no masking, no structured fields).

Summary Table

# Issue Severity Category
1 Server-side SSE per-event spans missing High Tracing Gap
2 Push notification delivery untraceable High Tracing Gap
3 No business-level semantic attributes High Semantic Gap
4 Task state transitions not tracked Medium Tracing Gap
5 No Metrics implementation in core module High Metrics Gap
6 Custom gen_ai.agent.a2a.* prefix vs official gen_ai.* SemConv High SemConv Misalignment
7 SSE streaming duration incomplete (span ends at initiation) High Tracing Gap
8 No structured payload logging Medium Logging Gap

Relevant log output

Code of Conduct

  • I agree to follow this project's Code of Conduct

Activity

  1. added theissue type on Sep 7, 2026
  2. ehsavoie commented on Sep 7, 2026

    @ehsavoie
    Collaborator

    Those don't make sense are they are not part of an A2A request:
    gen_ai.request.model
    gen_ai.response.model
    gen_ai.usage.input_tokens
    gen_ai.usage.output_tokens

    If you want to track those, it should be your agent doing that.

    This is already tracked by the server stack, so I'm not really sure of the value of this.
    server.address / server.port — ❌ Missing

    Can yo describe who is the client you are referring here:
    gen_ai.client.operation.duration — ❌ Missing
    gen_ai.client.token.usage — ❌ Missing

  3. khunj commented on Sep 7, 2026

    @khunj
    Author

    Those don't make sense are they are not part of an A2A request: gen_ai.request.model gen_ai.response.model gen_ai.usage.input_tokens gen_ai.usage.output_tokens

    If you want to track those, it should be your agent doing that.

    This is already tracked by the server stack, so I'm not really sure of the value of this. server.address / server.port — ❌ Missing

    Can yo describe who is the client you are referring here: gen_ai.client.operation.duration — ❌ Missing gen_ai.client.token.usage — ❌ Missing

    If this is the case, how can I associate the internal span with the A2A sendMessage span during the agent processing? I cannot obtain the corresponding spanId, and the sendMessage span ends directly after the first return.

  4. khunj commented on Sep 7, 2026

    @khunj
    Author

    Those don't make sense are they are not part of an A2A request: gen_ai.request.model gen_ai.response.model gen_ai.usage.input_tokens gen_ai.usage.output_tokens

    If you want to track those, it should be your agent doing that.

    This is already tracked by the server stack, so I'm not really sure of the value of this. server.address / server.port — ❌ Missing

    Can yo describe who is the client you are referring here: gen_ai.client.operation.duration — ❌ Missing gen_ai.client.token.usage — ❌ Missing

    Besides these genai-related issues, what do you think about the link tracing problems in issue 1/2/4/7 in my question?

  5. brunobat commented on Sep 8, 2026

    @brunobat

    This is my analysis:

    Issue 1 — Server-side per-event spans missing

    There seems to be a real problem, the span lifetime is bound to publisher construction, not stream lifetime.
    In relation to creating a span per SSE event, I disagree. Not even websockets work like what's requested.

    There is no semantic conventions for SSE events specifically but we can get help from the Messaging conventions and also how gen.ai working group deals with a similar problem... Unfortunately they have the Streaming chunks as TODO.
    The messaging SC does not provide concrete guidance on streams but has a section on handling attributes on batch events. I recommend something like:

    1. Have 1 span for the whole stream. This might cause problems for long lived sessions... We need to discuss if we really want this.
    2. Metric for the number of events sent by the stream
    3. Use span events to record attributes detailing the sent events.

    Same for the server but with links to the parent span from the client side.
    This will prevent parent/child ambiguity and flooding the system with spans from the stream.

    Issue 2 — Push notification delivery untraceable

    Could be a span with push.notification.url / notification.topic, linkable to the original request. No child/parent relationship because in the messaging conventions, a span can have only one parent, and delivery typically happens outside the original request's context — so correlation must be a span link, not a forced parent.

    Issue 3 — No business-level semantic attributes

    This is fair and a couple of things can be done:

    • Provide a public enrichment hook so users add attributes without reaching into internals.
    • Document a namespacing rule for user attributes to avoid collisions with a2a.* / gen_ai.*.

    Issue 4 — Task state transitions not tracked

    Transitions are discrete occurrences — model them as events. Matches the recommendation for Issue 1.

    Issue 5 — No metrics implementation

    It's a legitimate gap

    Issue 6 — Custom prefix vs. official SemConv

    I have mixed feelings here.
    We should not "rename everything to gen_ai.*.". We should use official keys only where they already exist in the semantic conventions.
    A distinct a2a namespace is correct for genuinely non-standard attributes.

    Issue 7 — SSE streaming duration incomplete

    The recommendation for Issue 1 should also fix this.

    Issue 8 — No structured payload logging

    Unbounded payloads don't belong as span attributes (cardinality/PII/size). They belong in log records or span events with truncation and masking. In both cases they are related with particular spans.

    Final Note

    A bug report with so many items is unaddressable. This should have been a discussion with multiple bugs spawning from it.

  6. changed the title [-][Bug]:[/-] [+][Bug]: some observability capabilities loss on A2A[/+] on Sep 9, 2026
  7. ehsavoie commented on Sep 9, 2026

    @ehsavoie
    Collaborator

    Issue 1 — Server-side per-event spans missing
    So the idea there is to create a closing span with a link to the span created on the request and add all the otel events as event to this closing span.

  8. self-assigned this
    on Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions