fix(google): report realtime usage metrics once per generation by BlueX888 · Pull Request #7896 · livekit/agents · GitHub
Skip to content

fix(google): report realtime usage metrics once per generation - #7896

Open
BlueX888 wants to merge 2 commits into
livekit:mainfrom
BlueX888:fix/google-live-usage-double-count
Open

BlueX888 wants to merge 2 commits into
livekit:mainfrom
BlueX888:fix/google-live-usage-double-count

Conversation

@BlueX888

Copy link
Copy Markdown
Contributor

What breaks

Every Gemini Live turn that produces two or more usageMetadata messages reports inflated token usage. The Live API documents that "the server will periodically send messages that include usageMetadata" during a session, and each of those messages carries the cumulative token totals (input tokens grow monotonically across turns in real traces). The Google realtime session emitted a metrics_collected event for every usageMetadata message, so a single generation that produced two or more of them was reported — and summed again by the usage collectors (ModelUsageCollector / UsageCollector do += per event) — once per message. A turn with a mid-turn report plus a final report on the turnComplete message was counted twice; long turns with several periodic reports were counted several times, each carrying the full running total.

Root cause

_handle_usage_metadata in livekit/plugins/google/realtime/realtime_api.py built and emitted a full RealtimeModelMetrics event on every usageMetadata message the receive loop forwarded to it:

if response.usage_metadata:
    self._handle_usage_metadata(response.usage_metadata)  # emits "metrics_collected" every time

There is no per-generation deduplication and no distinction between the periodic intermediate reports and the final one, so each cumulative report became a separate usage event attributed to the current generation.

For comparison, the built-in OpenAI realtime path reports usage exactly once per response (_handle_response_done in livekit/agents/llm/_realtime/openai.py), and #7231 addresses the same cumulative-usage semantics for the GPT-Live plugin.

The fix

Report usage once per generation, with the final cumulative value seen for it:

  • _ResponseGeneration records the latest usage_metadata it received.
  • While the generation is active, incoming usageMetadata messages only update that record (latest wins, nothing emitted).
  • When the generation completes (_mark_current_generation_done), the recorded usage is emitted as a single metrics_collected event. If the final usageMetadata arrives on the same message that ends the turn, it is emitted immediately instead; usage messages arriving after the generation is done are ignored with a debug log, so a mid-turn snapshot stands only in the rare case where the API splits the true final into a separate post-turn message (traces show it is normally bundled on turnComplete, which is handled correctly).
  • In _recv_task, usageMetadata is now recorded before the serverContent/toolCall handlers, which can end the generation. Without this reorder, a final usage value bundled on the turnComplete (or toolCall) message would be processed after the generation is already done and reported, and the stale mid-turn value would win.

Usage arriving while no generation exists keeps the previous warning behavior, and the emitted metrics payload (token counts, token details, duration, timestamps) is unchanged — only the emission frequency and the value's freshness change.

Tests

Added livekit-plugins/livekit-plugins-google/tests/test_realtime_usage_metrics.py (pure unit tests driving the real _recv_task with a scripted fake AsyncSession, no network):

  • test_usage_emitted_once_per_generation — one turn with an incremental and a final usageMetadata: asserts exactly one emission with the final value. Fails before the fix with two emissions.
  • test_final_usage_bundled_with_turn_complete_is_reported — the final usageMetadata arrives on the turnComplete message itself: asserts one emission carrying that final value (guards the handler ordering). Fails before the fix with two emissions.
  • test_usage_reported_once_per_turn_across_multiple_turns — two consecutive turns with several usageMetadata messages each: asserts one emission per turn, each with that turn's final value. Fails before the fix with four emissions.

Four existing parametrized unit tests in tests/test_plugin_google_realtime.py were updated to the new emission contract: test_ttft_is_unavailable_without_a_response_start_event[...] called _handle_usage_metadata directly on an active generation and pinned the old emit-on-call timing; they now complete the generation (session._mark_current_generation_done()) before asserting, and still verify exactly one usage event with ttft == -1 and the expected token counts. The full file (66 tests) passes with the fix.

The full livekit-plugins-google test suite (7 tests, including the existing STT timing tests) passes with the fix.

Gemini Live sends usageMetadata messages periodically during a turn, each
carrying the cumulative token totals, and a final one around turn
completion. The realtime session emitted a metrics_collected event for
every one of them, so usage collectors counted a single generation once
per usageMetadata message, inflating the reported token usage.

Record the latest usageMetadata on the current generation and emit the
usage metrics exactly once per generation: when the generation completes,
or right away when the final usage arrives on the message that ended the
turn (usage metadata is now recorded before the server_content/tool_call
handlers, which can end the generation).
@BlueX888
BlueX888 requested a review from a team as a code owner October 17, 2026 08:14

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment on lines +1637 to +1642

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Earlier turns counted again in usage

When a second turn completes, _report_usage_metrics emits session-cumulative tokens as that turn's usage. ModelUsageCollector adds both events, counting earlier turns again.

Learn more

Gemini Live usage snapshots accumulate across the connection, while the usage collector adds each metrics event to the conversation total. Emitting one full snapshot per generation still repeats the previous generations' usage. Track the last reported session snapshot across generations, including token modality details, and emit only the differences; handle connection resets and session resumption according to the provider's counter semantics.

Example: A first turn reports 100 input and 40 output tokens. A second turn reports cumulative totals of 150 input and 75 output tokens. The current code reports 250 input and 115 output tokens to the collector; the session consumed 150 input and 75 output tokens.

Recommended fix: Store the preceding cumulative UsageMetadata in RealtimeSession, calculate per-generation deltas in _report_usage_metrics, and update the baseline only after the metrics event is built. Test both consecutive turns and reconnection behavior.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +1624 to +1626
if current_gen._done and current_gen._usage_emitted:
logger.debug("ignoring usage metadata; already reported for this generation")
return

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Late final usage leaves incomplete token counts

When final usage arrives after turn_complete, _usage_emitted makes _handle_usage_metadata discard it. The earlier periodic snapshot remains the only reported usage, omitting tokens from the end of the turn.

Learn more

A generation can receive periodic usage before its turn_complete message. _mark_current_generation_done now emits that provisional snapshot immediately; a separate final usage message finds _usage_emitted set and is ignored. This loses the tokens added since the last periodic snapshot, even though _current_generation still points to the completed turn.

Example: An active generation receives a snapshot with 10 output tokens, followed by turn_complete, then a usage-only message with 40 output tokens. Listeners receive only 10 tokens instead of 40.

Recommended fix: Define a finalization strategy for unbundled metadata, such as deferring the completed generation's metrics until the next generation starts or the receive stream ends, while preserving support for final metadata on turn_complete and tool-call messages.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@BlueX888

Copy link
Copy Markdown
Contributor Author

@BlueX888
BlueX888 force-pushed the fix/google-live-usage-double-count branch from a16e525 to 56facff Compare October 17, 2026 15:00

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant