{{ message }}
Repository navigation
perf(observed): benchmark filtering and sampler overhead, compare filtering with tracing - #864
Draft
Evgeny Fedorov (evgenyfedorov2) wants to merge 8 commits into
Conversation
Add the `observed_benchmarks/emit_filtering` group. It measures one `emit!` of an info event that no processor receives, filtered out either by processors that are not interested or by a sampler that drops it, each with 1 and 3 processors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1927d8f8-25ed-4205-98ba-9632b7561696
Each iteration now emits the 5 info events that the `emit_pipeline` group delivers, so the filtered-out cost can be compared with the delivered cost of the same events. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1927d8f8-25ed-4205-98ba-9632b7561696
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #864 +/- ##
=========================================
Coverage 100.0% 100.0%
=========================================
Files 778 789 +11
Lines 90934 92087 +1153
=========================================
+ Hits 90934 92087 +1153 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
A filtered-out event is never built, so which event is emitted does not change the cost. One event shows the same comparison with less code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1927d8f8-25ed-4205-98ba-9632b7561696
Add a case that differs from `not_interested_one_processor` only by a sampler that keeps every event, so the difference between the two is what the sampler adds to each emit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1927d8f8-25ed-4205-98ba-9632b7561696
Add an `emit_delivered` benchmark that delivers the event to one processor, with and without a sampler that keeps it, so the sampler's overhead is also measured when the event is delivered. Rename `sampler_keeps_sink` to `not_interested_sampler_keeps_sink` to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1927d8f8-25ed-4205-98ba-9632b7561696
martin-kolinek
approved these changes
Oct 21, 2026
Add `tracing_filtered_out` and `tracing_delivered` benchmarks that run the same setups as `emit_filtered_out` and `emit_delivered` with `tracing::info!`: each processor is a do-nothing layer behind a per-layer filter asked on every event, and the sampler is an outermost layer. An extra case uses `LevelFilter::WARN`, the usual way tracing apps drop info logs. `tracing` and `tracing-subscriber` become dev-dependencies of `observed`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1927d8f8-25ed-4205-98ba-9632b7561696
…ework Rename the setup functions of the emit filtering benchmarks, for example `interested_sampler_keeps_sink` to `observed_interested_sampler_keeps` and `not_interested_sampler_keeps_subscriber` to `tracing_not_interested_sampler_keeps`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1927d8f8-25ed-4205-98ba-9632b7561696
tracing has no event sampler, so the layer that imitated one is removed together with its four cases. The tracing cases now cover per-layer filters that decline or accept every event, and a level filter at warn. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1927d8f8-25ed-4205-98ba-9632b7561696
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Apps that emit many info events may filter them out, either through processor interest or with an event sampler. This adds the
observed_benchmarks/emit_filteringgroup, which measures what oneemit!of an info event (SimpleLogEvent) costs when no processor receives it, what an event sampler adds to an emit, and what the same filtering costs withtracing.observed, filtered out (
emit_filtered_out):is_interestedreturnsfalse. Runs with 1 and 3 processors.EventSamplingDecision::Continue.EventSamplingDecision::Drop. Runs with 1 and 3 processors.observed, delivered (
emit_delivered) to one processor whoseprocessdoes nothing, so only observed's own delivery work is measured:tracing (
tracing_filtered_out,tracing_delivered): the same info log withtracing::info!. Each processor is a layer that does nothing, behind a per-layer filter that tracing asks on every event, as observed asks its processors. tracing has no event sampler, so only the setups without one are compared. One extra case,level_filtered_one_processor, usesLevelFilter::WARN, tracing's built-in filter by level. Each case installs its subscriber as the thread's scoped default, because the process-wide default can be set only once.Comparing each "sampler keeps" case with the same setup without a sampler gives the sampler's overhead. The sink calls its sampler once per emit, before asking any processor (
SingleSinkState::sample_and_ask_processorsincrates/observed/src/sink/core.rs). Without a sampler, or when the sampler keeps the event, the sink asks every processor; a dropping sampler stops before any processor is asked. The 3-processor cases show that difference.Results
Apple M3 Max, macOS 27.0.1, rustc 1.97.1. Time per emit is the median of four Criterion runs of the same build. The runs differed by up to 3.6 ns for observed's delivered cases, 1.5 ns for the tracing cases and 0.8 ns for observed's filtered-out cases. In earlier builds, adding or removing other benchmarks also moved identical cases by up to 0.5 ns. Small differences between cases are therefore not reliable.
warn"—" means the setup does not exist in that library: tracing has no sampler, and observed has no level filter. No case allocates. Delivered events go to a processor or layer that does nothing with them.
For reference, existing benchmarks of the same event from the same runs:
emit_pipeline/simple_log_2_fields: delivered to an OpenTelemetry loggersink_operations/emit_to_noop_sink:Sink::noop()Findings
tracing::info!first compares the event's level with the highest level any subscriber wants, before asking any filter. observed has no such check; it asks the sampler or every processor on each emit.Why delivery costs more in observed
For every delivered event, observed records the current time:
SingleSinkState::dispatch(crates/observed/src/sink/core.rs) callsself.clock.system_time(), which with the default clock isSystemTime::now()(SimpleClock::system_timeincrates/tick/src/simple_clock.rs). tracing reads no clock while delivering an event; that is left to the layer, and the layer in these benchmarks does nothing.Two throwaway builds, not part of this PR, measured each library with and without one
SystemTime::now()per event. The first added a tracing case whose layer callsSystemTime::now()for every event; the second also replaced observed's clock read with a fixed time. Each number is the average of two runs; observed without a clock read comes from the second build, the other three from the first. These runs are separate from the table above, so their numbers differ slightly from it.Doing the same work, the two cost the same: they differ by under 1 ns, less than these cases vary between runs. A real tracing log layer pays this cost too: by default, tracing-subscriber's
fmtlayer callsSystemTime::now()for every event (impl FormatTime for SystemTimein tracing-subscriber'ssrc/fmt/time/mod.rs).Validation
just anvil-clippy,just anvil-fmt,just anvil-spellcheckcargo bench -p observed --features test-util --bench observed_benchmarks -- --testcargo bench -p observed --features test-util --bench observed_benchmarks -- --criterion --criterion-arg 'emit_filtering|emit_to_noop_sink|emit_pipeline/simple_log_2_fields' --no-output --no-baseline(four runs)cargo bench -p observed --features test-util --bench observed_benchmarks -- --allocations --criterion-arg 'emit_filtering|emit_to_noop_sink|emit_pipeline/simple_log_2_fields' --no-output --no-baselineNotes
criterion_benchmarkspast Clippy's 100-line limit, so it now has#[expect(clippy::too_many_lines, ...)], as other bench registration functions in the repo do.tracingandtracing-subscriberbecome dev-dependencies ofobserved. Both were already workspace dependencies, soCargo.lockonly gains the two edges.