feat(collector): make ingestion capacity verifiable [TH-8687] by nik13 · Pull Request #3572 · future-agi/future-agi · GitHub
Skip to content

feat(collector): make ingestion capacity verifiable [TH-8687] - #3572

Open
nik13 wants to merge 69 commits into
devfrom
feat/th-8687-ingestion-benchmark
Open

nik13 wants to merge 69 commits into
devfrom
feat/th-8687-ingestion-benchmark

Conversation

@nik13

@nik13 nik13 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

AI use: Claude Code assisted with substantially all changed implementation, tests and documentation. Independent agent reviews and regression checks were used; human maintainer review remains required.

Refs TH-8687. Add a reproducible, bounded OTLP → real collector → ClickHouse measurement path, with independent ID reconciliation rather than treating HTTP acknowledgements as durable storage.

This PR adds opt-in bounded admission and fixes collector accounting for unresolved-project spans and shutdown behavior. Row admission remains disabled by default to preserve existing production behavior.

Reviewed implementation head: 64b4a62bbc12b2d70a83b6b713b04ff47b27ac91. Code-review findings have been addressed, but CI/admission, human review and the full-scale acceptance targets below are separate gates. This is not a capacity or merge-readiness claim.

Requirements and implementation

Requirement Implementation / evidence
Real ingestion path cmd/otlpbench, pkg/ingestbench; HTTP JSON OTLP traverses the production receiver, converter and writer. Existing direct-CH seeder is not called a collector benchmark.
Representative workload Deterministic synthetic payload-size distributions, configurable spans/trace and traces/session, root/child shapes, token metadata, error/cohort attributes. Compression assumptions are documented; no customer payloads.
Accounting Offered, scheduled, attempted, acknowledged and durable spans are distinct; read visibility is sampled. In-flight accounting, request-outcome decomposition and missing/duplicate/unexpected identity checks are separate. Ambiguous transport failure never becomes proof of rejection.
Independent storage verification Bounded exact-ID chunks, both trace and span identity checked, zero-padded native ClickHouse hexadecimal comparison. Duplicates cannot conceal missing IDs.
Visibility and reads Sentinel visibility bounds and censored-query lower bounds are disclosed. Direct database probes are separate from optional application-API probes; absent probe categories are reported as missing, not zero-latency.
Resources and reports Versioned JSON, streamed counter/resource telemetry, report command, measurement-input sizing worksheet with units and explicit extrapolation labels.
Backpressure Opt-in pending plus in-flight row cap, disabled by default. Explicit budgets include retry signalling, refusal diagnostics and batch-trigger compatibility. No claim of byte-bounded memory or a durable pre-ack queue.
Partial success Accurate rejected span counts, not ResourceSpan counts, for unresolved project data over JSON/protobuf/gRPC.
Safe execution Offline by default; explicit execution/remote-host/high-load confirmations, bounded duration/body bytes/concurrency, no redirects or implicit production target, no overwrite of result files.
Reproducible profiles Smoke plus runnable sustained and burst profiles; inputs, topology, hardware, versions and caveats recorded.

The full design, workload specification, commands, formulas, comparison references, failure semantics and acceptance checklist are in fi-collector/benchmark/README.rst.

Competitive basis

Primary-source research checked 2026-10-21; these are documented vendor capabilities, not comparable measurements:

  • Braintrust Brainstore distinguishes server flush from read-visible latency. This harness preserves that distinction rather than counting HTTP 200 as durable ingestion.
  • Braintrust Nitro publishes hardware/corpus and cold/warm methodology for query benchmarks. Query speed is not relabeled ingestion throughput here.
  • Opik scaling and Arize adb provide workload/architecture context. Stored totals, traces/day and events/s are not silently converted into spans/s or a ranking.
  • OTLP specification informs partial-success and retry handling.

Compatibility and rollout

  • No application-data migration or new production dependency. Collector /healthz metadata gains additive diagnostics; benchmark/report/checkpoint formats are versioned. The benchmark does not mutate the production schema.
  • Row admission is opt-in through a positive configured row budget; the shipped default preserves existing unbounded row admission. When enabled, callers must honor the documented refusal/retry semantics. Enabling a production default cap requires a separate rollout decision.
  • The benchmark never starts/stops production services. The disposable CI fixture is separate, project-scoped and bounded.
  • Resource limits and expected row counts are explicit. Row limits are not a process-memory guarantee; acknowledgement still precedes durable insertion in the existing collector architecture.
  • Rollback: revert the collector image/config change. Benchmark artifacts remain outside the runtime data path. No operator data deletion or automatic destructive cleanup.

Verification at reviewed head

  • Collector and compose CI completed successfully at 64b4a62bb: the collector, otlp-benchmark-smoke, replicated, integration and compose jobs all ran and succeeded.
  • Managed distributed E2E: the distributed E2E job and E2E Tests Pass aggregate both completed successfully at 64b4a62bb. This is current-head browser evidence, not a full-scale benchmark.
  • Local review fixes include exact namespace-reuse preflight, per-fault loss attribution, fair sentinel polling, bounded age-aware probe candidates and batch-consistent telemetry. Independent fix-removal checks and focused race-enabled tests support these correctness claims; virtual-time tests are not capacity measurements.
  • The publisher now distinguishes verified offered rate from measured capacity, preserves latency bounds and discloses missing measurements. See METHODOLOGY.rst.
  • Admission failed for a missing accepted GitHub issue and the previously absent AI-use line. This edit supplies the disclosure only; it does not satisfy or override the accepted-issue requirement.

Historical checks

The following original-implementation checks and prior-head links are historical context, not new exact-head qualification:

cd fi-collector
go test -p 1 ./...
go test -race ./pkg/ingestbench ./cmd/otlpbench ./pkg/server ./cmd/fi-collector
go vet ./pkg/ingestbench ./cmd/otlpbench ./pkg/server ./cmd/fi-collector
bash -n benchmark/run-profile.sh
  • Targeted mutation/fallibility checks demonstrated the queue-cap, rejected-span-count and probe-cancellation regressions before restoring the fixes.
  • Independent review found and corrected missing gRPC retry details, unpadded CH identity comparisons, and normal probe-window cancellation miscounted as server errors.
  • The mock pipeline artifact is explicitly mock storage evidence only, not SQL correctness or capacity evidence.
  • New hosted CI job otlp-benchmark-smoke pins ClickHouse 25.3.14.14, applies the actual canonical schema, sends 20 spans through the real collector/writer, requires a real PASS rather than a skipped test, and uploads JSON/log evidence. Passed on 9f5a481348: collector/integration/replication/real-CH smoke.
  • Existing successful-ingestion browser flows are the declared E2E coverage; protocol rejection behavior has direct integration/race tests. Managed distributed E2E passed on 9f5a481348, not merely from the coverage marker.

E2E: covered-by OBS-E2E-001, OBS-E2E-002

Scale evidence still required

The requested profiles remain targets, not results:

  • sustained 15,000 spans/s for a full 24 hours;
  • separate 60,000 spans/s burst;
  • independent zero-loss verification and measured backlog/visibility lag;
  • authenticated list/filter/session/dashboard performance on that populated system;
  • measured storage/index/rollup amplification and per-shard capacity;
  • replicated-cluster failure/recovery under load.

These need dedicated, approved capacity and fixture configuration. A 20-span smoke, mock test, sizing formula or short-run extrapolation does not satisfy them. Keep the tracking issue open for that acceptance evidence.

Merge gates

The current Collector and compose CI run is successful; see the exact-head verification section above for actual job outcomes. Managed E2E and admission must be read from their linked runs rather than inferred from the historical results. Admission still requires a maintainer-accepted GitHub issue; Refs TH-8687 is not such a link. A matching GitHub issue was not found in the tracking-key search, and no issue reference or acceptance label has been invented. Backend and standalone workflows reported success with their path-filtered test/install jobs skipped; those skipped jobs are not test qualification. A non-author approval is still required. Auto-merge is disabled, and no PR merge has been performed.

🤖 Generated with Claude Code

nik13 and others added 3 commits October 21, 2026 02:35
…verification

Add reproducible safe-default ingestion profiles, exact ID reconciliation, versioned telemetry, application read probes and explicit sizing inputs without claiming unrun full-scale results. Bound pending plus in-flight collector rows with retryable rejection and report unresolved-project span drops as OTLP partial success.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Add an isolated pinned ClickHouse CI smoke using the actual canonical schema, require real execution rather than a skipped integration test, and preserve artifacts. Keep client timeout failures distinct from normal read-window cancellation.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Replace the 20ms wall-clock race with a response-body-close handshake through the unchanged production probe path. Assert exact completed-response and success/error counters without treating normal censoring as application failure.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@nik13
nik13 marked this pull request as ready for review October 21, 2026 11:09
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

@github-actions github-actions Bot added the needs-admission Admission gate found template/issue/branch/hygiene problems label Oct 21, 2026
nik13 added a commit that referenced this pull request Oct 22, 2026
…s edit [TH-8743]

Same behaviour; the block now sits before FI_GRPC_ADDR so it merges
cleanly with the FI_MAX_PENDING_ROWS override in #3572.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 19 commits October 22, 2026 02:14
The only conflict was fi-collector/pkg/server/server.go: dev routes sandbox-key
requests (enqueueRouted, partialSuccessResponse) while this branch bounds the
pending row buffer. The merged handler admits a routed request, including one
split into hidden and visible parcels, as a single unit against the row budget
so a refused request that the client retries cannot store its first parcel
twice. Partial-success counts now combine routing drops and unresolvable
project drops.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…non-retryable

A request with more rows than max_pending_rows can never be admitted, yet it was
answered 503 / RESOURCE_EXHAUSTED with a retry hint, so a well-behaved client
would retry it forever. It is now refused with HTTP 413 (no Retry-After) or
gRPC INVALID_ARGUMENT (no RetryInfo) and a message naming the request size and
the limit. A request that merely meets a full buffer keeps the retryable 503 /
RESOURCE_EXHAUSTED with RetryInfo, and one that exactly fits an empty buffer is
admitted.

Admission is decided on the total rows of a request, including a routed request
split into hidden and visible parcels, so a refused request never leaves a parcel
behind for its retry to store twice.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s on /healthz

A benchmark that measures the path production SDKs use needs the collector to
accept gzip, which SDK exporters commonly enable and the collector refused:
OTLP/HTTP bodies with Content-Encoding gzip are now decoded with the 16 MiB cap
applied to both the wire bytes and the decoded bytes (other codings get 415, a
corrupt body 400, a decompression bomb 413), and the gRPC gzip compressor is
registered.

/healthz now also reports a build object (Go version, VCS revision, or
FI_BUILD_REVISION) and the admission limits in force, so a benchmark records the
revision and row budget of the system under test instead of assuming them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…hutdown

The test server drained into a nil writer on shutdown and panicked in a
goroutine that outlived the test. It now writes to a stub ClickHouse and waits
for Run to return.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The OTLP/HTTP specification requires every 4xx and 5xx response body to be a
google.rpc.Status, encoded like the request. Refusals were plain text, so an SDK
exporter could not report why an export failed. Every failure of the traces
handler (method, media type, content coding, decode, quota, routing, auth stamp,
conversion, oversize and full buffer) now answers with a Status in the request's
encoding (protobuf when the media type is not known yet). Retry-After and Accept
headers are kept and the HTTP status line stays authoritative.

Tests drive spec-following HTTP (JSON and protobuf) and gRPC exporters over real
listeners: an oversize export is sent exactly once (413 / INVALID_ARGUMENT, no
retry hint) and a full-buffer refusal is retried to success once the buffer
drains.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rification and a seeded workload

Result schema fi.otlpbench.v2. Counter meanings changed, so reports refuse v1.

Accounting: every request ends in exactly one outcome (acknowledged, partial
rejection, whole-request rejection retryable or not, ambiguous), and
attempted = acknowledged + partial rejected + whole-request rejected + ambiguous
is checked on every run. HTTP 429/503 and gRPC RESOURCE_EXHAUSTED with RetryInfo
are counted as whole-request rejections with their retry hint and requested
delay; 413 and other 4xx (and INVALID_ARGUMENT etc.) as non-retryable; transport
errors, timeouts, 500/502/504, malformed success bodies and client-producible
gRPC codes as ambiguous. Lag is sampled only for acknowledged spans.

Verification: a per-request outcome ledger lets missing spans be decomposed into
acknowledged-but-missing (data loss, including a lower bound inside partial
successes), rejected-and-missing (expected) and ambiguous-and-missing, and
rejected-but-persisted spans are flagged as a contract violation. The unbounded
whole-run totals scan is gone: chunks tile the run's trace-ID namespace (aligned
to traces), carry a start_time window from a per-second generation timeline so
ClickHouse can prune by the hourly primary key, group exact distinct spans by
request, and count unexpected rows per chunk; hourly windows count run rows
outside the namespace; the last chunk and incomplete-looking chunks are rescanned
and must not change. The verification budget (default grows with the run, 48h
cap), per-query server/client limit, memory limit and retries are configurable,
and an optional checkpoint makes it resumable in a new process
('otlpbench verify --checkpoint'). The reuse preflight reads the first 64 trace
IDs through the bloom-filter index instead of counting the whole project, and its
row count is no longer overwritten by the final verification.

Wire path: OTLP/HTTP protobuf (default), OTLP/HTTP JSON and OTLP/gRPC, each with
optional gzip; wire and uncompressed bytes are both recorded. Requests are built
on an ordered GOMAXPROCS worker pipeline at most one second ahead of schedule;
build latency and dispatch delay are reported so a generator bottleneck is
visible.

Workload: seeded, integer-only draws (same seed, sequence and timestamp give the
same bytes on any platform): text assembled from a seeded pool of distinct
blocks, lognormal sizes around the payload class, depth- and fan-out-bounded
trace trees with agent/chain/LLM/tool/retriever spans, configurable error rate
and session cohort split, token counts derived from text size. Realised
distributions are reported.

Reads and provenance: direct ClickHouse probes (trace by ID, session by ID,
attribute filter) during load and after settle, reported apart from application
API probes. Results record the redacted command line, generator build, generator
host hardware, ClickHouse version, table keys and cluster layout, the collector's
/healthz build and limits, and the system-under-test revision (collector-reported
when available, conflicts flagged); an executed run refuses to start without a
SUT revision. A heuristic row-budget check warns loudly when the collector's
max_pending_rows is smaller than the profile needs ('otlpbench collector-budget').

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…l ClickHouse in CI

The otlp-benchmark-smoke job now also runs TestLiveClickHouseBackpressureAccounting:
a real in-process collector with a 16-row budget (the equivalent of
FI_MAX_PENDING_ROWS=16) over HTTP protobuf, HTTP JSON+gzip and gRPC+gzip. Load
must produce whole-request 503 / RESOURCE_EXHAUSTED refusals carrying
Retry-After / RetryInfo, counted as rejected; requests larger than the whole
budget must be refused once each as non-retryable 413 / INVALID_ARGUMENT; and
the independent ClickHouse verification must find every acknowledged span, no
refused span, and attribute every missing span to a rejection. The job requires
an explicit pass of both live tests and all three sub-tests and uploads both
results. The 20-span smoke now also requires its direct ClickHouse reads to find
the acknowledged data.

An unfinished verification no longer concludes data loss from a partial scan: it
reports scanned and unscanned attempted spans (unscanned counted as missing) and
decomposes only the scanned part; data_loss_detected and
rejection_contract_violation are set only after the recheck succeeds.

OTLPBENCH_LIVE_SLOW_HOST=1 lengthens writer, settle and query timeouts for a
shared, overloaded developer ClickHouse without changing any assertion.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… the v2 accounting

run-profile.sh now prints the collector settings a profile needs before every plan
or run (FI_MAX_PENDING_ROWS of at least 900 / 43,000 / 141,000 for smoke /
sustained / burst by the documented heuristic, batch size within the budget, a
build revision on /healthz), runs 'otlpbench collector-budget' (a read-only GET of
/healthz only when BENCH_COLLECTOR_HEALTH_URL is set) and warns loudly when the
budget is smaller than the profile needs, the health URL is unset or a large
profile has no verification checkpoint. The sustained and burst profiles set
verification chunk, per-query and total budgets and direct-probe cadence. It no
longer records this checkout's revision as the system under test: that comes from
/healthz or BENCH_SUT_REVISION, and the checkout is recorded as the generator
revision. BENCH_ENCODING, BENCH_GZIP, BENCH_SEED and BENCH_VERIFY_CHECKPOINT are
passed through.

The runbook documents the wire paths, the seeded workload model, the outcome
classes and accounting identity, chunked/rechecked/resumable verification and its
budgets, direct ClickHouse reads, provenance, the live backpressure CI case, and a
sourced methodology comparison with public load generators, correctness tools and
vendor scale claims. It keeps every statement that the sustained and burst
profiles are unmeasured and makes no comparative performance claim.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… dry-run probes

verified_chunks and verified_namespace_windows now count everything verified so
far, including chunks finished before a resume (resumed_at_chunk says where this
process started), instead of only the current process's chunks. Direct ClickHouse
reads in a dry run are reported as not-executed when enabled, and not-configured
only when disabled.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… observed

On an overloaded shared ClickHouse a 16-row insert was seen to take up to four
minutes. A writer timeout shorter than that makes the collector writer retry an
insert the server later completes, storing rows twice. OTLPBENCH_LIVE_SLOW_HOST
now waits up to 15 minutes per insert, settles 8 minutes and allows 5 minutes per
verification or probe query. CI (no slow-host flag) is unchanged and every
assertion is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ark reads

ClickHouse can send HTTP 200 before a query finishes and report a later failure
or timeout inside the body, which surfaced on an overloaded server as an
unreadable 200 response during the reuse preflight. Every benchmark read now
sets wait_end_of_query=1, a read that hits the client deadline says so, and
preflight, provenance and lag queries use the verification query limit when it is
longer than 30 seconds.

The mock backpressure and oversize cases use the small payload class and a
400 ms flush interval, so refusals no longer depend on how fast the generator
builds gzip requests under the race detector; their assertions are unchanged.
The slow-host live chunk size stays within the validated 1,000,000 maximum.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The checkpoint embeds the run-length encoded outcome ledger, which can be large
for a day-long run with highly mixed outcomes; writing it after every chunk
could dominate verification. Progress between chunks is now saved at most every
10 seconds; a failure, a timeout and completion still always save, so only a
killed process can lose up to 10 seconds of verified chunks.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…as an invalid batch

A batch whose build was interrupted because the load window closed was
counted as generated and stopped the run as invalid-or-oversized-batch.
It now stops the load as duration-or-cancel and is not counted as
generated; a real build failure or an out-of-order batch is still invalid.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…t is dirty

'go run' does not stamp VCS state, so run-profile.sh recorded a checkout
with uncommitted generator changes as its clean HEAD revision.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ayloads

The trace-by-id and session-by-id ClickHouse probes only counted rows, so
they read the key columns and never the input/output payload columns a
trace or session view reads. They now aggregate length(input) +
length(output) server-side, which forces the payload read while keeping
the response small, and report the payload bytes each query read. The
attribute-filter probe stays a count. The mock and the real ClickHouse
test require non-empty payload reads after settle.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ment

BenchmarkBuildRequest on the same loaded 10-core host measured 4.9-8.7 ms per
500-span protobuf request and 169-185 ms with gzip (19x to 38x), about
2,000-3,000 spans/s per build worker with gzip. The text now gives that range
and the per-worker rates, and says they are not a generator capacity figure.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ipped

A trace or session probe whose every round lacked an acknowledged request
old enough was reported as not-executed, hiding that it was scheduled and
skipped; it is now skipped-no-candidate and the text report prints the
skipped count. The smoke profile lowers the probe minimum age to 1s,
because the 10s default exceeds its 5s load and the during-load trace and
session probes could never run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
nik13 and others added 30 commits October 22, 2026 22:09
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dent of mock poll speed

Under the race detector on a loaded host the mock's row-scanning sentinel
polls saw only 76 of 300 sentinels within a 220 ms settle, while the
coordinated-omission figures held (request p99 137 ms from the actual send,
1453 ms from the scheduled send). The test now settles for 2 s, still requires
at least 200 acknowledged sentinels, and requires the stalled request's own
sentinel (max >= 1400 ms) and p99 >= 1300 ms from the scheduled send.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aults

run-profile.sh recovery plans 15,000 spans/s for 3 hours and requires
BENCH_DECLARED_FAULTS (name@offset+duration, e.g. clickhouse-stop@30m+60m): the
operator performs each disruption on an isolated stack and the run records it
as declared. The script still never stops or restarts anything, warns when
the collector backlog cannot be followed, and points to otlpbench publish.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…isons it does not support

METHODOLOGY.rst defines the five stages (offered, scheduled, attempted,
accepted, durable) and read-visible sentinels, the outcome classes, every
latency figure and its bounds, the clock offset, the backlog summary, the
recovery figures and acceptance rule, and the publication headline rule. It
lists the invalid comparisons (acknowledged vs durable, actual vs scheduled
send, vendor units and definitions, plan limits, clock-limited lags, small
runs as capacity) with each vendor figure labelled as documented by that
vendor, and the small-run evidence with where it ran. The README covers the
v3 schema, the new tests, the recovery profile and publish; RST title
underlines that were one character short are fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fication budget

The derived default budget assumed one query per chunk; with the durable lag
there are two, so a large run on defaults could exhaust the budget before the
exact-ID scan finished. The per-span allowance doubles when the durable lag is
on (an explicit --verify-timeout is unchanged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…easurements

Under the race detector on a loaded host 4 of 5 sentinels became visible only
after the 3 s settle; all 20 spans were verified. A late sentinel is reported
as unseen (and withholds a publication headline), not a failure. The smoke
still requires every acknowledged request's sentinel, at least one seen
through the time-bounded poll, seen + unseen = acknowledged, both bounds for
each seen one and no poll error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k offset they measured

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dget

A request with more spans than the whole max_pending_rows budget was
answered HTTP 413 / gRPC INVALID_ARGUMENT, which OTLP clients drop
without retrying, although the previous unbounded buffer stored it. It
is now admitted whenever the buffer is below the budget, so the buffer
can exceed the budget by at most that one request (itself bounded by the
16 MiB transport caps); while it is over, every request gets the usual
retryable refusal (503 + Retry-After / RESOURCE_EXHAUSTED + RetryInfo).

Also on the admission path:
- batch_max_rows above max_pending_rows could never trigger a flush, so
  a full buffer drained only on the age timer while every request was
  refused; it is clamped to the budget with a startup warning.
- shutdown closes admission before the final drain, so an HTTP handler
  still running after the 5 s shutdown timeout is refused as retryable
  instead of acknowledging rows that no drain would flush.
- /healthz admission reports, since start, budget-full refusals
  (requests, rows), over-budget admissions and shutdown refusals.

The collector README lists the client-visible changes from the previous
collector as a release note. The benchmark's backpressure fixtures now
slow the insert path with a proxy instead of relying on a flush size
above the budget, the over-budget case expects the new rule, and the
crash-loss bound includes the one over-budget request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For OTLP/gRPC the build step marshalled the request only to measure it;
grpc-go then marshalled and gzip-compressed it again inside the timed
Export call, while it held a concurrency slot. gRPC request latency,
read-visible lag and recovery figures therefore included generator CPU
time that the HTTP paths exclude (about 0.5 s of gzip per 500-span
request on a loaded host).

The build workers now produce the protobuf bytes (gzip-compressed when
asked) for gRPC exactly as for an HTTP protobuf body, and the exporter
sends them through a pass-through codec and a pre-compressed gzip
compressor, so the timed send only transmits. A test sends a message
compressed at a level grpc-go never uses and checks that the server
received exactly those bytes and decoded every span; the build
benchmark gains a grpc-gzip case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… from lower edges

The latency histogram used 1 ms buckets to 10 s and 1 s buckets beyond,
so 10,001 ms was reported as 11,000 ms (10% high), and every series was
read at bucket upper edges, including the read-visible LOWER bound and
the durable lag (a lower bound on commit latency): lower bounds of
10,200 and 10,900 ms gave a median of 10,900 ms, above the true value.

Histograms are now log-linear over whole microseconds (HdrHistogram
layout): exact below 0.256 ms, then 128 sub-buckets per power of two to
38 h, at most 1/128 (0.79%) wide; the test measures a worst error of
0.78% from 1 ms to 24 h. Measured and upper-bound series keep upper
edges capped at the maximum; lower-bound series read lower edges, so
they never overstate. Buckets carry both edges, the per-hour lag
windows use the same histogram, each result states its quantisation
(latency_histogram), and the durable lag's server-side SQL bucketing is
exact-integer (log2 corrected by shifts) and checked value by value
against the Go bucketing on real ClickHouse. Histograms shrink from
about 750 KiB to 31 KiB. Result schema v4, checkpoint schema v3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…light count

in_flight_spans was computed as attempted minus every outcome class, so
"attempted = classes + in_flight" held by construction: with acknowledged
spans no longer counted, the final result reported a broken identity
while every telemetry line still "held". The per-line check could catch
only file tampering.

The generator now counts in-flight spans and requests on their own (add
when a request starts, subtract when it finishes, under the same lock as
the outcome classes) and writes both. Each progress line must satisfy
the span identity and the request identity, nothing may be negative,
and the final line (written after every request finished) must have
nothing in flight. A test shows an honest run holding, a request in
flight holding on a progress line, and a finished request missing from
its class breaking both lines; with the derived count it went
unnoticed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…klog drain

recoveryTime required the backlog to drain only when samples existed and
a pre-fault baseline was found, so an operator run without a collector
health URL, or a fault before the first /healthz sample, could be
reported as measured-recovery-complete with a recovery time that
ignored the drain, contrary to the stated rule that all three
components must be observed.

A fault without backlog samples, or without a sample before it, is now
not recovered, with the reason. The operator recovery profile requires
BENCH_COLLECTOR_HEALTH_URL instead of only warning. Tests cover each
case directly and a restart run without a health URL end to end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The headline rules require every offered span to be started,
acknowledged and found stored, so "verified sustained throughput"
always equalled the configured --rate while reading like a measured
capacity. It now says "sustained the offered N spans/s for D s
(verified)" and gives one measured figure next to it: acknowledged
spans over the measured load window (load start to the completion of
the last export), at most the offered rate. Publication schema v2.

The rule for unexpected rows and the rule for stored spans of refused
requests are now separate reasons, each with a test of its own;
disabling the combined rule previously left every publication test
passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
55dfc40 doubled the derived verification budget for the durable lag's
extra query per chunk, but an explicit --verify-timeout (the sustained
profile passes 12h) still made that query share the exact-ID scan's
budget: a slow durable query could end the verification as "budget
exhausted" (incomplete) instead of degrading only the durable lag.

--verify-timeout now bounds the exact-ID verification alone. The
durable lag has --durable-lag-timeout (default: the size of the exact-ID
budget); the time its queries take extends the exact-ID deadline, and
when its own budget runs out it stops with status "errors" and a reason
while the exact-ID verification completes. Both budgets and the durable
time spent are in the result; the profiles set both explicitly. A test
with a durable query slower than either 1 s budget completes the
exact-ID verification, which the shared budget did not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A direct (or application) probe query cut off by the end of its phase
was counted but its elapsed time was dropped. Those are the slowest
queries, so the measured-with-censoring latency understated the
database's latency; on a slow host every during-load probe was
censored.

Each censored query's elapsed time is now kept in its own histogram,
reported as a right-censored lower bound (read at bucket lower edges):
censored_query_elapsed_lower_bound for direct ClickHouse reads and
censored_response_elapsed_lower_bound for application reads. The text
report states it next to the successful latency. Tests check a query
cut off after 200 ms and an application request cut off by the window,
and that a client timeout is not recorded as censored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…me of day

Namespace verification windows are aligned to clock hours, so a run near
the top of an hour covers two windows. The canned server answers two
outside-namespace rows per window, and the test expected exactly two
rows in total; it failed whenever it ran near an hour boundary. It now
expects two rows per verified window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orage

Finish the interrupted real-ClickHouse fallibility fixture and CI pass gates. The proxy drops three acknowledged rows and stores three other spans twice, leaving the aggregate count unchanged. Exact-ID verification must identify both errors. Keep the restart fixture's finite row budget explicit.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Keep the previous production default instead of silently introducing a 20,000-row cap. Zero disables admission, positive values opt in, and negative configuration is rejected. Retain retryable large-batch handling and the shutdown gate. Test repeated 25,000-span exports over all transports and report disabled admission without inventing a crash-loss bound. The default regression fails before the change; recovery fixtures retain their explicit budget and integrity assertions.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Document independent telemetry accounting, bounded log-linear quantization, observed recovery drain, offered-rate headlines, separate durable-query budgets and right-censored query lower bounds. Remove unsupported claims about competitors lacking instrumentation. Explain opt-in row admission and mark historical v3 figures as insufficient release evidence without retained logs.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Bucketing still uses whole microseconds, but an upper quantile must not be capped at a maximum truncated below the measured duration. Cover fractional-microsecond maxima in dense and sparse histograms. Correct the collector overview's stale default and describe censoring bias without claiming all censored queries must be the slowest.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The histogram and JSON had correct lower edges, but whole-millisecond rendering turned a 1.536 ms lower bound into 2 ms. Render stored percentile values without additional rounding in publications and text reports, including censored elapsed bounds. The new regression reproduces the overstatement before this change.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Reject later-only deliveries from an earlier execution before sending load. Search the complete project/run tag with fail-closed overflow policies; disclose the scan cost and the existence count's lower-bound semantics. Cover sequential reuse and invalid preflight responses, with a real-ClickHouse regression.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Accumulate each fully acknowledged request's exact missing count in bounded fault-window counters, including chunk splits and checkpoint resumes. Keep attribution independent of diagnostic range truncation; older checkpoints without these counters report unknown rather than guessed zeroes. Cover unequal loss, end boundaries, unordered/equal fault ends, partial outcomes and resume.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Keep the per-tick query bound while advancing a sequence cursor across attempted chunks, including failed queries. Preserve unseen identities and last-miss bounds, and document the effective polling cadence. Cover more than two ticks of missing sentinels, query errors, wraparound and a real-storage visible tail.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Include the namespace-reuse, fault-attribution and polling-fairness tests in the live CI job and require explicit PASS events rather than accepting skipped tests.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Use test-owned temporary dead-letter paths for direct live writer fixtures, avoiding the production /var/lib default. Keep all regression assertions unchanged.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Use fixed monotonic time buckets plus a matured fallback instead of a request-count ring. Keep during-load trace/session probes eligible at sustained and burst rates, including idle/resume and out-of-order completion.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Count t=0 and partial final batches using the same exact nanosecond due times as dispatch. Freeze the producer horizon during request drain, and snapshot schedule, counters and time together. Cover boundaries, finite plans and independent in-flight accounting.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Document monotonic age buckets, idle/resume fallback, and the distinction between sampled ID-query dispatch proof and real service measurements.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Keep the candidate clock and retained samples in one snapshot so a delayed lock acquisition cannot use a stale age cutoff.

Co-Authored-By: Claude Code <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs infra needs-admission Admission gate found template/issue/branch/hygiene problems

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant