{{ message }}
Repository navigation
Conversation
…verification Add reproducible safe-default ingestion profiles, exact ID reconciliation, versioned telemetry, application read probes and explicit sizing inputs without claiming unrun full-scale results. Bound pending plus in-flight collector rows with retryable rejection and report unresolved-project span drops as OTLP partial success. Co-Authored-By: Claude Code <noreply@anthropic.com>
Add an isolated pinned ClickHouse CI smoke using the actual canonical schema, require real execution rather than a skipped integration test, and preserve artifacts. Keep client timeout failures distinct from normal read-window cancellation. Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Replace the 20ms wall-clock race with a response-body-close handshake through the unchanged production probe path. Assert exact completed-response and success/error counters without treating normal censoring as application failure. Co-Authored-By: Claude Code <noreply@anthropic.com>
nik13
marked this pull request as ready for review
October 21, 2026 11:09
nik13
added a commit
that referenced
this pull request
Oct 22, 2026
…s edit [TH-8743] Same behaviour; the block now sits before FI_GRPC_ADDR so it merges cleanly with the FI_MAX_PENDING_ROWS override in #3572. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The only conflict was fi-collector/pkg/server/server.go: dev routes sandbox-key requests (enqueueRouted, partialSuccessResponse) while this branch bounds the pending row buffer. The merged handler admits a routed request, including one split into hidden and visible parcels, as a single unit against the row budget so a refused request that the client retries cannot store its first parcel twice. Partial-success counts now combine routing drops and unresolvable project drops. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…non-retryable A request with more rows than max_pending_rows can never be admitted, yet it was answered 503 / RESOURCE_EXHAUSTED with a retry hint, so a well-behaved client would retry it forever. It is now refused with HTTP 413 (no Retry-After) or gRPC INVALID_ARGUMENT (no RetryInfo) and a message naming the request size and the limit. A request that merely meets a full buffer keeps the retryable 503 / RESOURCE_EXHAUSTED with RetryInfo, and one that exactly fits an empty buffer is admitted. Admission is decided on the total rows of a request, including a routed request split into hidden and visible parcels, so a refused request never leaves a parcel behind for its retry to store twice. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s on /healthz A benchmark that measures the path production SDKs use needs the collector to accept gzip, which SDK exporters commonly enable and the collector refused: OTLP/HTTP bodies with Content-Encoding gzip are now decoded with the 16 MiB cap applied to both the wire bytes and the decoded bytes (other codings get 415, a corrupt body 400, a decompression bomb 413), and the gRPC gzip compressor is registered. /healthz now also reports a build object (Go version, VCS revision, or FI_BUILD_REVISION) and the admission limits in force, so a benchmark records the revision and row budget of the system under test instead of assuming them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…hutdown The test server drained into a nil writer on shutdown and panicked in a goroutine that outlived the test. It now writes to a stub ClickHouse and waits for Run to return. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The OTLP/HTTP specification requires every 4xx and 5xx response body to be a google.rpc.Status, encoded like the request. Refusals were plain text, so an SDK exporter could not report why an export failed. Every failure of the traces handler (method, media type, content coding, decode, quota, routing, auth stamp, conversion, oversize and full buffer) now answers with a Status in the request's encoding (protobuf when the media type is not known yet). Retry-After and Accept headers are kept and the HTTP status line stays authoritative. Tests drive spec-following HTTP (JSON and protobuf) and gRPC exporters over real listeners: an oversize export is sent exactly once (413 / INVALID_ARGUMENT, no retry hint) and a full-buffer refusal is retried to success once the buffer drains. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rification and a seeded workload
Result schema fi.otlpbench.v2. Counter meanings changed, so reports refuse v1.
Accounting: every request ends in exactly one outcome (acknowledged, partial
rejection, whole-request rejection retryable or not, ambiguous), and
attempted = acknowledged + partial rejected + whole-request rejected + ambiguous
is checked on every run. HTTP 429/503 and gRPC RESOURCE_EXHAUSTED with RetryInfo
are counted as whole-request rejections with their retry hint and requested
delay; 413 and other 4xx (and INVALID_ARGUMENT etc.) as non-retryable; transport
errors, timeouts, 500/502/504, malformed success bodies and client-producible
gRPC codes as ambiguous. Lag is sampled only for acknowledged spans.
Verification: a per-request outcome ledger lets missing spans be decomposed into
acknowledged-but-missing (data loss, including a lower bound inside partial
successes), rejected-and-missing (expected) and ambiguous-and-missing, and
rejected-but-persisted spans are flagged as a contract violation. The unbounded
whole-run totals scan is gone: chunks tile the run's trace-ID namespace (aligned
to traces), carry a start_time window from a per-second generation timeline so
ClickHouse can prune by the hourly primary key, group exact distinct spans by
request, and count unexpected rows per chunk; hourly windows count run rows
outside the namespace; the last chunk and incomplete-looking chunks are rescanned
and must not change. The verification budget (default grows with the run, 48h
cap), per-query server/client limit, memory limit and retries are configurable,
and an optional checkpoint makes it resumable in a new process
('otlpbench verify --checkpoint'). The reuse preflight reads the first 64 trace
IDs through the bloom-filter index instead of counting the whole project, and its
row count is no longer overwritten by the final verification.
Wire path: OTLP/HTTP protobuf (default), OTLP/HTTP JSON and OTLP/gRPC, each with
optional gzip; wire and uncompressed bytes are both recorded. Requests are built
on an ordered GOMAXPROCS worker pipeline at most one second ahead of schedule;
build latency and dispatch delay are reported so a generator bottleneck is
visible.
Workload: seeded, integer-only draws (same seed, sequence and timestamp give the
same bytes on any platform): text assembled from a seeded pool of distinct
blocks, lognormal sizes around the payload class, depth- and fan-out-bounded
trace trees with agent/chain/LLM/tool/retriever spans, configurable error rate
and session cohort split, token counts derived from text size. Realised
distributions are reported.
Reads and provenance: direct ClickHouse probes (trace by ID, session by ID,
attribute filter) during load and after settle, reported apart from application
API probes. Results record the redacted command line, generator build, generator
host hardware, ClickHouse version, table keys and cluster layout, the collector's
/healthz build and limits, and the system-under-test revision (collector-reported
when available, conflicts flagged); an executed run refuses to start without a
SUT revision. A heuristic row-budget check warns loudly when the collector's
max_pending_rows is smaller than the profile needs ('otlpbench collector-budget').
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…l ClickHouse in CI The otlp-benchmark-smoke job now also runs TestLiveClickHouseBackpressureAccounting: a real in-process collector with a 16-row budget (the equivalent of FI_MAX_PENDING_ROWS=16) over HTTP protobuf, HTTP JSON+gzip and gRPC+gzip. Load must produce whole-request 503 / RESOURCE_EXHAUSTED refusals carrying Retry-After / RetryInfo, counted as rejected; requests larger than the whole budget must be refused once each as non-retryable 413 / INVALID_ARGUMENT; and the independent ClickHouse verification must find every acknowledged span, no refused span, and attribute every missing span to a rejection. The job requires an explicit pass of both live tests and all three sub-tests and uploads both results. The 20-span smoke now also requires its direct ClickHouse reads to find the acknowledged data. An unfinished verification no longer concludes data loss from a partial scan: it reports scanned and unscanned attempted spans (unscanned counted as missing) and decomposes only the scanned part; data_loss_detected and rejection_contract_violation are set only after the recheck succeeds. OTLPBENCH_LIVE_SLOW_HOST=1 lengthens writer, settle and query timeouts for a shared, overloaded developer ClickHouse without changing any assertion. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… the v2 accounting run-profile.sh now prints the collector settings a profile needs before every plan or run (FI_MAX_PENDING_ROWS of at least 900 / 43,000 / 141,000 for smoke / sustained / burst by the documented heuristic, batch size within the budget, a build revision on /healthz), runs 'otlpbench collector-budget' (a read-only GET of /healthz only when BENCH_COLLECTOR_HEALTH_URL is set) and warns loudly when the budget is smaller than the profile needs, the health URL is unset or a large profile has no verification checkpoint. The sustained and burst profiles set verification chunk, per-query and total budgets and direct-probe cadence. It no longer records this checkout's revision as the system under test: that comes from /healthz or BENCH_SUT_REVISION, and the checkout is recorded as the generator revision. BENCH_ENCODING, BENCH_GZIP, BENCH_SEED and BENCH_VERIFY_CHECKPOINT are passed through. The runbook documents the wire paths, the seeded workload model, the outcome classes and accounting identity, chunked/rechecked/resumable verification and its budgets, direct ClickHouse reads, provenance, the live backpressure CI case, and a sourced methodology comparison with public load generators, correctness tools and vendor scale claims. It keeps every statement that the sustained and burst profiles are unmeasured and makes no comparative performance claim. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… dry-run probes verified_chunks and verified_namespace_windows now count everything verified so far, including chunks finished before a resume (resumed_at_chunk says where this process started), instead of only the current process's chunks. Direct ClickHouse reads in a dry run are reported as not-executed when enabled, and not-configured only when disabled. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… observed On an overloaded shared ClickHouse a 16-row insert was seen to take up to four minutes. A writer timeout shorter than that makes the collector writer retry an insert the server later completes, storing rows twice. OTLPBENCH_LIVE_SLOW_HOST now waits up to 15 minutes per insert, settles 8 minutes and allows 5 minutes per verification or probe query. CI (no slow-host flag) is unchanged and every assertion is unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ark reads ClickHouse can send HTTP 200 before a query finishes and report a later failure or timeout inside the body, which surfaced on an overloaded server as an unreadable 200 response during the reuse preflight. Every benchmark read now sets wait_end_of_query=1, a read that hits the client deadline says so, and preflight, provenance and lag queries use the verification query limit when it is longer than 30 seconds. The mock backpressure and oversize cases use the small payload class and a 400 ms flush interval, so refusals no longer depend on how fast the generator builds gzip requests under the race detector; their assertions are unchanged. The slow-host live chunk size stays within the validated 1,000,000 maximum. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The checkpoint embeds the run-length encoded outcome ledger, which can be large for a day-long run with highly mixed outcomes; writing it after every chunk could dominate verification. Progress between chunks is now saved at most every 10 seconds; a failure, a timeout and completion still always save, so only a killed process can lose up to 10 seconds of verified chunks. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…as an invalid batch A batch whose build was interrupted because the load window closed was counted as generated and stopped the run as invalid-or-oversized-batch. It now stops the load as duration-or-cancel and is not counted as generated; a real build failure or an out-of-order batch is still invalid. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…t is dirty 'go run' does not stamp VCS state, so run-profile.sh recorded a checkout with uncommitted generator changes as its clean HEAD revision. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ayloads The trace-by-id and session-by-id ClickHouse probes only counted rows, so they read the key columns and never the input/output payload columns a trace or session view reads. They now aggregate length(input) + length(output) server-side, which forces the payload read while keeping the response small, and report the payload bytes each query read. The attribute-filter probe stays a count. The mock and the real ClickHouse test require non-empty payload reads after settle. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ment BenchmarkBuildRequest on the same loaded 10-core host measured 4.9-8.7 ms per 500-span protobuf request and 169-185 ms with gzip (19x to 38x), about 2,000-3,000 spans/s per build worker with gzip. The text now gives that range and the per-worker rates, and says they are not a generator capacity figure. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ipped A trace or session probe whose every round lacked an acknowledged request old enough was reported as not-executed, hiding that it was scheduled and skipped; it is now skipped-no-candidate and the text report prints the skipped count. The smoke profile lowers the probe minimum age to 1s, because the 10s default exceeds its 5s load and the during-load trace and session probes could never run. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dent of mock poll speed Under the race detector on a loaded host the mock's row-scanning sentinel polls saw only 76 of 300 sentinels within a 220 ms settle, while the coordinated-omission figures held (request p99 137 ms from the actual send, 1453 ms from the scheduled send). The test now settles for 2 s, still requires at least 200 acknowledged sentinels, and requires the stalled request's own sentinel (max >= 1400 ms) and p99 >= 1300 ms from the scheduled send. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aults run-profile.sh recovery plans 15,000 spans/s for 3 hours and requires BENCH_DECLARED_FAULTS (name@offset+duration, e.g. clickhouse-stop@30m+60m): the operator performs each disruption on an isolated stack and the run records it as declared. The script still never stops or restarts anything, warns when the collector backlog cannot be followed, and points to otlpbench publish. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…isons it does not support METHODOLOGY.rst defines the five stages (offered, scheduled, attempted, accepted, durable) and read-visible sentinels, the outcome classes, every latency figure and its bounds, the clock offset, the backlog summary, the recovery figures and acceptance rule, and the publication headline rule. It lists the invalid comparisons (acknowledged vs durable, actual vs scheduled send, vendor units and definitions, plan limits, clock-limited lags, small runs as capacity) with each vendor figure labelled as documented by that vendor, and the small-run evidence with where it ran. The README covers the v3 schema, the new tests, the recovery profile and publish; RST title underlines that were one character short are fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fication budget The derived default budget assumed one query per chunk; with the durable lag there are two, so a large run on defaults could exhaust the budget before the exact-ID scan finished. The per-span allowance doubles when the durable lag is on (an explicit --verify-timeout is unchanged). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…easurements Under the race detector on a loaded host 4 of 5 sentinels became visible only after the 3 s settle; all 20 spans were verified. A late sentinel is reported as unseen (and withholds a publication headline), not a failure. The smoke still requires every acknowledged request's sentinel, at least one seen through the time-bounded poll, seen + unseen = acknowledged, both bounds for each seen one and no poll error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k offset they measured Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dget A request with more spans than the whole max_pending_rows budget was answered HTTP 413 / gRPC INVALID_ARGUMENT, which OTLP clients drop without retrying, although the previous unbounded buffer stored it. It is now admitted whenever the buffer is below the budget, so the buffer can exceed the budget by at most that one request (itself bounded by the 16 MiB transport caps); while it is over, every request gets the usual retryable refusal (503 + Retry-After / RESOURCE_EXHAUSTED + RetryInfo). Also on the admission path: - batch_max_rows above max_pending_rows could never trigger a flush, so a full buffer drained only on the age timer while every request was refused; it is clamped to the budget with a startup warning. - shutdown closes admission before the final drain, so an HTTP handler still running after the 5 s shutdown timeout is refused as retryable instead of acknowledging rows that no drain would flush. - /healthz admission reports, since start, budget-full refusals (requests, rows), over-budget admissions and shutdown refusals. The collector README lists the client-visible changes from the previous collector as a release note. The benchmark's backpressure fixtures now slow the insert path with a proxy instead of relying on a flush size above the budget, the over-budget case expects the new rule, and the crash-loss bound includes the one over-budget request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For OTLP/gRPC the build step marshalled the request only to measure it; grpc-go then marshalled and gzip-compressed it again inside the timed Export call, while it held a concurrency slot. gRPC request latency, read-visible lag and recovery figures therefore included generator CPU time that the HTTP paths exclude (about 0.5 s of gzip per 500-span request on a loaded host). The build workers now produce the protobuf bytes (gzip-compressed when asked) for gRPC exactly as for an HTTP protobuf body, and the exporter sends them through a pass-through codec and a pre-compressed gzip compressor, so the timed send only transmits. A test sends a message compressed at a level grpc-go never uses and checks that the server received exactly those bytes and decoded every span; the build benchmark gains a grpc-gzip case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… from lower edges The latency histogram used 1 ms buckets to 10 s and 1 s buckets beyond, so 10,001 ms was reported as 11,000 ms (10% high), and every series was read at bucket upper edges, including the read-visible LOWER bound and the durable lag (a lower bound on commit latency): lower bounds of 10,200 and 10,900 ms gave a median of 10,900 ms, above the true value. Histograms are now log-linear over whole microseconds (HdrHistogram layout): exact below 0.256 ms, then 128 sub-buckets per power of two to 38 h, at most 1/128 (0.79%) wide; the test measures a worst error of 0.78% from 1 ms to 24 h. Measured and upper-bound series keep upper edges capped at the maximum; lower-bound series read lower edges, so they never overstate. Buckets carry both edges, the per-hour lag windows use the same histogram, each result states its quantisation (latency_histogram), and the durable lag's server-side SQL bucketing is exact-integer (log2 corrected by shifts) and checked value by value against the Go bucketing on real ClickHouse. Histograms shrink from about 750 KiB to 31 KiB. Result schema v4, checkpoint schema v3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…light count in_flight_spans was computed as attempted minus every outcome class, so "attempted = classes + in_flight" held by construction: with acknowledged spans no longer counted, the final result reported a broken identity while every telemetry line still "held". The per-line check could catch only file tampering. The generator now counts in-flight spans and requests on their own (add when a request starts, subtract when it finishes, under the same lock as the outcome classes) and writes both. Each progress line must satisfy the span identity and the request identity, nothing may be negative, and the final line (written after every request finished) must have nothing in flight. A test shows an honest run holding, a request in flight holding on a progress line, and a finished request missing from its class breaking both lines; with the derived count it went unnoticed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…klog drain recoveryTime required the backlog to drain only when samples existed and a pre-fault baseline was found, so an operator run without a collector health URL, or a fault before the first /healthz sample, could be reported as measured-recovery-complete with a recovery time that ignored the drain, contrary to the stated rule that all three components must be observed. A fault without backlog samples, or without a sample before it, is now not recovered, with the reason. The operator recovery profile requires BENCH_COLLECTOR_HEALTH_URL instead of only warning. Tests cover each case directly and a restart run without a health URL end to end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The headline rules require every offered span to be started, acknowledged and found stored, so "verified sustained throughput" always equalled the configured --rate while reading like a measured capacity. It now says "sustained the offered N spans/s for D s (verified)" and gives one measured figure next to it: acknowledged spans over the measured load window (load start to the completion of the last export), at most the offered rate. Publication schema v2. The rule for unexpected rows and the rule for stored spans of refused requests are now separate reasons, each with a test of its own; disabling the combined rule previously left every publication test passing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
55dfc40 doubled the derived verification budget for the durable lag's extra query per chunk, but an explicit --verify-timeout (the sustained profile passes 12h) still made that query share the exact-ID scan's budget: a slow durable query could end the verification as "budget exhausted" (incomplete) instead of degrading only the durable lag. --verify-timeout now bounds the exact-ID verification alone. The durable lag has --durable-lag-timeout (default: the size of the exact-ID budget); the time its queries take extends the exact-ID deadline, and when its own budget runs out it stops with status "errors" and a reason while the exact-ID verification completes. Both budgets and the durable time spent are in the result; the profiles set both explicitly. A test with a durable query slower than either 1 s budget completes the exact-ID verification, which the shared budget did not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A direct (or application) probe query cut off by the end of its phase was counted but its elapsed time was dropped. Those are the slowest queries, so the measured-with-censoring latency understated the database's latency; on a slow host every during-load probe was censored. Each censored query's elapsed time is now kept in its own histogram, reported as a right-censored lower bound (read at bucket lower edges): censored_query_elapsed_lower_bound for direct ClickHouse reads and censored_response_elapsed_lower_bound for application reads. The text report states it next to the successful latency. Tests check a query cut off after 200 ms and an application request cut off by the window, and that a client timeout is not recorded as censored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…me of day Namespace verification windows are aligned to clock hours, so a run near the top of an hour covers two windows. The canned server answers two outside-namespace rows per window, and the test expected exactly two rows in total; it failed whenever it ran near an hour boundary. It now expects two rows per verified window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orage Finish the interrupted real-ClickHouse fallibility fixture and CI pass gates. The proxy drops three acknowledged rows and stores three other spans twice, leaving the aggregate count unchanged. Exact-ID verification must identify both errors. Keep the restart fixture's finite row budget explicit. Co-Authored-By: Claude Code <noreply@anthropic.com>
Keep the previous production default instead of silently introducing a 20,000-row cap. Zero disables admission, positive values opt in, and negative configuration is rejected. Retain retryable large-batch handling and the shutdown gate. Test repeated 25,000-span exports over all transports and report disabled admission without inventing a crash-loss bound. The default regression fails before the change; recovery fixtures retain their explicit budget and integrity assertions. Co-Authored-By: Claude Code <noreply@anthropic.com>
Document independent telemetry accounting, bounded log-linear quantization, observed recovery drain, offered-rate headlines, separate durable-query budgets and right-censored query lower bounds. Remove unsupported claims about competitors lacking instrumentation. Explain opt-in row admission and mark historical v3 figures as insufficient release evidence without retained logs. Co-Authored-By: Claude Code <noreply@anthropic.com>
Bucketing still uses whole microseconds, but an upper quantile must not be capped at a maximum truncated below the measured duration. Cover fractional-microsecond maxima in dense and sparse histograms. Correct the collector overview's stale default and describe censoring bias without claiming all censored queries must be the slowest. Co-Authored-By: Claude Code <noreply@anthropic.com>
The histogram and JSON had correct lower edges, but whole-millisecond rendering turned a 1.536 ms lower bound into 2 ms. Render stored percentile values without additional rounding in publications and text reports, including censored elapsed bounds. The new regression reproduces the overstatement before this change. Co-Authored-By: Claude Code <noreply@anthropic.com>
Reject later-only deliveries from an earlier execution before sending load. Search the complete project/run tag with fail-closed overflow policies; disclose the scan cost and the existence count's lower-bound semantics. Cover sequential reuse and invalid preflight responses, with a real-ClickHouse regression. Co-Authored-By: Claude Code <noreply@anthropic.com>
Accumulate each fully acknowledged request's exact missing count in bounded fault-window counters, including chunk splits and checkpoint resumes. Keep attribution independent of diagnostic range truncation; older checkpoints without these counters report unknown rather than guessed zeroes. Cover unequal loss, end boundaries, unordered/equal fault ends, partial outcomes and resume. Co-Authored-By: Claude Code <noreply@anthropic.com>
Keep the per-tick query bound while advancing a sequence cursor across attempted chunks, including failed queries. Preserve unseen identities and last-miss bounds, and document the effective polling cadence. Cover more than two ticks of missing sentinels, query errors, wraparound and a real-storage visible tail. Co-Authored-By: Claude Code <noreply@anthropic.com>
Include the namespace-reuse, fault-attribution and polling-fairness tests in the live CI job and require explicit PASS events rather than accepting skipped tests. Co-Authored-By: Claude Code <noreply@anthropic.com>
Use test-owned temporary dead-letter paths for direct live writer fixtures, avoiding the production /var/lib default. Keep all regression assertions unchanged. Co-Authored-By: Claude Code <noreply@anthropic.com>
Use fixed monotonic time buckets plus a matured fallback instead of a request-count ring. Keep during-load trace/session probes eligible at sustained and burst rates, including idle/resume and out-of-order completion. Co-Authored-By: Claude Code <noreply@anthropic.com>
Count t=0 and partial final batches using the same exact nanosecond due times as dispatch. Freeze the producer horizon during request drain, and snapshot schedule, counters and time together. Cover boundaries, finite plans and independent in-flight accounting. Co-Authored-By: Claude Code <noreply@anthropic.com>
Document monotonic age buckets, idle/resume fallback, and the distinction between sampled ID-query dispatch proof and real service measurements. Co-Authored-By: Claude Code <noreply@anthropic.com>
Keep the candidate clock and retained samples in one snapshot so a delayed lock acquisition cannot use a stale age cutoff. Co-Authored-By: Claude Code <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
AI use: Claude Code assisted with substantially all changed implementation, tests and documentation. Independent agent reviews and regression checks were used; human maintainer review remains required.
Refs TH-8687. Add a reproducible, bounded OTLP → real collector → ClickHouse measurement path, with independent ID reconciliation rather than treating HTTP acknowledgements as durable storage.
This PR adds opt-in bounded admission and fixes collector accounting for unresolved-project spans and shutdown behavior. Row admission remains disabled by default to preserve existing production behavior.
Reviewed implementation head:
64b4a62bbc12b2d70a83b6b713b04ff47b27ac91. Code-review findings have been addressed, but CI/admission, human review and the full-scale acceptance targets below are separate gates. This is not a capacity or merge-readiness claim.Requirements and implementation
cmd/otlpbench,pkg/ingestbench; HTTP JSON OTLP traverses the production receiver, converter and writer. Existing direct-CH seeder is not called a collector benchmark.The full design, workload specification, commands, formulas, comparison references, failure semantics and acceptance checklist are in
fi-collector/benchmark/README.rst.Competitive basis
Primary-source research checked 2026-10-21; these are documented vendor capabilities, not comparable measurements:
Compatibility and rollout
/healthzmetadata gains additive diagnostics; benchmark/report/checkpoint formats are versioned. The benchmark does not mutate the production schema.Verification at reviewed head
64b4a62bb: thecollector,otlp-benchmark-smoke,replicated,integrationandcomposejobs all ran and succeeded.64b4a62bb. This is current-head browser evidence, not a full-scale benchmark.METHODOLOGY.rst.Historical checks
The following original-implementation checks and prior-head links are historical context, not new exact-head qualification:
otlp-benchmark-smokepins ClickHouse 25.3.14.14, applies the actual canonical schema, sends 20 spans through the real collector/writer, requires a real PASS rather than a skipped test, and uploads JSON/log evidence. Passed on9f5a481348: collector/integration/replication/real-CH smoke.9f5a481348, not merely from the coverage marker.E2E: covered-by OBS-E2E-001, OBS-E2E-002
Scale evidence still required
The requested profiles remain targets, not results:
These need dedicated, approved capacity and fixture configuration. A 20-span smoke, mock test, sizing formula or short-run extrapolation does not satisfy them. Keep the tracking issue open for that acceptance evidence.
Merge gates
The current Collector and compose CI run is successful; see the exact-head verification section above for actual job outcomes. Managed E2E and admission must be read from their linked runs rather than inferred from the historical results. Admission still requires a maintainer-accepted GitHub issue;
Refs TH-8687is not such a link. A matching GitHub issue was not found in the tracking-key search, and no issue reference or acceptance label has been invented. Backend and standalone workflows reported success with their path-filtered test/install jobs skipped; those skipped jobs are not test qualification. A non-author approval is still required. Auto-merge is disabled, and no PR merge has been performed.🤖 Generated with Claude Code