What two runs of the same four filings agree on by WaylandYang · Pull Request #769 · deeplethe/utopia · GitHub
Skip to content

What two runs of the same four filings agree on - #769

Merged
WaylandYang merged 2 commits into
devfrom
docs/what-extraction-still-owes
Sep 19, 2026
Merged

WaylandYang merged 2 commits into
devfrom
docs/what-extraction-still-owes

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Two runs of the same four filings, one configuration, so that the remaining open marks in the
extraction section of prior-work are either answered or waiting on
something nameable.

What a run costs when nothing changes

Same four NVDA filings, same 14 classes and 28 properties, same 55 chunks, extracted twice against
the same endpoint with the same contract:

first second
open statements 885 824
entities 468 445
judged misworded 3.7% 4.2%
judged not stated 0.0% 0.2%
reads alone 10.7% 8.2%
figures in prose reaching no statement 0/35 1/35
figures in tables reaching no statement 7/451 (2%) 18/451 (4%)

A run is worth about seven per cent of its statements and half a point of misworded. The judge's
own spread is 1.4 points on one graph judged twice, so the two together are the band a claim has to
clear before it means anything. The reasoning cap measured in #759 stays rejected on this reading:
729 statements and 16% of table figures lost are outside both bands, not near them.

This closes the "run-to-run stability number" half of item 9.

The second look at figures is measured, and not built

Item 10 has been waiting on the second-look pass of item 5, for the figures a dense sentence drops.
The four filings are the densest text in the corpora, 25 figures per thousand characters against 7
for the next one, and figures.mjs reads no model: it counts every figure a passage writes, against
every statement drawn from that passage, table rows apart from prose.

Over the two runs, prose loses 0 and 1 figure of 35, tables 7 and 18 of 451. A second pass would
chase a dozen table cells at the price of a second call on every chunk, where a chunk already costs
150 seconds. It is not built, and the item now says what would change that: a corpus where prose
figures are lost often enough for the pass to move the number.

Two things the bench sections did not say

figures.mjs counts a figure only when it carries a currency symbol, a percent sign, a unit word or
a thousands separator, so a bare digit cannot be matched by accident; and it counts a figure as
reached when it appears in a statement drawn from its own passage, including that statement's
qualifiers and time mentions, which is where a period or an amount often lands.

The judge counts a statement once per piece of evidence, so its denominator is statement-evidence
pairs rather than statements: 845 against 824 on the second run. The same on both sides of any
comparison, but worth writing down before someone reconciles two numbers that were never the same
number.

Also

The extraction page listed alignment producing typed facts and time interpretation as proposed and
not built. Both are built, in 0044 cut 2 and 0045 cuts 1 to 3. Identity profiles and the layer
marker in the interface stay on that list.

🤖 Generated with Claude Code

WaylandYang and others added 2 commits September 19, 2026 11:41
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <wayland0916@gmail.com>
@WaylandYang
WaylandYang merged commit 44b1b94 into dev Sep 19, 2026
@WaylandYang
WaylandYang deleted the docs/what-extraction-still-owes branch September 19, 2026 04:05
@WaylandYang WaylandYang mentioned this pull request Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant