Let a source push statements in the open contract by WaylandYang · Pull Request #884 · deeplethe/utopia · GitHub
Skip to content

Let a source push statements in the open contract - #884

Merged
WaylandYang merged 7 commits into
devfrom
feat/statements-source
Sep 24, 2026
Merged

WaylandYang merged 7 commits into
devfrom
feat/statements-source

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Record 0054, cut 1. Surfaced by #875: a system that already holds {thing, relation, value, when} could only get it in by spelling it into prose, chunking it on a model-sized budget, and paying a model call to read the prose back into a statement.

What lands

  • A statements source kind (next to api, same push token, same identity/version/tombstone semantics, same run history) and POST /sources/{id}/statements.
  • The body is the contract. The request carries the open extraction shape verbatim (e / s / n, the same positions the model is asked to fill) inside the api push's envelope (external_id, doc_time, deleted). No second representation.
  • The payload is the document, in one piece. Stored as the document's content and its single chunk; the chunker is not consulted (its budget is sized for a model's attention, and no model reads this).
  • No model, same path. For a document under a statements source, extraction parses the chunk instead of prompting for it and then continues unchanged (identity, described things, name facts, evidence, time mentions, qualifiers, drop signals). The job runs with no chat model configured.
  • The item is its own evidence. Statements carry no quote; the evidence row names the chunk and phrase with null offsets, which is what fact_evidence already means by "not located" (0061).
  • No slot for a type. Keys outside the envelope and the contract are refused at the door (422), so a client cannot believe it wrote a typed fact. Pushed statements reach the typed graph only through alignment.
  • Events, not state; an update marks, it does not close. Both stated in the record and in the in-app guide, because the first caller will try with a table and the second will expect the old observation to close.
  • Library: the source appears like an api source (status, token dialog, guide link); Braces icon; both language packs; the ingest guide has a new section and its chooser table a new column.

No migration: sources.kind is unconstrained text.

Not in this cut

Batch pushes, a typed write, a table importer (0036), synthesised quotes, per-item confidence. Listed with reasons in the record.

Verified

  • cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings: clean.
  • a_source_kind_is_listed_once (frontend list ↔ backend enum, order included): passes.
  • New tests (sources_statements_tests.rs): the door refuses unknown keys / a non-null quote / a short item / a blank identity / an empty s (no DB); against pgvector/pgvector:pg16 on a fresh database: a pushed statement becomes one open fact with phrase is on, both entities, and an evidence row on the payload's own chunk with null quote and offsets, with no chat model configured; a second push under the same identity is unchanged then updated with a second version; the route answers 422 / 404 (an api source) / 401.
  • cargo test -p utopia-server on that fresh database: 398 passed, 0 failed.
  • pnpm guard, pnpm typecheck, pnpm test (132), pnpm build: pass.

One thing worth knowing if you run the DB tests locally: a dev server connected to the same database will have its job worker pick up the process_document jobs the route tests enqueue and fail them against its own data dir. Point tests at a database the server is not using.

Refs #875.

🤖 Generated with Claude Code

WaylandYang and others added 2 commits September 23, 2026 21:08
…model must read back

A statements source takes the extraction contract itself (e/s/n) on
POST /sources/{id}/statements with the api push's identity, versions and
tombstones; the payload is one chunk and extraction parses it instead of
prompting, so a pushed statement is an open statement and reaches the typed
graph only through alignment. Record 0054.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
… are still computed

Only the chat client is absent when a chunk is the contract itself; the
embedding model that #877 uses for name vectors comes from the same settings
row and applies to pushed names as much as extracted ones.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
@WaylandYang

Copy link
Copy Markdown
Contributor Author

…se an unlisted subject at the door

`locate_time` begins with `quote?`, and the door requires a pushed
statement's quote to be null, so every `when` / `ended` on a pushed
statement fell into the TIME_NOT_IN_QUOTE branch: the fact landed with
only doc_time and a drop signal, and no time mention was ever recorded.
On the pushed path the chunk is the payload, so the words are looked up
in the chunk directly. The DB test now asserts the mention lands; it
fails without this change.

The door now resolves each statement's subject and each name's entity
against `e`, folding whitespace and case as `name_key` does, and answers
422 instead of letting extraction drop the item silently as UNKNOWN_REF.
An object not in `e` still lands as a literal value, as it does for a
model's reply. The route carries a body limit of four payloads so the
2 MiB default is no longer buffered before the token check; the
documented 64 KiB refusal stays a 422 with its message.

Tests added: tombstone marks and a new push revives, a tombstone for an
unknown identity is a no-op, the missing Authorization header, the
64 KiB limit over HTTP, the 200-item limit, an `n` entity not in `e`,
a non-null `n` quote, and the folded subject that must pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
@WaylandYang

Copy link
Copy Markdown
Contributor Author

Updated in a0b3875, all three points.

Time words (blocking). On the pushed path the words are now located in the chunk directly (the chunk is the payload), instead of going through locate_time, which returns at quote? before it searches. a_pushed_statement_reaches_the_open_graph_without_a_model now asserts that one time mention (08:14:03, role when) lands on the fact; I ran it with the fix stashed and it fails there, so it does guard the path.

Unlisted subject / n entity. The door resolves s[i][1] and n[i][0] against e with the same folding as name_key (whitespace and case only) and answers 422 naming the slot. An object not in e still lands as a literal value, as it does for a model's reply; the record and the guide say both.

Tests. Added: tombstone marks and a new push revives (missing_since set, then cleared, document never deleted); a tombstone for an identity never pushed is a no-op that still answers marked_missing; the missing Authorization header; the 64 KiB limit over HTTP (still 422 with the message, not a 413); the 200-item limit; an n entity not in e; a non-null n quote; and a subject that differs only by case and whitespace, which must pass.

The route also carries DefaultBodyLimit of four payloads now, so the 2 MiB default is no longer buffered ahead of the token check.

Local: clippy clean, the five statements tests and the extraction_open unit tests pass on a fresh database.

@WaylandYang

Copy link
Copy Markdown
Contributor Author

Re-reviewed a0b3875. All three points are addressed: the pushed branch locates time words in the chunk itself (extraction_open.rs:680) instead of through locate_time, and the DB test now asserts the 08:14:03 mention lands; the door resolves s[i][1] and n[i][0] against e with the same name_key folding and answers 422 naming the slot; the new tests cover tombstone and revival, an unknown-identity tombstone, the missing Authorization header, the 64 KiB limit over HTTP, the 200-item limit, an n entity not in e, a non-null n quote and a folded subject that must pass, and the route carries a DefaultBodyLimit. Approved on my side once CI is green; merging waits for Wayland's explicit go, as asked.

WaylandYang and others added 3 commits September 24, 2026 23:10
…i source's

Found by running the server end to end: the token came back once at
creation, then GET /token and POST /rotate-token answered 404 because
both still gated on kind == "api". One predicate, has_push_token, now
names the two kinds that carry a push token, and creation, view and
rotation use it. A test signs in as the editor, reads the token back,
rotates it, and checks that the old token is refused and the new one
pushes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
@WaylandYang

Copy link
Copy Markdown
Contributor Author

@WaylandYang
WaylandYang merged commit 22c21ea into dev Sep 24, 2026
@WaylandYang
WaylandYang deleted the feat/statements-source branch September 24, 2026 16:22
WaylandYang added a commit that referenced this pull request Sep 25, 2026
…o end (#928)

A reader could learn the philosophy and the feature list and still not
answer which problem they would bring (#890). Both READMEs gain a Use
cases section ahead of Features: four scenarios in the shape of the
question someone brings, each with its input, what comes back, what the
base adds over search, and its status, drawn from the corpora and
guides already in the repository.

docs/walkthrough.md runs the observations scenario from sample data to
answer with evidence in curl: a class and a functional attribute, a
Statements source, two dated pushes, the phrase meeting the ontology,
the timeline that closes the earlier value, and what the base does not
do. It is the run that #884 and #911 were verified with.

Closes #890

Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant