Clarify or expose a stable structured read contract for facts, evidence, proofs, history, and conflicts · Issue #550 · deeplethe/utopia · GitHub
Skip to content

Clarify or expose a stable structured read contract for facts, evidence, proofs, history, and conflicts #550

Closed

Description

@ansonnmm

Summary

Utopia already stores and exposes a rich epistemic chain internally and through several structured HTTP endpoints:

entity
→ asserted fact
→ evidence / quote
→ source document
→ validity
→ history / correction
→ derived fact
→ proof
→ conflict / review state

However, the documented MCP read surface currently returns much of this information as formatted human-readable text rather than identity-rich structured data.

I would like to clarify what Utopia considers its supported external machine-readable contract for this information.

Tested version

Tested against stock Utopia at:

b03a0f20a3d053323a3ec2669c7c9b4aeb9a8e14

No source, schema, or migration modifications were made.

What I verified

Using the existing HTTP and MCP interfaces, I was able to verify that Utopia has enough underlying information to reconstruct:

  • entity identity;
  • asserted fact identity and world-time validity;
  • evidence quote and chunk identity;
  • source document identity and version;
  • correction / history events;
  • derived fact identity;
  • ordered proof premises;
  • conflicts / review items.

Most of this information is already available through structured /api/v1/... HTTP routes.

The uncertainty is whether those routes are intended to be stable external integration contracts.

Current MCP limitation

The documented MCP interface is clearly intended for external clients, but tools such as entity_facts primarily return formatted text.

For example, the result can describe validity and identify a derived result with something similar to:

[rule: ...]

but the external caller does not receive a machine-composable structure containing identities such as:

fact_id
evidence_id
document_id
derived_fact_id
proof / premise ids
conflict identity / status

Similarly, change/history output contains useful human-readable information, but does not expose enough stable identities to reliably join it back to the fact/evidence/proof chain.

This makes the MCP surface useful for an interactive agent, but difficult to use as a durable machine-to-machine knowledge interface without parsing formatted prose.

Questions for maintainers

Could you clarify which of the following is the intended direction?

1. Existing structured /api/v1 routes are public external contracts

If so:

  • Are integrations expected to depend on the entity, fact, evidence, history, proof, graph, and review routes?
  • Is there a compatibility or deprecation policy for those response shapes?
  • Are opaque entity/fact/evidence/derived IDs intended to remain usable across versions?

2. MCP is the intended public integration contract

If so, would you consider exposing structured content alongside the human-readable text, including stable opaque identities and lineage relationships?

3. Another external API layer is planned

If neither of the above is intended as the durable contract, is there a planned structured read API for this use case?

Minimal information that seems sufficient

A useful structured external contract would not need to expose database rows or internal implementation details.

Opaque IDs plus typed relationships would be sufficient. For example:

EntityRef {
  id
  name
  type
}

AssertedFact {
  id
  subject
  predicate
  object / value
  valid_from
  valid_to
}

EvidenceRef {
  id
  fact_id
  quote
  chunk_id
  document_id
}

DerivedFact {
  id
  subject
  predicate
  object / value
  rule
}

Proof {
  derived_fact_id
  premises[]
}

HistoryEvent {
  id
  timestamp
  entity_id
  fact_id?
  event_type
}

Conflict {
  id
  fact_ids[]
  kind
  status
}

The exact schema above is only illustrative.

The important properties are:

  • stable opaque identity;
  • machine-composable lineage;
  • asserted and derived facts remain distinguishable;
  • evidence and source provenance remain recoverable;
  • history/corrections can be joined back to the affected facts;
  • conflicts can be addressed as structured objects rather than reconstructed from text.

Why this may be useful beyond one integration

One of Utopia's differentiating properties is that it preserves more than the current graph state.

It also preserves:

  • evidence;
  • provenance;
  • temporal validity;
  • changing belief/history;
  • derivations and proof chains;
  • conflicts;
  • auditability.

A structured external read contract would allow downstream agents and applications to preserve those guarantees instead of flattening the result into text and then attempting to reconstruct structure from prose.

It would also provide a clean boundary between Utopia's internal schema—which can continue to evolve—and external integrations that should not depend directly on PostgreSQL tables, migrations, or Rust implementation details.

Related documentation

Current MCP documentation at the tested commit:

Request

The immediate request is primarily clarification:

What external structured read surface, if any, should integrations treat as a supported compatibility boundary?

If MCP is intended to be that boundary, structured identity-rich responses for the epistemic chain above would close the current gap.

If the existing structured HTTP routes are already intended to be that boundary, documenting their compatibility expectations would also resolve the uncertainty.

Activity

  1. WaylandYang commented on Sep 9, 2026

    @WaylandYang
    Contributor

    Thanks for laying this out so carefully. The short answer to your question: none of the three surfaces is documented as a compatibility boundary today, and that is the gap. Here is what each one is meant to be, and what changes.

    The RDF export is the supported machine-readable read contract for the chain you describe. GET /api/v1/kbs/{id}/export?format=turtle|jsonld streams the whole base (decision record 0020). Every entity and fact has an addressable IRI, minted under urn:utopia:kb:{kb_id}:... from the row's uuid, so two exports a year apart line up; ?base= overrides the namespace for a deployment with a public address. A fact is an rdf:Statement carrying world time (schema:validFrom / validThrough with precision), record time (prov:generatedAtTime / prov:invalidatedAtTime), confidence, supersedes for a correction, the quoted sentence, and prov:wasDerivedFrom the documents it came from. A derived fact is a statement flagged derived, prov:wasGeneratedBy its rule, with prov:used on each premise, asserted or derived. Classes and relations keep the IRI they were imported with. That file is the boundary between our schema and an integration: the tables and the Rust types can move, the export cannot without a record saying why. It is written for a triple store, and it deliberately uses vocabularies a reader already has (PROV-O, schema.org, RDF reification) instead of a private JSON. We will say this in the docs, next to the MCP page you linked, so the question does not have to be asked again.

    Three things on your list are missing from the export, and each becomes its own issue: conflicts and review state, the chunk identity behind a quote (today the quote and the document are there, the chunk is not), and fetching one entity's chain on demand (the export is whole-base; a SPARQL endpoint is the planned answer and is not built).

    MCP stays the agent surface, and structured content beside the text is the right addition. The prose is deliberate: it is what a model reads well, and entity_facts is tuned for that. But the protocol version we speak (2025-06-18) defines structuredContent, and the read tools could return it alongside content: entity ids from find_entities; fact, derived and document ids with both clocks from entity_facts; document and chunk ids from search_chunks; fact ids from changes. Same opaque uuids as the export, so a client can join the two. It is a bounded change and we would take a PR for it: keep the text as it is, add the structure next to it, and cover each tool with a test that reads the ids back.

    /api/v1 is the UI's API and is not a compatibility promise. Its shapes change with the interface, there is no OpenAPI description and no deprecation policy, and we would rather not pretend otherwise. The ids inside those responses are the same uuids the export and MCP use, so an id you keep stays usable; the response shapes around it do not.

    On the derived-facts point you raised in #549: fixed in #557, and the record now says what a rewound proof can and cannot show.

  2. ansonnmm commented on Sep 9, 2026

    @ansonnmm
    ContributorAuthor

    Thanks — this clarifies the integration boundary cleanly.

    I’ll treat:

    • RDF export as the supported machine-readable read contract;
    • MCP as the agent-facing surface;
    • /api/v1 as UI-internal rather than a compatibility boundary.

    I’ve opened #564 for the conflict/review-state gap you identified. I’ll leave chunk identity, MCP structuredContent, and per-entity/SPARQL access separate for now rather than expanding this thread.

    Thanks again for the detailed clarification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions