LDE – Linked Data Elements

Coherent, composable building blocks for your Linked Data apps and pipelines.

📖 Documentation – guide, package reference and design decisions.

Every organization working with Linked Data ends up building the same infrastructure from scratch: endpoint management, data import, transformation pipelines, dataset discovery.

LDE covers the full Linked Data lifecycle – from discovery and ingestion through transformation to publication – as an open-source toolkit of composable building blocks for Node.js.

Data transformations are expressed as plain SPARQL queries: portable, transparent and free of vendor lock-in.

Key capabilities

• Discover datasets from DCAT-AP 3.0 registries.
• Download and import data dumps to a local SPARQL endpoint for querying.
• Transform datasets with pure SPARQL CONSTRUCT queries: composable stages with fan-out item selection.
• Analyze datasets with VoID statistics and SPARQL monitoring.
• Publish results to SPARQL endpoints or local files.
• Serve RDF data over HTTP with content negotiation (Fastify plugin).

Standards

Standard	Usage
DCAT-AP 3.0 (EU)	Dataset discovery and registry queries
SPARQL 1.1	Data transformations, dataset queries and endpoint management
SHACL	Per-class sampling (@lde/pipeline-shacl-sampler), validation (@lde/pipeline-shacl-validator) and documentation generation (@lde/docgen)
VoID	Statistical analysis of RDF datasets (@lde/pipeline-void)
RDF/JS	Internal data model (N3)
LDES (EU)	Event stream consumption and publication (planned)

Quick example

import {
  Pipeline,
  Stage,
  SparqlConstructReader,
  SparqlItemSelector,
  SparqlUpdateWriter,
  ManualDatasetSelection,
} from '@lde/pipeline';

const pipeline = new Pipeline({
  datasetSelector: new ManualDatasetSelection([dataset]),
  stages: [
    new Stage({
      name: 'per-class',
      itemSelector: new SparqlItemSelector({
        query: 'SELECT DISTINCT ?class WHERE { ?s a ?class }',
      }),
      readers: new SparqlConstructReader({
        query:
          'CONSTRUCT { ?class a <http://example.org/Class> } WHERE { ?s a ?class }',
      }),
    }),
  ],
  writers: new SparqlUpdateWriter({
    endpoint: new URL('http://localhost:7200/repositories/lde/statements'),
  }),
});

await pipeline.run();

Packages

Discovery – Find and retrieve dataset descriptions from registries
@lde/dataset	Core dataset and distribution objects
@lde/dataset-registry-client	Retrieve dataset descriptions from DCAT-AP 3.0 registries
Processing – Transform, enrich and analyse datasets with SPARQL pipelines
@lde/pipeline	Build pipelines that query, transform and enrich Linked Data
@lde/pipeline-shacl-sampler	Per-class sampling stages derived from SHACL shapes
@lde/pipeline-shacl-validator	SHACL validation for pipeline stages
@lde/pipeline-void	VoID statistical analysis for RDF datasets
@lde/distribution-downloader	Download distributions for local processing
@lde/distribution-health	Derive distribution usability from reachability and RDF validity
@lde/distribution-probe	Probe distributions for availability and metadata
@lde/iiif-validator	Validate that a URL resolves to a valid IIIF Presentation Manifest
@lde/resolver	Resolve external references with a durable store, bounds, retry and isolation
@lde/sparql-importer	Import data dumps to a local SPARQL endpoint for querying
@lde/sparql-anything	Convert tabular and other non-RDF sources to RDF with the SPARQL Anything CLI
Publication – Serve and document your data
@lde/fastify-rdf	Fastify plugin for RDF content negotiation and request body parsing
@lde/docgen	Generate documentation from RDF such as SHACL shapes
@lde/search	The search core: projects RDF into engine-agnostic search documents (framing + a declarative field spec)
@lde/search-api-graphql	Engine- and domain-agnostic GraphQL surface for search: builds an executable GraphQL schema from a SearchSchema at runtime and serves it as a framework-agnostic fetch handler with a self-contained playground
@lde/search-api-server	The served search API as a bootable process and prebuilt Docker image: mounts a schema-declaration module, binds the GraphQL handler to a Typesense engine, and serves /graphql plus /health from environment config
@lde/search-indexer	The search indexer as a bootable process and prebuilt Docker image: mounts the same schema-declaration module as the API server, selects datasets from a registry, and rebuilds the Typesense collections from environment config
@lde/search-pipeline	Applies the @lde/search projection inside an @lde/pipeline run, fanning each document out to the transactional engine writer for its type’s collection (owns no projection itself)
@lde/search-typesense	Typesense engine adapter: transactional Blue/green (build fresh, swap atomically) and In-place (update the live index) rebuild writers
@lde/text-normalization	Text folding (diacritic stripping and transliteration) for search index and query normalization
Monitoring – Observe pipeline runs and endpoint health
@lde/distribution-monitor	Monitor DCAT distributions (SPARQL endpoints and data dumps) with periodic probes
@lde/pipeline-console-reporter	Console progress reporter for pipelines
Infrastructure – Manage SPARQL servers and run tasks
@lde/host-limiter	Concurrent map bounded by a global and a per-host cap
@lde/local-sparql-endpoint	Quickly start a local SPARQL endpoint for testing and development
@lde/sparql-server	Start, stop and control SPARQL servers
@lde/sparql-qlever	QLever SPARQL adapter for importing and serving data
@lde/wait-for-sparql	Wait for a SPARQL endpoint to become available
@lde/task-runner	Task runner core classes and interfaces
@lde/task-runner-docker	Run tasks in Docker containers
@lde/task-runner-native	Run tasks natively on the host system
@lde/task-runner-http	Run tasks in another container of the pod, such as a sidecar

Architecture

See the architecture documentation for how the packages fit together, including the full dependency diagram.

Who uses LD Elements

 Netwerk Digitaal Erfgoed – Dutch national digital heritage infrastructure, commissioned by the Ministry of Education, Culture and Science

 DC4EU – Dutch Collections for Europe, the accredited national aggregator connecting Dutch heritage institutions with Europeana

Linked Open Limburg – Coöperatie Erfgoed Limburg’s regional heritage programme, built on LDE’s pipeline components

Comparison

See when to use LDE for how LDE compares to TriplyETL, rdf-connect and OpenLDES/LDI.

Development

Prerequisites: Node.js (LTS) and npm.

npm install
npx nx run-many -t build
npx nx run-many -t test
npx nx affected -t lint test typecheck build  # only changed packages

The documentation site lives in docs/; serve it locally with npm run docs:dev.

See CONTRIBUTING.md for the full development workflow.

License

MIT – see LICENSE.

Acknowledgements

LD Elements originated at the Dutch national infrastructure for digital heritage (NDE).