Interleave reproduces race conditions in PostgreSQL application code. It records the order of SQL commands that broke your data and keeps that order as a regression test.
Two requests read the same row, both write, and one update disappears. It shows up in production and never in your test suite, because the tests never hit the one order that breaks it. Interleave runs your actual operations against a real PostgreSQL server and decides the order in which their SQL commands reach it, so you can produce that order deliberately and repeat it.
Take an ordinary read-then-write operation:
// counter.mjs — your application code, unchanged
export async function incrementCounter(client, id) {
const { rows } = await client.query('SELECT value FROM counters WHERE id = $1', [id]);
const value = rows[0].value + 1;
await client.query('UPDATE counters SET value = $1 WHERE id = $2', [value, id]);
return value;
}A scenario names the concurrent operations (actors), sets up the database and states the rule that must hold. Each actor gets its own connection string and calls your code:
// scenario.mjs
import assert from 'node:assert/strict';
import { Client } from 'pg';
import { defineScenario } from '@pavangupta352/interleave';
import { incrementCounter } from './counter.mjs';
async function increment({ connectionString }) {
const client = new Client({ connectionString });
client.on('error', () => {}); // a dropped connection must not end the run
await client.connect();
try { return { value: await incrementCounter(client, 1) }; }
finally { await client.end(); }
}
export default defineScenario({
name: 'application-counter',
async setup({ db }) {
await db.query('CREATE TABLE counters (id int PRIMARY KEY, value int NOT NULL)');
await db.query('INSERT INTO counters VALUES (1, 0)');
},
actors: { alice: increment, bob: increment },
async invariant({ db }) {
const { rows } = await db.query('SELECT value FROM counters WHERE id = 1');
assert.equal(rows[0].value, 2, 'Both increments must be retained');
},
});Then let Interleave find the order that breaks it, replay it and shrink it. This
output comes from a real run, shortened where the CLI repeats the search's scope
note, each run's summary and the assertion's detail (1 !== 2):
$ npx interleave run scenario.mjs --docker --out failure.json
application-counter: 1 attempted, 1 completed; violations: 1; failure.
Search: fifo; pending prefixes: 0; maximum attempted depth: 0.
Recorded release units: 4; actor switches: 3.
$ npx interleave replay scenario.mjs failure.json --docker
application-counter: violation (replay); 4 commands; cleanup complete.
Both increments must be retained
$ npx interleave minimize scenario.mjs failure.json --docker --out minimal.json
Reduced 4 choices to 0 in 4 attempts; locally-minimal.
$ npx interleave report failure.json --out failure.html
Wrote offline evidence report: failure.htmlExit codes are meant for CI: 0 checked success, 1 invariant violation, 2
usage, actor or harness error, 3 incompatible replay, 4 inconclusive or
budget exhausted, and 128 plus the signal number after an interrupted run cleans
up (130 for Ctrl+C). --docker starts a
throwaway PostgreSQL container and removes it afterwards; you can point Interleave
at a dedicated test server instead.
alice ──► proxy A ──┐ ┌──► real PostgreSQL
├──► scheduler: releases one ──────┤ (executes, locks,
bob ──► proxy B ──┘ command at a time, observes │ reports waits)
completions and lock waits └──► invariant check
- Each actor connects to its own local proxy. The proxy forwards the original protocol bytes and never rewrites or re-executes SQL.
- The scheduler holds each command until it chooses to release it, then waits
for PostgreSQL to finish it or for a lock wait that PostgreSQL itself reports.
With
--overlap pairs, a search can also release two commands together and let PostgreSQL interleave them, which reaches races inside one statement. - A bounded search tries different release orders. When the invariant fails, the order, SQL, results and waits are saved as a JSON artifact.
- The artifact records what makes the run repeatable: your source files, installed dependencies, Node.js and PostgreSQL versions, the starting database and the connection settings. If any of them changed, exact replay reports the run as incompatible instead of executing it. After you fix the code, a guided rerun or a fresh search checks the change as new evidence.
| Command | What it does |
|---|---|
run |
Explores command orders and saves the first failing one |
replay |
Reruns a saved failure exactly, or --guided against changed code |
minimize |
Removes ordering choices while keeping the same failure |
report |
Writes a standalone HTML evidence viewer that works offline |
export |
Packages the failing source, lockfile and runtime so the failure replays elsewhere, offline |
doctor, demo |
Checks your setup; runs a built-in oversell example |
init |
Scaffolds a scenario in an existing project |
The same workflow is available from JavaScript: explore, runOnce,
runScenarioFile, replay, minimize, exportRegression and renderReport.
The compatibility matrix lists exact versions and the limits of each profile. A qualified workload does not establish support for every feature of its driver or ORM. The case studies measure Interleave against ordinary concurrency, manual barriers and PostgreSQL's isolation tester on three historical bugs in Knex, node-pg-migrate and Sequelize.
- A search that passes is evidence about the orders it explored, not proof that no race exists.
- Interleave releases one command at a time unless PostgreSQL reports a lock
wait, or you enable
--overlap pairs. Without it, races that need two statements executing at the same instant are missed; the case studies include one such miss, which--overlap pairsfinds. With it, PostgreSQL chooses how a released pair interleaves, so replaying such a failure can succeed only some of the time. - By default each actor holds one command-producing connection at a time; the
multi-producer-v1profile schedules several.COPY ... FROM STDINandCOPY ... TO STDOUTstream through as part of one command. Cursors and pipelines that wait on Flush need--protocol-profile flush-stages-v1; under the default profile they fail explicitly, as replication's COPY BOTH always does. Pipelined queries that end each cycle with Sync are scheduled one cycle at a time. - A cancel request reaches PostgreSQL only when it names a connection of the actor that sent it, and the record lists every request. The application's own timers decide when a request arrives, so replaying one can find a different command running.
- Clocks, randomness and external services are not controlled.
- On Windows,
exportis refused, and a run killed without a signal (such astaskkill /F) skips cleanup.
Interleave needs Node.js 22.18 or later on Linux, macOS or Windows, and Docker if
you use --docker.
Install it with node-postgres 8.23.0, the qualified driver version:
npm install --save-dev @pavangupta352/interleave pg@8.23.0
npx --no-install interleave doctor --docker--no-install runs the copy you just installed. Each
GitHub release also carries
the npm archive with SHA256SUMS; getting started shows
how to check and install that file, and covers the source checkout. The unscoped
interleave package on npm is unrelated.
Start with getting started, then read the concepts and the application guide. The CLI and API references cover every option; regression checks and CI shows how to keep a failure and check a repair; troubleshooting explains stopped runs. The examples pair a real race with its repair for each supported driver, ORM and language. Report problems through issues and vulnerabilities through the security policy.
PostgreSQL's isolation tester explores interleavings of hand-written SQL sessions. Interleave works on existing application operations and keeps replayable evidence. determined provides deterministic TypeScript simulation, and Antithesis controls a whole execution environment.
MIT © Pavan Gupta. Vendored example source and bundled dependencies keep their own license notices, including neveroversell and pghybrid.