supabase db reset intermittently fails: temporary Realtime init job races with the running Realtime service on "schema_migrations_pkey" · Issue #6915 · supabase/cli · GitHub
Skip to content

supabase db reset intermittently fails: temporary Realtime init job races with the running Realtime service on "schema_migrations_pkey" #6915

Open

Description

@rjuanluis

CLI version: 2.118.0 (Linux, Docker Engine, self-hosted GitHub Actions runner), Postgres 17 local stack.

Describe the bug

supabase db reset (local) fails intermittently at the "Initialising schema" stage with error running container: exit 1. It happens mostly on the second reset of a CI job, when the stack has been running for a while. With --debug, the output of the temporary Realtime init container shows:

Resetting local database...
Recreating database...
Initialising schema...
+ echo 'Running migrations'
+ sudo -E -u nobody /app/bin/migrate
** (Ecto.ConstraintError) constraint error when attempting to insert struct:

    * "schema_migrations_pkey" (unique_constraint)
...
    (ecto_sql 3.14.0) lib/ecto/migrator.ex:337: anonymous fn/6 in Ecto.Migrator.async_migrate_maybe_in_transaction/7
error running container: exit 1

Why it happens (reading internal/db/reset/reset.go and internal/db/start/start.go at v2.118.0)

resetDatabase15:

  1. removes and recreates the db container and volume;
  2. runs SetupLocalDatabase → initSchema15, which runs one-off containers for Realtime (/app/bin/migrate, seeds, health_check), Storage (migrate-call.js) and GoTrue (gotrue migrate);
  3. only then calls restartServices (storage, gotrue, realtime, pooler).

During step 2, the persistent supabase_realtime_<project> container is still running against the freshly recreated database. When it restarts itself (it loses its DB connection while the db container is recreated), its entrypoint runs the same Realtime migrations concurrently with the one-off init job. Both insert the same version into _realtime.schema_migrations, and the one-off job crashes on schema_migrations_pkey.

Workaround

Stop the persistent Realtime container before supabase db reset (docker stop supabase_realtime_<project>). restartServices starts it again at the end.

Suggested fix

In resetDatabase15, stop the service containers that run their own migrations (at least Realtime; probably Storage and GoTrue too) before recreating the database, and start them again in restartServices. Alternatively, run the one-off init jobs only while those services are stopped.

To reproduce

  1. supabase start with Realtime enabled (default config).
  2. Run supabase db reset --debug and, as soon as it prints Initialising schema..., run docker restart supabase_realtime_<project>. This simulates the service restarting on its own after losing the database.
  3. The reset fails with the error above. The same sequence with the Realtime container stopped beforehand succeeds.

Observed locally (CLI 2.118.0, 2026-09-30):

  • Realtime running and restarted at Initialising schema...: 1 of 3 resets failed with the exact error above; the one-off Realtime container was the only one that exited 1.
  • Realtime stopped before the reset: 3 of 3 succeeded, and restartServices left it running.
  • Plain resets (control): 2 of 2 succeeded.

Activity

  1. self-assigned this
    on Sep 30, 2026
  2. RobertKoval commented on Oct 26, 2026

    @RobertKoval

    We see the same race on CLI 2.112.0 (Realtime image v2.123.5, Postgres 17.6.1.158, self-hosted runners). On loaded CI hosts it hit 4 of about 30 jobs in one run. Some details that may help with the fix:

    • Why the persistent container comes back. supabase_realtime_<project> is created with restartPolicy: "unless-stopped". When resetDatabase15 removes the db container, Realtime exits 1 and Docker restarts it about 0.3 s later. Its entrypoint then runs /app/bin/migrate and, because SEED_SELF_HOST=true, Realtime.Release.seeds against the new database. Meanwhile initSchema15 runs the one-off Realtime job.
    • Docker events from one reset, in seconds after the reset started:
      +2.8  die     supabase_db         exit=137
      +3.6  start   supabase_db
      +4.8  die     supabase_realtime   exit=1
      +5.1  start   supabase_realtime   (restart policy)
      +14.8 create  <one-off realtime>
      +20.7 die     <one-off realtime>  exit=0
      +28.0 stop/start supabase_realtime (restartServices)
      
    • A second failure mode besides schema_migrations_pkey: Ecto.InvalidChangesetError ... external_id: has already been taken (tenants_external_id_index) in Realtime.Release.seeds, often with relation "realtime.schema_migrations" does not exist in the Postgres log.
    • The workaround holds. docker stop supabase_realtime_<project> before the reset has kept our runs clean: a container stopped this way is not brought back by unless-stopped, and restartServices starts it again. Stopping with ContainerStop before removing the db, for every service that restartServices restarts afterwards, would close this race for storage and auth as well (compare db reset: intermittent "error running container: exit 1" at "Initialising schema" (one-off migration jobs race the running storage service) #6445).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions