You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
apply_rls: one subscription whose RLS evaluation raises silently kills postgres_changes delivery for ALL subscribers #2093
One subscriber whose stored claims make RLS evaluation raise causes postgres_changes delivery to stop for every subscriber on the project — silently. We hit this in production on 2026-08-13 and lost all realtime delivery for ~2.5 hours, platform-wide.
Environment
Hosted Supabase, eu-north-1 (project ref available via support ticket on request)
For ~2 hours (coinciding with a platform-side restart of our project's Auth + Realtime at ~05:04 UTC), GoTrue-issued access tokens carried the string"null" instead of JSON null in nullable app_metadata claims (e.g. app_id). Our access-token hook stamps proper JSON nulls and was byte-identical on our dev project, which was unaffected — the mangling was platform-side and stopped on its own at ~07:19 UTC. That is a separate Auth issue; it matters here only as the trigger.
Our RLS policies call a helper that does NULLIF(auth.jwt() -> 'app_metadata' ->> 'app_id', '')::uuid. With the claim being the string "null", this raises 22P02 invalid input syntax for type uuid: "null".
realtime.apply_rls() evaluates policies per subscription using the claims stored at subscribe time. When evaluation for one subscription raises, the error aborts processing of the WAL batch — and no subscriber receives anything, including service_role subscriptions and subscriptions on unrelated tables.
Observed symptoms
WebSocket connects (101), channels reach SUBSCRIBED, rows appear in realtime.subscription with correct claims/filters — everything looks healthy.
The replication slot stays active with confirmed_flush_lsn advancing to pg_current_wal_lsn() — WAL is consumed and acknowledged.
Zero postgres_changes frames are delivered to any subscriber (verified with a service_role client subscribed with and without filters, on multiple tables).
Reproduced directly: calling realtime.apply_rls() with a wal2json-shaped record while the poisoned subscriptions were present raises 22P02 with context SQL statement "execute walrus_rls_stmt"; after hardening our helper to not raise, the same call returns records and delivery resumed instantly for all subscribers.
Expected behavior
An error while evaluating RLS for one subscription should disqualify that subscription (arguably with an error surfaced to that channel), not abort delivery for every subscriber on the project. JWT claims are external input from the subscriber's perspective of the pipeline — any upstream serializer defect (as happened here, in the platform's own Auth layer) becomes a total, silent, project-wide realtime outage that no health signal exposes: sockets up, subscriptions registered, slot advancing, nothing delivered.
Workaround for anyone else hitting this
Make every function reachable from an RLS policy raise-proof against malformed claims — validate instead of casting:
CREATE OR REPLACEFUNCTIONpublic.claim_uuid(p_claim text)
RETURNS uuid LANGUAGE sql IMMUTABLE SET search_path =''AS $$
SELECT CASE
WHEN p_claim ~*'^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
THEN p_claim::uuid
END;
$$;
Diagnosis query that found our poisoned subscriptions, in case it helps others:
SELECT jsonb_typeof(claims->'app_metadata'->'app_id'), count(*)
FROMrealtime.subscriptionGROUP BY1;
-- 'string' where you expect 'null' = poisoned tokens
Happy to provide timestamps, the project ref, and full reproduction detail through a support ticket — please reference this issue.
I went looking at apply_rls after reading this and managed to reproduce it locally, so here is what I found in case it is useful.
The per-subscription check runs execute 'execute walrus_rls_stmt' with nothing around it, so when a policy raises the exception leaves apply_rls and takes the whole list_changes call with it. Reproduced on supabase/postgres:17.6.1.127 with three subscriptions on one table: two anon (one with app_metadata.app_id set to the string "null", one without the claim at all) and one authenticated whose policy is just using (true).
ERROR: invalid input syntax for type uuid: "null"
CONTEXT: SQL statement "execute walrus_rls_stmt"
PL/pgSQL function realtime.apply_rls(jsonb,integer) line 219 at EXECUTE
SQL function "list_changes" statement 1
The authenticated subscription cannot fail its own policy and still receives nothing, which lines up with what @Raundahl described.
One detail worth adding to the report: the batch is not retried either. confirmed_flush_lsn moves to pg_current_wal_lsn() even though the call errored, and the next poll comes back empty:
### poll 1
ERROR: invalid input syntax for type uuid: "null"
slot_name | confirmed_flush_lsn | pg_current_wal_lsn
-------------+---------------------+--------------------
verify_slot | 0/2EEB228 | 0/2EEB228
### poll 2 - is the record still there?
subscription_ids | details
------------------+---------
|
So while a subscription with unevaluable claims exists, every decoded change is dropped rather than replayed, and nothing in the slot or connection state shows it.
I have a patch for this and will open a PR against it: the per-subscription check goes in its own block, the subscription that raised is disqualified on its own and gets an Error 500 row carrying only schema/table/action (the same shape as the existing 400/401 rows, so a policy that could not be evaluated does not end up disclosing the row it was meant to gate), and everyone else is unaffected. The extra subtransaction costs about 3% with 100 subscriptions on a single record, measured both ways on the same box.
Activity
edgurgel commentedon Aug 13, 2026
Hi yes please let's open a support ticket!
edgurgel commentedon Aug 13, 2026
Thanks for the report and I think you are right. We should not give up processing other subscriptions if that's what is happening.
hamodywe commentedon Aug 14, 2026
I went looking at
apply_rlsafter reading this and managed to reproduce it locally, so here is what I found in case it is useful.The per-subscription check runs
execute 'execute walrus_rls_stmt'with nothing around it, so when a policy raises the exception leavesapply_rlsand takes the wholelist_changescall with it. Reproduced onsupabase/postgres:17.6.1.127with three subscriptions on one table: twoanon(one withapp_metadata.app_idset to the string"null", one without the claim at all) and oneauthenticatedwhose policy is justusing (true).The
authenticatedsubscription cannot fail its own policy and still receives nothing, which lines up with what @Raundahl described.One detail worth adding to the report: the batch is not retried either.
confirmed_flush_lsnmoves topg_current_wal_lsn()even though the call errored, and the next poll comes back empty:So while a subscription with unevaluable claims exists, every decoded change is dropped rather than replayed, and nothing in the slot or connection state shows it.
I have a patch for this and will open a PR against it: the per-subscription check goes in its own block, the subscription that raised is disqualified on its own and gets an
Error 500row carrying only schema/table/action (the same shape as the existing 400/401 rows, so a policy that could not be evaluated does not end up disclosing the row it was meant to gate), and everyone else is unaffected. The extra subtransaction costs about 3% with 100 subscriptions on a single record, measured both ways on the same box.