Memory grows linearly without release eventually hitting container ceiling and causing crash (CPU time soft limit reached) · Issue #740 · supabase/edge-runtime · GitHub
Skip to content

Memory grows linearly without release eventually hitting container ceiling and causing crash (CPU time soft limit reached) #740

Open

Description

@BlackdreamIO

Bug report

  • I confirm this is a bug with Supabase, not with my own application.
  • I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

Running supabase/edge-runtime as a Docker container, memory usage climbs
continuously and never releases — including across isolate terminations
that are logged by the runtime itself.

I isolated this down to a minimal reproduction : a handler that does nothing
but return a response, with no application
logic no database or Redis connections inside the handler and no timers
or listeners of my own. the function is invoked on a short interval via
cron (HTTP call scheduled from terminal) therefore each invocation is
a fresh independent HTTP request to the function not a chain of promises
or in process recursion.

Under this workload:

  • Memory climbs continuously in a linear pattern. from the moment the
    container starts.
  • The runtime periodically logs an isolate termination (CPU time soft limit reached / early termination has been triggered) followed by a new
    isolate starting for the next invocation.
  • Immediately after each such termination/restart, memory jumps up on
    the very next request rather than resetting to the pre-climb baseline,
    and then resumes the same linear climb from that new, higher floor.
  • This repeats indefinitely each restart's post-jump floor is higher than
    the last and total memory usage ratchets upward until it approaches the
    containers configured memory limit.

Because this reproduces with an inert handler. application level causes
(connection pooling, event listeners, timers, retained closures) are ruled
out. The behavior appears to originate in the edge-runtime worker/isolate
lifecycle itself specifically that terminating an isolate is not fully
releasing the memory associated with it back to the container before the
next isolate is created.

This may be related to the general class of issue described in #212,
though that report covers a different code path (supabase functions serve's dev-server wrapper) iamm not certain the underlying cause is the
same but the symptom category — worker/isolate memory not being reclaimed
looks similar.

To Reproduce

  1. Run the supabase/edge-runtime image locally via Docker with a memory
    limit set (i used 128 MB; also reproduces at higher limits. just
    takes proportionally longer).
  2. Deploy a minimal Edge Function whose handler:
    • Logs the current isolate ID and uptime.
    • Returns a simple JSON response.
  3. Trigger the function on a short interval (roughly once per second) via
    an external scheduled trigger (e.g. cron, or any
    external scheduler) so each invocation is an independent HTTP request.
  4. Watch container memory via docker stats or an equivalent dashboard
    (Portainer in my case) over 10–20 minutes of continuous invocation.
  5. Observe:
    • Memory starts low 16–25 MB and climbs roughly linearly.
    • The runtime logs isolate termination events (CPU time soft limit reached: isolate: <uuid>, early termination has been triggered: isolate: <uuid>) at intervals.
    • On the next request after each termination, memory jumps to a level
      higher than the pre termination baseline then resumes climbing
      linearly from there.
    • This continues until the container approaches its configured memory
      limit.

Expected behavior

When an isolate is terminated by the runtime all memory associated with
that isolate should be released back to the container so the memory
baseline after a termination/restart stays consistent across cycles rather
than compounding upward on each restart a function invoked repeatedly on
an interval should be able to run indefinitely at a stable memory
footprint. the same way it does when deployed to hosted Supabase i have run
equivalent repeatedly invoked functions on hosted Supabase for extended
periods without hitting memory limits, only the expected wall clock
function timeout.

Screenshots

Cpu usage stays consistent only spikes during new isolate spin ups, but the memory linearly accumulates as time goes on

Image Image Image Image

System information

  • OS: Windows
  • Docker version: 29.6.2, build dfc4efb
  • supabase/edge-runtime image tag/version: public.ecr.aws/supabase/edge-runtime:v1.68.4
  • Docker memory limit used: 128 MB (also reproduced at higher limits)
  • Docker CPU allocation used: 0.2 core
  • Version of Deno: 2.7.3

Additional context

  • Termination logs consistently show CPU time soft limit reached rather
    than an explicit OOM kill even though the memory graph shows usage
    approaching the containers memory ceiling immediately before
    termination. Unclear whether the CPU soft limit accounting is a
    downstream effect of memory pressure-GC churn or an independent cause.
  • Happy to provide the minimal reproduction handler/Dockerfile/compose
    config if that speeds up triage.consistent

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions