Intro

I noticed this problem at the end of last year already, but forgot to report it at the end. So here it is.
I detected increasing RAM consumption of my app after I trigger dozens of thousands of workflows. I made a workaround in my project at that time, but I must run my services for months/years without interruptions, so I would be interested in fixing this permanently.

My workaround: a temporary hosted service reflects into WorkflowHost._backgroundTasks and, under _runningTasksLock, call List<Task>.RemoveAll(t => t.IsCompleted). That bounds the leak without changing execution. Keeps the RAM stable, but it not sustainable.

I used Grok to summarize the topic for me, the response is below. It did a bit of research and I think it is quite accurate.

Describe the bug

QueueConsumer keeps every ExecuteItem task in _runningTasks until the host stops. The list exists so shutdown can await tasks that are still running, but completed tasks are never removed. Each WorkflowConsumer, EventConsumer, and IndexConsumer therefore grows for the life of the process.

MaxConcurrentItems bounds in-flight work, not this list. Its size is the number of items processed since Start(). Each completed Task keeps the ExecuteItem state machine alive, and that state machine keeps the per-item ManualResetEvent alive. Those wait handles are never disposed, so this retains both managed memory and OS handles.

To Reproduce

1. Run a host on 3.17.0 or later (master still has this code).
2. Complete a "large" number of workflows.
3. Inspect QueueConsumer._runningTasks on WorkflowConsumer, EventConsumer, and IndexConsumer.
4. The count stays equal to the number of items processed since startup, not the number in flight. After the items have finished, the completed Task objects and their ManualResetEvent handles are still referenced.

The relevant code in src/WorkflowCore/Services/BackgroundTasks/QueueConsumer.cs is:

var task = ExecuteItem(item, waitHandle, activity);
lock (_runningTasksLock)
{
    _runningTasks.Add(task);
}

ExecuteItem removes the item from _activeTasks in its finally. It never touches _runningTasks. The list is read only after the dispatch loop exits:

lock (_runningTasksLock)
{
    tasksToAwait = _runningTasks.ToArray();
}
await Task.WhenAll(tasksToAwait);

Expected behavior

_runningTasks should hold only tasks that are still running. Once ExecuteItem has completed, that task should be removed. Shutdown should still await any task that has not finished. Task.WhenAll on an already completed task returns immediately, so removing finished tasks does not weaken that wait.

Additional context

#645 reported that StopAsync returned before a running async step finished. PR #1401, merged 8 October 2025 and shipped in 3.17.0 on 11 October 2025, added _runningTasks so shutdown could await Task.WhenAll those tasks. The PR says the existing ManualResetEvent path already synchronized completion, and that the list made that wait explicit. The list has no removal path.

The shutdown behavior added there should be preserved. The retained completed tasks are not part of that behavior. #1402, opened after 3.17.0, reports that host exit can still finish before in-step work stops gracefully. This leak should not be treated as a necessary part of the #645 fix, and fixing it should not turn into another shutdown rewrite.

Out of scope: cancellation semantics, #1402, greylist handling, and the persistence-provider leaks.

This report was drafted with Grok 4.7 agent assistance and checked against QueueConsumer on master and PR #1401.