Reusing PreparedStatement Causes Memory Leak Due to State Accumulation · Issue #21089 · duckdb/duckdb · GitHub
Skip to content

Reusing PreparedStatement Causes Memory Leak Due to State Accumulation #21089

Closed

Description

@wzharies

What happens?

Repeatedly reusing the same PreparedStatement causes continuous memory growth due to accumulated PhysicalResultCollector instances in the physical_plan->ops vector. Each execution creates a new collector but never cleans up previous ones, leading to memory leaks in long-running applications.
Root cause:

  • PhysicalResultCollector::GetResultCollector (src/execution/operator/helper/physical_result_collector.cpp:30-47) calls physical_plan.Make<T>() on every execution
  • PhysicalPlan::Make (src/include/duckdb/execution/physical_plan_generator.hpp:41) always appends to ops vector via ops.push_back(*ptr)
  • These collectors are never removed until the PreparedStatement is destroyed
    After N executions: physical_plan->ops = [query_operators..., Collector₁, Collector₂, ..., Collector_N]

We are using version 1.4 and have considered potential fixes, but have not been able to come up with a good solution.
Our use case involves using a Table Function as the source to perform the same computation pattern on different data. In order to maximize performance, we chose to reuse PreparedStatements.
This issue was introduced in v1.4. In previous versions, PhysicalResultCollector::GetResultCollector returned a unique_ptr, which did not have the problem of operators growing indefinitely.

To Reproduce

#include "duckdb.hpp"
#include <iostream>

int main() {
    duckdb::DuckDB db(nullptr);
    duckdb::Connection conn(db);

    // Prepare a statement
    auto stmt = conn.Prepare("SELECT 1+1");

    // Reuse the PreparedStatement 10000 times
    for (int i = 0; i < 10000; i++) {
        auto result = stmt->Execute(i);
        result->Fetch();

        if (i % 1000 == 0) {
            std::cout << "Execution count: " << i << std::endl;
            // Memory keeps growing here, physical_plan.ops is growing
        }
    }

    return 0;
}

OS:

x86_64

DuckDB Version:

v1.4

DuckDB Client:

C++

Hardware:

No response

Full Name:

Zhenghao Wang

Affiliation:

Alibaba

Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

  • Yes, I have

Did you include all code required to reproduce the issue?

  • Yes, I have

Did you include all relevant data sets for reproducing the issue?

Not applicable - the reproduction does not require a data set

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions