You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Reusing PreparedStatement Causes Memory Leak Due to State Accumulation #21089
Repeatedly reusing the same PreparedStatement causes continuous memory growth due to accumulated PhysicalResultCollector instances in the physical_plan->ops vector. Each execution creates a new collector but never cleans up previous ones, leading to memory leaks in long-running applications. Root cause:
PhysicalResultCollector::GetResultCollector (src/execution/operator/helper/physical_result_collector.cpp:30-47) calls physical_plan.Make<T>() on every execution
PhysicalPlan::Make (src/include/duckdb/execution/physical_plan_generator.hpp:41) always appends to ops vector via ops.push_back(*ptr)
These collectors are never removed until the PreparedStatement is destroyed
After N executions: physical_plan->ops = [query_operators..., Collector₁, Collector₂, ..., Collector_N]
We are using version 1.4 and have considered potential fixes, but have not been able to come up with a good solution.
Our use case involves using a Table Function as the source to perform the same computation pattern on different data. In order to maximize performance, we chose to reuse PreparedStatements.
This issue was introduced in v1.4. In previous versions, PhysicalResultCollector::GetResultCollector returned a unique_ptr, which did not have the problem of operators growing indefinitely.
To Reproduce
#include"duckdb.hpp"
#include<iostream>intmain() {
duckdb::DuckDB db(nullptr);
duckdb::Connection conn(db);
// Prepare a statementauto stmt = conn.Prepare("SELECT 1+1");
// Reuse the PreparedStatement 10000 timesfor (int i = 0; i < 10000; i++) {
auto result = stmt->Execute(i);
result->Fetch();
if (i % 1000 == 0) {
std::cout << "Execution count: " << i << std::endl;
// Memory keeps growing here, physical_plan.ops is growing
}
}
return0;
}
OS:
x86_64
DuckDB Version:
v1.4
DuckDB Client:
C++
Hardware:
No response
Full Name:
Zhenghao Wang
Affiliation:
Alibaba
Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?
Yes, I have
Did you include all code required to reproduce the issue?
Yes, I have
Did you include all relevant data sets for reproducing the issue?
Not applicable - the reproduction does not require a data set
Activity
Fix memory leak when reusing PreparedStatement (duckdb#21089)
Fix memory leak when reusing PreparedStatement (duckdb#21089)
Fix memory leak when reusing PreparedStatement (duckdb#21089)
Fix memory leak when reusing PreparedStatement (duckdb#21089)
Fix memory leak when reusing PreparedStatement (duckdb#21089)
Fix memory leak when reusing PreparedStatement (duckdb#21089)
Fix memory leak when reusing PreparedStatement (#21089) (#21104)
vendor: Update vendored sources to duckdb/duckdb@4c959e2
vendor: Update vendored sources to duckdb/duckdb@4c959e2 (#2285)
vendor: Update vendored sources to duckdb/duckdb@4c959e2 (#2285)
vendor: Update vendored sources to duckdb/duckdb@4c959e2 (#2285)
vendor: Update vendored sources to duckdb/duckdb@4c959e2 (#2285)
vendor: Update vendored sources to duckdb/duckdb@4c959e2
vendor: Update vendored sources to duckdb/duckdb@4c959e2
vendor: Update vendored sources to duckdb/duckdb@4c959e2
vendor: Update vendored sources to duckdb/duckdb@4c959e2
databases/{duckdb,py-duckdb}: Update to 1.5.3
databases/{duckdb,py-duckdb}: Update to 1.5.3