Speed up server shutdown for databases with many tables by jaymebrd · Pull Request #104969 · ClickHouse/ClickHouse · GitHub
Skip to content

Speed up server shutdown for databases with many tables#104969

Open
jaymebrd wants to merge 13 commits into
ClickHouse:masterfrom
jaymebrd:parallel-table-shutdown
Open

Speed up server shutdown for databases with many tables#104969
jaymebrd wants to merge 13 commits into
ClickHouse:masterfrom
jaymebrd:parallel-table-shutdown

Conversation

@jaymebrd

@jaymebrd jaymebrd commented May 14, 2026

Copy link
Copy Markdown
Member

Table shutdown runs sequentially per table, which can make server shutdown take many minutes to 1hr+ on clusters with many thousands of tables.

This PR parallelises DatabaseWithOwnTablesBase::shutdown:

  • src/Databases/DatabasesCommon.cpp: run flushAndPrepareForShutdown and flushAndShutdown across tables concurrently via a dedicated thread pool. Logs per-database elapsed time.

The speedup is best measured by SIGTERM-to-exit wall time on many-table clusters.

Changelog category (leave one):

  • Performance Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Speed up server shutdown for databases with many tables by running per-table shutdown in parallel. Controlled by the new database_catalog_shutdown_table_concurrency setting (default 16).

@clickhouse-gh

clickhouse-gh Bot commented May 14, 2026

Copy link
Copy Markdown
Contributor

@clickhouse-gh clickhouse-gh Bot added the pr-performance Pull request with some performance improvements label May 14, 2026
Comment thread src/Databases/DatabasesCommon.cpp Outdated
Comment thread src/Core/ServerSettings.cpp

@serxa serxa left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me.

Comment thread src/Databases/DatabasesCommon.cpp Outdated
@jaymebrd jaymebrd requested a review from serxa May 24, 2026 10:22
@serxa serxa enabled auto-merge June 2, 2026 11:09
@serxa

serxa commented Jun 4, 2026

Copy link
Copy Markdown
Member

@groeneai can you check what prevents merging and help?

Comment thread src/Core/ServerSettings.cpp
@clickhouse-gh

clickhouse-gh Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

📊 Cloud Performance Report

⚠️ AI verdict: not_sure2 query(s) regressed out of 38 analysed

This is a sync PR mirroring public #104969; both diffs are identical and only add a parallel table-shutdown thread pool that runs during server stop, never during query execution. Because the query hot path is untouched, none of the flagged ClickBench/TPC-H deltas — improvements or regressions — can be attributed to this change, so 8 of them are downgraded as off-path variance. Two TPC-H regressions (Q7 +800%, Q8 +30%) are large and consistent enough to trip the hard-trust override and are retained, but given the shutdown-only scope they most likely reflect environment/measurement noise and warrant a manual look rather than code review.

clickbench

⚠️ 6 inconclusive

Flagged queries (6 of 43)
Query Verdict Baseline median (ms) PR median (ms) Change q-value Hint
⚠️ 15 not_sure 249 218 -12.4% <0.0001 PR only changes server-shutdown teardown; query path untouched, so -12% is run-to-run variance.
⚠️ 24 not_sure 212 57 -73.1% <0.0001 Shutdown-only diff can't touch query execution; -73% is off-path variance (5 vs 30 runs).
⚠️ 26 not_sure 215 56 -74.0% <0.0001 Same as Q24: shutdown-only diff, -74% not attributable to this PR.
⚠️ 27 not_sure 201 157 -21.9% <0.0001 Shutdown-path change only; -22% is off-path variance.
⚠️ 30 not_sure 240 282 +17.7% <0.0001 Shutdown-only diff; +18% not caused by this PR, off-path variance.
⚠️ 31 not_sure 301 376 +24.8% <0.0001 Shutdown-only diff; +25% not caused by this PR, off-path variance.

q-value = BH-FDR adjusted p; smaller is stronger evidence. MIRAI flags a query when q < fdr_q (default 0.10) — the value the verdict is based on.

tpch_adapted_1_official

🔴 2 regressed · ⚠️ 2 inconclusive

Flagged queries (4 of 22)
Query Verdict Baseline median (ms) PR median (ms) Change q-value Hint
🔴 7 regression 71 640 +800.7% <0.0001 +800% is large and consistent but the diff only touches shutdown; retained by override, review environment.
🔴 8 regression 166 215 +29.5% <0.0001 +30% retained by hard override; diff touches shutdown teardown only, so verify against environment.
⚠️ 4 not_sure 2284 980 -57.1% <0.0001 Parallel-shutdown diff doesn't touch query execution; -57% is off-path variance.
⚠️ 16 not_sure 101 50 -50.5% <0.0001 Shutdown-only diff; -50% not attributable to this PR, off-path variance.

q-value = BH-FDR adjusted p; smaller is stronger evidence. MIRAI flags a query when q < fdr_q (default 0.10) — the value the verdict is based on.

Debug info
  • StressHouse run: f05d0e58-c2f6-47d3-9e54-c7401a37f23e
  • MIRAI run: 55541232-c05b-4a51-bbb9-019736c05dac
  • PR check IDs:
    • clickbench_621720_1782912842
    • clickbench_621734_1782912842
    • clickbench_621740_1782912842
    • tpch_adapted_1_official_621757_1782912842
    • tpch_adapted_1_official_621769_1782912842

@groeneai

Copy link
Copy Markdown
Contributor

@serxa diagnosis for #104969 (head 3d4725f0): the only thing blocking merge is the CH Inc sync check (tests failed). It is the sole failure in the combined commit status, which is why mergeStateStatus = BLOCKED. It carries no public target_url, so external contributors cannot see its logs. Could you check it on the CH Inc side to tell whether it is a real internal-test failure or a transient sync-infra flake that just needs a re-run?

Everything else is green:

  • mergeable: MERGEABLE, no conflicts (master merged in on 06-09).
  • 152/152 public checks pass; Mergeable Check, PR, and Style check statuses are all green.
  • All 4 review threads resolved. The two real bot findings are fixed: the max_threads=0 hang (now clamped via std::max<UInt64>(1, ...) in Server + LocalServer) and the over-strict non-UUID chassert in DatabaseWithOwnTablesBase::shutdown (removed). The latest NonZeroUInt64-vs-document-0 nit you already dismissed as acceptable safety margins.
  • Your 2026-06-01 approval stands.

Nothing on the author's side looks blocking. Happy to dig further once the sync logs are visible.

pull Bot pushed a commit to peijunlin2008/ClickHouse that referenced this pull request Jun 11, 2026
`AddDefaultDatabaseVisitor::visit(ASTSelectQuery &, ASTPtr &)` guards its
WITH-traversal on `select.recursive_with` but then dereferences
`select.with()->children` unconditionally. The parser sets the flag together with
a non-empty WITH expression, but AST mutation after parsing (server-side
`BuzzHouse` fuzzing, in particular) can leave the flag set while the expression
is absent, producing a null pointer dereference inside `IStorageCluster::read`.
Surfaced by `BuzzHouse (amd_tsan)` as `STID: 3440-3393` on PRs ClickHouse#104969 and
ClickHouse#105332 (two unrelated PRs in five days).

Guard the dereference with the actual presence of the WITH expression, matching
the pattern already used in `ASTSelectQuery::formatImpl` and
`QueryTreeBuilder::buildSelectQuery`. Valid `WITH RECURSIVE x AS ...` queries
are unaffected.

Adds a `gtest` that constructs the inconsistent AST shape, runs the visitor, and
asserts no fatal failure; the test reliably aborts on the previous code with the
`boost::intrusive_ptr` null-deref assertion.

Closes ClickHouse#105370

CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=105332&sha=ac51d1463d5def7f7ba0354f581e078cbf10d6c6&name_0=PR&name_1=BuzzHouse%20%28amd_tsan%29

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@groeneai

Copy link
Copy Markdown
Contributor

Hi @serxa, gentle ping. PR #104969 is green on public CI and all review threads are resolved; the only thing still blocking merge is the internal CH Inc sync -> Check public build check (status tests failed, no public target_url), which only CH Inc can inspect or re-run. Could you check whether that is a real internal failure or a transient sync-infra flake? Thanks.

@serxa serxa disabled auto-merge June 16, 2026 13:59
@serxa

serxa commented Jun 16, 2026

Copy link
Copy Markdown
Member

@groeneai Let's merge master into this branch to take new fixes into account and into sync PR

@groeneai

Copy link
Copy Markdown
Contributor

@serxa branch is ~3.9k commits behind master but mergeable is MERGEABLE, so merging master in is conflict-free. I can't push it myself: @groeneai is an external contributor with no write access to ClickHouse/ClickHouse, and "Allow edits from maintainers" grants fork-branch push to CH Inc maintainers (you), not to external contributors.

@jaymebrd could you merge master into parallel-table-shutdown? It's conflict-free, so the PR "Update branch" button works. That refreshes the internal sync PR (currently pending) with current master. A CH Inc maintainer can also do it via the same button (maintainerCanModify is true).

@serxa

serxa commented Jun 18, 2026

Copy link
Copy Markdown
Member

@groeneai check recent CI failures? do we have fixes for them? are they merged in master?

@groeneai

Copy link
Copy Markdown
Contributor

@serxa Triaged the 5 public failures at head 39a0842. None are caused by this PR; all are pre-existing trunk flakiness unrelated to the shutdown-parallelism change. No fix is merged in master for either, because neither needs one on the author side.

1. Integration tests (4 shards: asan_ubsan / msan / tsan / arm_binary) - test_format_cannot_allocate_thread::test_it
All four fail identically inside cluster.shutdown(), not in the test body. It is a pre-existing data race on the global getActivePartsLoadingThreadPool() singleton (src/IO/SharedThreadPools.cpp): main-thread teardown (StaticThreadPool::shutdown via the Server::main scope guard, line 82) races a SystemLog flush thread that lazily creates a system table on the way down (SystemLog<...>::prepareTable -> StorageMergeTree ctor -> loadDataParts -> StaticThreadPool::get(), line 100). This has nothing to do with DatabaseWithOwnTablesBase::shutdown from this PR. CIDB confirms: 0 master failures in 30 days; it only surfaces in unrelated PRs (streaming-pool, profile-events, signal-handler work) where the deliberate cannot_allocate_thread failpoint perturbs teardown timing. Caused by this PR: no. Fix exists/merged: no. Re-running the jobs clears it.

2. BuzzHouse (amd_debug) - Inconsistent AST formatting LOGICAL_ERROR (STID 1941-1bfa)
A fuzzer-generated query using caseWithExpression; a generic parser/formatter roundtrip bug (the formatted SQL re-parses to a different AST, tripping the debug-only consistency check in executeQueryImpl). Unrelated to shutdown. The Inconsistent AST formatting class hits master roughly 1-3 times/day every day; this specific variant is fuzzer-only and does not reproduce deterministically. Caused by this PR: no. Fix merged: no.

3. CH Inc sync - internal check, no public target_url; only CH Inc can inspect or re-run (unchanged from prior runs).

Recommendation: nothing on the author side blocks merge. The shutdown code is not implicated by any of the five. Re-running BuzzHouse + the Integration shards should clear them; the only real gate remains the internal sync.

Public CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=104969&sha=39a0842660da13a14f507feaa9d2f9f6a3edaf24&name_0=PR

@serxa

serxa commented Jun 23, 2026

Copy link
Copy Markdown
Member

@groeneai Failed test test_format_cannot_allocate_thread/test.py::test_it looks suspicious. Is it related to the changes in this PR?

@groeneai

Copy link
Copy Markdown
Contributor

@serxa Yes, it looks related to this PR. I need to correct my 06-18 triage: I lumped it in with a pre-existing teardown race, but the hard signature on the current runs is specific to this branch.

Evidence:

  • The crash is LOGICAL_ERROR: The MergeTreePartsLoaderThreadPool is not initialized on the non-TSan shards (asan/msan/coverage/arm_binary), plus a TSan data race on the same pool. CIDB: 11 hits in 90 days, all on this branch (parallel-table-shutdown), 0 on master and 0 on any other PR. The test is green on master (1532 OK / 0 FAIL in 30d). It first appears at head 39a0842 (the 06-18 master merge), not on the earlier 06-10 head.
  • It fails in cluster.shutdown() (module teardown), not in the test body, so it is the server shutdown that aborts. test_format_cannot_allocate_thread is just the canary that happens to restart the node.
  • Stack (identical on the hard-error shards and the TSan shard):
    StaticThreadPool::get() SharedThreadPools.cpp:100 <- MergeTreeData::loadDataParts MergeTreeData.cpp:2413 <- StorageMergeTree::StorageMergeTree StorageMergeTree.cpp:206 <- make_shared<StorageMergeTree> (registerStorageMergeTree.cpp:1079) <- a SystemLog flush thread lazily creating system.background_schedule_pool_log during shutdown.

Mechanism: a SystemLog flush thread lazily CREATES its backing MergeTree table during shutdown; the StorageMergeTree ctor calls loadDataParts -> getActivePartsLoadingThreadPool().get(). This PR moves DatabaseWithOwnTablesBase::shutdown onto a worker pool, which changes the timing and concurrency of the shutdown sequence so that this lazy table creation now overlaps with the global ActivePartsLoading static pool being torn down (StaticThreadPool::shutdownAll() in the Server::main SCOPE_EXIT). On non-TSan builds the pool is already reset -> hard not initialized; on TSan the read of the pool instance races with its reset. The serial shutdown on master does not hit this window.

So it is a latent shutdown-ordering fragility (system logs can lazily create their tables during shutdown and reach a torn-down static pool), and this PR's parallel shutdown is the first thing to surface it.

Report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=104969&sha=9d7ff6e40da97cd60ee5769c279c2850e40d5991&name_0=PR

Possible directions: skip lazy system-log table creation when the catalog is shutting down (just drop the unflushed entries, since the table was never materialized), or make sure the new parallel-shutdown work is fully joined before system logs flush so loadDataParts never runs against a pool that is being reset. I can dig into the exact ordering and prepare a fix in a separate PR off master if you want it. Since this is jaymebrd's branch I will not push to it.

Comment thread programs/server/Server.cpp Outdated
Comment thread src/Databases/DatabasesCommon.cpp Outdated
@jaymebrd

Copy link
Copy Markdown
Member Author

@groeneai Failed test test_format_cannot_allocate_thread/test.py::test_it looks suspicious. Is it related to the changes in this PR?

It looks like this PR introduced the race #107291 and our parallel shutdown exposes it

@serxa

serxa commented Jun 24, 2026

Copy link
Copy Markdown
Member

@groeneai Shutting down multiple interdependent pools is tricky. In easy cases, it could be resolved just by the proper ordering. Try to prepare the fix in a separate PR before we can merge this PR. The goal is to add necessary dependencies between startup and shutdown procedures, such that issue you found here is resolved. I hope you will not face circular dependencies; be careful with that (if nothing can prevent them, report back here)

@groeneai

Copy link
Copy Markdown
Contributor

@serxa Will do. Preparing the fix in a separate PR off master.

Plan: make SystemLogs teardown happen-before the static ActivePartsLoading pool teardown on all paths. Concretely, in DatabaseCatalog::shutdownImpl the shutdown_system_logs() call (which joins the system-log flush threads) is currently skipped if a user database shutdown() throws, so a flush thread can still lazily create its MergeTree backing table after StaticThreadPool::shutdownAll() reset the pool, giving the MergeTreePartsLoaderThreadPool is not initialized abort. Ordering the dependency (always tear down system logs before the global/static pools, even on a throwing path) closes that window.

No circular dependency expected: system logs depend on the static pools and on the system database (shut down last), not the other way around. Will report back here if a cycle shows up. This is a pre-existing latent issue on master that this PR's parallel shutdown exposes, so the fix belongs in its own master PR; I will link it here once it is up.

@groeneai

Copy link
Copy Markdown
Contributor

@serxa prepared the fix in a separate PR off master: #108417. No circular dependency. Two parts: DatabaseCatalog::shutdownImpl now keeps going when a database shutdown() throws (so shutdown_system_logs() always runs and joins the flush threads before the static pools are reset, which is the MergeTreePartsLoaderThreadPool is not initialized crash), and DatabaseWithOwnTablesBase::shutdown() releases its tables' references even on a throw so the storages are destroyed during shutdown instead of at atexit. Includes a regression test using a new failpoint. Once it merges, #104969 should be clean.

@serxa

serxa commented Jun 28, 2026

Copy link
Copy Markdown
Member

@groeneai fix has been merged, update this PR please

@groeneai

Copy link
Copy Markdown
Contributor

@serxa #108417 is merged to master (commit e5a7294, 2026-06-28 16:17 UTC), so the shutdown-crash fix is now on master.

Two things about updating this PR:

1. The branch update has to be done by @jaymebrd or a CH Inc maintainer. I'm an external contributor with no write access to jaymebrd's fork, and "Allow edits from maintainers" only grants push to CH Inc maintainers, not to other external contributors. maintainerCanModify is true, so a maintainer can do it too.

2. It is now a real conflict, not a clean "Update branch". #108417 touched the same two files this PR does:

  • src/Databases/DatabasesCommon.cpp: both rewrite DatabaseWithOwnTablesBase::shutdown().
  • src/Common/FailPoint.cpp: both add a failpoint in the same REGULAR(...) block.

So merging master needs manual conflict resolution in those two files.

3. On the clickhouse-gh[bot] shutdown-ordering finding: the system-log-race half (the MergeTreePartsLoaderThreadPool is not initialized / TSan race when a SystemLog flush thread outlives the static-pool teardown) is now handled by #108417 at the DatabaseCatalog::shutdownImpl level: database->shutdown() is wrapped in try/catch and shutdown_system_logs() always runs, even if a user-DB shutdown throws or parallel scheduling fails. After master is merged in, that part of the finding is addressed without making the prepare phase no-throw.

One thing to carry forward when resolving the DatabasesCommon.cpp conflict, so the other crash (the atexit Poco LoggerDeleter abort) does not return: the parallel shutdown() should still release every table's UUID mapping and clear tables even when a table throws, rethrowing the first error only at the very end. The current parallel version rethrows from the first waitForAllToFinishAndRethrowFirstError(), which on a prepare-phase throw skips the shutdown phase, the removeUUIDMapping calls and tables.clear(), so a failed database's storages would survive to ~DatabaseCatalog at process exit. Collecting the first error across both phases and clearing/rethrowing at the end keeps both the parallelism and that robustness.

jaymebrd and others added 5 commits June 29, 2026 12:02
DatabaseWithOwnTablesBase ran shutdown sequentially per table, which
made server shutdown take many minutes on clusters with thousands of
tables. Fan flushAndPrepareForShutdown and flushAndShutdown across a
dedicated thread pool (database_catalog_shutdown_table_concurrency,
default 16) and log per-database elapsed time. The parallel path is
exercised by a new integration test gated behind a test-only failpoint.
The lambda captured db_name and database_uuid only to assert a
database-level invariant. Under NDEBUG the assert is a no-op, so clang
flagged both as unused captures and -Werror failed Fast test. Move the
assertion out of the per-table lambda — it only needs to run once per
database — and drop the unused captures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DatabaseWithOwnTablesBase is also used by Memory-backed databases like
INFORMATION_SCHEMA and system in clickhouse-local, where getUUID() is
Nil but the snapshot is not empty. The previous unconditional chassert
would trip on those in debug/ASan builds. Gate the check on whether the
tables themselves carry UUIDs, mirroring the original assertion that
was guarded by table_id.hasUUID().

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
If the setting is 0, the pool accepts tasks but has no worker to run
them, so DatabaseWithOwnTablesBase::shutdown blocks forever on
waitForAllToFinishAndRethrowFirstError. Bump to 1 at init time.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the predefined-database check into the fiu_do_on block so the
test-only logic lives inside the failpoint scope rather than being
hoisted into shutdown()'s top-level variables.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@jaymebrd jaymebrd force-pushed the parallel-table-shutdown branch from 9d7ff6e to b36e6bb Compare June 29, 2026 13:09
Comment thread src/Databases/DatabasesCommon.cpp Outdated
jaymebrd and others added 3 commits June 29, 2026 14:38
enqueueAndKeepTrack can throw before the task's try/catch runs, skipping
per-table UUID mapping removal and the tail cleanup. Run the task inline
on schedule failure.
Both can throw (removeUUIDMapping has an explicit LOGICAL_ERROR throw if
the mapping is already gone). When they're outside the inner try, the
throw escapes into the task future, which waitForAllToFinish discards.
Move them inside the try so record_error captures them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment thread src/Databases/DatabasesCommon.cpp Outdated
jaymebrd and others added 2 commits June 30, 2026 14:49
Move getStorageID, flushAndShutdown, and removeUUIDMapping into three
separate try blocks. UUID mapping cleanup must run even when
flushAndShutdown throws, otherwise the catalog keeps the
UUID->StoragePtr mapping past Context::shutdown and the storage is
destroyed only at process exit (after loggers/static pools are gone).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Capture shared error state as shared_ptr<ErrorRecorder> and the logger
by value, dropping `this` and by-ref captures. An orphan task — rare
case where the pool accepts the job but tasks.emplace_back throws —
can outlive shutdown() and would otherwise dangle into the stack frame.

Also read recorder->first_error under the recorder's mutex in the tail,
since an orphan task can still write to it after waitForAllToFinish.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment thread src/Databases/DatabasesCommon.cpp
jaymebrd and others added 2 commits June 30, 2026 16:12
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
enqueueAndKeepTrack schedules the task, then tracks it with
tasks.emplace_back. If that emplace_back reallocated and threw after the
task was already scheduled, the catch ran the task inline a second time
while the scheduled copy stayed untracked (and unwaited). Reserve the
tracking vector up front so emplace_back never reallocates; then a throw
from enqueueAndKeepTrack always means the task was never scheduled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread src/Databases/DatabasesCommon.cpp Outdated
Parallelizing flushAndPrepareForShutdown was unsafe: some tables flush
into other tables there (a Buffer table writes to its destination), and
running that concurrently could hit a destination already in
shutdown-prepared read-only mode and silently drop rows.

Keep the prepare phase sequential (unchanged from master), and
parallelize only the raw shutdown() phase — the expensive,
table-independent drain (ZooKeeper sessions, interserver endpoints,
background pools) that was the actual bottleneck for databases with many
tables. Since prepare already ran, call shutdown() directly instead of
flushAndShutdown() to avoid re-running the flush.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@clickhouse-gh

clickhouse-gh Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

LLVM Coverage Report

Metric Baseline Current Δ
Lines 85.40% 85.50% +0.10%
Functions 92.60% 92.70% +0.10%
Branches 77.60% 77.70% +0.10%

Changed lines: Changed C/C++ lines covered: 105/110 (95.45%) · Uncovered code

Full report · Diff report

@jaymebrd

jaymebrd commented Jul 1, 2026

Copy link
Copy Markdown
Member Author

@serxa serxa left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rework feels okay to me, but now I think we may need one more opinion on this.
@evillique PTAL, maybe we need something similar in SharedCatalog, or that's completely unrelated/orthogonal?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-performance Pull request with some performance improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants