Abort on `std::logic_error` in CI by tavplubix · Pull Request #51907 · ClickHouse/ClickHouse · GitHub
Skip to content

Abort on std::logic_error in CI#51907

Merged
tavplubix merged 5 commits into
masterfrom
tavplubix-patch-7
Sep 4, 2023
Merged

Abort on std::logic_error in CI#51907
tavplubix merged 5 commits into
masterfrom
tavplubix-patch-7

Conversation

@tavplubix

Copy link
Copy Markdown
Member

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Usually std::logic_error indicates a bug, just like our LOGICAL_ERROR.
See also #51135

@robot-clickhouse-ci-2 robot-clickhouse-ci-2 added the pr-not-for-changelog This PR should not be mentioned in the changelog label Jul 6, 2023
@robot-clickhouse-ci-2

robot-clickhouse-ci-2 commented Jul 6, 2023

Copy link
Copy Markdown
Contributor

@tavplubix

Copy link
Copy Markdown
Member Author

AST fuzzer (asan) — Logical error: 'std::exception. Code: 1001, type: std::__1::future_error, e.what() = The associated promise has been destructed prior to ...
Details

265308:2023.07.07 16:27:44.570993 [ 393 ] {af7c4afd-dd32-445f-b4a4-844482a1a709::all_1_1_0_5} <Fatal> : Logical error: 'std::exception. Code: 1001, type: std::__1::future_error, e.what() = The associated promise has been destructed prior to the associated state becoming ready. (version 23.7.1.1)'.

2023.07.07 16:27:44.607442 [ 520 ] {} <Fatal> BaseDaemon: ########################################
2023.07.07 16:27:44.607538 [ 520 ] {} <Fatal> BaseDaemon: (version 23.7.1.1, build id: 948C87E8DC8453D03931D5428BB5D00E77687495, git hash: 67c9cf811db59824f13a763aecb35741483ddbaa) (from thread 393) (query_id: af7c4afd-dd32-445f-b4a4-844482a1a709::all_1_1_0_5) (query: ) Received signal Aborted (6)
2023.07.07 16:27:44.607626 [ 520 ] {} <Fatal> BaseDaemon: 
2023.07.07 16:27:44.607702 [ 520 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007f6525ef2a7c 0x00007f6525e9e476 0x00007f6525e847f3 0x0000562601408972 0x00005626014152a8 0x00005626014189f1 0x0000562618af00ab 0x0000562618aecce8 0x00005626155a40de 0x0000562617b44503 0x0000562617b49c80 0x0000562617b7b15e 0x0000562617b6f2b2 0x0000562617b3a764 0x000056261972390b 0x00005626197223cb 0x000056261982baab 0x00005626196e1f12 0x00005626196deec1 0x000056261526171a 0x000056261554cda2 0x0000562618273d76 0x0000562618271e5f 0x00005626185fec5a 0x0000562617c0166d 0x0000562617c034b9 0x00005626016b719c 0x00005626016c1600 0x00005626016adc92 0x00005626016bbf6d 0x00007f6525ef0b43 0x00007f6525f82a00
2023.07.07 16:27:44.607814 [ 520 ] {} <Fatal> BaseDaemon: 3. pthread_kill @ 0x00007f6525ef2a7c in ?
2023.07.07 16:27:44.607877 [ 520 ] {} <Fatal> BaseDaemon: 4. raise @ 0x00007f6525e9e476 in ?
2023.07.07 16:27:44.607927 [ 520 ] {} <Fatal> BaseDaemon: 5. abort @ 0x00007f6525e847f3 in ?
2023.07.07 16:27:44.759891 [ 520 ] {} <Fatal> BaseDaemon: 6. ./build_docker/./src/Common/Exception.cpp:43: DB::abortOnFailedAssertion(String const&) @ 0x000000002442b972 in /workspace/clickhouse
2023.07.07 16:27:44.921252 [ 520 ] {} <Fatal> BaseDaemon: 7. ./build_docker/./src/Common/Exception.cpp:430: DB::getCurrentExceptionMessageAndPattern(bool, bool, bool) @ 0x00000000244382a8 in /workspace/clickhouse
2023.07.07 16:27:45.116058 [ 520 ] {} <Fatal> BaseDaemon: 8.1. inlined from ./build_docker/./src/Common/Exception.cpp:0: DB::getCurrentExceptionMessage(bool, bool, bool)
2023.07.07 16:27:45.116136 [ 520 ] {} <Fatal> BaseDaemon: 8. ./build_docker/./src/Common/Exception.cpp:608: DB::ExecutionStatus::fromCurrentException(String const&, bool) @ 0x000000002443b9f1 in /workspace/clickhouse
2023.07.07 16:27:45.210398 [ 520 ] {} <Fatal> BaseDaemon: 9. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x000000003bb130ab in /workspace/clickhouse
2023.07.07 16:27:45.237963 [ 520 ] {} <Fatal> BaseDaemon: 10. ./build_docker/./src/Processors/Executors/CompletedPipelineExecutor.cpp:106: DB::CompletedPipelineExecutor::execute() @ 0x000000003bb0fce8 in /workspace/clickhouse
2023.07.07 16:27:45.403681 [ 520 ] {} <Fatal> BaseDaemon: 11. ./build_docker/./src/Interpreters/PreparedSets.cpp:0: DB::FutureSetFromSubquery::buildOrderedSetInplace(std::shared_ptr<DB::Context const> const&) @ 0x00000000385c70de in /workspace/clickhouse
2023.07.07 16:27:46.701408 [ 520 ] {} <Fatal> BaseDaemon: 12. ./build_docker/./src/Storages/MergeTree/KeyCondition.cpp:0: DB::KeyCondition::tryPrepareSetIndex(DB::RPNBuilderFunctionTreeNode const&, DB::KeyCondition::RPNElement&, unsigned long&) @ 0x000000003ab67503 in /workspace/clickhouse
2023.07.07 16:27:48.032312 [ 520 ] {} <Fatal> BaseDaemon: 13. ./build_docker/./src/Storages/MergeTree/KeyCondition.cpp:0: DB::KeyCondition::extractAtomFromTree(DB::RPNBuilderTreeNode const&, DB::KeyCondition::RPNElement&) @ 0x000000003ab6cc80 in /workspace/clickhouse
2023.07.07 16:27:49.308516 [ 520 ] {} <Fatal> BaseDaemon: 14. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848: DB::RPNBuilder<DB::KeyCondition::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) @ 0x000000003ab9e15e in /workspace/clickhouse
2023.07.07 16:27:50.575274 [ 520 ] {} <Fatal> BaseDaemon: 15. ./build_docker/./src/Storages/MergeTree/RPNBuilder.h:0: DB::RPNBuilder<DB::KeyCondition::RPNElement>::RPNBuilder(DB::ActionsDAG::Node const*, std::shared_ptr<DB::Context const>, std::function<bool (DB::RPNBuilderTreeNode const&, DB::KeyCondition::RPNElement&)> const&) @ 0x000000003ab922b2 in /workspace/clickhouse
2023.07.07 16:27:51.619680 [ 520 ] {} <Fatal> BaseDaemon: 16.1. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:818: ~__policy_func
2023.07.07 16:27:51.619743 [ 520 ] {} <Fatal> BaseDaemon: 16.2. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1174: ~function
2023.07.07 16:27:51.619764 [ 520 ] {} <Fatal> BaseDaemon: 16. ./build_docker/./src/Storages/MergeTree/KeyCondition.cpp:840: DB::KeyCondition::KeyCondition(std::shared_ptr<DB::ActionsDAG>, std::shared_ptr<DB::Context const>, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::ExpressionActions> const&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>, bool, bool) @ 0x000000003ab5d764 in /workspace/clickhouse
2023.07.07 16:27:52.488522 [ 520 ] {} <Fatal> BaseDaemon: 17. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1202: DB::buildIndexes(std::optional<DB::ReadFromMergeTree::Indexes>&, std::shared_ptr<DB::ActionsDAG>, DB::MergeTreeData const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&) @ 0x000000003c74690b in /workspace/clickhouse
2023.07.07 16:27:53.152665 [ 520 ] {} <Fatal> BaseDaemon: 18.1. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: ~shared_ptr
2023.07.07 16:27:53.152720 [ 520 ] {} <Fatal> BaseDaemon: 18. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1306: DB::ReadFromMergeTree::applyFilters() @ 0x000000003c7453cb in /workspace/clickhouse
2023.07.07 16:27:53.217330 [ 520 ] {} <Fatal> BaseDaemon: 19. ./build_docker/./src/Processors/QueryPlan/Optimizations/optimizeTree.cpp:204: DB::QueryPlanOptimizations::optimizeTreeThirdPass(DB::QueryPlan::Node&, std::list<DB::QueryPlan::Node, std::allocator<DB::QueryPlan::Node>>&) @ 0x000000003c84eaab in /workspace/clickhouse
2023.07.07 16:27:53.423949 [ 520 ] {} <Fatal> BaseDaemon: 20. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:487: DB::QueryPlan::optimize(DB::QueryPlanOptimizationSettings const&) @ 0x000000003c704f12 in /workspace/clickhouse
2023.07.07 16:27:53.583038 [ 520 ] {} <Fatal> BaseDaemon: 21.1. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__memory/compressed_pair.h:36: __compressed_pair_elem
2023.07.07 16:27:53.583094 [ 520 ] {} <Fatal> BaseDaemon: 21.2. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__memory/compressed_pair.h:108: __compressed_pair<std::__value_init_tag, std::__value_init_tag>
2023.07.07 16:27:53.583123 [ 520 ] {} <Fatal> BaseDaemon: 21.3. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:168: unique_ptr<true, void>
2023.07.07 16:27:53.583144 [ 520 ] {} <Fatal> BaseDaemon: 21. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:169: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000003c701ec1 in /workspace/clickhouse
2023.07.07 16:27:54.364049 [ 520 ] {} <Fatal> BaseDaemon: 22. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::execute() @ 0x000000003828471a in /workspace/clickhouse
2023.07.07 16:27:55.015773 [ 520 ] {} <Fatal> BaseDaemon: 23. ./build_docker/./src/Interpreters/MutationsInterpreter.cpp:0: DB::isStorageTouchedByMutations(DB::MergeTreeData&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::vector<DB::MutationCommand, std::allocator<DB::MutationCommand>> const&, std::shared_ptr<DB::Context const>) @ 0x000000003856fda2 in /workspace/clickhouse
2023.07.07 16:27:55.434840 [ 520 ] {} <Fatal> BaseDaemon: 24. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1782: DB::MutateTask::prepare() @ 0x000000003b296d76 in /workspace/clickhouse
2023.07.07 16:27:55.830902 [ 520 ] {} <Fatal> BaseDaemon: 25. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1690: DB::MutateTask::execute() @ 0x000000003b294e5f in /workspace/clickhouse
2023.07.07 16:27:55.901178 [ 520 ] {} <Fatal> BaseDaemon: 26. ./build_docker/./src/Storages/MergeTree/MutatePlainMergeTreeTask.cpp:90: DB::MutatePlainMergeTreeTask::executeStep() @ 0x000000003b621c5a in /workspace/clickhouse
2023.07.07 16:27:55.997770 [ 520 ] {} <Fatal> BaseDaemon: 27. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:281: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000003ac2466d in /workspace/clickhouse
2023.07.07 16:27:56.097679 [ 520 ] {} <Fatal> BaseDaemon: 28.1. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: ~shared_ptr
2023.07.07 16:27:56.097717 [ 520 ] {} <Fatal> BaseDaemon: 28. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:336: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000003ac264b9 in /workspace/clickhouse
2023.07.07 16:27:56.199090 [ 520 ] {} <Fatal> BaseDaemon: 29.1. inlined from ./build_docker/./base/base/../base/wide_integer_impl.h:809: bool wide::integer<128ul, unsigned int>::_impl::operator_eq<wide::integer<128ul, unsigned int>>(wide::integer<128ul, unsigned int> const&, wide::integer<128ul, unsigned int> const&)
2023.07.07 16:27:56.199158 [ 520 ] {} <Fatal> BaseDaemon: 29.2. inlined from ./build_docker/./base/base/../base/wide_integer_impl.h:1482: bool wide::operator==<128ul, unsigned int, 128ul, unsigned int>(wide::integer<128ul, unsigned int> const&, wide::integer<128ul, unsigned int> const&)
2023.07.07 16:27:56.199198 [ 520 ] {} <Fatal> BaseDaemon: 29.3. inlined from ./build_docker/./base/base/../base/strong_typedef.h:42: StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag>::operator==(StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag> const&) const
2023.07.07 16:27:56.199226 [ 520 ] {} <Fatal> BaseDaemon: 29.4. inlined from ./build_docker/./src/Common/OpenTelemetryTraceContext.h:65: DB::OpenTelemetry::Span::isTraceEnabled() const
2023.07.07 16:27:56.199245 [ 520 ] {} <Fatal> BaseDaemon: 29. ./build_docker/./src/Common/ThreadPool.cpp:429: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000246da19c in /workspace/clickhouse
2023.07.07 16:27:56.273603 [ 520 ] {} <Fatal> BaseDaemon: 30. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000246e4600 in /workspace/clickhouse
2023.07.07 16:27:56.361254 [ 520 ] {} <Fatal> BaseDaemon: 31.1. inlined from ./build_docker/./base/base/../base/wide_integer_impl.h:809: bool wide::integer<128ul, unsigned int>::_impl::operator_eq<wide::integer<128ul, unsigned int>>(wide::integer<128ul, unsigned int> const&, wide::integer<128ul, unsigned int> const&)
2023.07.07 16:27:56.361322 [ 520 ] {} <Fatal> BaseDaemon: 31.2. inlined from ./build_docker/./base/base/../base/wide_integer_impl.h:1482: bool wide::operator==<128ul, unsigned int, 128ul, unsigned int>(wide::integer<128ul, unsigned int> const&, wide::integer<128ul, unsigned int> const&)
2023.07.07 16:27:56.361354 [ 520 ] {} <Fatal> BaseDaemon: 31.3. inlined from ./build_docker/./base/base/../base/strong_typedef.h:42: StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag>::operator==(StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag> const&) const
2023.07.07 16:27:56.361386 [ 520 ] {} <Fatal> BaseDaemon: 31.4. inlined from ./build_docker/./src/Common/OpenTelemetryTraceContext.h:65: DB::OpenTelemetry::Span::isTraceEnabled() const
2023.07.07 16:27:56.361413 [ 520 ] {} <Fatal> BaseDaemon: 31. ./build_docker/./src/Common/ThreadPool.cpp:429: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000246d0c92 in /workspace/clickhouse
2023.07.07 16:27:56.473973 [ 520 ] {} <Fatal> BaseDaemon: 32.1. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: std::unique_ptr<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>, std::default_delete<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>>::reset[abi:v15000](std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>*)
2023.07.07 16:27:56.474045 [ 520 ] {} <Fatal> BaseDaemon: 32.2. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259: ~unique_ptr
2023.07.07 16:27:56.474093 [ 520 ] {} <Fatal> BaseDaemon: 32. ./build_docker/./contrib/llvm-project/libcxx/include/thread:297: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000246def6d in /workspace/clickhouse
2023.07.07 16:27:56.474133 [ 520 ] {} <Fatal> BaseDaemon: 33. ? @ 0x00007f6525ef0b43 in ?
2023.07.07 16:27:56.474152 [ 520 ] {} <Fatal> BaseDaemon: 34. ? @ 0x00007f6525f82a00 in ?
2023.07.07 16:27:56.474176 [ 520 ] {} <Fatal> BaseDaemon: Integrity check of the executable skipped because the reference checksum could not be read.
2023.07.07 16:28:08.343011 [ 520 ] {} <Fatal> BaseDaemon: This ClickHouse version is not official and should be upgraded to the official build.
2023.07.07 16:28:08.343170 [ 520 ] {} <Fatal> BaseDaemon: Changed settings: log_queries = true, table_function_remote_max_addresses = 200, max_execution_time = 10., max_memory_usage = 10000000000, allow_introspection_functions = true

@tavplubix tavplubix marked this pull request as ready for review September 1, 2023 18:05
@tavplubix

Copy link
Copy Markdown
Member Author

Integration tests (asan, analyzer) [1/6] - test_parallel_replicas_skip_shards was broken in master
Integration tests (release) [3/4] - test_storage_azure_blob_storage/test.py::test_schema_inference_cache is flaky
Upgrade check (debug) - #51586

@tavplubix tavplubix merged commit 973f4de into master Sep 4, 2023
@tavplubix tavplubix deleted the tavplubix-patch-7 branch September 4, 2023 10:51
Comment thread src/Common/Exception.cpp

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you mean

if (with_stacktrace)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, if with_stacktrace is true, then we already have the stacktrace printed to stream. And we need to print it if it's false

pull Bot pushed a commit to AKJUS/ClickHouse that referenced this pull request Jun 25, 2026
In Coordination::ZooKeeper::sendThread, after a request is popped from the
queue the local RequestInfo is the sole owner of its callback until the request
is inserted into operations (from which receiveEvent()/finalize() later satisfy
it). Async callers such as ZooKeeper::asyncTryExistsNoThrow capture a
shared_ptr<std::promise<...>> in that callback and hand the std::future to a
waiting caller before the request is processed.

If anything in the pop->insert window throws (the OpenTelemetry span finalize,
addRootPath, or the operations map insert, all of which can allocate and throw
under memory pressure), the local RequestInfo is destroyed while unwinding and
the callback never runs. The captured std::promise is then destroyed unsatisfied,
so the waiter's future.get() observes a std::future_error (broken promise). That
future_error is a std::logic_error, which the server reports as a LOGICAL_ERROR
and aborts on (see PR ClickHouse#51907). This matches the reported crash (STID 2508-34fb,
ZooKeeper-client variant): ~promise<ExistsResponse> running from sendThread's
teardown.

Arm a SCOPE_EXIT guard over that window: if the request is dropped before its
callback ownership transfers to operations, satisfy the callback with an error
response (ZCONNECTIONLOSS if it was probably sent, otherwise ZSESSIONEXPIRED)
instead of abandoning the promise. This mirrors how finalize() drains
outstanding operations and the already-merged sibling fixes for the same error
class (TestKeeper, PR ClickHouse#73570; the thread-pool callback runner, PR ClickHouse#107383). The
fix is at the send layer, so it covers every async request type uniformly.

Add a unit test reproducing the window contract: without the guard the future
carries a broken-promise future_error; with the guard it carries a normal
ZSESSIONEXPIRED response and nothing aborts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-not-for-changelog This PR should not be mentioned in the changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants