iframe-proxy

alexey-milovidov · 2019-07-23T14:34:35Z

For changelog. Remove if this is non-significant change.

Category (leave one):

New Feature

Short description (up to few sentences):
Merging #4247

alexey-milovidov · 2019-07-23T14:48:18Z

alexey-milovidov · 2019-07-23T18:58:58Z

Now we can get usable results:

SELECT 
    count(), 
    arrayStringConcat(arrayMap(x -> symbolizeAddress(x), trace), '\n') AS trace
FROM system.trace_log
GROUP BY trace
ORDER BY count() DESC
LIMIT 10
FORMAT Vertical

Row 1:
──────
count(): 2767
trace:   StackTrace::StackTrace(ucontext_t const&)



DB::DataTypeString::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const
DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)
DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)
DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)
DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeBaseSelectBlockInputStream::readFromPart()
DB::MergeTreeBaseSelectBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::FilterBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ExpressionBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::loop(unsigned long)
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)


clone

Row 2:
──────
count(): 1201
trace:   StackTrace::StackTrace(ucontext_t const&)



LZ4::decompress(char const*, char*, unsigned long, unsigned long, LZ4::PerformanceStatistics&)
DB::ICompressionCodec::decompress(char const*, unsigned int, char*) const
DB::CompressedReadBufferBase::decompress(char*, unsigned long, unsigned long)
DB::CompressedReadBufferFromFile::nextImpl()

DB::DataTypeString::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const
DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)
DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)
DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)
DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeBaseSelectBlockInputStream::readFromPart()
DB::MergeTreeBaseSelectBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::FilterBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ExpressionBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::loop(unsigned long)
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)


clone

Row 3:
──────
count(): 959
trace:   StackTrace::StackTrace(ucontext_t const&)


read
DB::ReadBufferFromFileDescriptor::nextImpl()
DB::CompressedReadBufferBase::readCompressedData(unsigned long&, unsigned long&)
DB::CompressedReadBufferFromFile::nextImpl()
DB::CompressedReadBufferFromFile::seek(unsigned long, unsigned long)
DB::MergeTreeReaderStream::seekToMark(unsigned long)

DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, unsigned long, DB::IDataType::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::IDataType::DeserializeBinaryBulkState>&) const
DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)
DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)
DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)
DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeBaseSelectBlockInputStream::readFromPart()
DB::MergeTreeBaseSelectBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::FilterBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ExpressionBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::loop(unsigned long)
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)


clone

Row 4:
──────
count(): 931
trace:   StackTrace::StackTrace(ucontext_t const&)


pthread_cond_wait
std::condition_variable::wait(std::unique_lock<std::mutex>&)
Poco::Event::wait()
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::wait()
DB::ParallelAggregatingBlockInputStream::execute()
DB::ParallelAggregatingBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ExpressionBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ExpressionBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::AsynchronousBlockInputStream::calculate()

ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::_List_iterator<ThreadFromGlobalPool>)
ThreadFromGlobalPool::ThreadFromGlobalPool<ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::function<void ()>, int, std::optional<unsigned long>)::{lambda()#3}>(ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::function<void ()>, int, std::optional<unsigned long>)::{lambda()#3}&&)::{lambda()#1}::operator()() const
ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)


clone

Row 5:
──────
count(): 531
trace:   StackTrace::StackTrace(ucontext_t const&)


DB::DataTypeArray::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, unsigned long, DB::IDataType::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::IDataType::DeserializeBinaryBulkState>&) const
DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)
DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)
DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)
DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeBaseSelectBlockInputStream::readFromPart()
DB::MergeTreeBaseSelectBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::FilterBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ExpressionBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::loop(unsigned long)
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)


clone

Row 6:
──────
count(): 525
trace:   StackTrace::StackTrace(ucontext_t const&)



LZ4::decompress(char const*, char*, unsigned long, unsigned long, LZ4::PerformanceStatistics&)
DB::ICompressionCodec::decompress(char const*, unsigned int, char*) const
DB::CompressedReadBufferBase::decompress(char*, unsigned long, unsigned long)
DB::CompressedReadBufferFromFile::readBig(char*, unsigned long)
DB::DataTypeNumberBase<int>::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const
DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)
DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)
DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)
DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeBaseSelectBlockInputStream::readFromPart()
DB::MergeTreeBaseSelectBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::FilterBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ExpressionBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::loop(unsigned long)
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)


clone

Row 7:
──────
count(): 300
trace:   StackTrace::StackTrace(ucontext_t const&)


read
DB::ReadBufferFromFileDescriptor::nextImpl()
DB::CompressedReadBufferBase::readCompressedData(unsigned long&, unsigned long&)
DB::CompressedReadBufferFromFile::nextImpl()

DB::DataTypeString::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const
DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)
DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)
DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)
DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeBaseSelectBlockInputStream::readFromPart()
DB::MergeTreeBaseSelectBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::FilterBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ExpressionBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::loop(unsigned long)
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)


clone

Row 8:
──────
count(): 288
trace:   StackTrace::StackTrace(ucontext_t const&)



LZ4::decompress(char const*, char*, unsigned long, unsigned long, LZ4::PerformanceStatistics&)
DB::ICompressionCodec::decompress(char const*, unsigned int, char*) const
DB::CompressedReadBufferBase::decompress(char*, unsigned long, unsigned long)
DB::CompressedReadBufferFromFile::readBig(char*, unsigned long)
DB::DataTypeNumberBase<unsigned int>::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const
DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)
DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)
DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)
DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeBaseSelectBlockInputStream::readFromPart()
DB::MergeTreeBaseSelectBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::FilterBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ExpressionBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::loop(unsigned long)
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)


clone

Row 9:
───────
count(): 288
trace:   StackTrace::StackTrace(ucontext_t const&)



LZ4::decompress(char const*, char*, unsigned long, unsigned long, LZ4::PerformanceStatistics&)
DB::ICompressionCodec::decompress(char const*, unsigned int, char*) const
DB::CompressedReadBufferBase::decompress(char*, unsigned long, unsigned long)
DB::CompressedReadBufferFromFile::nextImpl()
DB::CompressedReadBufferFromFile::seek(unsigned long, unsigned long)
DB::MergeTreeReaderStream::seekToMark(unsigned long)

DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, unsigned long, DB::IDataType::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::IDataType::DeserializeBinaryBulkState>&) const
DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)
DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)
DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)
DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeBaseSelectBlockInputStream::readFromPart()
DB::MergeTreeBaseSelectBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::FilterBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ExpressionBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::loop(unsigned long)
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)


clone

Row 10:
───────
count(): 236
trace:   StackTrace::StackTrace(ucontext_t const&)



LZ4::decompress(char const*, char*, unsigned long, unsigned long, LZ4::PerformanceStatistics&)
DB::ICompressionCodec::decompress(char const*, unsigned int, char*) const
DB::CompressedReadBufferBase::decompress(char*, unsigned long, unsigned long)
DB::CompressedReadBufferFromFile::nextImpl()
DB::DataTypeArray::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, unsigned long, DB::IDataType::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::IDataType::DeserializeBinaryBulkState>&) const
DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)
DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)
DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)
DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
DB::MergeTreeBaseSelectBlockInputStream::readFromPart()
DB::MergeTreeBaseSelectBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::FilterBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ExpressionBlockInputStream::readImpl()
DB::IBlockInputStream::read()
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::loop(unsigned long)
DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)


clone

alexey-milovidov · 2019-07-23T19:13:21Z

In gdb type

handle SIGUSR1 noprint
handle SIGUSR2 noprint

to not stop on profiling signals.

alexey-milovidov · 2019-07-23T19:17:00Z

It still can cause segfault:

Thread 65 "ParalInputsProc" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff232fa700 (LWP 565980)]
0x000000000b900c1d in libunwind::DwarfInstructions<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::stepWithDwarf (addressSpace=..., pc=<optimized out>, fdeStart=<optimized out>, 
    registers=...) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
34        return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest));
(gdb) bt
#0  0x000000000b900c1d in libunwind::DwarfInstructions<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::stepWithDwarf (addressSpace=..., pc=<optimized out>, 
    fdeStart=<optimized out>, registers=...) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
#1  0x000000000b8fc6f7 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::stepWithDwarfFDE (this=0x7fff232f3870)
    at ../contrib/libunwind/src/Registers.hpp:343
#2  libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::step (this=0x7fff232f3870) at ../contrib/libunwind/src/UnwindCursor.hpp:1972
#3  libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::step (this=0x7fff232f3870) at ../contrib/libunwind/src/UnwindCursor.hpp:1960
#4  __unw_step (cursor=0x7fff232f3870) at ../contrib/libunwind/src/libunwind.cpp:161
#5  0x000000000b8fc907 in unw_backtrace (buffer=buffer@entry=0x7fff232f3a68, size=size@entry=32) at ../contrib/libunwind/src/libunwind.cpp:291
#6  0x0000000007a48d22 in StackTrace::tryCapture (this=0x7fff232f3a60) at /usr/include/c++/8/array:234
#7  StackTrace::StackTrace (this=0x7fff232f3a60, signal_context=...) at ../libs/libcommon/src/StackTrace.cpp:183
#8  0x0000000003ae7c83 in DB::(anonymous namespace)::writeTraceInfo(DB::TimerType, int, siginfo_t*, void*) [clone .isra.40] () at ../dbms/src/Common/QueryProfiler.cpp:42
#9  <signal handler called>
#10 0x0000000007ab6cf8 in extent_heap_first (ph=ph@entry=0x7fff1e002688) at ../contrib/jemalloc/src/extent.c:289
#11 0x0000000007ab915b in extents_best_fit_locked (tsdn=0x7fff232f6f70, arena=0x7fff1e0008c0, size=<optimized out>, extents=0x7fff1e002540) at ../contrib/jemalloc/src/extent.c:460
#12 extents_fit_locked (tsdn=0x7fff232f6f70, arena=0x7fff1e0008c0, alignment=64, esize=1052672, extents=0x7fff1e002540) at ../contrib/jemalloc/src/extent.c:516
#13 extent_recycle_extract (slab=false, growing_retained=false, alignment=64, pad=4096, size=1048576, new_addr=0x0, extents=0x7fff1e002540, rtree_ctx=0x7fff232f6fa0, 
    r_extent_hooks=0x7fff232f4d30, arena=0x7fff1e0008c0, tsdn=0x7fff232f6f70) at ../contrib/jemalloc/src/extent.c:943
#14 extent_recycle (tsdn=tsdn@entry=0x7fff232f6f70, arena=arena@entry=0x7fff1e0008c0, r_extent_hooks=r_extent_hooks@entry=0x7fff232f4d30, extents=extents@entry=0x7fff1e002540, 
    new_addr=new_addr@entry=0x0, size=size@entry=1048576, pad=4096, alignment=64, slab=false, szind=60, zero=0x7fff232f4d97, commit=0x7fff232f4d2f, growing_retained=false)
    at ../contrib/jemalloc/src/extent.c:1123
#15 0x0000000007aba194 in extents_alloc (tsdn=tsdn@entry=0x7fff232f6f70, arena=arena@entry=0x7fff1e0008c0, r_extent_hooks=r_extent_hooks@entry=0x7fff232f4d30, 
    extents=extents@entry=0x7fff1e002540, new_addr=new_addr@entry=0x0, size=size@entry=1048576, pad=4096, alignment=64, slab=false, szind=60, zero=0x7fff232f4d97, commit=0x7fff232f4d2f)
    at ../contrib/jemalloc/src/extent.c:558
#16 0x0000000007a8f81b in arena_extent_alloc_large (tsdn=tsdn@entry=0x7fff232f6f70, arena=arena@entry=0x7fff1e0008c0, usize=usize@entry=1048576, alignment=alignment@entry=64, 
    zero=zero@entry=0x7fff232f4d97) at ../contrib/jemalloc/src/arena.c:375
#17 0x0000000007a73155 in large_palloc (tsdn=tsdn@entry=0x7fff232f6f70, arena=0x7fff1e0008c0, usize=usize@entry=1048576, alignment=alignment@entry=64, zero=false, zero@entry=243)
    at ../contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h:13
#18 0x0000000007a7361e in large_malloc (tsdn=tsdn@entry=0x7fff232f6f70, arena=<optimized out>, usize=usize@entry=1048576, zero=zero@entry=243) at ../contrib/jemalloc/src/large.c:17
#19 0x0000000007a926c8 in arena_malloc_hard (tsdn=tsdn@entry=0x7fff232f6f70, arena=<optimized out>, arena@entry=0x0, size=size@entry=1048576, ind=ind@entry=60, zero=zero@entry=false)
    at ../contrib/jemalloc/include/jemalloc/internal/sz.h:207
#20 0x0000000007a6d152 in arena_malloc (slow_path=false, tcache=0x7fff232f7138, zero=false, ind=60, size=1048576, arena=0x0, tsdn=<optimized out>)
    at ../contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h:151
#21 iallocztm (slow_path=false, arena=0x0, is_internal=false, tcache=0x7fff232f7138, zero=false, ind=60, size=1048576, tsdn=<optimized out>)
    at ../contrib/jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_c.h:53
#22 imalloc_no_sample (ind=60, usize=1048576, size=1048576, tsd=<optimized out>, dopts=<synthetic pointer>, sopts=<synthetic pointer>) at ../contrib/jemalloc/src/jemalloc.c:1840
#23 imalloc_body (tsd=<optimized out>, dopts=<synthetic pointer>, sopts=<synthetic pointer>) at ../contrib/jemalloc/src/jemalloc.c:2036
#24 imalloc (dopts=<synthetic pointer>, sopts=<synthetic pointer>) at ../contrib/jemalloc/src/jemalloc.c:2145
#25 malloc_default (size=1048576) at ../contrib/jemalloc/src/jemalloc.c:2175
#26 0x0000000007a6d65a in malloc (size=<optimized out>) at ../contrib/jemalloc/src/jemalloc.c:2274
#27 0x000000000667b953 in AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>::allocNoTrack(unsigned long, unsigned long) [clone .constprop.351] ()
    at ../dbms/src/Common/Allocator.h:81
#28 0x000000000667e85c in AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>::alloc (alignment=0, size=1048576, this=0x7fff2c00a1c0) at ../dbms/src/Common/Allocator.h:115
#29 DB::PODArrayBase<8ul, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul>::alloc<>(unsigned long) (bytes=1048576, this=0x7fff2c00a1c0)
    at ../dbms/src/Common/PODArray.h:108
#30 DB::PODArrayBase<8ul, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul>::alloc_for_num_elements (num_elements=<optimized out>, this=0x7fff2c00a1c0)
    at ../dbms/src/Common/PODArray.h:102
#31 DB::PODArray<unsigned long, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul>::PODArray (n=<optimized out>, this=0x7fff2c00a1c0)
    at ../dbms/src/Common/PODArray.h:296
#32 DB::ColumnVector<unsigned long>::ColumnVector (n=<optimized out>, this=0x7fff2c00a1b0) at ../dbms/src/Columns/ColumnVector.h:110
#33 COWHelper<DB::ColumnVectorHelper, DB::ColumnVector<unsigned long> >::create<unsigned long&> () at ../dbms/src/Common/COW.h:292
#34 DB::NumbersBlockInputStream::readImpl (this=0x7fff2c000290) at ../dbms/src/Storages/System/StorageSystemNumbers.cpp:27
#35 0x000000000674a0d8 in DB::IBlockInputStream::read() () at ../dbms/src/DataStreams/IBlockInputStream.cpp:56
---Type <return> to continue, or q <return> to quit---
#36 0x0000000006e9db0d in DB::ExpressionBlockInputStream::readImpl() () at /usr/include/c++/8/bits/shared_ptr_base.h:1018
#37 0x000000000674a0d8 in DB::IBlockInputStream::read() () at ../dbms/src/DataStreams/IBlockInputStream.cpp:56
#38 0x0000000006ec9344 in DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::loop (this=this@entry=0x7fff2c19e6a0, thread_num=thread_num@entry=8)
    at /usr/include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:778
#39 0x0000000006ec9ac9 in DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread (this=0x7fff2c19e6a0, thread_group=..., thread_num=8)
    at /usr/include/c++/8/bits/atomic_base.h:390
#40 0x0000000006ec9f8f in std::__invoke_impl<void, void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>* const&, std::shared_ptr<DB::ThreadGroupStatus> const&, unsigned long const&> (
    __t=@0x7fff1de00038: 0x7fff2c19e6a0, __f=
    @0x7fff1de00010: (void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler> * const, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)) 0x6ec98c0 <DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)>) at /usr/include/c++/8/ext/atomicity.h:96
#41 std::__invoke<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>* const&, std::shared_ptr<DB::ThreadGroupStatus> const&, unsigned long const&> (__fn=
    @0x7fff1de00010: (void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler> * const, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)) 0x6ec98c0 <DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)>) at /usr/include/c++/8/bits/invoke.h:95
#42 std::__apply_impl<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), std::tuple<DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long> const&, 0ul, 1ul, 2ul> (__t=..., __f=
    @0x7fff1de00010: (void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler> * const, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)) 0x6ec98c0 <DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)>) at /usr/include/c++/8/tuple:1678
#43 std::apply<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), std::tuple<DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long> const&> (__t=..., __f=
    @0x7fff1de00010: (void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler> * const, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)) 0x6ec98c0 <DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)>) at /usr/include/c++/8/tuple:1687
#44 ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const (this=0x7fff1de00000) at ../dbms/src/Common/ThreadPool.h:147
#45 0x0000000003b0753f in std::function<void ()>::operator()() const (this=0x7fff232f5610) at /usr/include/c++/8/bits/std_function.h:682
#46 ThreadPoolImpl<std::thread>::worker (this=<optimized out>, thread_it=...) at ../dbms/src/Common/ThreadPool.cpp:177
#47 0x000000000b8d0f2f in execute_native_thread_routine () at ../contrib/libunwind/src/DwarfParser.hpp:78
#48 0x00007ffff79b56db in start_thread (arg=0x7fff232fa700) at pthread_create.c:463
#49 0x00007ffff713c88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

alexey-milovidov · 2019-07-23T20:13:16Z

0x000000000b8bc0b1 in libunwind::LocalAddressSpace::get64 (this=0x10471a08 <libunwind::LocalAddressSpace::sThisAddressSpace>, addr=18446744073709551576)
    at ../contrib/libunwind/src/AddressSpace.hpp:203
203         memcpy(&val, (void *)addr, sizeof(val));
(gdb) bt
#0  0x000000000b8bc0b1 in libunwind::LocalAddressSpace::get64 (this=0x10471a08 <libunwind::LocalAddressSpace::sThisAddressSpace>, addr=18446744073709551576)
    at ../contrib/libunwind/src/AddressSpace.hpp:203
#1  0x000000000b8bc1b1 in libunwind::LocalAddressSpace::getRegister (this=0x10471a08 <libunwind::LocalAddressSpace::sThisAddressSpace>, addr=18446744073709551576)
    at ../contrib/libunwind/src/AddressSpace.hpp:241
#2  0x000000000b8bf654 in libunwind::DwarfInstructions<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::getSavedRegister (addressSpace=..., registers=..., cfa=16, savedReg=...)
    at ../contrib/libunwind/src/DwarfInstructions.hpp:89
#3  0x000000000b8bec45 in libunwind::DwarfInstructions<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::stepWithDwarf (addressSpace=..., pc=128409963, fdeStart=251834728, 
    registers=...) at ../contrib/libunwind/src/DwarfInstructions.hpp:191
#4  0x000000000b8be21d in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::stepWithDwarfFDE (this=0x7fff4c1f38a0)
    at ../contrib/libunwind/src/UnwindCursor.hpp:929
#5  0x000000000b8bdc42 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::step (this=0x7fff4c1f38a0) at ../contrib/libunwind/src/UnwindCursor.hpp:1972
#6  0x000000000b8ba524 in __unw_step (cursor=0x7fff4c1f38a0) at ../contrib/libunwind/src/libunwind.cpp:161
#7  0x000000000b8ba7d9 in unw_backtrace (buffer=0x7fff4c1f3a68, size=32) at ../contrib/libunwind/src/libunwind.cpp:291

alexey-milovidov · 2019-07-23T20:20:44Z

#0  0x000000000b8bc0b1 in libunwind::LocalAddressSpace::get64 (this=0x10471a08 <libunwind::LocalAddressSpace::sThisAddressSpace>, addr=18446744073709551576)
    at ../contrib/libunwind/src/AddressSpace.hpp:203
        val = 19721751088857089
#1  0x000000000b8bc1b1 in libunwind::LocalAddressSpace::getRegister (this=0x10471a08 <libunwind::LocalAddressSpace::sThisAddressSpace>, addr=18446744073709551576)
    at ../contrib/libunwind/src/AddressSpace.hpp:241
No locals.
#2  0x000000000b8bf654 in libunwind::DwarfInstructions<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::getSavedRegister (addressSpace=..., registers=..., cfa=16, savedReg=...)
    at ../contrib/libunwind/src/DwarfInstructions.hpp:89
        __func__ = "getSavedRegister"
#3  0x000000000b8bec45 in libunwind::DwarfInstructions<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::stepWithDwarf (addressSpace=..., pc=128409963, fdeStart=251834728, 
    registers=...) at ../contrib/libunwind/src/DwarfInstructions.hpp:191
        i = 3
        cfa = 16
        newRegisters = {_registers = {__rax = 140734424459136, __rbx = 140734470507632, __rcx = 6, __rdx = 29, __rdi = 140734424426120, __rsi = 1052671, __rbp = 0, __rsp = 140734470507024, 
            __r8 = 1048576, __r9 = 0, __r10 = 60, __r11 = 268435456, __r12 = 140734470516592, __r13 = 0, __r14 = 0, __r15 = 140734470507823, __rip = 128409963, __rflags = 11589455512659454976, 
            __cs = 140734470502576, __fs = 193710667, __gs = 140737339403848}}
        returnAddress = 0
        lastReg = 32
        prolog = {cfaRegister = 6, cfaRegisterOffset = 16, cfaExpression = 0, spExtraArgSize = 0, codeOffsetAtStackDecrement = 1, registersInOtherRegisters = false, sameValueUsed = false, 
          savedRegisters = {{location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {
              location = libunwind::kRegisterInCFA, value = -56}, {location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {
              location = libunwind::kRegisterInCFA, value = -16}, {location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {
              location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {
              location = libunwind::kRegisterInCFA, value = -48}, {location = libunwind::kRegisterInCFA, value = -40}, {location = libunwind::kRegisterInCFA, value = -32}, {
              location = libunwind::kRegisterInCFA, value = -24}, {location = libunwind::kRegisterInCFA, value = -8}, {location = libunwind::kRegisterUnused, value = 0} <repeats 16 times>}}
        fdeInfo = {fdeStart = 251834728, fdeLength = 48, fdeInstructions = 251834745, pcStart = 128407904, pcEnd = 128411909, lsda = 0}
        cieInfo = {cieStart = 247242904, cieLength = 24, cieInstructions = 247242921, pointerEncoding = 27 '\033', lsdaEncoding = 255 '\377', personalityEncoding = 0 '\000', 
          personalityOffsetInCIE = 0 '\000', personality = 0, codeAlignFactor = 1, dataAlignFactor = -8, isSignalFrame = false, fdesHaveAugmentationData = true, 
          returnAddressRegister = 16 '\020'}

alexey-milovidov · 2019-07-23T22:36:14Z

It's also reproduced: in debug build; in build with clang; without jemalloc.

alexey-milovidov · 2019-07-23T23:05:24Z

It happens not only inside malloc, also in normal C++ code:

#0  0x000000001402be70 in libunwind::LocalAddressSpace::get64 (this=0x14175fc0 <libunwind::LocalAddressSpace::sThisAddressSpace>, addr=524288) at ../contrib/libunwind/src/AddressSpace.hpp:203
#1  0x000000001402cecd in libunwind::LocalAddressSpace::getRegister (this=0x14175fc0 <libunwind::LocalAddressSpace::sThisAddressSpace>, addr=524288) at ../contrib/libunwind/src/AddressSpace.hpp:241
#2  0x000000001402b4fb in libunwind::DwarfInstructions<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::getSavedRegister (addressSpace=..., registers=..., cfa=524304, savedReg=...) at ../contrib/libunwind/src/DwarfInstructions.hpp:89
#3  0x000000001402ae3a in libunwind::DwarfInstructions<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::stepWithDwarf (addressSpace=..., pc=303046867, fdeStart=209557416, registers=...) at ../contrib/libunwind/src/DwarfInstructions.hpp:192
#4  0x000000001402aaa2 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::stepWithDwarfFDE (this=0x7fff0cfebb78) at ../contrib/libunwind/src/UnwindCursor.hpp:929
#5  0x0000000014029a66 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::step (this=0x7fff0cfebb78) at ../contrib/libunwind/src/UnwindCursor.hpp:1972
#6  0x000000001402758b in __unw_step (cursor=0x7fff0cfebb78) at ../contrib/libunwind/src/libunwind.cpp:161
#7  0x0000000014027a25 in unw_backtrace (buffer=0x7fff0cfebde8, size=32) at ../contrib/libunwind/src/libunwind.cpp:291
#8  0x0000000013555aa7 in StackTrace::tryCapture (this=0x7fff0cfebde0) at ../libs/libcommon/src/StackTrace.cpp:202
#9  0x0000000013555ae7 in StackTrace::StackTrace (this=0x7fff0cfebde0, signal_context=...) at ../libs/libcommon/src/StackTrace.cpp:183
#10 0x000000000d1e5ff0 in DB::(anonymous namespace)::writeTraceInfo (timer_type=DB::TimerType::Cpu, info=0x7fff0cfec9f0, context=0x7fff0cfec8c0) at ../dbms/src/Common/QueryProfiler.cpp:42
#11 0x000000000d1e5ea8 in DB::QueryProfilerCpu::signalHandler (sig=12, info=0x7fff0cfec9f0, context=0x7fff0cfec8c0) at ../dbms/src/Common/QueryProfiler.cpp:136
#12 <signal handler called>
#13 std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<unsigned long, DB::QuotaForInterval>, std::__1::__tree_node<std::__1::__value_type<unsigned long, DB::QuotaForInterval>, void*>*, long> >::operator-- (this=0x7fff0cfecfc0)
    at /home/milovidov/work/ClickHouse/contrib/libcxx/include/map:812
#14 0x00000000121020d3 in std::__1::reverse_iterator<std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<unsigned long, DB::QuotaForInterval>, std::__1::__tree_node<std::__1::__value_type<unsigned long, DB::QuotaForInterval>, void*>*, long> > >::operator++ (this=0x7fff0cfecfc0) at /home/milovidov/work/ClickHouse/contrib/libcxx/include/iterator:694
#15 0x000000001210007d in DB::QuotaForIntervals::checkAndAddReadRowsBytes (this=0x7fffa40014d8, current_time=1563923020, rows=65536, bytes=524288) at ../dbms/src/Interpreters/Quota.cpp:234
#16 0x0000000011a2da04 in DB::IBlockInputStream::progressImpl (this=0x7fffa400e7a0, value=...) at ../dbms/src/DataStreams/IBlockInputStream.cpp:373
#17 0x000000000f639e2e in DB::IBlockInputStream::progress (this=0x7fffa400e7a0, value=...) at ../dbms/src/DataStreams/IBlockInputStream.h:154
#18 0x0000000011a2c64a in DB::IBlockInputStream::read (this=0x7fffa400e7a0) at ../dbms/src/DataStreams/IBlockInputStream.cpp:82
#19 0x0000000011f608c6 in DB::ExpressionBlockInputStream::readImpl (this=0x7fffa400e370) at ../dbms/src/DataStreams/ExpressionBlockInputStream.cpp:33
#20 0x0000000011a2c470 in DB::IBlockInputStream::read (this=0x7fffa400e370) at ../dbms/src/DataStreams/IBlockInputStream.cpp:56
#21 0x0000000011f84f5d in DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::loop (this=0x7fffa400f510, thread_num=0) at ../dbms/src/DataStreams/ParallelInputsProcessor.h:270
#22 0x0000000011f827fc in DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread (this=0x7fffa400f510, thread_group=..., thread_num=0) at ../dbms/src/DataStreams/ParallelInputsProcessor.h:208

alexey-milovidov · 2019-07-23T23:06:57Z

#0  0x000000001402be70 in libunwind::LocalAddressSpace::get64 (this=0x14175fc0 <libunwind::LocalAddressSpace::sThisAddressSpace>, addr=524288) at ../contrib/libunwind/src/AddressSpace.hpp:203
        val = 0
#1  0x000000001402cecd in libunwind::LocalAddressSpace::getRegister (this=0x14175fc0 <libunwind::LocalAddressSpace::sThisAddressSpace>, addr=524288) at ../contrib/libunwind/src/AddressSpace.hpp:241
No locals.
#2  0x000000001402b4fb in libunwind::DwarfInstructions<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::getSavedRegister (addressSpace=..., registers=..., cfa=524304, savedReg=...) at ../contrib/libunwind/src/DwarfInstructions.hpp:89
No locals.
#3  0x000000001402ae3a in libunwind::DwarfInstructions<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::stepWithDwarf (addressSpace=..., pc=303046867, fdeStart=209557416, registers=...) at ../contrib/libunwind/src/DwarfInstructions.hpp:192
        i = 6
        cfa = 524304
        newRegisters = {_registers = {__rax = 140733411413960, __rbx = 0, __rcx = 140735944868104, __rdx = 0, __rdi = 140733411413960, __rsi = 0, __rbp = 524288, __rsp = 140733411413856, __r8 = 140735944856840, __r9 = 135804566, __r10 = 0, __r11 = 6, __r12 = 140733411418048, 
            __r13 = 0, __r14 = 140736213305280, __r15 = 140733428197728, __rip = 303046867, __rflags = 0, __cs = 140733411409216, __fs = 140733411410432, __gs = 8192}}
        returnAddress = 0
        lastReg = 32
        prolog = {cfaRegister = 6, cfaRegisterOffset = 16, cfaExpression = 0, spExtraArgSize = 0, codeOffsetAtStackDecrement = 1, registersInOtherRegisters = false, sameValueUsed = false, savedRegisters = {{location = libunwind::kRegisterUnused, value = 0}, {
              location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {
              location = libunwind::kRegisterInCFA, value = -16}, {location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {
              location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {location = libunwind::kRegisterUnused, value = 0}, {
              location = libunwind::kRegisterInCFA, value = -8}, {location = libunwind::kRegisterUnused, value = 0} <repeats 16 times>}}
        fdeInfo = {fdeStart = 209557416, fdeLength = 32, fdeInstructions = 209557433, pcStart = 303046832, pcEnd = 303046884, lsda = 0}
        cieInfo = {cieStart = 192896504, cieLength = 24, cieInstructions = 192896521, pointerEncoding = 27 '\033', lsdaEncoding = 255 '\377', personalityEncoding = 0 '\000', personalityOffsetInCIE = 0 '\000', personality = 0, codeAlignFactor = 1, dataAlignFactor = -8, 
          isSignalFrame = false, fdesHaveAugmentationData = true, returnAddressRegister = 16 '\020'}
#4  0x000000001402aaa2 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::stepWithDwarfFDE (this=0x7fff0cfebb78) at ../contrib/libunwind/src/UnwindCursor.hpp:929
No locals.
#5  0x0000000014029a66 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::step (this=0x7fff0cfebb78) at ../contrib/libunwind/src/UnwindCursor.hpp:1972
        result = 0
#6  0x000000001402758b in __unw_step (cursor=0x7fff0cfebb78) at ../contrib/libunwind/src/libunwind.cpp:161
        co = 0x7fff0cfebb78
#7  0x0000000014027a25 in unw_backtrace (buffer=0x7fff0cfebde8, size=32) at ../contrib/libunwind/src/libunwind.cpp:291
        context = {data = {140733411409384, 0, 140733411413064, 32, 140733411409024, 32, 140733411409216, 140733411408736, 0, 135804566, 140733411408736, 514, 140733411418048, 0, 140736213305280, 140733428197728, 335706590, 0, 140733411409216, 140733411410432, 8192}}
        cursor = {data = {180194192, 337076160, 140733411413960, 0, 140735944868104, 0, 140733411413960, 0, 524288, 140733411413856, 140735944856840, 135804566, 0, 6, 140733411418048, 0, 140736213305280, 140733428197728, 303046867, 0, 140733411409216, 140733411410432, 8192, 
            303046832, 303046884, 0, 0, 0, 0, 137506062336, 209557416, 220028928, 140733411360768}}
        ip = 303046867
        current = 7

alexey-milovidov · 2019-07-23T23:11:32Z

Note that the address that libunwind wants to read (and it think that it's the value of rbp register) corresponds to the value of function argument a few stack frames above:

#0 0x000000001402be70 in libunwind::LocalAddressSpace::get64 (this=0x14175fc0 libunwind::LocalAddressSpace::sThisAddressSpace, addr=524288) at ../contrib/libunwind/src/AddressSpace.hpp:203
#1 0x000000001402cecd in libunwind::LocalAddressSpace::getRegister (this=0x14175fc0 libunwind::LocalAddressSpace::sThisAddressSpace, addr=524288) at ../contrib/libunwind/src/AddressSpace.hpp:241
#2 0x000000001402b4fb in libunwind::DwarfInstructions<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::getSavedRegister (addressSpace=..., registers=..., cfa=524304, savedReg=...) at ../contrib/libunwind/src/DwarfInstructions.hpp:89
#3 0x000000001402ae3a in libunwind::DwarfInstructions<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::stepWithDwarf (addressSpace=..., pc=303046867, fdeStart=209557416, registers=...) at ../contrib/libunwind/src/DwarfInstructions.hpp:192
#4 0x000000001402aaa2 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::stepWithDwarfFDE (this=0x7fff0cfebb78) at ../contrib/libunwind/src/UnwindCursor.hpp:929
#5 0x0000000014029a66 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::step (this=0x7fff0cfebb78) at ../contrib/libunwind/src/UnwindCursor.hpp:1972
#6 0x000000001402758b in __unw_step (cursor=0x7fff0cfebb78) at ../contrib/libunwind/src/libunwind.cpp:161
#7 0x0000000014027a25 in unw_backtrace (buffer=0x7fff0cfebde8, size=32) at ../contrib/libunwind/src/libunwind.cpp:291
#8 0x0000000013555aa7 in StackTrace::tryCapture (this=0x7fff0cfebde0) at ../libs/libcommon/src/StackTrace.cpp:202
#9 0x0000000013555ae7 in StackTrace::StackTrace (this=0x7fff0cfebde0, signal_context=...) at ../libs/libcommon/src/StackTrace.cpp:183
#10 0x000000000d1e5ff0 in DB::(anonymous namespace)::writeTraceInfo (timer_type=DB::TimerType::Cpu, info=0x7fff0cfec9f0, context=0x7fff0cfec8c0) at ../dbms/src/Common/QueryProfiler.cpp:42
#11 0x000000000d1e5ea8 in DB::QueryProfilerCpu::signalHandler (sig=12, info=0x7fff0cfec9f0, context=0x7fff0cfec8c0) at ../dbms/src/Common/QueryProfiler.cpp:136
#12
#13 std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<unsigned long, DB::QuotaForInterval>, std::__1::__tree_node<std::__1::__value_type<unsigned long, DB::QuotaForInterval>, void*>, long> >::operator-- (this=0x7fff0cfecfc0)
at /home/milovidov/work/ClickHouse/contrib/libcxx/include/map:812
#14 0x00000000121020d3 in std::__1::reverse_iterator<std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<unsigned long, DB::QuotaForInterval>, std::__1::__tree_node<std::__1::__value_type<unsigned long, DB::QuotaForInterval>, void>*, long> > >::operator++ (this=0x7fff0cfecfc0) at /home/milovidov/work/ClickHouse/contrib/libcxx/include/iterator:694
#15 0x000000001210007d in DB::QuotaForIntervals::checkAndAddReadRowsBytes (this=0x7fffa40014d8, current_time=1563923020, rows=65536, bytes=524288) at ../dbms/src/Interpreters/Quota.cpp:234

alexey-milovidov · 2019-07-24T00:43:49Z

Probably it will work if we limit DWARF version to 3 for ClickHouse build.

alexey-milovidov · 2019-07-24T00:46:01Z

Probably it will work if we limit DWARF version to 3 for ClickHouse build.

Not true.

alexey-milovidov · 2019-07-24T01:22:50Z

Deadlock is possible if:

signal handler is invoked while malloc mutex is hold;
the trace info pipe becomes full because there are many traces.

…rge-profiler

alexey-milovidov · 2019-07-25T19:32:01Z

00568_compile_catch_throw failed as expected.

alexey-milovidov · 2019-07-26T00:20:10Z

TODO: timer offset randomization. For example, we may want to set up timer interval of one second but be able to sample some queries with duration less than a second.

alexey-milovidov · 2019-07-27T21:40:34Z

Note that the address that libunwind wants to read (and it think that it's the value of rbp register) corresponds to the value of function argument a few stack frames above:

This probably requires -fasynchronous-unwind-tables option.

alexey-milovidov · 2019-07-27T21:43:56Z

But it's by default.

alexey-milovidov · 2019-07-28T14:50:12Z

Backported changes to query profiler to 19.12, because it was in unusable state there.

laplab · 2019-07-28T11:15:11Z

        throw;
    }
+#else
+    throw Exception("QueryProfiler cannot work with stock libunwind", ErrorCodes::NOT_IMPLEMENTED);


Can we make it a compile time error?

There are alternative build variants for ClickHouse:

build for Debian repository;

build for Arcadia source code repository in Yandex;

These builds must work, and we simply disable query profiler in that cases.

But maybe better to hide all QueryProfiler class altogether.

laplab · 2019-07-28T11:21:43Z

+        throw Exception("QueryProfiler cannot be used without PHDR cache, that is not available for TSan build", ErrorCodes::NOT_IMPLEMENTED);
+
+    /// Too high frequency can introduce infinite busy loop of signal handlers. We will limit maximum frequency (with 1000 signals per second).
+    if (period < 1000000)


This check should be performed after period_rand is selected. Otherwise we still can get very small periods

period_rand is the offset. It affects only the first signal. And it is needed for sampling of short queries.

alexey-milovidov · 2019-07-30T23:25:29Z

@laplab
PS. If you are interested, it was continued in #6201

alexey-milovidov · 2019-07-31T03:46:06Z

alexey-milovidov added 6 commits July 23, 2019 15:46

Changed symbolizeTrace to symbolizeAddress

2d895c4

Every function in its own file

97103b9

Fixed error

398a280

Added comment

bd3173b

Added demangling to function symbolizeAddress

0755d1e

Fixed error with initialization of query profiler

1b00e99

alexey-milovidov added 2 commits July 23, 2019 17:59

Added cache for symbolizeAddress function

1913bfa

Simple protection from too frequent profiler signals

02fb67d

Fixed warning from PVS-Studio

a3f6b70

alexey-milovidov added 2 commits July 24, 2019 03:23

Mitigate segfault in libunwind #6124

99be894

Addition to prev. revision

dc06885

alexey-milovidov added 5 commits July 24, 2019 03:46

Lower bound on profiler period

378fff2

Better cache for symbolizeAddress function

36746d8

Addition to prev. revision

33ee957

Added a test

c97a4e1

Updated test

72d543f

Avoid deadlock in TraceCollector

b9105e1

alexey-milovidov added 7 commits July 24, 2019 22:04

Added ProfileEvent for dropped traces due to pipe full

6e8258f

Merge branch 'merge-profiler' of github.com:yandex/ClickHouse into me…

63764af

…rge-profiler

Added ProfileEvents for signal handler overruns

d9976ab

Update settings description

4b93b45

Added information about thread number to trace log

cfb2066

Added information about revision to trace log

4c57405

Trigger CI

f358064

alexey-milovidov added 10 commits July 25, 2019 22:54

Allow to update PHDRCache

1d499bc

Merge branch 'master' into merge-profiler

050db07

Updated comment

f5c1dfc

Updating PHDRCache after dlopen

472a61c

More strict memory order just in case

9b8375a

Merge branch 'master' into merge-profiler

eb2533f

Merge remote-tracking branch 'origin/master' into merge-profiler

0cce86d

Disable PHDR Cache with TSan

673fb05

Sanity and configuration checks

5a47b0b

Reverted bad modification

cf67765

Add timer offset randomization

da3806b

alexey-milovidov merged commit 2442e65 into master Jul 26, 2019

laplab reviewed Jul 30, 2019

View reviewed changes

KochetovNicolai added the pr-feature Pull request with new product feature label Sep 19, 2019

CurtizJ mentioned this pull request Jul 30, 2020

ClickHouse server silently crashed. #13098

Closed

filimonov mentioned this pull request Apr 22, 2024

libunwind: getSavedRegister: invalid read from memory #60460

Closed

Sunbelt Computer Software

PL/B Language Development and Support

Uh oh!

Conversation

alexey-milovidov commented Jul 23, 2019

Uh oh!

alexey-milovidov commented Jul 23, 2019

Uh oh!

alexey-milovidov commented Jul 23, 2019

Uh oh!

alexey-milovidov commented Jul 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov commented Jul 23, 2019

Uh oh!

alexey-milovidov commented Jul 23, 2019

Uh oh!

alexey-milovidov commented Jul 23, 2019

Uh oh!

alexey-milovidov commented Jul 23, 2019

Uh oh!

alexey-milovidov commented Jul 23, 2019

Uh oh!

alexey-milovidov commented Jul 23, 2019

Uh oh!

alexey-milovidov commented Jul 23, 2019

Uh oh!

alexey-milovidov commented Jul 24, 2019

Uh oh!

alexey-milovidov commented Jul 24, 2019

Uh oh!

alexey-milovidov commented Jul 24, 2019

Uh oh!

alexey-milovidov commented Jul 25, 2019

Uh oh!

alexey-milovidov commented Jul 26, 2019

Uh oh!

alexey-milovidov commented Jul 27, 2019

Uh oh!

alexey-milovidov commented Jul 27, 2019

Uh oh!

alexey-milovidov commented Jul 28, 2019

Uh oh!

laplab Jul 28, 2019

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Jul 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

laplab Jul 28, 2019

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov commented Jul 30, 2019

Uh oh!

alexey-milovidov commented Jul 31, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alexey-milovidov commented Jul 23, 2019 •

edited

Loading

alexey-milovidov Jul 30, 2019 •

edited

Loading