Unknown TF crash on OSX in C++ Application, works fine on another machine, other operating systems · Issue #19426 · tensorflow/tensorflow · GitHub
Skip to content

Unknown TF crash on OSX in C++ Application, works fine on another machine, other operating systems #19426

Closed
#80243
Closed
Unknown TF crash on OSX in C++ Application, works fine on another machine, other operating systems#19426
#80243
Assignees
robieta

Description

@samhodge
samhodge
opened on May 20, 2018

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
    Yes there is a bunch of private OpenFX plugin and C++ hosting of the network involved.

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
    macOS 10.12

  • TensorFlow installed from (source or binary):
    source

  • TensorFlow version (use command below):
    r1.6

  • Python version:
    na

  • Bazel version (if compiling from source):
    0.6

  • GCC/Compiler version (if compiling from source):
    clang 8

  • CUDA/cuDNN version:
    NA

  • GPU model and memory:
    NVIDIA GTX 780M

  • Exact command to reproduce:
    Download alpha trial of software, install according to instructions.

Launch Nuke as follows:
ROTOBOT_MODEL_DIR=$PWD OFX_PLUGIN_PATH=$OFX_PLUGIN_PATH:$PWD /Applications/Nuke10.5v7/Nuke10.5v7.app/Contents/MacOS/Nuke10.5v7

Create read node, create Rotobot-MaskRCNN node, connect source to read node

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.

Same neural network work fine on hardware without GPU installed

Works OK under Linux 64, macOS

I am unsure of what is causing this crash.

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

Thread 36 Crashed:
0   libtensorflow_cc.so           	0x000000013c6ce622 Eigen::internal::TensorExecutor<Eigen::TensorAssignOp<Eigen::TensorMap<Eigen::Tensor<float, 3, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorSlicingOp<Eigen::DSizes<long, 3> const, Eigen::DSizes<long, 3> const, Eigen::TensorMap<Eigen::Tensor<float const, 3, 1, long>, 16, Eigen::MakePointer> const> const> const, Eigen::DefaultDevice, true>::run(Eigen::TensorAssignOp<Eigen::TensorMap<Eigen::Tensor<float, 3, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorSlicingOp<Eigen::DSizes<long, 3> const, Eigen::DSizes<long, 3> const, Eigen::TensorMap<Eigen::Tensor<float const, 3, 1, long>, 16, Eigen::MakePointer> const> const> const&, Eigen::DefaultDevice const&) + 1682
1   libtensorflow_cc.so           	0x000000013ca4eb15 tensorflow::functor::Split<Eigen::ThreadPoolDevice, float>::operator()(Eigen::ThreadPoolDevice const&, Eigen::TensorMap<Eigen::Tensor<float, 3, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorMap<Eigen::Tensor<float const, 3, 1, long>, 16, Eigen::MakePointer>, Eigen::DSizes<long, 3> const&, Eigen::DSizes<long, 3> const&) + 101
2   libtensorflow_cc.so           	0x000000013c6cde31 tensorflow::SplitOpCPU<float>::Compute(tensorflow::OpKernelContext*)::'lambda'(long long, long long)::operator()(long long, long long) const + 481
3   libtensorflow_cc.so           	0x000000013c6cd7db tensorflow::SplitOpCPU<float>::Compute(tensorflow::OpKernelContext*) + 1099
4   libtensorflow_cc.so           	0x000000013e3f7442 tensorflow::grappler::ConstantFolding::EvaluateNode(tensorflow::NodeDef const&, tensorflow::gtl::InlinedVector<tensorflow::TensorValue, 4> const&, tensorflow::gtl::InlinedVector<tensorflow::TensorValue, 4>*) const + 562
5   libtensorflow_cc.so           	0x000000013e3f78bc tensorflow::grappler::ConstantFolding::EvaluateOneFoldable(tensorflow::NodeDef const&, std::__1::vector<tensorflow::NodeDef, std::__1::allocator<tensorflow::NodeDef> >*) + 604
6   libtensorflow_cc.so           	0x000000013e3f82d2 tensorflow::grappler::ConstantFolding::FoldNode(tensorflow::NodeDef*, tensorflow::GraphDef*) + 402
7   libtensorflow_cc.so           	0x000000013e3fa54d tensorflow::grappler::ConstantFolding::FoldGraph(tensorflow::GraphDef*) + 973
8   libtensorflow_cc.so           	0x000000013e400f83 tensorflow::grappler::ConstantFolding::RunOptimizationPass(tensorflow::grappler::Cluster*, tensorflow::grappler::GrapplerItem const&, tensorflow::GraphDef*) + 1523
9   libtensorflow_cc.so           	0x000000013e401925 tensorflow::grappler::ConstantFolding::Optimize(tensorflow::grappler::Cluster*, tensorflow::grappler::GrapplerItem const&, tensorflow::GraphDef*) + 1829
10  libtensorflow_cc.so           	0x000000013e3db066 tensorflow::grappler::MetaOptimizer::Optimize(tensorflow::grappler::Cluster*, tensorflow::grappler::GrapplerItem const&, tensorflow::GraphDef*) + 2822
11  libtensorflow_cc.so           	0x000000013e3dc2dd tensorflow::grappler::RunMetaOptimizer(tensorflow::grappler::GrapplerItem const&, tensorflow::RewriterConfig const&, tensorflow::DeviceBase*, tensorflow::grappler::Cluster*, tensorflow::GraphDef*) + 109
12  libtensorflow_cc.so           	0x000000013e3d6dac tensorflow::GraphExecutionState::OptimizeGraph(tensorflow::BuildGraphOptions const&, std::__1::unique_ptr<tensorflow::Graph, std::__1::default_delete<tensorflow::Graph> >*) + 3564
13  libtensorflow_cc.so           	0x000000013e3d4686 tensorflow::GraphExecutionState::BuildGraph(tensorflow::BuildGraphOptions const&, std::__1::unique_ptr<tensorflow::ClientGraph, std::__1::default_delete<tensorflow::ClientGraph> >*) + 166
14  libtensorflow_cc.so           	0x000000013e3c0d37 tensorflow::DirectSession::CreateGraphs(tensorflow::BuildGraphOptions const&, std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::unique_ptr<tensorflow::Graph, std::__1::default_delete<tensorflow::Graph> >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::unique_ptr<tensorflow::Graph, std::__1::default_delete<tensorflow::Graph> > > > >*, std::__1::unique_ptr<tensorflow::FunctionLibraryDefinition, std::__1::default_delete<tensorflow::FunctionLibraryDefinition> >*, tensorflow::DirectSession::RunStateArgs*, tensorflow::gtl::InlinedVector<tensorflow::DataType, 4>*, tensorflow::gtl::InlinedVector<tensorflow::DataType, 4>*) + 375
15  libtensorflow_cc.so           	0x000000013e3bb469 tensorflow::DirectSession::GetOrCreateExecutors(tensorflow::gtl::ArraySlice<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, tensorflow::gtl::ArraySlice<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, tensorflow::gtl::ArraySlice<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, tensorflow::DirectSession::ExecutorsAndKeys**, tensorflow::DirectSession::RunStateArgs*) + 4585
16  libtensorflow_cc.so           	0x000000013e3b893c tensorflow::DirectSession::Run(tensorflow::RunOptions const&, std::__1::vector<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, tensorflow::Tensor>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, tensorflow::Tensor> > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::vector<tensorflow::Tensor, std::__1::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*) + 908
17  libtensorflow_cc.so           	0x000000013e3b8437 tensorflow::DirectSession::Run(std::__1::vector<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, tensorflow::Tensor>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, tensorflow::Tensor> > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::vector<tensorflow::Tensor, std::__1::allocator<tensorflow::Tensor> >*) + 103
18  rotobotmaskrcnn.ofx           	0x000000013b805f21 drawMasks(OpenImageIO::v1_6::ImageBuf&, std::__1::unique_ptr<tensorflow::Session, std::__1::default_delete<tensorflow::Session> >&, OpenImageIO::v1_6::ImageBuf&, bool) + 6737
19  rotobotmaskrcnn.ofx           	0x000000013b8038af RotobotMaskRCNNPlugin::computeMask(OFX::Image*) + 3823
20  rotobotmaskrcnn.ofx           	0x000000013b80234c RotobotMaskRCNNPlugin::setupAndProcess(ImageScalerBase&, OFX::RenderArguments const&) + 1628
21  rotobotmaskrcnn.ofx           	0x000000013b808df9 RotobotMaskRCNNPlugin::render(OFX::RenderArguments const&) + 409
22  rotobotmaskrcnn.ofx           	0x000000013b84b2a6 OFX::Private::renderAction(OfxImageEffectStruct*, OFX::PropertySet) + 86
23  rotobotmaskrcnn.ofx           	0x000000013b84750b OFX::Private::mainEntryStr(char const*, void const*, OfxPropertySetStruct*, OfxPropertySetStruct*, char const*) + 9115
24  rotobotmaskrcnn.ofx           	0x000000013b81b970 OFX::FactoryMainEntryHelper<RotobotMaskRCNNPluginFactory>::mainEntry(char const*, void const*, OfxPropertySetStruct*, OfxPropertySetStruct*) + 320
25  libnuke-10.5.7.dylib          	0x000000010702e9ac OFX::Host::ImageEffect::Instance::mainEntry(char const*, void const*, OFX::Host::Property::Set*, OFX::Host::Property::Set*) + 284
26  libnuke-10.5.7.dylib          	0x000000010703072d OFX::Host::ImageEffect::Instance::renderAction(double, std::string const&, OfxRectI const&, OfxPointD) + 973
27  libnuke-10.5.7.dylib          	0x0000000106b25e21 Nofx::NofxIop::renderBlock(unsigned long, DD::Image::ChannelSet const&) + 1393
28  libnuke-10.5.7.dylib          	0x0000000106b21df1 Nofx::NofxIop::engine(int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 657
29  libDDImage.dylib              	0x000000010850873c DD::Image::Cache::fill_line(int, DD::Image::Row&, DD::Image::ChannelSet const&, DD::Image::ChannelSet&, DD::Image::ChannelSet&) + 556
30  libDDImage.dylib              	0x0000000108508d78 DD::Image::Cache::load_line_incr(int, DD::Image::Row&, DD::Image::ChannelSet const&, DD::Image::ChannelSet&, DD::Image::ChannelSet&) + 488
31  libDDImage.dylib              	0x00000001085a5fef DD::Image::Iop::get(int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 1519
32  libDDImage.dylib              	0x0000000108631757 DD::Image::PixelIop::engine(int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 183
33  libDDImage.dylib              	0x000000010850b079 DD::Image::Cache::run_engine(DD::Image::Iop*, int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 137
34  libDDImage.dylib              	0x00000001085a6225 DD::Image::Iop::get(int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 2085
35  libDDImage.dylib              	0x0000000108631757 DD::Image::PixelIop::engine(int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 183
36  libDDImage.dylib              	0x000000010850b079 DD::Image::Cache::run_engine(DD::Image::Iop*, int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 137
37  libDDImage.dylib              	0x00000001085a6225 DD::Image::Iop::get(int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 2085
38  libDDImage.dylib              	0x0000000108631757 DD::Image::PixelIop::engine(int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 183
39  libDDImage.dylib              	0x000000010850b079 DD::Image::Cache::run_engine(DD::Image::Iop*, int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 137
40  libDDImage.dylib              	0x00000001085a6225 DD::Image::Iop::get(int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 2085
41  libDDImage.dylib              	0x0000000108631757 DD::Image::PixelIop::engine(int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 183
42  libDDImage.dylib              	0x000000010850b079 DD::Image::Cache::run_engine(DD::Image::Iop*, int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 137
43  libDDImage.dylib              	0x00000001085a6225 DD::Image::Iop::get(int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 2085
44  libDDImage.dylib              	0x0000000108631757 DD::Image::PixelIop::engine(int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 183
45  libDDImage.dylib              	0x000000010850b079 DD::Image::Cache::run_engine(DD::Image::Iop*, int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 137
46  libDDImage.dylib              	0x00000001085a6225 DD::Image::Iop::get(int, int, int, DD::Image::ChannelSet const&, DD::Image::Row&) + 2085
47  libnuke-10.5.7.dylib          	0x0000000106f954bd Viewer_Image::thread_proc() + 813
48  libnuke-10.5.7.dylib          	0x0000000106f95186 Viewer_Image::_thread_proc(void*) + 6
49  libnuke-10.5.7.dylib          	0x000000010668cc0e thread_func(unsigned int, unsigned int, void*) + 14
50  libDDImage.dylib              	0x0000000108690ad8 DD::Image::thread_proc(void*) + 120
51  libsystem_pthread.dylib       	0x00007fffc282093b _pthread_body + 180
52  libsystem_pthread.dylib       	0x00007fffc2820887 _pthread_start + 286
53  libsystem_pthread.dylib       	0x00007fffc282008d thread_start + 13

To see the expected result please visit:

https://www.linkedin.com/feed/update/urn:li:activity:6403107128722190336

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions