Ruby google-protobuf linux precompiled gem crash with Segmentation fault on linux-musl · Issue #16853 · protocolbuffers/protobuf · GitHub
Skip to content

Ruby google-protobuf linux precompiled gem crash with Segmentation fault on linux-musl #16853

Closed
Closed
Ruby google-protobuf linux precompiled gem crash with Segmentation fault on linux-musl#16853
Assignees
JasonLunn
Labels
packaging & distributionplatform relatedAny issue releated to specific platform or OSrubywait for user action

Description

@ntkme
ntkme
opened on May 15, 2024

What version of protobuf and what language are you using?
Version: 3.22.2 or higher on aarch64-linux-musl / 4.29.3 or higher on x86_64-linux-musl
Language: Ruby

What operating system (Linux, Windows, ...) and version?
Alpine Linux any version

What runtime / compiler are you using (e.g., python version or gcc version)
Any ruby version

What did you do?
Steps to reproduce the behavior:

/ # gem install google-protobuf
Successfully installed google-protobuf-4.26.1-aarch64-linux
1 gem installed
/ # ruby -e "require 'google/protobuf'"
<internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': cannot load such file -- google/protobuf_c (LoadError)
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf_native.rb:15:in `rescue in <top (required)>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf_native.rb:12:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:57:in `<module:Protobuf>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:15:in `<module:Google>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:14:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:135:in `require'
	from -e:1:in `<main>'
<internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': Error loading shared library ld-linux-aarch64.so.1: No such file or directory (needed by /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/3.3/protobuf_c.so) - /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/3.3/protobuf_c.so (LoadError)
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf_native.rb:13:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:57:in `<module:Protobuf>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:15:in `<module:Google>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:14:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:135:in `require'
	from -e:1:in `<main>'
<internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': cannot load such file -- google/protobuf (LoadError)
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from -e:1:in `<main>'

What did you expect to see
google-protobuf should work on linux-musl just like how it works on linux-gnu

What did you see instead?
google-protobuf prebuilt library is strictly linked to glibc and it's not compatible with musl-libc

Anything else we should know about your project / environment
Please see rake-compiler's documentation regarding recent changes in rubygems in regarding to -linux, -linux-gnu, and -linux-musl platforms: https://github.com/rake-compiler/rake-compiler-dock?tab=readme-ov-file#linux-gnu-and-musl-important-details

Activity

  1. ntkme
    added
    untriagedauto added to all issues by default when created.
    on May 15, 2024

  2. ntkme commented on May 15, 2024

    @ntkme
    ntkme
    on May 15, 2024
    ContributorAuthor

    As a workaround, uninstalling the prebuilt gem and reinstall platform ruby to build from source fixes the issue:

    / # gem uninstall google-protobuf
Successfully uninstalled google-protobuf-4.26.1-aarch64-linux
/ # apk add alpine-sdk
...
/ # gem install --platform ruby google-protobuf
Fetching google-protobuf-4.26.1.gem
Building native extensions. This could take a while...
Successfully installed google-protobuf-4.26.1
1 gem installed
/ # ruby -e "require 'google/protobuf'"

    For bundler, the following in Gemfile would work:

    gem 'google-protobuf', force_ruby_platform: true if RUBY_PLATFORM.include?('linux-musl')

  3. ntkme commented on May 15, 2024

    @ntkme
    ntkme
    on May 15, 2024
    ContributorAuthor

    There are a few options here:

    1. Do not ship aarch64-linux gem, and let both aarch64-linux-gnu and aarch64-linux-musl platforms fallback to platform ruby and compile from source.
    2. Ship the current flavor, but label it as aarch64-linux-gnu, so that it does not get installed incorrectly on aarch64-linux-musl.
    3. Upgrade rake-compiler, and ship separate aarch64-linux-gnu and aarch64-linux-musl gems.

    Again, please refer to rake-compiler's documentation on details: https://github.com/rake-compiler/rake-compiler-dock?tab=readme-ov-file#linux-gnu-and-musl-important-details

  4. JasonLunn
    added
    ruby
    on Oct 1, 2024

  5. JasonLunn commented on Oct 1, 2024

    @JasonLunn
    JasonLunn
    on Oct 1, 2024
    Member

    @dazuma - what's the expected platform support for linux-musl?

  6. JasonLunn
    added
    platform relatedAny issue releated to specific platform or OS
    and removed
    untriagedauto added to all issues by default when created.
    on Oct 1, 2024

  7. ntkme commented on Oct 15, 2024

    @ntkme
    ntkme
    on Oct 15, 2024
    ContributorAuthor

    As said early, even if we don't "officially" support linux-musl, we can avoid it from breaking out of box. - Just ship the gem with label -linux-gnu instead of -linux using latest rake-compiler.

  8. henrahmagix commented on Jan 10, 2025

    @henrahmagix
    henrahmagix
    on Jan 10, 2025

    I'd really appreciate if @ntkme's comment could be actioned pls ☺️

    ship the gem with label -linux-gnu instead of -linux using latest rake-compiler

  9. ntkme
    changed the title [-]Ruby google-protobuf-4.26.1-aarch64-linux precompiled gem is not compatible with linux-musl[/-] [+]Ruby google-protobuf linux precompiled gem is not compatible with linux-musl[/+] on Jan 10, 2025

  10. ntkme commented on Jan 10, 2025

    @ntkme
    ntkme
    on Jan 10, 2025
    ContributorAuthor

    This issue used to be aarch64-linux-musl only.

    However, with 4.29.3 release, that x86_64-linux prebuilt now triggers Segmentation fault on x86_64-linux-musl:

    This issue can be worked around by force compiling from source by adding the following in Gemfile:

    gem 'google-protobuf', force_ruby_platform: true if RUBY_PLATFORM.include?('linux-musl')

    @JasonLunn I highly suggest that we take action now to update rake-compiler and split linux-gnu and linux-musl gems as soon as possible.

  11. ntkme commented on Jan 10, 2025

    @ntkme
    ntkme
    on Jan 10, 2025
    ContributorAuthor

    The statistics from my own gem that has 10M+ total download shows that roughly 10% of x86_64-linux are from linux-musl, which is definitely not a small number. E.g. As of writing, the download counts for the latest version show:

    The statistics for google-protobuf might be different, but the total download for google-protobuf is way higher that even only 5% would affect tons of users. I know lots of users in ruby community uses docker.io/library/ruby:alpine image, some even use alpine for production, thus it's critical to not break linux-musl out of box even if we decide to not ship linux-musl prebuilt - an error from bundler asking user to install a compiler is way better than seeing Segmentation fault.

  12. ntkme commented on Jan 10, 2025

    @ntkme
    ntkme
    on Jan 10, 2025
    ContributorAuthor

    I was probably the earliest adaptor of splitting linux-gnu and linux-musl prebuilt, as I introduced it way before rake-compiler added support, using my own custom cross-platform build scripts. I have been through lots of trouble with rubygems/bundler's bug around this area in terms of platform detection and even helped fixing a few bugs in rubygems/bundler.

    From my own experience with it, I can say that as of now it's reliable with ruby >=3.1 and the default rubygems/bundler or higher version from them. In fact, nokogiri, one of the most downloaded gem of all time, introduced split -linux-gnu and -linux-musl with 1.18.0 release this month: https://rubygems.org/gems/nokogiri/versions
    sparklemotion/nokogiri@4c15c5c

    Now it's time to adopt -linux-gnu and -linux-musl gems.

  13. dazuma commented on Jan 14, 2025

    @dazuma
    dazuma
    on Jan 14, 2025
    Contributor

    I share the desire to split the builds into -linux-gnu and -linux-musl as soon as possible. However, we cannot until April 2025 because our support policy doesn't drop Ruby 3.0 until then. Once we're able to drop support for Ruby 3.0, we'll definitely split the builds with the next protobuf release.

  14. ntkme commented on Jan 15, 2025

    @ntkme
    ntkme
    on Jan 15, 2025
    ContributorAuthor

    @dazuma In fact shipping separate -gnu and -musl does not require dropping Ruby 3.0.

    The way it works is that you can set different ruby version and rubygems version requirements on the platform ruby gem and native prebuilt gems.

    For example:

    This allow older version of ruby and rubygems/bundler to install the gem by compiling from source. For google-protobuf, we can require 3.0 on the platform ruby gem, and require ruby 3.0 AND rubygems 3.3.22.

    You can learn more here about in this issue: rake-compiler/rake-compiler-dock#117 and why ruby 3.2 instead of rubygems 3.3.22 on native gem might be a better idea in this comment rake-compiler/rake-compiler-dock#117 (comment)

  15. 3 remaining items

  16. jeffque commented on Jan 31, 2025

    @jeffque
    jeffque
    on Jan 31, 2025

    The current suggested workaround had some problems to me, as in my CI the Gemfile.lock was frozen.

    For me, this was the workaround:

    gem 'google-protobuf', force_ruby_platform: RUBY_PLATFORM.include?('linux-musl')

    Pretty similar to the one suggested by @ntkme , but the gem is added regardless of platform. In my case, the gem was added as a transitive dependency from sass-embedded gem.

    Running with my workaround: https://gitlab.com/computaria/blog/-/jobs/9012424269

    Running with the workaround from @ntkme: https://gitlab.com/computaria/blog/-/jobs/9012354715

    Notice the error:

    Installing bundler 2.5.11
Bundler is unlocking ruby, but the lockfile can't be updated because frozen mode
is set
You have added to the Gemfile:
* google-protobuf
Run `bundle install` elsewhere and add the updated Gemfile to version control.

  17. ntkme commented on Mar 14, 2025

    @ntkme
    ntkme
    on Mar 14, 2025
    ContributorAuthor

    @dazuma April is coming soon. Can we formally put this on the roadmap of next v31 release?

  18. ntkme
    closed this as completedon Mar 14, 2025
  19. ntkme
    reopened this on Mar 14, 2025

  20. JasonLunn commented on Mar 17, 2025

    @JasonLunn
    JasonLunn
    on Mar 17, 2025
    Member

    Work to drop Ruby 3.0 and update to a newer bundler version is in progress, stay tuned for further updates.

  21. JasonLunn
    self-assigned this
    on Mar 17, 2025
  22. JasonLunn
    added
    packaging & distribution
    on Mar 17, 2025

  23. ntkme commented on Mar 18, 2025

    @ntkme
    ntkme
    on Mar 18, 2025
    ContributorAuthor

    I see that we merged the commit to drop Ruby 3.0. Next step would be update rake-compiler / rake-compiler-dock, so that we can relabel the -linux gem to -linux-gnu, and optionally publish a separate -linux-musl gem.

  24. ball-hayden commented on Mar 18, 2025

    @ball-hayden
    ball-hayden
    on Mar 18, 2025

    Could I please add a request that -linux-musl be published.
    Using Alpine as a base image is reasonably common (Alpine using MUSL rather than GNU).

  25. JasonLunn commented on Apr 21, 2025

    @JasonLunn
    JasonLunn
    on Apr 21, 2025
    Member

    Please try out the latest release, 4.31.0-rc1, and confirm that it has resolved this issue for you, @ntkme. The release includes binary packages for -gnu and -musl.

  26. JasonLunn
    added
    wait for user action
    on Apr 21, 2025

  27. ntkme commented on Apr 21, 2025

    @ntkme
    ntkme
    on Apr 21, 2025
    ContributorAuthor

    @JasonLunn Worked as expected!

  28. JasonLunn
    closed this as completedon Apr 21, 2025

  29. henrahmagix commented on Apr 21, 2025

    @henrahmagix
    henrahmagix
    on Apr 21, 2025

    @JasonLunn could we kindly get a v3.x.x release of the separate -musl binary if possible please, when it's out of release-candidate? Thanks in advance! 🤞

  30. ntkme commented on Apr 21, 2025

    @ntkme
    ntkme
    on Apr 21, 2025
    ContributorAuthor

    @henrahmagix Unfortunately, based on #16853 (comment) back-porting is probably not going to happen, because they cannot drop support of ruby <= 3.0 in a patch release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

packaging & distributionplatform relatedAny issue releated to specific platform or OSrubywait for user action

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions