Appveyor - Create Windows pre-compiled gems and test#854
Conversation
sodabrew
left a comment
sodabrew
left a comment
There was a problem hiding this comment.
This is awesome! A few q's and nits.
There was a problem hiding this comment.
Nit: remove test from the directory name - and more broadly, making clear with the file names which run the tests vs. which build the output gems.
There was a problem hiding this comment.
There are three 'code' files in that folder. One is utilities, one is the file for creating the gems, and one is for testing. That's where the files you mentioned in the next comment are placed.
The other items listed on lines 26 thru 31 are items created by running the build locally. I set it up so all the artifacts from a local build are untracked files (is that the correct term?).
| TESTOPTS: -v --no-show-detail-immediately | ||
| ri_file: none | ||
| matrix: | ||
| - ruby_version: "200" |
There was a problem hiding this comment.
Without the ruby_version matrix, what is the default Ruby that AppVeyor uses?
There was a problem hiding this comment.
The file win_gem_test/eventmachine.ps1, lines 16-20 defines the lowest ruby version and how trunk (with & without JIT) is handled.
See PR comment.
| # download shared script files | ||
| - ps: >- | ||
| if ( !(Test-Path -Path ./shared -PathType Container) ) { | ||
| $uri = 'https://ci.appveyor.com/api/projects/MSP-Greg/av-gem-build-test/artifacts/shared.7z' |
There was a problem hiding this comment.
This may not be hosted indefinitely. Is there another canonical source? I can set up another repo under github.com/eventmachine just to host these as Releases, for example.
There was a problem hiding this comment.
This may not be hosted indefinitely
In the future I suppose I could win the lottery...
Maybe you could fork it and run an Appveyor build on it to create the 7z file?
An example of why I thought it best that you use my repo is how about updates to OpenSSL packages? I can go into the repo, update the package urls, then test my fork of EM, Of course, I'd post an issue or contact you to let you know of the change.
|
I've done a lot of work with Windows Ruby. I've maintained the ruby-loco trunk build for well over a year, and this and other repos are using it for trunk testing. I believe it is the only fully tested trunk version available. I'm also somewhat active in Ruby Core, and occasionally test patches before they are committed. I've helped with Appveyor/Windows testing here, in Puma, and others. You've provided pre-compiled gems (without OpenSSL?); Puma has not. Both gems are very popular, and often used with Rails. Many users have issues compiling/building both with SSL. Since Windows Ruby versions use several OpenSSL versions/packages, there is a lot messy code for all the different OpenSSL packages, and the Appveyor files get pretty crazy. So, between the SSL issue, and the pre-compiled gem issue, I stepped back and thought that since compiling is needed for testing, why not build the gems at the same time? You asked about a version matrix. AFAIK, both Appveyor & Travis jobs are independent, so it's difficult to share files between them. Also, I've seen Travis spin up 5 jobs at once, Appveyor runs one job at a time. So, since the gem files need the *.so files from every Ruby version they work with, there are only two jobs, a 32 bit and a 64 bit. I don't know if you've looked much at the logs, but first the build system updates MSYS2/MinGW as needed, then it loops thru all ruby versions, compiles them, and saves the so files. It then builds the gems and saves them as artifacts. Lastly, the version loop starts again, dependent gems are installed/updated if needed, the gem is installed, and tests are run. Re testing, I've added code in the shared files to parse the test results and the code adds messages to the 'Messages' & 'Tests' tab of the job web page. ruby-loco has similar code parsing the tests, as every build is saved, but anyone using the build can be certain that the build url always returns the most recent good build, even if several of the more recent builds have failed... I expect to update the build/test framework as soon as new Ruby releases happen in December. Lastly, the idea of gem building/testing bounced around in my head for a while, and I worked with it using several extension gems. It does require changing the typical job layout that is used for Ruby CI testing. There are a lot of software packages that are built & tested on Appveyor. Maybe it's time for Ruby extension gems to do the same... Thanks, I hope this made sense, Greg |
|
Not that it's necessarily something you'd consider, but Puma decided to start using the framework for Appveyor CI, as it also compiles with OpenSSL. This build is using Ruby 2.0 forward. Windows Ruby builds from 2.3 and back use an older build system (MSYS) that is no longer maintained, and building them with the newer MSYS2 system would result in quite a support mess as far as packages, etc. Because of that, Ruby 2.0 & 2.1 use OpenSSL 1.0.1. Hence, I could update this to start with Ruby 2.2 (which uses OpenSSL 1.0.2) if you wish. Thanks, Greg |
|
Whatever you want to do here, I'm fine with. I would just like the EM pre-compiled gem built with OpenSSL. We can bring the three files in here and I can do PR's as I revise them. I set it up this way so you don't have to deal with it. Puma is using it, and they don't have to do anything to add Ruby 2.6... |
Would it simplify the code here to drop Windows Ruby 2.0 and 2.1, or even 2.2 as well? |
|
Not really. OpenSSL changes are not occurring for Window Ruby versions 2.3 and lower. It's written, it works, and meta-data in the code affecting 2.3 and lower is not changing. AFAIK, there are two popular gems built with OpenSSL, EM & Puma. Puma is using the same code, so I'd prefer to keep both identical, as it makes it easier for me to update/improve. At present, Puma is getting the 'shared' code from Appveyor, but it isn't an issue embedding it here. JFYI, EM has been using my trunk build since 2017-12-24. That was when I jumped into code with OpenSSL, other repos have been using it for longer. Although a year or more isn't that long, it still goes to "I'm not going anywhere"... Also, Puma is now testing against Ruby 2.6, and they didn't have change anything. If you were building a pre-compiled gem, that still requires gems/code and the build systems. That code has been around longer than this code, but this code integrates testing & gem building, uses the build systems on Appveyor, handles the various OpenSSL versions, and also tests against the gem, not the repo. Forgive me, but I'm not really sure what issues you have. If it would help, I can strip the non-windows specific changes in the fork branch and do a PR? That will get all the code for OpenSSL 1.1.1, test changes, etc into EM... https://github.com/MSP-Greg/eventmachine/commits/update-for-1.1.1-tests Thanks, Greg |
|
You've been exceedingly patient. Thank you. I'm working through landing this and getting 1.3.0 built. This PR doesn't have all of the commits that are on |
Great. I love it when a plan comes together...
Thank you. You're easy and fair to work with. Some maintainer/owners seem to expect contributors to totally re-write code in their coding style. As far as I'm concerned, if my code passes RuboCop and is green, that's about 90% of the work I'll do...
I'm home the evening. I've always been a theory guy, hence not the best recall. I may not be able to do a PR now, as the Hopefully I can cherry pick them once this is merged and create a new PR. That would be very quick, otherwise I can also create the PR with patch files... Is that ok? BTW, you've mentioned dropping Ruby versions. The last appveyor build took 47 minutes, that's 2.0 thru trunk. It is a long wait... |
|
Thanks. I was looking thru the code, and was reminded of a test skip that was added to the test shown below. There are other skips, but I think they're all 32 bit Windows, and I don't recall what versions they affected. The block for line 194 ( eventmachine/tests/test_attach.rb Lines 175 to 199 in c0c2b6f |
2 participants
This PR changes Appveyor CI to build pre-compiled Windows gems, then uses them for testing all Ruby versions. To use the build/test system, a new folder (win_gem_test) is added, which contains three files specific to EM. The packaged gems include a new rakefile and the test files.
All versions are built & tested with the correct version of OpenSSL.
The 2nd commit 'Update Tests for Windows', modifies current tests with 'pends' to allow passing on both 32 & 64 bit, using Ruby 2.0 thru trunk. Not being that familiar with EM, these should be reviewed.
Current Appveyor build on my fork is here, and passing.