Add support for platform-dependant pre-compiled gems #68
Conversation
Allows the source to contribute parameters outside of the "source"
lavoiesl
changed the title
Allows to tweak the version depending on the platform
Only mock the fetching part such that the internal logic is tested
Knowing the exact platform of a compiled gem is important to download the correct gem and because different binaries have different hashes. Note that this change makes the gemset platform dependant.
lavoiesl
force-pushed
the
seb-platform
branch
from
89e56d5
to
b86d7c5
Compare
|
Is there anything else I can do to help support this PR? This is blocking our use of bundix |
|
I don't think we can have platform-dependent gemsets. What do you
envision a workflow looking like in that case? You have to run Bundix
seperately on each platform you might want to build on?
|
|
Correct. It's worth nothing that the gem's platform is tied to the ruby platform, not the host platform. So, if two people are on darwin and are working on a project that mandates using a particular ruby version, that ruby would come from Nix, compiled for a particular platform, so it will be consistent across devices. e.g. depending on ruby 2.6.5, which is compiled on x86_64-darwin-17 in Nixpkgs, would result in needing darwin-17 gems, regardless of the exact platform version (macOS version) of the user. Because of that, as long as devs are on the same platform family and are using the same ruby version, they can share a gemset.nix It is indeed unfortunate that this PR makes the gemset platform dependant if using platform dependant gems, but I feel it's a step in the right direction since currently, platform dependant gems are flat out not supported. |
|
It is indeed unfortunate that this PR makes the gemset platform
dependant if using platform dependant gems, but I feel it's a step in
the right direction since currently, platform dependant gems are flat
out not supported.
Oh, I see. What does this look like in Bundler if you have some users
on GNU/Linux and some on Darwin (or Musl/Linux, etc.)? I assume such a
thing is supported?
|
|
Yeah, it wouldn't be supported for now. Users could either generate the gemset themselves or maintain a per-platform gemset. In the future, we could change it such that a single gemset could hold multiple platforms, but this has the pitfall of having to detect all platforms and identify which ones are relevant to the user. There is also the added complexity of figuring out the correct platform to install within Nix. All that feels like a can of worms that I’d rather open on a later day. |
|
@lavoiesl do you know why given a gemfile like this: source 'https://rubygems.org' do
gem 'libv8'
end
I notice Gemfile.lock also doesn't mention darwin, which is guess is the real source of the problem: How is libv8 packaged differently compared to sorbet-static? |
|
Anyone still interested in this? Building on this PR, I've hacked together a proof-of-concept which allows for multi-platform gemsets here: 6a921d0 It generates gemsets like: {
libv8-node = {
groups = ["default"];
platforms = [];
source = {
+ platform = "ruby";
remotes = ["https://rubygems.org"];
sha256 = "0k2cqxnbm7lm284fa65bf4b18w7k6977mh6anyyai34r43g3vm34";
type = "gem";
};
+ nativeSources = [{
+ platform = "arm64-darwin-20";
+ remotes = ["https://rubygems.org"];
+ sha256 = "04qnpw0bm8j82km2whx5zhw59lm8isd3i540g1xfyc0mqw0vynhp";
+ type = "gem";
+ } {
+ platform = "x86_64-darwin-20";
+ remotes = ["https://rubygems.org"];
+ sha256 = "0r65dwxjrk6s9q4lf1jf7kg344qg3fn8d9r6nad91swmabpzd2bg";
+ type = "gem";
+ } {
+ platform = "x86_64-linux";
+ remotes = ["https://rubygems.org"];
+ sha256 = "04avzccvrjk4nbi7926cvbdkpkgc6a2m0gyz1s1k4x7q06lkdnnk";
+ type = "gem";
+ }];
version = "15.14.0.0";
};
}so Then I've tweaked nixpkgs' composeGemAttrs to look up the relevant source for the hostPlatform here: jdelStrother/nixpkgs@e7316af Still unresolved:
pkgs.bundlerEnv {
gemdir = ../.;
gemConfig = (builtins.removeAttrs pkgs.defaultGemConfig ["nokogiri" "libv8" "mini_racer"]);
...
}
{
ruby = {
nokogiri = {
dependencies = ["mini_portile2"];
source = { ... source for the ruby version of the gem ... };
};
mini_portile2 = { ... };
};
x86_64-linux = {
nokogiri = {
dependencies = [];
source = { ... source for the x86-linux version of the gem ... };
};
};
x86_64-darwin19 = {
nokogiri = {
dependencies = [];
source = { ... source for the x86-darwin version of the gem ... };
};
};
}It would lead to a gemsets being much bigger though. If we took that approach, it seems like you might as well just have multiple gemsets.nix - forget about my |
|
I was naively thinking about the gemset per arch approach too.
Edit: We switched to using this PR and generating a gitignored gemset.nix. |
|
So, it looks like currently there are two approaches –
But there is another possibility – generate different gemsets for different platforms. That way @omnibs will be able exclude from gitignore. Any thoughts? |
|
IMO, we should generate a separate gemset file for each platform. The mess of files it'll generate is a little inelegant, but it's conceptually simpler than my universal gemset approach, and doesn't require changes to nixpkgs. |
|
Still interested in progress here. |
|
There really is nothing preventing you from just using different So generate that per platform (admittedly not particularly elegant) and pass different gemsets to the |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
|
A bit late to the scene (week 2~3 learning Nix). I had just published a flake that is built on top of @jdelStrother and @lavoiesl changes here. It supports native sources, among other things. |
|
@sagittaros would you be interested in becoming maintainer of bundix as well? I'm hardly doing anything with Ruby these days and feel we need someone with actual stake here. |
|
Thanks for the offer @manveru , I am unable to take this on now due to my lack of bandwidth. I am happy to help close a couple issues though. |
|
No worries, any help is better than the current state of affairs :) |
n.b. bundler config is workaround for nix-community/bundix#68
For example:
Problem 1 - download url
Trying to start a shell with that would result in the wrong gem url trying to be downloaded:
The URL should include the platform; namely https://rubygems.org/gems/sorbet-static-0.4.4821-universal-darwin-14.gem
This is what #67 was attempting to fix.
Problem 2 - sha256
The version recorded in the
gemset.nixends up being for whatever platform recorded was in theGemfile.lock. In practice, we need to download a gem that is compatible with our current platform.Knowing the exact platform of a compiled gem is important because we need the correct URL and different binaries have different hashes.
Changes in this PR
PR is broken down in smaller commits for easier review
Superseeds #67; this PR doesn't assume that the gem exists with a version for the exact local platform. Instead, it finds the first matching platform in available gems.
Testing the PR
(With Gemfile above)
Known caveats
bundix/cc @burke @Morriar