Support fetching source/files via additional methods (git, svn...) - not just HTTP and FTP

[dekz]

How do you feel about Open-uri support for installers, it seems currently http is assumed as the default. If open-uri support was accepted it should hopefully achieve a lot more easily (local files at least).

  source 'ftp://test:abcd.tar.gz'

This should also remove the dependency on wget.

[joshgoebel]

I'm aware of no issues with wget, it's freely available on most distros (if it's not there it's only an apt-get or rpm away) and I tend to trust it more than a Ruby built-in... although in 98% of cases I'm sure there is no effective difference. I think this is one of those things where if it isn't broken, don't fix it.

[dekz]

I didn't mean to imply there was something wrong with wget, it was more to do with allowing the source installer to handle everything open-uri can handle in ruby, rather than just http.

What I was originally attempting to solve was the fact that I wanted a :source installation of a local file, rather than a transfer with a custom install command. If that makes sense.

source '/tmp/jruby.tar'
source 'http://jruby.com/jruby.tar'
source 'ftp://jruby.com/jruby.tar'

Could all follow the same code paths as they all do the same thing, apart from the download_command.

[joshgoebel]

Can you explain a bit more how this would work? Ruby runs locally so using OpenURI it has to download a large file twice... once from the Internet locally (using open-URI) and then again it has to push that file to the server somehow (to every server in the case of multiple server)... many users have much slower up pipes than their servers have down pipes. So you could be downloading once and pushing a huge file to 20 servers. It sounds like a huge step backwards to me just to solve a small edge case. FTP and HTTP already work fine with wget.

It seems we should be trying to solve the edge case instead (Sprinkle already has all the tools to do so) rather than trying to change the entire process.

[joshgoebel]

The file has to come from somewhere though - it just can't really just magically exist on all your boxes (thinking in the bootstrap case)... but if you truly want to say "it's there, so deal with it" (assuming an existing state or some pre-boostrap not done with Sprinkle) then I'd suggest a small change to the source installer than detects a local path and in that case just cancels the download step and does everything else the same. (perhaps extract needs to change too if we're extracting the file in place to another location).

If that's the case I'd open a new issue for THAT specific problem and lets solve it.

[joshgoebel]

If what you're wanting to say is (as I originally understood) it's local on MY filesystem (vs the remote) then providing an exact path would be ambiguous... does it mean a local file, or remote file? So some further discussion is necessary there. My existing comment was thinking it referred to a REMOTE file... but doesn't fit the context of the whole discussion. Typically paths are assumed to be REMOTE unless you're in the context of a file or transfer installer - whose goal is to move files from local to remote... otherwise all paths, etc. should be thought of as references to local files on the remote server.

I guess what I'm saying above is I believe in clear separation of concerns... if you have a LOCAL local file you want installed via source you'd need to use a file or transfer installer to move that file to the remote and then we'd need an updated source installer that knows how to work with an existing file vs downloading.

[joshgoebel]

Ok, done writing book now. :)

[dekz]

Chapter 2: A more sensible explanation

You touched on what I was meaning in that last post. I didn't exactly mean use ruby and open-uri itself, but have a mechanism similar to detect which code path to follow based on protocols passed in. (http wget, local file, ftp, git?). This stems from my experiments with implementing version control system installers. I currently have a local branch with git and cvs support which then uses the underlying source installer with a custom 'download' command. I do perfer the push methodology rather than the pull methodology, ie transfer the file then call source install on local file.

This is the idea I was playing around with when it came to local source repositories, it may not be possible to easily give the remote machine credentials to pull from a repository, but I can alleviate that by transferring some local files and build them similar to source installer.

SO it was file:// protocol functionality support in :source which sparked the issue, then extrapolated to handling other protocols.

Hope this rambling made sense!

[joshgoebel]

Ok, I get the concept now... Sure, I could see some type of Fetcher class or module that download_commands and extract_commands delegated to. To allow for git, svn, etc. urls. I'd think the download/extract functionality has to be bundled since some tools they are one thing (git clone, etc).

Code is worth a 1000 words. If you have some to look at (even if it's rough, open a pull request).

[dekz]

Yeah will do mate. Most of these version control installers are simply extensions of source, with like I said, a custom downloader (git clone, cvs co).

Have you given much thought to exporting environment settings in a generic way? Such as having :env attribute down in source or installer itself? I'd like to hear your thoughts on having something like that which is propogated to every command. For example:

source 'http://a/b.tar' do
  env 'CC=gcc'
end

#...
# Would propagate to configure command and all other commands.
# cd /usr/local/build/ && CC=gcc ./configure
#...

[joshgoebel]

THREAD POLICE Please open another issue to discuss a different idea.

[joshgoebel]

Yeah... we'd want to abstract it more like the lines of your idea in the prior thread:

source "[email protected]:sprinkle-tool/sprinkle.git" do
end

And then that would simply fetch via git vs wget... wait, I guess git and svn don't really have their own distinct URL prefixes then I'd suggestion something along the lines of:

source "https://github.com/sprinkle-tool/sprinkle.git" do
   fetch_using :git, :fetch_submodules => false
end

If fetch_using isn't passed we can try and sniff out the best protocol and then decide which fetcher to use.

[joshgoebel]

Now that installers can run other installers this almost begs for a fetch installer that knows how to fetch things from various places (broken out):

package :mysql do
   fetch  "https://github.com/mysql/mysql-server.git", :into => download_dir, :via => "git"
   source "#{download_dir}/mysql-server/"
end

And then if source was asked to fetch something via github it would just do a pre { fetch }. The advantage being you could then use fetch as a standalone installer (vs some magic sprinkle internal)... perhaps to setup a box for development, etc.

[joshgoebel]

Something like this:

https://github.com/sprinkle-tool/sprinkle/tree/feature-fetch-installer

I don't have time to flesh it out right now but that's the general idea.