add support for git repos

[jmcclelland]

I'm trying to add support for something like this:

"downloads": {
  "bar": {
    "url": "https://github.com/foo/bar@5a15e18a98e21c6e73b88f2af3114654dc613554", 
    "path": "web/extensions/bar", 
    "type": "git"
  }
}

I'm really not sure if this PR makes sense, so feel free to close if I'm way off base.

I'm actively working on converting my old drupal 7 drush-make build system do a drupal 8+ composer system and am struggling with converting a drush make file with about 40 or so CiviCRM extensions (most of them without a composer.json file) specified as git addresses plus git commits.

This composer plugin seems to be doing the same but for zip files.

Does adding git support make sense? Is there a better way to do what I'm trying for?

[jmcclelland]

Just for completeness, it seems one alternative to this patch is outlined in this blog in the section "Require Any Git Repository with Composer". In your project's main composer.json file you can specify something like the following:

{
  "name": "daggerhart/my-project-that-uses-composer",
  "repositories": [
    {
      "type": "package",
      "package": {
        "name": "daggerhart/my-custom-theme",
        "version": "1.2.3",
        "type": "drupal-theme",
        "source": {
          "url": "https://github.com/daggerhart/my-custom-theme.git",
          "type": "git",
          "reference": "master"
        }
      }
    }
  ],
  "require": {
    "daggerhart/my-custom-theme": "^1",
    "composer/installers": "^1"
  }
}

That should work for a git repo that does not have a composer.json file.

However, the composer documentation seems to discourage this approach and notes some limitations:

Note: This repository type has a few limitations and should be avoided whenever possible:

Composer will not update the package unless you change the version field.
Composer will not update the commit references, so if you use master as reference you will have to delete the package to force an update, and will have to deal with an unstable lock file.

[totten]

For both L71 and L54 -- I think git remote update is a newer command? We don't have a good way to control the git version here. It might be safer to use something like:

• git fetch <remote> <refspec>
• git fetch --depth 1 <remote> <refspec>

[jmcclelland]

Yup - makes sense.

[totten]

@jmcclelland That's a neat idea. The patch looks pretty clean. I think the patch is generally welcome.

I'm adding a couple comments / suggestions. One other general suggestion - there should be some kind of coverage in the test-suite. The smallest possible thing would be to update tests/SniffTest.php to include a git example.

Just for completeness, it seems one alternative to this patch is outlined in this blog in the section "Require Any Git Repository with Composer".

That's up to you. If I were you, I would decide based on the feelings about transitive dependencies:

• If it seems ideal for the related packages (daggerhart/my-custom-theme, et al) to have their own PHP dependencies, then go with your second example -- use a require statement and use the repositories section to backfill metadata. (The repositories may be long right now because you have to backfill composer data -- but in the future, that could be migrated into each repo.)
• If it seems ideal for the related packages (daggerhart/my-custom-theme, et al) to be fairly flat (with no transitive PHP dependencies), then go with extra.downloads. It should be simpler/faster.

Composer will not update the package unless you change the version field.
Composer will not update the commit references, so if you use master as reference you will have to delete the package to force an update, and will have to deal with an unstable lock file.

So one part sounds similar; another part sounds different.

• Similar: It sounds like they want a hard/concrete pin (version#/commit#/tag-name) rather than a soft/abstract one (branch-name). I believe that this guard encodes a similar expectation. And personally, I'm fine with that -- it means fewer edge-cases to think about.
• Different: To apply an update, it sounds like the repositories[type==package] approach requires you to update two fields (version and reference). The extra.downloads approach in the PR only requires you to update one field (url). That sounds better...

[totten]

I haven't tested, but I see these two code-paths (based on file_exists()), and it makes me imagine the following use-case:

1. In top-level composer.json, add extra.downloads definition with a git URL.
2. Run composer install.
3. In top-level composer.json, edit extra.downloads and change the git commit/tag.
4. Run composer install.

I guess the idea is that step 2 makes a new folder, and step 4 updates that same folder.

But what if that folder doesn't have any .git data? For example:

1. In top-level composer.json, add extra.downloads definition with a zip URL.
2. Run composer install.
3. In top-level composer.json, edit extra.downloads definition to a git URL.
4. Run composer install.

At step 4, shouldn't we clear out the folder before doing any git operations?

(IIRC, this only matters for extra.downloads in the top-level folder. In other contexts, there's a higher level rm -rf.)

[jmcclelland]

Thanks for the feedback! And yes, I agree, I think adding git support here is a better way forward than adding the alternative.

I can work on extending the test coverage.

Regarding the file_exists code paths - I added it without much thought in the interest of speed and efficiency (if we already have the directory in place and it's checked out to the same commit, we move on very quickly). However, it does sacrifice correctness in the use case you provide (and perhaps others we haven't even thought of).

I think, in the context of this extension, the .composer-downloads/<id>-<hash>.json is kinda like the composer.lock file. So, on composer install we could test for the existance of that file. If it doesn't exist and we have a folder, than rm -rf the folder and install from scratch. If it does exist, we could assume the contents properly describe the folder and run git fetch (if someone mucked with the folder git will return an error and that seems reasonable).

On composer update we would need to compare composer.json with .composer-downloads/<id>-<hash>.json. If we are moving from zip to git, then rm -rf the folder and start from scratch. If the repo URL changes, rm -rf and start from scratch, and otherwise git fetch.

Having spelled all this out, I'm leaning towards your initial suggestion of just rm -rf. I'm not sure all the added complexity is worth the small efficiency gain. Open to writing the more complex approach though if you prefer.