Support packages hosted on GitLab

[Sherlouk]

Add support for URLs in the PackageList to come from GitLab as well as GitHub.

---

Original issue text:

What limitations/problems are currently blocking from us adding packages that are hosted on other services (not GitHub)?

Here's some completely random libraries I've found that we may be able to test with:
Bitbucket
GitLab

[daveverwer]

I think it may be a little early to prioritise this. The more hosting providers we support early, the larger the changes we need to make when touching that part of the system and with the package registry potentially being a thing soon™️ this area of the system may go through changes.

I would definitely like to get to this one day, but it feels like it's a low priority for now. How do you feel about it @finestructure?

[Sherlouk]

Oh I definitely agree there Dave, this is low priority. I was more asking for my own understanding and so that we have it written down.

While it doesn't necessarily change our approach to the way we develop things, it doesn't hurt to have in the back of your mind approaches that could be avoided or done in such a strategic way that doesn't make this incredibly difficult in the future!

[daveverwer]

Tagging @Mordil here as he brought this up on Twitter recently and may potentially want to work on GitLab support.

Before it gets to PR, it's probably worth a quick discussion here of the approach with @finestructure involved.

[finestructure]

Whoops, I never actually replied to this thread it seems - apologies!

Just in terms of procedure in case someone were to tackle this: we've currently got three dependencies reaching out to Github (see AppEnvironment.swift):

    var fetchLicense: (_ client: Client, _ package: Package) -> EventLoopFuture<Github.License?>
    var fetchMetadata: (_ client: Client, _ package: Package) -> EventLoopFuture<Github.Metadata>
    var fetchReadme: (_ client: Client, _ package: Package) -> EventLoopFuture<Github.Readme?>

and they're all used in Intestor.swift. The three types Github.Metadata, Github.License, Github.Readme show what it is that we're pulling down.

It's not a whole lot but it would obviously have to be mapped to the repsective vendor APIs. I know Gitlab has a GraphQL API, so that should make it easier for that particular vendor.

We're also already talking to the Gitlab API for the build system but we'd need to be careful with respect to API keys and rate limits to ensure the two systems don't rate limit each other. (Although that's going to be mitigated by a lot fewer packages on Gitlab and on Github.)

None of this is terribly complicated but it's a fair amount of leg work to be done. In addition, there's a bit of operational risk if there were lots of Gitlab packages and API limits.

There's also the maintenance aspect to be kept in mind, both when it comes to additional properties we're pulling in and that need to be added across multiple providers as well as operationally when errors happen. (And Gitlab does have a bit of a history with instability...)

I think it'd be really important to get a feel for how many packages live on other hosting providers before embarking on this. And while I'm not a fan to have just a single provider, it would certainly make our lives easier if mirroring to Github was an option if it's just a few.

[daveverwer]

I'm going to rename this to "Add support for GitLab". That's a concrete thing we want to do and that's the most likely source of a new package right now, I know @Mordil has some packages on GitLab that he'd like to add.

Concrete tasks are more likely to get implemented, and we can add an issue for "Add BitBucket support" after this is closed.

[Mordil]

What's the high level of effort on this these days, now that a bunch of (awesome) features have been added to the Index?

It's becoming a question again after considering dropping the GitHub mirror for my GitLab packages: https://forums.swift.org/t/redistack-and-repository-mirroring/56518/10

[daveverwer]

Predictably, the situation has only got worse 😬 and we are more tightly coupled to GitHub than we were at the start. We definitely give consideration to this as we add features, but without a framework in which to cope with different source-code hosting platforms, more and more is coded specifically for GitHub.

We would still like SPI to support alternate source code hosting systems, but it's certainly not going to be a trivial task to implement and it's hard to make it a priority compared to some of the other things we have planned.

I think step one would be for someone to start to put together a definitive list of things that would need doing for multiple source-code hosting platforms to become a possibility. That way, we'd actually have a concrete list of things we need to do rather than "oh this is a hard task".

Is this feature something you'd be interested in getting involved with or were you just looking to check in on it?

[Mordil]

Both, I suppose - depending on my availability and priorities against other things I need to do (like actually doing features in my own OSS projects 😅)

[daveverwer]

Well, we can certainly help if you want to take the lead on it. I think at the point that it's a concrete list of things it can become an issue rather than a discussion (we have an epic tag for that issue) and can then be split into smaller issues to organise the work on specific bits.