Build failure notification

[finestructure]

I'm relaying a feature request here that I've received to see if there's wider interest in supporting this use case.

Users have noticed that we're sometimes displaying build failures for their packages in our compatibility tests which covers platforms they're not testing for in their CI. While we don't aim to be a project's CI, we still do cover a wider range of platforms/Swift versions than most people will have set up and there is useful feedback we could provide in case of failures.

The problem is that unless users happen to check their package page routinely they won't notice these build failures. It would be interesting if we could provide feedback to the authors on build failures.

A difficulty on our end with this sort of feature is that we're not currently dealing with any sort of accounts on SPI. We'd like to avoid having to handle the technical and legal complexities of it for as long as possible.

However, it's been suggested that a field in .spi.yml where authors provide a feedback email could work around this.

I think that's a great idea. We might even start by supporting a social network like a private mention on Mastodon for example, because delivery would be much easier than via email in that case.

[finestructure]

One thing to bear in mind with this sort of feature is that it can turn spammy very quickly. In fact, unless this feature came with an include/exclude list it would be pretty much too spammy for almost all projects, given that we currently run 25 builds per version for a package.

So if you're only targeting, say, Swift 5.8 and 5.9, you'd get a lot of notifications unless you had a way to also say "ignore Swift < 5.8 builds".

[heckj]

While I love that this exists and provides the verification of compatibility with versions, I worry that the concept is stepping too far and encouraging people to rely on Swift Package Index for CI, which (as you know) is a WHOLE ball of wax. I guess I lean towards "If an author wants to know about a build, they're responsible for adding it into CI somewhere". There's fine arguments to be made for a public service CI for open-source, but that's also what GitHub and others provide - and require a LOT more people-effort to support over time.

If anything, I think a smoother path to seeing the results (as a package author) would provide the most benefit there, something designed up front to highlight errors and failures, or make it easy as an author to dig into that detail - but I'm feeling like it's not all that difficult today, either.

Notification of the documentation build failures might be my only exception to that thought, since the build results are directly hosted by the index, and it seems a bit more core (to me, anyway) than CI/build feedback in general.

In any case, if something like this was offered, I think it would be best exposed as an explicit "opt-in" from the author due to its potentially spammy nature.

[daveverwer]

I can see both sides of this. You’re right, @heckj, that we don’t want to be seen as a CI system, but I can also see it being a really nice feature for package authors to be notified if their package compatibility changes.

Thinking about this from a user perspective and also to avoid being seen as a CI system, I wonder if we should think about this as a notification when the overall compatibility matrix changes rather than thinking about this in terms of individual builds. So, maybe on a nightly basis, if a package has a different compatibility matrix than it did yesterday, that’s what triggers the notification. It would reduce noise and provide genuinely useful information.

I must admit, I’d rather we encourage package authors to set up their own CI, though. I wonder if the first step is to add some wording and maybe even some example GitHub workflow files to the maintainer’s page to make that process easier for package authors.