Notify library maintainers of failures

[abraham]

As a maintainer of a NPM package, I'd love to get notified when my releases break dependant projects.

My ideal flow would be something like this:

A new release abraham/nutmeg#241 gets made and is published.

Dependant projects using Renovate get update PRs

• Update dependency @nutmeg/cli to v0.11.0 abraham/github-repository#8
• Update dependency @nutmeg/cli to v0.11.0 abraham/twitter-status#21
• Update dependency @nutmeg/cli to v0.11.0 abraham/twitter-user#12

But one or more of the updates fail in CI.

A single comment gets added to abraham/nutmeg#241 listing all the projects that failed with the new release.

This would allow me as maintainer to fix a lot of new release errors really quickly.

Is this a feature that Renovate would be interested in providing? I'd be happy to help out with the implementation with some pointers of where to start.

[rarkins]

@abraham this is definitely a feature I'm interested in. You may also be interested in the discussion at https://github.com/renovateapp/renovate/issues/1453, but I don't consider this a duplicate because you have pointed out a valid use-case for post release notifications, which is probably a good start.

Let's start with discussing the requirements and "how it would work" and then I can tell you if I think the codebase is clean enough for you to work on a PR on your own or if I should get it started or at least refactor some things first!

[rarkins]

Some thoughts:

• Must be opt-in by library maintainers, not opt-out. Perhaps they wouldn't even need to be running Renovate themselves, so long as there's a way for them to indisputably have opted-in.

• No opt-in should be necessary for downstream dependents, if they are also a public package

• Skip private downstream repositories for now

• Probably need a way to help filter out "false negatives", e.g. you probably only want to be notified if Travis tests fail, rather than "any possible status check" they may have enabled (CLA checks, etc). Where/how to configure that though? Could be painful as a maintainer

• If their master was previously broken, you also probably don't care either?

• Need a way to filter out certain downstream projects? Perhaps configurable white list or blacklist?

• How to handle if it "goes green"? e.g. downstream admin retries same Travis build and it passes the second time

• How to handle if it is upgraded as part of a group? e.g. if someone has configured Renovate to upgrade all packages together in one PR weekly, you're probably best to ignore that repository completely or you get too many false negatives. But would you ignore all groups?

• Need a convention for how to determine which issue/PR to add the comment(s) in. I don't think everyone has a separate PR per-release, for example.

[abraham]

For filtering out "false negatives" I would start with a default allow list of something like CircleCI, TravisCI, and AppVeyor. As new CI services come up they could be added to the default. Having a config for a maintainer to override this in the future would be nice.

Tracking if a project is already red sounds like a pain. Maybe try and find out CI status of the parent commit?

If a commit goes green maybe leave the comment but strike the specific project from it.

Configuring Renovate to group upgrades is not the default so I would ignore those and then allow maintainers to opt-in if they decide it worth the noise. Although any reasonably used library would likely have a number of ungrouped downstream dependents so ignoring a couple wouldn't be that big of a deal.

It might be that in order for maintainers to take advantage of this feature they have to have separate PR for each release. Or if one isn't found create a new issue in the project. Maybe just create issues and ignore PRs on the upstream project altogether.

Otherwise it all sounds good to me.