Only log warnings when BCD is known to be wrong and not fixable.

[foolip]

This is restating a problem with the update-bcd script in foolip/mdn-bcd-collector#571, but also suggesting a specific solution.

The current design of the update-bcd script is, somewhat simplified:

• Build a "support matrix" mapping a BCD entry + browser release to a support status (true/false/null) (this also resolves contradictory results for a single BCD entry)
• For every BCD entry:
  • Infer a BCD-style support statement from the per-version data in the support matrix. Where we have incomplete information, ranged (≤) versions are used.
  • Update the existing BCD entry using the inferred entries. For some complex scenarios a warning is logged and the entry is skipped.

The warnings and how to address them are my concern. Sample warnings:

warn: api.AbortController skipped for safari due to multiple default statements
warn: api.ApplicationCache skipped for chrome due to added+removed statement
warn: api.BeforeInstallPromptEvent.userChoice skipped for firefox due to multiple inferred statements

Currently, npm run update-bcd logs 1426 such warnings.

---

I think this approach would be more robust:

• Build the support matrix (same as current, but any representation of all the test results)
• For every BCD entry:
  • Iterate the true and false (not null) support claims for the BCD entry in the support matrix. Test each of those against the current BCD data and see if there is any contradiction. If there is not, continue to the next BCD entry.
  • If there was a contradiction, proceed to update the existing support statement to resolve it. Initially this can simply be the existing code.
  • Look for contradictions between the BCD entry and the support matrix again. If there's still a contradiction, log a warning and a reason.

This will probably remove a lot of the current warnings where there is no problem, and creates a burndown list of verified contradictions that need to be address, either by improving the script, or manualy.

[ChrisC]

@foolip I made good progress today on implementing these suggested changes and should have a draft PR very soon. However, I just want to confirm that you're not suggesting any changes to the way the Support Matrix currently gets built?

I think this approach would be more robust:

• Build the support matrix (same as current, but any representation of all the test results)

If you are suggesting some change to the support matrix, can you say a bit more?

[foolip]

@ChrisC I'm not suggesting any changes, but at the back of my mind I was wondering how much work could be deferred until after we know there's a contradiction to resolve.

Building an inferred support statement for a single entry could be deferred, I'm quite sure.

But building the support matrix can resolve some contradictions in the test results (differences between platforms) and deferring that would require other changes too. So not for this issue.

Also, if you have ideas for further improvement while working on this, please do share!