Reviewing CI failures

[dabd]

Since the most common use case for snapshot testing is asserting on large values, what techniques do you use to communicate failures in CI ? Given the default setting for INSTA_UPDATE is auto, I suppose you have to replay the tests locally but what if you can’t reproduce it easily?

[mitsuhiko]

I never tried this but you could try to upload the .new snapshots into github artifacts and then download them. If that sort of workflow works, it could potentially be automated.

[max-sixty]

I'm not sure this would work, but writing an annotation or suggestion from GHA would be quite cool.

It would require parsing the insta output, and then writing it out as an error message annotation. Or writing a problem matcher, which would be easier but less featureful.

I don't think annotations support suggestions, but with some GHA permission wrangling, it could instead be written out as a comment containing a suggestion, which then someone could click "Accept" on in GitHub...

[dabd]

BTW, I was trying to understand in detail (from the documentation) the behaviour of each INSTA_UPDATE option.

Are the following flowcharts correct?

They highlight that the underlying workflows are not that simple (at least for me), for example Unseen and Always are potentially a bit like power user options, bypassing review in case of mismatches, since they write .snap files directly, instead of .snap.new files for subsequent review.

[max-sixty]

(not my library but responding regardless)

I think these are correct!

for example Unseen and Always are potentially a bit like power user options, bypassing review in case of mismatches, since they write .snap files directly, instead of .snap.new files for subsequent review.

That's correct — that's very much their intention IIUC. Personally, I generally use --accept, similar to always¹, and then use git to review changes.

Footnotes

1. these could do with a consolidation between env vars & flags; touched on in https://github.com/mitsuhiko/insta/issues/368 ↩