Collect error rate data for tools

[abretaud]

It would nice to identify tools that are probably bugged: we can (somehow) look into the job exit codes on different usegalaxy.* servers, and tag tools with a "to fix" label when we see most of the jobs are failing.

I guess a tool can fail on a specific server and work fine on another one, not sure how to handle that (only tag when it fails everywhere? warn specific server admin where it fails?)

I recall @neoformit had some magic sql queries to identify problematic tools, maybe it could be used somehow?

[neoformit]

We did work on these scripts that enumerate jobs states and perform some crude clustering in an attempt to enumerate each tool-error. Maybe you can find them useful:

https://github.com/usegalaxy-au/tools-au/tree/master/scripts/broken-tools (check the README)

After some discussion at last GCC it seemed like the Galaxy Sentry server should be taken advantage of for this purpose. It seemed like modifying the sentry handler a little could provide most of the functionality above.

[paulzierep]

Thanks for the hint @neoformit that looks very interesting !!