job chat: save and report anonymised chat logs (opt-in)

[josephjclark]

If there's an opt-in flag on the payload, the job chat service should log/report/save the question and answer somewhere.

Questions and answers can be reviewed by openfn staff and used to train/improve/debug the system.

This must be opt-out by default. The payload must explicitly say "hey, it's cool to save and use my data". Of course it's client tools (like Lightning and the CLI) which are responsible for setting that flag.

An easy win for this is just to pipe questions and answers to slack. But its hard to manage threading and stuff.

User information should be removed from the question and answer. Should we save the project ID or anything? How anonymous do we need to be?

We could consider using questions and answers as a training set down the line (if we went in and optimised the answers manually).

Questions & Design

• Do we post/save the whole chat history? Or do we save a UUID which points to the previous message? Do we even know what the previous message is?

• Maybe each chat needs to generate a reproducible name - like the date and first sentence of the chat. That name is re-used and the chat is updated for every message.

• Is slack the right place? Good for live reviewing but awful for historical analysis. And is it hard to extract the data for training later? So maybe we post to slack and some database somewhere (probably lighting)