Alert when Orka cluster is unavailable

[finestructure]

The Orka cluster underwent maintenance but we're not getting any alerts about it being unavailable. We need to put a mechanism in place for the Orka orchestrator to report back connectivity issues.

[finestructure]

I think the only way this could work is if we add Prometheus metrics to the Orka driver and scrape it from the prod Prometheus.

[finestructure]

The problem with that is that the Orka orchestrator is spawned via Gitlab per job, i.e. it is not a long-running, scrapable process. We'd have to have it push metrics to a push gateway.

We do that already for the batch jobs like reconciliation and analysis but that push gateway is local to the docker swarm network and currently not accessible from outside the network.

We could make it available but then we'd need to deal with some form of authentication. It's probably better/easier to bring up another push gateway next to the Orka orchestrator that we then push to from the orchestrator and scrape from Prometheus.