Hydra integration with RabbitMQ Proposal

[disassembler]

Hydra Integration with RabbitMQ Proposal

This proposal is something IOHK is looking at doing to improve hydra notifications. It solves a number of our problems even in phase 1 of the proposal. I'm opening this up for community feedback before we start work on the tasks listed below.

Details of implementation

• RabbitMQ as a queuing system for hydra to send messages to
  • start of eval
  • end of eval
  • cached eval
  • start of build
  • end of build
  • cached build already completed
  • build failed
  • step starting/ending
  • improve metadata when builds finish
• Modify every build in eval running hydra-notify perl script to push messages RabbitMQ
• Write independent consumers that watch the queue and do something
  • GitHub Notifier
  • Docker Hub Image Push
  • Application Signing
  • SMTP notifier
  • Slack/Mattermost notifier
  • cachix push

Phases

1. RabbitMQ added to pipeline for notifications

• Convert hydra notify to a daemon that watches RabbitMQ for notifications
• Change perl/C code to not run hydra-notify and send messages to RabbitMQ
• implement notifications before/after eval and before/after builds matching regex

2. Rewrite portions of hydra-notify in something other than perl

• IOHK would rewrite GitHub and SMTP notifier code

3. Change polling/webhook plugins to be producers to RabbitMQ and hydra to watch for eval and build requests

• split perl script evaluator into 2 pieces.
  • One that receives events via polling repo status and publishes message to RabbitMQ
  • One that parses the output from RabbitMQ, runs evaluation and updates database
• Possibly rewrite one or both pieces as independent non-perl plugins

[edolstra]

The downside is that it adds another moving part (RabbitMQ). We're already using PostgreSQL's LISTEN/NOTIFY for notifications in some places (e.g. for communication between the web interface and the evaluator, and between the evaluator and the queue runner), so maybe that could be used instead? So consumers like the GitHub notifier would do listen build_finished; to be woken up when something happens.

[grahamc]

I am +1 in limiting the dependencies and infrastructure required for hydra. If there were a way to optionally hook in rabbitmq, that could be interesting.

Rabbitmq has an advantage over postgres events for some usecases, like streaming build logs: rabbitmq can easily handle many thousands of messages per second, and has a STOMP plugin shipped with core to send those build log messages to a web browser.

[edolstra]

PR #668 changes hydra-notify to be based on PostgreSQL notifications. Other services could use these notifications directly, or maybe use a bridge like https://github.com/subzerocloud/pg-amqp-bridge.