Lots of step-results can still cause deadlocks

[flosell]

Problem

PR #143 shows, writing a lot of step results (i.e. writing more events than can be processed by event-listeners), can still cause deadlocks, so clearly #135 and #140 have not entirely solved the problem.

Contributing Factors

• Reading and writing on the same event-bus from the same go-loop: All events write and read from the same event bus; if an event-listener gets stuck writing to the event-bus, it also can't read (and therefore unblock) the event-bus, causing a deadlock
• One shared event-bus: All steps and all builds share the same event-bus, so lots of steps/builds active at the same time increase the likelihood of the problem appearing
• Inheritance: Step-Result updates in a child can trigger a step-result update in a parent (e.g. chaining), duplicating the number of update events
• Slow/Blocking listeners

Work so far

• Large output in shell-script breaks pipelines in chaining-macro (since 0.9.4) #135, Reduce number of events generated by step updates #140: Reduce number of events sent on a per-step basis; if there are lot's of steps active in parallel (or in parallel builds), this doesn't help

Potential Workarounds

• Persistence: Decouple step-updates from flushing to disk to improve performance/resource usage #137: decouple flushing to disk as this is probably the most expensive/slow event listener
• Increase buffers to decrease deadlock probability #143: Increase the buffer for pushing step-result updates to pipeline state

Solutions

• Decouple topics from each other so a full topic does not affect our ability to write to another topic (see comments below)

[flosell]

After further investigation, reading/writing on the same event-bus seems to be the root cause, specifically, the publisher-channel that is shared by all topics and pipelines. This way, the following could occur:

1. Lots of :step-result-updated events clog the event-bus (i.e. the publisher-channel)
2. The pipeline-state-updater now tries to process such an event
3. After processing an event, the pipeline-state-updater tries to send a :step-result-update-consumedevent but is blocked because the event-publisher channel is full
4. Since the pipeline-state-updater is now blocked, no updates are processed anymore, resulting in a deadlock

I now rewrote the event-bus to use separate publisher-channels per topic. This way, components that subscribe to a topic and write to another should no longer deadlock.

Components that write to the same topic they read from (e.g. step result inheritance) could still be an issue. However, the fixes for #135 and #140 seem to have worked for now.

[flosell]

The new event-bus is still not active default (activate with :use-new-event-bus true).

Keeping this issue open until we have some evidence it's good in production and we can make it the default.

Also, the workarounds are probably still a smart idea to reduce overall resource consumption

[flosell]

Workarounds and new default have their own issues, closing this one.