Gossip Reliability

[andydunstall]

Issue to track improvements to gossip reliability:

Netsplit Recovery

Each node should periodically gossip with seed nodes that it doesn't know about. This ensures that if there is a long partition between cluster nodes, once the partition recovers they'll re-discover each other.

Testing

Can extend piko test to support proxying traffic then drop messages, add latency, add partitions... Then make sure the cluster recovers (or use toxiproxy)