track lowest allowed timestamp per persistence ID

[leviramsey]

References #108

Socializing this before writing tests, but the idea is:

• on replay, record the minimum acceptable next timestamp for this pid
• on write, check the desired timestamp (from InstantFactory.now()) against the acceptable timestamp, choose the later of the two. If the later one is the minimum acceptable, then update the minimum acceptable for next time.
• periodically, the minimum acceptable next timestamps are checked against InstantFactory.now() (local monotonic clock): earlier ones are evicted because monotonicity ensures that the next write will be after the minimum.

Net performance impact should be minimal:

• in a recovery, we'll only write for the last event: this write is done as a local ask: it would need a high level of contention to add noticeable latency to recoveries
• when writing events (more common), the check is done against a ConcurrentHashMap view, so should be fast. In the unlikely event we need to use a late timestamp than we want (viz. we've detected clock skew), there will be a local ask: this is like the recovery case, but happens more often, so it might have an impact but this should be "transitory": the minimum acceptable timestamp will advance by a microsecond per event, and it's unreasonable to expect thousands of events for a persistence ID per second (so InstantFactory.now() should be moving a few orders of magnitude faster than the minimum acceptable)
• contention for updates to minimum acceptable timestamp is ameliorated by partitioning by persistence plugin and slice range (with a number of slice ranges at least the detected number of CPUs)

[pvlugter]

Could be useful to start with just some detection, while this is under development and discussion.

[pvlugter]

May just be me, but the implementation looks more complex than what we need. Feel that it could be simpler, but will think through it some more too.

The journal is an actor, so could also track directly. Also see how writesInProgress are tracked in the journal implementation.

[patriknw]

Haven't looked at the details here. Would it be enough to increase the time in InstantFactory when detecting clock skew? At the point the warning is logged in #110

[pvlugter]

Yeah, was thinking the same. We could just bump in InstantFactory. That will then be monotonically increasing micros until the current time catches up and it reverts to regular timestamps again. And probably check against a configurable tolerance setting, so it can only be skewed by so much, otherwise error.