Subscription doesn't get notified of new elements when reconnecting too early on EventStore

[samidalouche]

Problem

When EventStore.JVM client subscribes too early before EventStore is fully functional (has built all of its indexes, etc), the subscription never gets notified of new additions to the stream.

The problem might be in the client, or might be in the server, but since I am not familiar with the internals, it is hard for me to gauge.

Versions

• akka:2.5.1
• akka-http:10.0.5
• eventstore-client:4.1.0
• eventstore:4.0.1

How to reproduce

See: https://gist.github.com/samidalouche/4eaeb891dcd9c51ad987c1f87386b6d2

1. Start a fresh EventStore instance
2. In parallel

• Execute ListenForChanges => will display just the 1,000,000th element to acknowledge it saw everything that was created by PushData
• Execute PushData => will create 1,000,000 elements

3. Stop EventStore
4. rm -rf /var/lib/eventstore/index
5. Start EventStore
6. Wait for EventStore to have recreated all of its indexes (tail -f /var/log/eventstore/DATE/*.log)
7. Execute PushData => will create another 1,000,000 elements in the stream

At that point, ListenForChanges should be displaying all of the new elements. But nothing happens.

Alternative execution paths:

• The code will work fine in the following cases if PushData is re-executed without steps 3 to 6 (restart EventStore and remove indexes)
• The code will generally (not sure how reliably) work fine if the indexes are not removed but EventStore is restarted. Note the generally, since I am quite sure I have observed the same issue in production without manually removing indexes (but EventStore might have decided to rebuild its indexes for some reason).

[t3hnar]

@samidalouche could you please check this with v4.1.1 ?

[samidalouche]

Let me check right now with 4.1.1, but I have seen this problem with 3.0.7, so I doubt it is a minor recent regression

[samidalouche]

OK, so something related to this problem definitely changed in 4.1.1

This is what I get now:

Exception in thread "main" eventstore.OperationTimeoutException: Operation hasn't got response from server for PackOut(SubscribeTo(Stream(mystream),true),858149ff-a163-4d30-8846-5806f7e8a7c7,Some(UserCredentials(admin,***)))

Is that what you expect?

That definitely solves the issue on our side, since we would retry on any exception. Thanks!

[t3hnar]

good. The issue still exists, however it is not silent anymore.