[WIP] Subscriptions & projection engine #213
Conversation
| ) { | ||
| } | ||
|
|
||
| public function handle(ProjectionId $projectionId, Checkpoint $initialCheckpoint): void |
There was a problem hiding this comment.
this method probably need a strategy to deal with consumer failures
There was a problem hiding this comment.
Hmm yeah, agree. Not sure what the best strategy would be 🤔
Initial thought would be to wrap the consumer->handle in a try-catch, and on failure commit the checkpoint at that point in time (before the message was processed) to the repository store.
Calculating the checkpoint would be hard/impossible though, due to the partitioning of subscribers.
Any ideas on what strategy would work best here?
There was a problem hiding this comment.
When I dealt with this, I stored the last successfully handled offset and then initiated a retry mechanism. We also have eventsauce/back-off, which can help with this, it supplies some (sophisticated) strategies for retries.
There was a problem hiding this comment.
Yeah, was planning to introduce back-off in the "getLock" part as well.
Only problem is calculating the checkpoint; If we have a stream:
1: A
2: B
3: A
4: B
4: A
5: A
And we partition by letter, so A and B are both different partitions. Than the projection would receive:
A,A,A,A as events with new checkpoint: offset: 5.
When applying the second event, something fails, how do we calculate the right checkpoint to store?
Two things I can think off:
- Drop the idea of partitioning
- Store a "sub offset" to the last checkpoint. If projection has subOffset, then we first need to deal with that. Gets the job done, but introduces complexity..
|
I'm wondering if the Checkpoint should contain the ProjectionId since it's always for that specific, releasing it would then only need to contain the checkpoint. WDYT, @Robertbaelde ? |
I see checkpoints as part of a subscription to a stream, and not per-se projection related. So those subscriptions could be used for other stuff than projections (Reactors, process managers, message publisher or other stuff). Happy to discuss further though. |
This PR introduces subscriptions and projections to EventSauce.
Subscriptions
Subscriptions are a subscription to a set of messages from the MessageStorage. The simplest form of subscription would be the offset subscription, returning all events since a specific offset. More complex subscriptions can be created, for example on aggregateId or on a specific partition of a stream.
Projections
Currently there are some message consumer interfaces and there is a replay functionality in EventSauce. However oftenly you'd want a projector that is autonomous, and keeps track of its own offset to the stream. This gives more flexibility with replaying projectors, and handling projections in parallel.
Todo: