Add ability to customize task repository

[marcellustavares]

hey @kagkarlsson, thanks for the project!

I was wondering what you think about adding the ability to customize task repositories so we can switch to non-JDBC compliant stores.

Cheers

[kagkarlsson]

Sorry, missed replying to this one.

I think that might be possible, but that task repository would need to provide the same guarantees as the jdbc one. We are currently using the version field for optimistic locking, for example to ensure that only a single scheduler will be able to "lock" an execution. But it should be possible to implement similar guarantees in other stores..

[kagkarlsson]

What store did you have in mind?

[marcellustavares]

yea, i thought about elasticsearch initially.

[kagkarlsson]

I think this might be doable. The TaskRepository would have to be a bit formalized and come with a contract that needs to be fulfilled to work properly with the Scheduler. The contract would be tested by a suite of integrationtests run against a testcontainer-instance of the data store in question.

[rafal-kowalski]

Hi, I would really like to have similar feature. I need to override JdbcTaskRepository.getDue and explicitly add limit to sql statement, due to how Postgres planner works internally, now if there is a lot of due tasks (millions) it uses sequential scan instead of index, because query is suppose to return all of the rows. With explicit limit in that case it will use index because it knows that we need only a small subset.
I could make a PR, but I guess I need some input from you how to approach TaskRepository contract. Do you have some idea in mind?

[kagkarlsson]

Hi, I would really like to have similar feature. I need to override JdbcTaskRepository.getDue and explicitly add limit to sql statement, due to how Postgres planner works internally, now if there is a lot of due tasks (millions) it uses sequential scan instead of index, because query is suppose to return all of the rows. With explicit limit in that case it will use index because it knows that we need only a small subset.
I could make a PR, but I guess I need some input from you how to approach TaskRepository contract. Do you have some idea in mind?

Hi! This is a bit of a different use-case, would you please raise this in a separate issue? 😄

[kagkarlsson]

No description provided.

[kagkarlsson]

Added an issue to track this feature, #173