PoC for experimental async support

[kagkarlsson]

A variant of #304 which adds async-support without breaking anything. A bit hacky implementation since it relies on special interfaces and instanceof checking.

AsyncExecutePicked is just a modified duplication of ExecutePicked.

Current problems:

• We are not waiting for all CompletableFuture before shutdown. ExecutorService no longer handles this.
• ManualScheduler used in tests breaks now that everything is async. It relied on executeAnyDue running the same thread, i.e. blocking until execution is done. How to handle this now? ManualScheduler is used to clients as well.
• There are more test failures. Some concurrency tests seem to be struggling on GA. Needs to be investigated..

Options/thoughts:

• Change Task to return ExecutionHandler instead of implementing it. Might make it easier to add async-variants to existing builder-classes
• Could this work without changing interface of ExecutionHandler? What about an async-mode that uses CompletableFuture under-the-hood for execution and completion, similar to now?

[kagkarlsson]

For example, why is this happening:

2023-04-17T10:53:08.0880209Z 10:53:08.086 INFO  [main   ] com.github.kagkarlsson.scheduler.ClusterTest - Starting test_concurrency_optimistic_locking
2023-04-17T10:53:08.0999835Z 10:53:08.099 INFO  [main   ] c.g.k.scheduler.jdbc.AutodetectJdbcCustomization - Detected database PostgreSQL.
2023-04-17T10:53:08.1101036Z 10:53:08.109 INFO  [main   ] c.g.k.scheduler.jdbc.AutodetectJdbcCustomization - Using PostgreSQL jdbc-overrides.
2023-04-17T10:53:08.1162889Z 10:53:08.110 INFO  [main   ] com.github.kagkarlsson.scheduler.SchedulerBuilder - Creating scheduler with configuration: threads=10, pollInterval=0s, heartbeat=2s enable-immediate-execution=false, table-name=scheduled_tasks, name=scheduler1
2023-04-17T10:53:08.1169288Z 10:53:08.116 INFO  [main   ] com.github.kagkarlsson.scheduler.Scheduler - Using polling-strategy: type=FETCH, lowerLimit=0.0, upperLimit=30.0
2023-04-17T10:53:08.1174426Z 10:53:08.117 INFO  [main   ] c.g.k.scheduler.jdbc.AutodetectJdbcCustomization - Detected database PostgreSQL.
2023-04-17T10:53:08.1183535Z 10:53:08.118 INFO  [main   ] c.g.k.scheduler.jdbc.AutodetectJdbcCustomization - Using PostgreSQL jdbc-overrides.
2023-04-17T10:53:08.1192180Z 10:53:08.118 INFO  [main   ] com.github.kagkarlsson.scheduler.SchedulerBuilder - Creating scheduler with configuration: threads=10, pollInterval=0s, heartbeat=2s enable-immediate-execution=false, table-name=scheduled_tasks, name=scheduler2
2023-04-17T10:53:08.1197859Z 10:53:08.119 INFO  [main   ] com.github.kagkarlsson.scheduler.Scheduler - Using polling-strategy: type=FETCH, lowerLimit=0.0, upperLimit=30.0
2023-04-17T10:53:08.1202056Z 10:53:08.120 INFO  [main   ] com.github.kagkarlsson.scheduler.Scheduler - Starting scheduler.
2023-04-17T10:53:08.1311968Z 10:53:08.130 INFO  [main   ] com.github.kagkarlsson.scheduler.Scheduler - Starting scheduler.
2023-04-17T10:53:17.3675406Z 10:53:17.366 INFO  [hread-3] com.github.kagkarlsson.scheduler.Scheduler - Found dead execution. Delegating handling to task. Execution: Execution: task=Custom, id=7, executionTime=2023-04-17T10:53:08.138Z, picked=true, pickedBy=scheduler2, lastHeartbeat=2023-04-17T10:53:08.461Z, version=2
2023-04-17T10:53:17.3841535Z 10:53:17.366 INFO  [hread-3] c.g.k.s.t.DeadExecutionHandler$ReviveDeadExecution - Reviving dead execution: Execution: task=Custom, id=7, executionTime=2023-04-17T10:53:08.138Z, picked=true, pickedBy=scheduler2, lastHeartbeat=2023-04-17T10:53:08.461Z, version=2 to 2023-04-17T10:53:17.366Z
2023-04-17T10:53:17.3843200Z 10:53:17.367 INFO  [hread-3] com.github.kagkarlsson.scheduler.Scheduler - Found dead execution. Delegating handling to task. Execution: Execution: task=Custom, id=19, executionTime=2023-04-17T10:53:08.142Z, picked=true, pickedBy=scheduler2, lastHeartbeat=2023-04-17T10:53:08.462Z, version=2
2023-04-17T10:53:17.3844554Z 10:53:17.367 INFO  [hread-3] c.g.k.s.t.DeadExecutionHandler$ReviveDeadExecution - Reviving dead execution: Execution: task=Custom, id=19, executionTime=2023-04-17T10:53:08.142Z, picked=true, pickedBy=scheduler2, lastHeartbeat=2023-04-17T10:53:08.462Z, version=2 to 2023-04-17T10:53:17.367Z
2023-04-17T10:53:22.5735767Z 10:53:22.438 ERROR [hread-1] com.github.kagkarlsson.scheduler.ClusterTest - Error during completion!
2023-04-17T10:53:22.5738164Z com.github.kagkarlsson.scheduler.exceptions.ExecutionException: Expected one execution to be removed, but removed 0. Indicates a bug. (task name: Custom, instance id: 7)
2023-04-17T10:53:22.5738939Z    at com.github.kagkarlsson.scheduler.jdbc.JdbcTaskRepository.remove(JdbcTaskRepository.java:236)
2023-04-17T10:53:22.5739603Z    at com.github.kagkarlsson.scheduler.task.ExecutionOperations.remove(ExecutionOperations.java:43)
2023-04-17T10:53:22.5752377Z    at com.github.kagkarlsson.scheduler.task.ExecutionOperations.stop(ExecutionOperations.java:39)
2023-04-17T10:53:22.5753776Z    at com.github.kagkarlsson.scheduler.ClusterTest$RecordResultAndStopExecutionOnComplete.complete(ClusterTest.java:175)
2023-04-17T10:53:22.5754530Z    at com.github.kagkarlsson.scheduler.ExecutePicked.complete(ExecutePicked.java:104)
2023-04-17T10:53:22.5755576Z    at com.github.kagkarlsson.scheduler.ExecutePicked.executePickedExecution(ExecutePicked.java:88)
2023-04-17T10:53:22.5760758Z    at com.github.kagkarlsson.scheduler.ExecutePicked.run(ExecutePicked.java:68)
2023-04-17T10:53:22.5761274Z    at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
2023-04-17T10:53:22.5761857Z    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2023-04-17T10:53:22.5772872Z    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2023-04-17T10:53:22.5773360Z    at java.lang.Thread.run(Thread.java:750)
2023-04-17T10:53:22.5774129Z 10:53:22.439 ERROR [hread-1] com.github.kagkarlsson.scheduler.ClusterTest - Error during completion!
2023-04-17T10:53:22.5774856Z com.github.kagkarlsson.scheduler.exceptions.ExecutionException: Expected one execution to be removed, but removed 0. Indicates a bug. (task name: Custom, instance id: 19)
2023-04-17T10:53:22.5775578Z    at com.github.kagkarlsson.scheduler.jdbc.JdbcTaskRepository.remove(JdbcTaskRepository.java:236)
2023-04-17T10:53:22.5776214Z    at com.github.kagkarlsson.scheduler.task.ExecutionOperations.remove(ExecutionOperations.java:43)
2023-04-17T10:53:22.5776855Z    at com.github.kagkarlsson.scheduler.task.ExecutionOperations.stop(ExecutionOperations.java:39)
2023-04-17T10:53:22.5777508Z    at com.github.kagkarlsson.scheduler.ClusterTest$RecordResultAndStopExecutionOnComplete.complete(ClusterTest.java:175)
2023-04-17T10:53:22.5778136Z    at com.github.kagkarlsson.scheduler.ExecutePicked.complete(ExecutePicked.java:104)
2023-04-17T10:53:22.5778767Z    at com.github.kagkarlsson.scheduler.ExecutePicked.executePickedExecution(ExecutePicked.java:88)
2023-04-17T10:53:22.5779366Z    at com.github.kagkarlsson.scheduler.ExecutePicked.run(ExecutePicked.java:68)
2023-04-17T10:53:22.5780127Z    at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
2023-04-17T10:53:22.5791090Z    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2023-04-17T10:53:22.5791693Z    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2023-04-17T10:53:22.5792145Z    at java.lang.Thread.run(Thread.java:750)
2023-04-17T10:53:44.9971356Z 10:53:44.995 INFO  [main   ] com.github.kagkarlsson.scheduler.ClusterTest - Failed to execute all for 10s. ok=17, failed=0, errors=0
2023-04-17T10:53:44.9972044Z 10:53:44.995 INFO  [main   ] com.github.kagkarlsson.scheduler.Scheduler - Shutting down Scheduler.
2023-04-17T10:53:45.0186459Z 10:53:45.012 INFO  [main   ] com.github.kagkarlsson.scheduler.Executor - Letting running executions finish. Will wait up to 2xPT30M.
2023-04-17T10:53:45.1277434Z 10:53:45.121 INFO  [hread-3] com.github.kagkarlsson.scheduler.FetchCandidates - Scheduler has been shutdown. Skipping fetched due execution: Custom_205
2023-04-17T10:53:45.1278695Z 10:53:45.121 INFO  [hread-3] com.github.kagkarlsson.scheduler.FetchCandidates - Scheduler has been shutdown. Skipping fetched due execution: Custom_278
2023-04-17T10:53:45.1378678Z 10:53:45.121 INFO  [hread-3] com.github.kagkarlsson.scheduler.FetchCandidates - Scheduler has been shutdown. Skipping fetched due execution: Custom_279
2023-04-17T10:53:45.1379515Z 10:53:45.121 INFO  [hread-3] com.github.kagkarlsson.scheduler.FetchCandidates - Scheduler has been shutdown. Skipping fetched due execution: Custom_281
2023-04-17T10:53:45.1380230Z 10:53:45.121 INFO  [hread-3] com.github.kagkarlsson.scheduler.FetchCandidates - Scheduler has been shutdown. Skipping fetched due execution: Custom_280
2023-04-17T10:53:45.1380928Z 10:53:45.122 INFO  [hread-3] com.github.kagkarlsson.scheduler.FetchCandidates - Scheduler has been shutdown. Skipping fetched due execution: Custom_282

Possibly unbounded loop for fetch of executions to run?

2023-04-17T10:54:52.8425398Z [INFO] Results:
2023-04-17T10:54:52.8425995Z [INFO]
2023-04-17T10:54:52.8426285Z [ERROR] Errors:
2023-04-17T10:54:52.8439288Z [ERROR]   ClusterTest.test_concurrency_optimistic_locking » Completion java.lang.OutOfMe...
2023-04-17T10:54:52.8439737Z [INFO]
2023-04-17T10:54:52.8440000Z [ERROR] Tests run: 186, Failures: 0, Errors: 1, Skipped: 3