Background service improvements

[mratsim]

#136 is an initial implementation of Weave as a background service
with facilities to submit jobs from threads foreign to Weave.

Here are parts that were sacrificed and can be improved:

• The job queue is a MPSC queue accessible on Weave side by a single manager/root thread.

  • It can be working and prevents jobs from entering scheduling on steal request
    • This was already true and is not a problem if tasks are fine-graind or with loadBalance calls in jobs.
  • If all workers are sleeping and 10 tasks are enqueued, only the direct child threads will be woken up. But those will work on the stolen task instead of requesting more for their children. This is the same problem as load balancing on backed off child wake-up. #90 solved
  • Alternatively, the job queue can be made MPMC however MPMC queue design is hard, they also suffer from contention which hurts latency and may not be better than having a MPSC queue and a thread redistributing work.
  • Furthermore the manager thread may become dedicated to the following tasks in the future:
    • Distributed computing communication Distributed computing #73
    • Managing the GPU work Cuda scheduling #133 OpenCL scheduling #134
    • Managing the external job submission

• Foreign thread waitFor is implemented via exponential backoff

  weave/weave/datatypes/flowvars.nim

  Lines 234 to 249 in 42360c3

  	proc waitFor*[T](p: Pending[T]): T {.inline.} =
  	## Wait for a pending value
  	## This blocks the thread until the value is ready
  	## and then returns it.
  	preCondition: onSubmitterThread
  	var backoff = 1
  	while not p.isReady:
  	sleep(backoff)
  	backoff *= 2
  	if backoff > 16:
  	backoff = 16
  	let ok = p.fv.tryComplete(result)
  	ascertain: ok
  	cleanup(p.fv)

  . Instead it should use the event notifier to not use any CPU resource at all and save on energy. This would probably require a new flag in task/job and storing a pointer to the whole Pending instead of just to the channel. Hopefully checking this flag is costless due to branch prediction.

• the parallel_jobs and parallel_tasks module are 90% the same code. This should be factorized in parallel_macros

• There is no metrics and profiling for jobs

• There is almost no example for Weave as a background service

• There is no benchmark for Weave as a background service