Parallelism in metadata export with a Task Queue

[adrianq]

• Parallelize calls to groups and groups sets up to a max number

@tokland said:

There are some Promise.all(...) in the code, a general comment about it.

When we have limited/known number of requests in a Promise.all, it's ok. However, for size-unbounded requests, it can be dangerous, DHIS2 servers tend to reply with 500 when they get many parallel requests.

As you say in a comment, if we want efficiency, we will need a pool of promises (Bluebird provides one, with a {concurrency: N} option for Promise.map). A simple option is to use a sequential promise mapper. For example, I use this in our projects:

https://gist.github.com/tokland/71c483c89903da417d7062af009da571

Your code needs to be refactored a little bit to use the abstraction, you don't map async promises but the arguments (non-async) that will be used to create the promise. This is the only way, as when a promise is created, it starts doing work.

The good thing about using this abstraction is that whenever we want we can replace it for a concurrent implementation without changing the caller code, just the function in the utils file.

All this is a bit tricky, we can talk about it if you have any doubt.