backend choice with joblib multiprocessing

[fcharras]

The multi-processing example given is the repository or in the documentation seems to insist on using the multiprocessing backend.

It seems unclear as to why or what are the risks with the default loky backend, is this documented ? if not, could you elaborate on the amount of risks taken by using loky ?

Which page or section is this issue related to?

https://github.com/explosion/spaCy/blob/master/examples/pipeline/multi_processing.py#L42

[honnibal]

I didn't understand the problem very well, but it seemed that Loky was selecting a different pickling strategy that failed on our models. If you try out Loky and it works then feel free to use it. I'd like to understand the problem better and ideally get it fixed.

[fcharras]

I'm only using the tokenizer at the moment. It works with loky but there's a worrying (loky) warning stating that some queues were closed even though tasks were still being dispatched to them. (will paste if I encounter it again). However no batch of task seems to be lost.

[BramVanroy]

For reference: related issue #3193 that was indeed solved by switching to multiprocessing.

Not sure on which hardware @honnibal tested in that issue, but for me on Windows I had to use multiprocessing as well.