spacy.cli.download doesn't work for transformer model

[Vasniktel]

How to reproduce the behaviour

1. Create a new virtual env (e.g. in pyenv).
2. pip install spacy[transformers]==3.7.2.
3. Start python REPL and:

import spacy
spacy.cli.download('en_core_web_trf')
nlp = spacy.load('en_core_web_trf')

4. At this point you'll get the following error:

ValueError: [E002] Can't find factory for 'curated_transformer' for language English (en). This usually happens when spaCy calls nlp.create_pipe with a custom component name that's not registered on the current language class. If you're using a Transformer, make sure to install 'spacy-transformers'. If you're using a custom component, make sure you've added the decorator @Language.component (for function components) or @Language.factory (for class components).

Apparently, spacy.cli.download installed curated-transformers as a dependency of en-core-web-trf but couldn't load it. Everything works fine if you reenter the REPL and try spacy.load again.

Your Environment

• spaCy version: 3.7.2
• Platform: macOS-14.0-arm64-arm-64bit
• Python version: 3.9.16
• Pipelines: en_core_web_trf (3.7.2)

[shadeMe]

Thanks for the report! Some context as to why this happens: The spacy[transformers] extra package only installs spacy-transformers whereas the models in 3.7.x depend on spacy-curated-transformers. So, the entry points in the latter are not initialized during the first import spacy call. The spacy.cli.download invocation installs the dependency, but it's too late to reload the entry points, which is why one needs to reload the interpreter/REPL.

We'll look into updating the error message to make it clarify this error case.

[Vasniktel]

Would it make sense to include spacy-curated-transformers into spacy[transformers] package for spacy>=3.7.x?

[adrianeboyd]

I am not sure it makes sense to include both in the same extra, since each package's requirements may diverge in the future and always trying to install both at the same time may lead to headaches.

[rjurney]

Was there a resolution to this? Do we just use spacy-curated-transformers?