Install Prepacked Transformer model

[linasamnegard]

Hi!

I am trying to install prepacked transformer models from huggingface. I have spacy version 3.0.0rc5, pytorch 1.8.1 and have installed spacy transformers. When running the command python -m spacy download en_trf_bertbaseuncased_lg, I get an error message saying "No compatible package found for 'en_trf_distilbertbaseuncased_lg' (spaCy v3.0.0rc5)".

Anyone know what to do?

Thanks in advance!

[svlandeg]

It would be a good idea to upgrade spaCy to an actual release instead of the Release Candiate 5 - the current version is 3.0.6. After upgrading, you should be able to run

python -m spacy download en_core_web_trf

Let us know if you run into any more issues!

[linasamnegard]

Hi!

Thank you! I upgraded Spacy to 3.0.6, and I can run the line you wrote. However, I want to use a BERT model from huggingface, but it does not work install the prepacked models, such as en_trf_bertbaseuncased_lg. I get the same error message as before. Do you know why?

[svlandeg]

Happy to hear it works. For spaCy v2, we had a few different pretrained pipelines like en_trf_bertbaseuncased_lg and en_trf_robertabase_lg. For spaCy v3, we've reduced the transformer-based pretrained pipelines for English to just en_core_web_trf, which uses roberta-base.

If you want to use any other model from Huggingface, you can assemble your own config file, define the name of the HF model in [components.transformer.model], and train that on your specific component.

To get started with that in v3, have a look at these sections in the docs: https://spacy.io/usage/training#config and https://spacy.io/usage/embeddings-transformers#transformers. You can also run something like

spacy init config -p "ner" --gpu ner_gpu.cfg

to produce an initial config file with reasonable defaults - in this case the pipeline will include a transformer & ner component that you can train from scratch, and the transformer will build upon a pretrained HF model. Again, this is roberta-base by default, but you can change that in your config before training.

Not sure if that's helpful - let us know if you have more specific questions around usage of transformers in v3!

[linasamnegard]

Thank you for the answer!

We have tried it this way, with code from https://spacy.io/api/transformer:

import spacy
from spacy_transformers import Transformer
from spacy import displacy
nlp=spacy.load("en_core_web_trf", exclude=["transformer"])

name = "bert-base-swedish-cased-ner"
path = "C:\UserData\z0049ujz\Documents\bert-base-swedish-cased-ner"
config = {
"model": {
"@architectures": "spacy-transformers.TransformerModel.v1",
"name": "bert-base-swedish-cased-ner",
"tokenizer_config": {"use_fast": True}
}
}
trf = nlp.add_pipe("transformer", config=config)

We thought that the transformer pipe could be excluded and instead add the swedish bert model from Huggingface. Is this the correct way? It doesn´t work to call doc=nlp("...").

Thank you in advance!

[svlandeg]

I'm afraid this won't work. The pretrained pipelines like en_core_web_trf have a trained transformer component, which is being used by other components in the pipeline that were trained on top of it. Simply plugging in a new HF model, will break the whole pipeline.

If you want to use a different HF model, you'll need to create a pipeline from scratch. Have a look at the training & config docs I cited before :-)

[linasamnegard]

Thank you, now it worked after a second look on the links you sent before!

However, now I get a different error... I am trying to create a custom NER, and the nlp.update()-function throws error: Argument 'reference' has incorrect type (expected spacy.tokens.doc.Doc, got dict)

The code is:
nlp.begin_training()

Loop for 40 iterations

for itn in range(40):
# Shuffle the training data
random.shuffle(TRAINING_DATA)
losses = {}
for batch in spacy.util.minibatch(TRAINING_DATA, size=2):
for text, annotations in batch:
# create Example
doc = nlp.make_doc(text)
example = Example.from_dict(doc, annotations)
# Update the model
nlp.update([example], losses=losses, drop=0.3)

The training data is on the form TRAIN_DATA= ("Training Sentence", {entities: ( ), ( )})