Managing large number of models

[rolisz]

I'm working on a project where we are using Spacy to extract information from documents. We are looking for different kinds of information, so we are using several Spancat projects and several textcat projects. On top of this, we want to solve the problem for several languages. I expect that for most languages, the workflows for projects would be quite similar.

What is the best way to handle this using Spacy projects? Create m (number of models) x n (number of languages) projects? Is there a way to reuse similar code between projects? Or to somehow parametrize projects and do something like spacy project run train_model_X english and then spacy project run train_model_X german?

[polm]

Projects are actually designed with this kind of abstraction in mind. There are two layers of abstraction it's helpful to know about:

1. project file overrides
2. config file overrides

These are quite similar, but they do technically happen in different places.

For project file overrides, you can make a vars block and then specify values there on the command line and interpolate them in your commands.

For the config file overrides, these are specified on the command line - in many of our tutorials we have arguments like --paths.train train.spacy, but --paths.train here isn't special, it's actually just looked up as a config path.

So putting this all together you can have bits like this in project.yml:

vars:
  # these can be default values
  lang: "de"

commands:
  - name: train
    help: "Train a model"
    script:
      - "spacy train configs/x.cfg --nlp.lang ${vars.lang}"

And then you can call spacy project run train --vars.lang en, or iterate over a variety of languages.

It's easy to miss, but this is mentioned in the documentation in this section.