Language Code for languages not listed in ISO 639-1

[gurudevrao]

Currently spaCy uses the two letter ISO 639-1 language codes. However that list does not cover all the languages.

As more and more language models are added, we may come across situation where the language does not have a two letter code, or the model is trained on a specific dialect. So, I guess we should switch to ISO 639-3 language codes, or find some way to use both?

[ines]

Thanks for this issue – those are definitely valid points. I'll also go ahead and merge the Norwegian issue #2339 and the older issue #1308, because they're all related to the same considerations.

We should probably consider extending the two-letter language codes and allow using hyphenated forms like pt-br or even no-nb. This would also keep things flexible going forward – e.g. for when we add Chinese models or if a user wants to train more specific models and have en-us. (This shouldn't be a problem internally – but we might have to adjust the documentation in some places.) In cases like these, main module name could then be spacy.lang.pt, with two submodules pt and br.

[ines]

Quick upodate, copying over my reply from #4187:

Thanks everyone for your input! It does sound like this is still a bigger question, so I guess we'll hold off on this plan until we've solved the "sub-sub" language classes.

I do like the idea of no.nb or de.ch. There are two main questions here:

• Is this going to mess up any assumptions we currently make about nlp.lang and the downstream usage of it (like, model names etc)? I do think we just treat it as an arbitrary unicode sequence and dynamically import from "spacy.lang.{lang}", so it should all keep working as expected.
• How should we organise the file strucutre? In theory, we could just have a lang/no/nb/__init__.py defining a NorwegianBokmal class, and a lang/no/__init__.py that sets Norwegian = NorwegianBokmal (similar to what I did in the PR). But I am slightly worried about bloat and code duplication 🤔

[goodmami]

This issue is a bit old but I just stumbled upon it. If you haven't yet committed to a language code format, might I suggest against defining your own format and instead using a standard? BCP 47 is the format used in computing already (e.g., as the value of an xml:lang attribute in XML) and it has a specification (RFC 5646). It composes other existing standards, such as ISO 639-1 (alpha-2) and ISO 639-3 (alpha-3) codes for language names, ISO 3166-1 alpha-2 codes for countries/territories, and ISO 15924 for script names.

One big benefit is that you only need to be as specific as necessary, so if a language has an ISO 639-1 alpha-2 code and you don't care about the region, script, etc., you can just use the 2-letter code, meaning it would be backward compatible with spaCy's existing codes. There may be some edge cases: zh is a macro-language encompassing all Chinese languages (including Cantonese, Min Nan, etc.), and cmn is specifically Mandarin, cmn-Hans is Mandarin with simplified characters, cmn-Hant-TW is Mandarin with traditional characters as spoken in Taiwan, etc. Again, you only need to be as specific as is necessary for your data.

There are also good libraries for working with BCP 47; langcodes is one with a compatible license (MIT).