I try to load word vectors from a bin file in the directory but I get an error

[noam267]

I run the command in terminal in pycharm:
python -m spacy init vectors en PubMed-and-PMC-w2v.bin output/

and I get the error:
i Creating blank nlp object for language 'en'
[2022-01-18 13:22:13,399] [INFO] Reading vectors from PubMed-and-PMC-w2v.bin
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf2 in position 18: invalid continuation byte

do I need to change the encoding and how do I do that?

[pmbaumgartner]

Are you using the vectors listed here? http://evexdb.org/pmresources/vec-space-models/

One thing to try would be converting from the binary format to the text file format. You'll need gensim for this, according to this SO post.

from gensim.models.keyedvectors import KeyedVectors

model = KeyedVectors.load_word2vec_format('PubMed-and-PMC-w2v.bin', binary=True)
model.save_word2vec_format('PubMed-and-PMC-w2v.txt', binary=False)

Then, try the same command, but point it at the new .txt file, like so:

python -m spacy init vectors en PubMed-and-PMC-w2v.txt output/

I'm still downloading the vectors myself to see if this is indeed the solution, but I'm inferring from the docs that the txt format is correct from looking at the vectors_loc arg:

Location of vectors. Should be a file where the first row contains the dimensions of the vectors, followed by a space-separated Word2Vec table. File can be provided in .txt format or as a zipped text file in .zip or .tar.gz format.

[noam267]

Thank you so much. it works.

[sushantsp]

@pmbaumgartner Any ideaa how to download it fast ? it is taking days to download. If these vectors are present anywhere else, I can download from there too.