GloVe Integration

[tmkontra]

Feature description

I am interested in integrating GloVe closer to the python ecosystem. There currently exists 2 public re-implementations of the GloVe algorithm in the Python runtime (pure python: https://github.com/hans/glove.py and cython: https://github.com/maciejkula/glove-python). I am interested in providing a thin Cython wrapper around the stanfordnlp distribution of glove (a C program).

My proposed implementation would refactor the extant source to accept files instead of stdin/stdout, and a call to my GloVe.train() class would return a Path object to the file-on-disk containing the word vectors. I think this would integrate nicely with the from_glove() method in spacy.

I am wondering:

• if this would be seen as valuable to the spacy ecosystem
• or if this has been tried before
• if there are obvious barriers to making this performant that I am missing, as I am relatively new to Cython. (i.e. if the stanfordnlp implementation of the algorithm is not entirely efficient)
• if there are suggestions for the interface

Could the feature be a custom component or spaCy plugin?

If so, we will tag it as project idea so other users can take it on.

I believe this would would make sense as a plugin, with an emphasis on compatibility with other scientific packages (numpy, pandas, nltk, etc)

[honnibal]

Hi @ttymck ,

Some disorganised thoughts on this:

I really like self-contained packages that provide C-extensions. So, your wrapper should include the C source files, which would be linked into the Cython interface. You can see an example of using static linking for this in my Blis wrapper: https://github.com/explosion/cython-blis .

The other question around this is, why GloVe as opposed to FastText? One thing I really like about GloVe is the two phase process, where first we count and then the algorithm runs. I think there's always been a huge missed opportunity around this. We should be using approximate counting, so that we can count a huge text corpus with a limited memory budget. RaRe's bounter library provides good approximate counting: https://github.com/RaRe-Technologies/bounter

The other thing that would be cool to get right is the text preprocessing, and integration between that and spaCy. Especially for more exotic preprocessing, like merging phrases, lemmatization, etc. I don't know exactly what the solution to this should look like.

[BramVanroy]

As a user, I would be very interested in a package like this.

[tmkontra]

I want to add here that I came across mittens from Roam Analytics: https://github.com/roamanalytics/mittens

From the README:

By vectorizing the GloVe objective function, we deliver massive speed gains over other Python implementations (10x on CPU; 60x on GPU). See the Speed section below.

The caveat is that our implementation is only suitable for modest vocabularies (up to ~20k tokens should be fine) since the co-occurrence matrix must be held in memory.

I will still be pursuing a thin wrapper around C executables.

Thank you, @honnibal for your timely feedback:

...So, your wrapper should include the C source files, which would be linked into the Cython interface.

...We should be using approximate counting

...text preprocessing, and integration between that and spaCy.

I'm working on a first pass of this, and will keep all of these in mind when doing so!

[tmkontra]

Managed to get a prototype working: https://github.com/ttymck/crucyble/blob/master/test/test_glove.py
(the c source is in crucyble/src/lib/glove and the cython defs are one directory above). Wanted to mention it here so passers-by can offer feedback/suggestions.

Aside from re-implementing the cooccur to allow an approximate counter/alternate algorithm, I'm hoping to achieve an elegant interface between crucyble.glove.train() --> spacy from_glove() (suggestions appreciated).

Other considerations include:

• allowing non-local files/in-memory file pointers as input arguments
• re-implementing a more robust shuffle binary (I get segfaults when requesting to use too much memory).

However I fear that pursuing features as such will result in straying too far from the "pure" stanford implementation...haven't yet decided if that is a primary concern or not...opinions welcome.

[DonaldTsang]

#2154 for Fasttext, it is better to have both GloVe and Fasttext (and other common embed vectors)