Export annotation results to pandas dataframe

[hmswaffles]

Feature description

Consider the following use case:
A user is able to call nlp("").to_array(*attrs).to_dataframe(), which will produce an array that contains the string forms of the annotation features.

Currently, the way to get this behavior is to combine the results of many list comprehensions, a la:

Words = [t.text for t in X]
POStags = [t.tag_ for t in X]
POS = [t.pos_ for t in X]

df = pd.DataFrame([Words,POS,POStags])

The issue with .to_array() is that the mappings of the resulting numbers are not easy to find.

[ines]

The issue with .to_array() is that the mappings of the resulting numbers are not easy to find.

For efficiency, to_array stores the hash IDs of the values and not the strings. You can always convert them to strings by looking them up in the StringStore, e.g. nlp.vocab.strings[some_hash].

I think some helpers for pandas could be a nice spaCy plugin? We wouldn't want to ship anything that depends on pandas in the core library, but I can totally see a little helper library that depends on spaCy and pandas and includes useful functions to represent a spaCy Doc as a dataframe.

I think a good basis would be the Doc.to_json method, which represents the document and its linguistic annotations as strings and a JSON-serializable dict. (This method will also be the standard format for creating spaCy training data in the future, so I think it makes sense to build on top of it.)

[hmswaffles]

Good thinking.

It turns out what I was looking for was actually pretty simple:

processed_text = nlp("<text>") List_of_attributes = [[T.text,T.tag_,T.pos_] for T in processed_text] df = pd.DataFrame(List_of_attributes)

For whatever reason, I didn't realize that the same list comprehension could be used to get multiple attributes. I had been using a for loop, because I needed to do additional computations on certain attributes for every word. However, that's doable with the aforementioned list comprehension.

As I progress with my project, if I spot an opportunity for more pandas helpers, I'll code them up.

[ines]

Ah, that's nice to hear! 👍 I think this thread could also be a good place for other users to share their pandas helpers. And maybe some day, someone wants to go ahead and package them all into a mini library.

[teoh]

As I progress with my project, if I spot an opportunity for more pandas helpers, I'll code them up.

@hmswaffles, if you've found any new ones since your last post, could you share a use case or two of your pandas helpers? I'm interested in starting a small plugin for this :)