Tokenizing named entities as a single token

[Fourthought]

Feature description

Hi There,

I'm looking to tokenize documents so that named entities appear as single tokens for onward vector representation. For example:

doc = 'New York is a city in the United States of America"

Would be tokenized as:

['New York', 'is', 'a', 'city', 'in', 'the', 'United States of America']

Do you have a way of doing this please?

Steve

[ines]

Named entities are Span objects, so you can iterate over the doc.ents and then merge them into a single token. spaCy also ships with a handy component you can plug into your pipeline that takes care of this automatically:

from spacy.pipeline import merge_entities

nlp = spacy.load("en_core_web_sm")  # or any other model
nlp.add_pipe(merge_entities)