Matcher IS_PUNCT does not match on periods.

[eaporetsky]

How to reproduce the behaviour

For some reason the Matcher IS_PUNCT does not think . is a punctuation. Is there a specific reason for that?

import spacy
from spacy.matcher import Matcher

nlp = spacy.load("en_core_web_sm")
matcher = Matcher(nlp.vocab)
pattern = [{"lower": "t"}, {"is_punct": True}]
matcher.add("tv", None, pattern)

doc = nlp("t.")
matches = matcher(doc)
for match_id, start, end in matches:
    string_id = nlp.vocab.strings[match_id]  # Get string representation
    span = doc[start:end]  # The matched span
    print(start, end, span.text)

Your Environment

spaCy version 2.3.0
Location env/lib/python3.7/site-packages/spacy
Platform Darwin-19.6.0-x86_64-i386-64bit
Python version 3.7.7
Models en_core_web_sm

[adrianeboyd]

Remember to check that your pattern aligns with the underlying tokenization. The default English tokenizer tokenizes t. as one token, which is why this pattern isn't matching like you'd expect. See the highlighted note near the top of this section: https://spacy.io/usage/rule-based-matching#adding-patterns

You can either change the matcher pattern or the tokenizer. In this case, there's a tokenizer exception that handles t., so that's where you'd need to look if you want to modify the tokenizer. See the docs starting around here for more details: https://spacy.io/usage/linguistic-features#tokenizer-debug