Training coreference resolver on Italian Ontonotes produces low scores

[ghidav]

I transformed Ontonotes in a format readable by Spacy but the training doesn't overcome 25% in the SCORE metric.
Data should be ok, so the problems could be the following two:

• Does spacy coreference resolution module support Italian language?
• My cluster.cfg contains errors (see below...)

[paths]
train = "train.spacy"
dev = "dev.spacy"
init_tok2vec = null
vectors = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "it"
pipeline = ["transformer", "coref"]
batch_size = 2
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.coref]
factory = "experimental_coref"
span_cluster_prefix = "coref_clusters"

[components.coref.model]
@architectures = "spacy-experimental.Coref.v1"
distance_embedding_size = 20
dropout = 0.3
hidden_size = 1024
depth = 1
antecedent_limit = 100
antecedent_batch_size = 512

[components.coref.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 0.5
upstream = "transformer"

[components.coref.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.coref.scorer]
@scorers = "spacy-experimental.coref_scorer.v1"
span_cluster_prefix = "${components.coref.span_cluster_prefix}"

[components.transformer]
factory = "transformer"

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "dbmdz/electra-base-italian-xxl-cased-discriminator"
tokenizer_config = {"use_fast": true}


[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 400
stride = 350

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
gold_preproc = false
path = "dev.spacy"
max_length = 0
limit = 0

[corpora.train]
@readers = "spacy.Corpus.v1"
gold_preproc = false
path = "train.spacy"
max_length = 0
limit = 0

[training]
dev_corpus = "dev.spacy"
train_corpus = "train.spacy"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.3
accumulate_gradient = 1
patience = 0
max_epochs = 20
max_steps = 200000
eval_frequency = 4000
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_sequence.v1"
size = 1
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = true
eps = 1e-8

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 5e-5

[training.score_weights]
coref_f = 1.0
coref_p = 0
coref_r = 0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

[rmitsch]

Hi @DvdGhl78, sorry for the late response!

Which corpus are you using exactly? Can you provide a link? I was under the impression that OntoNotes isn't available in Italian.

Does spacy coreference resolution module support Italian language?

Yes.

My cluster.cfg contains errors (see below...)

Can you clarify why you think your config contains errors?

Also, can you tell me what exactly you changed in your config compared to https://github.com/explosion/projects/blob/v3/experimental/coref?

[ghidav]

There's no public OntoNotes for IT but I requested a translated version from here.
The data are in json format in the following structure:

{"docs": 
    {"bc/cctv/00/cctv_0000": 
        {"parts": 
            {"000": 
                {"utts": [
                    {
                            "index": x,
                            "text": "...",
                            "corefs": [
                                {
                                    "label": "...",
                                    "text": "...",
                                    "start": x,
                                    "end": y
                                }
                            ],
                            "tags": [
                                "..."
                            ],
                            "tokens": [
                                "..."
                            ],
                            "speakers": x
}, ... }}}}

Then I converted these files into spacy docbins with:

from spacy.tokens import DocBin
import os
import spacy
import re
from tqdm import tqdm
import numpy as np
from transformers import ElectraTokenizer

def map_to_ids(x):
  value_to_id = {}  
  y = []  

  for value in x:
      if value not in value_to_id:
          value_to_id[value] = len(value_to_id)
      y.append(value_to_id[value])

  return y

def logger(a, b, filename):
    with open(filename, 'w') as file:
        for element_a, element_b in zip(a, b):
            file.write(f"{element_a}\t{element_b}\n")

# Function to correct inequalities in tokens because of '
def token_map(tokens): 
  map = []
  tokens_split = []
  for i, tok in enumerate(tokens):
    split = re.split(r"(?=')", tok)
    split = [x for x in split if x != '']

    tokens_split += split
    map += list(np.ones(len(split), dtype=np.int32) * i)

  return tokens_split, map

### PROCESSING FUNCTION 
def preprocess(data_set, max_len=512):
  db = DocBin()
  for document in tqdm(data_set['docs']):
    for part in data_set['docs'][document]['parts']:

      i = 0 # to keep position

      spans = {}

      corefs = []
      tokens = []
      labels = []
      speakers = []

      for utt in data_set['docs'][document]['parts'][part]['utts']:

        toks, map = token_map(utt['tokens']) # tokens correction

        if len(tokenizer(' '.join(tokens + toks))['input_ids']) < max_len: # Truncation at max len
          tokens += toks

          speakers += [utt['speakers'] for i in range(len(map))]

          for coref in utt['corefs']:
            start = i + np.where(np.array(map) == coref['start'])[0][0]
            end = i + np.where(np.array(map) == coref['end'])[0][-1]
            lab = coref['label']

            corefs.append([start, end])
            labels.append(lab)

          i = len(tokens)

        else:
          break

      corefs = [[start, end, lab] for (start, end), lab in zip(corefs, map_to_ids(labels))]

      doc = nlp(' '.join(tokens))

      for coref in corefs:
        start, end, lab = coref

        if f"coref_clusters_{lab}" in spans:
          spans[f"coref_clusters_{lab}"].append(doc[start:end+1])
        else:
          spans[f"coref_clusters_{lab}"] = [doc[start:end+1]]

      for cluster in spans:
        doc.spans[cluster] = spans[cluster]

     
      try:  # compatibility check
        assert [tok.text for tok in doc] == tokens, f"Spacy and OntoCoref tokens don't match! [{len(tokens)}-{len([tok.text for tok in doc])} {logger(tokens, [tok for tok in doc], fn)}]"
        db.add(doc)
      except Exception as e:
        print(e)

  return db

# Loading tokenizer
pretrained_generator = "dbmdz/electra-base-italian-xxl-cased-generator"
tokenizer = ElectraTokenizer.from_pretrained(pretrained_generator)

# Loadig spacy pipeline
nlp = spacy.load("it_core_news_lg")

# Loading data
path = '...'

dev_set = load_set(path, set_='dev')
train_set = load_set(path, set_='train')
test_set = load_set(path, set_='test')

fn = 'tokens.txt'

prep_dev = preprocess(dev_set)
prep_train = preprocess(train_set)
prep_test = preprocess(test_set)

prep_train.to_disk(os.path.join(path, 'docbin/train.spacy'))
prep_dev.to_disk(os.path.join(path, 'docbin/dev.spacy'))
prep_test.to_disk(os.path.join(path, 'docbin/test.spacy'))

After checking that all was correct, I started the pipeline with the config above. I don't really know if the error is in the config or in the data conversion. Since I couldn't find a solution, in the mean time I tried the implementation of coref-hoi and achieved a 73% F1.

Iterating the docbins I get the following sample output:

for doc in prep_dev.get_docs(nlp.vocab): 
    doc.text --> "Il quinto parco Disney del mondo sarà presto aperto al pubblico qui . ..."
    [doc.spans[span] for span in doc.spans] --> [[Disney, Disney, è], [Hong Kong, Hong Kong, Hong Kong, Hong Kong], ...]
    break

[rmitsch]

Thanks! Some follow-up questions:

• Can you tell me what exactly you changed in your config compared to https://github.com/explosion/projects/blob/v3/experimental/coref?
• Have you modified the score weights? SCORE is a weighted average of other metrics, not just F1.
• Could you provide an actual screenshot of your training run output?