textcat vs textcat_multilabel

[pedrovidal]

Hi!

I'm trying to train a textcat model on a small dataset, to make sure my code is working properly, but I don't think it is.

This dataset has 8 categories, with only one per document, but when I try training a textcat pipeline I get results like this:

Pipeline: ['textcat']
Initial learn rate: 0.001
E    #       LOSS TEXTCAT  CATS_SCORE  SCORE
---  ------  ------------  ----------  ------
  0       0          0.01        0.00    0.00
  1     171          1.59       19.67    0.20
  2     342          1.16       33.16    0.33
  4     513          0.56       52.57    0.53
  5     684          0.33       53.85    0.54
  6     855          0.34       63.28    0.63
  8    1026          0.24       58.52    0.59
  9    1197          0.23       67.93    0.68
 10    1368          0.20       58.78    0.59
 12    1539          0.20       67.14    0.67
 13    1710          0.19       69.81    0.70
 14    1881          0.14       79.04    0.79
 16    2052          0.15       66.25    0.66
 17    2223          0.14       65.72    0.66
 18    2394          0.16       65.40    0.65

And it doesn't get much better than this. But when I change it to a textcat_multilabel pipeline I get results like this:

Pipeline: ['textcat_multilabel']
Initial learn rate: 0.001
E    #       LOSS TEXTC...  CATS_SCORE  SCORE 
---  ------  -------------  ----------  ------
  0       0           0.01       54.04    0.54
  1     171           1.65       78.35    0.78
  2     342           0.90       83.07    0.83
  4     513           0.64       89.29    0.89
  5     684           0.38       93.54    0.94
  6     855           0.28       92.95    0.93
  8    1026           0.21       95.97    0.96
  9    1197           0.21       95.07    0.95
 10    1368           0.19       96.39    0.96
 12    1539           0.16       96.01    0.96
 13    1710           0.17       96.18    0.96
 14    1881           0.15       96.51    0.97
 16    2052           0.18       96.31    0.96
 17    2223           0.15       96.55    0.97
 18    2394           0.12       96.97    0.97

Is this behaviour expected? Am I doing something wrong? Any help or insight on what might be causing this problem would be appreciated!
Thank you in advance!

I'm using the same train (1136 examples) and validation (286 examples), and this is my code for converting inputs to examples and config, if it is of any help.

def text_to_examples(
    nlp: Language,
    input_texts: List[str],
    input_categories: List[str],
    dataset_categories: List[str],
) -> List[Example]:
    examples: List[Example] = []

    for text, category in zip(input_texts, input_categories):
        doc_spacy: Doc = nlp.tokenizer(text)
        doc_reference: Doc = doc_spacy.copy()

        score_cats = {cat: 0.0 for cat in dataset_categories}
        score_cats[category] = 1.0

        doc_reference.cats = score_cats

        example = Example(doc_spacy, doc_reference)
        examples.append(example)
    return examples

config_textcat = {
            "model": {
                "@architectures": "spacy.TextCatCNN.v1",
                "exclusive_classes": True,
                "tok2vec": {
                    "@architectures": "spacy.HashEmbedCNN.v1",
                    "pretrained_vectors": None,
                    "width": self.width,
                    "depth": self.depth,
                    "embed_size": 2000,
                    "window_size": self.window_size,
                    "maxout_pieces": self.maxout_pieces,
                    "subword_features": True,
                },
            }
        }

[svlandeg]

Hm, interesting question.

Are you using the same configuration with "exclusive_classes": True for both the textcat experiment as well as the textcat_multilabel? Because in that case, you really do expect to see the exact same results for both models. And it might be so that your models are indeed exactly the same, even though they look different from the printed table. This happens because the scoring is implemented slightly differently between the two textcat options. The function score_cats is used in both cases, but textcat sets the flag multi_label to False. If you look into the code, you'll notice that this flag also has an influence on the default threshold value that is used for a prediction (between 0 and 1) to be calculated as "correct". There are some more details I'm happy to discuss further if you're interested ;-)

So while this might seem confusing, this might not be a bug or even a problem at all. You just have to make sure to use the correct component that matches your data & configuration, I guess.

[pedrovidal]

Hi @svlandeg !

Thank you for your reply!

To answer your question, yes, I used the same config on both runs.

I tried debugging the score_cats, and it seems that the default threshold for textcat and textcat_multilabel is the same, but I saw that when multi_label = False the score is the cats_macro_f, and when it is true the score is the cats_macro_auc, so I guess this explains why the results look different.

Thank you very much!

[svlandeg]

Yes, that's exactly it. Your model looks worse, but it really isn't ;-)