It's classify same label when I use textcat_multilabel pipe.

[wuye251]

Hi! I am new to using spacy. Now I have some mistake that want to found help .

• [Background]: I want to train a textcat-multi model/pipe base on my dataset.
• [Current Trouble]: When I predict content with trained textcat-multi model/pipe, it's predict same label and similar confidence.
  I want trouble shooting with it but I can't know how to debug or fix it. So I'm coming here want to find help. I will put my code down there. I'm not going to go into the details here because I'm afraid I'm not describing it clearly. And I add some comment with code.

Thanks for your response and help!

Spacy Version: 3.6.1

import spacy
from spacy.training import Example
from spacy.lang.zh import Chinese
from spacy.pipeline.textcat import DEFAULT_SINGLE_TEXTCAT_MODEL
import random
import json

# use the pretrained model will get error when i'm train with new dataset.
# optimizer = nlp.initialize() will get pkusge error:
# Unable to load pkuseg model from: assets/pkuseg_model
# FileNotFoundError: [Errno 2] No such file or directory: 'assets/pkuseg_model/unigram_word.txt'
# Maybe i need to download it?
# nlp = spacy.load("zh_core_web_sm") 

# so i'm use blank model to train and test.
# i don't know which of the two is better.
nlp = spacy.blank("zh")

# add textcat component to the pipeline
textcat = nlp.add_pipe("textcat_multilabel", config={"model": DEFAULT_SINGLE_TEXTCAT_MODEL})


# add lable.
# my dataset data.
d = json.loads('[{"label_name":"好评","content":"真不错，挺好的","threshold":0.7},{"label_name":"好评","content":"很好很好很好很好很好很好很好很好很好很好很好很好很好很好","threshold":0.7},{"label_name":"好评","content":"很喜欢","threshold":0.7},{"label_name":"好评","content":"nice","threshold":0.7},{"label_name":"好评","content":"很棒","threshold":0.7},{"label_name":"好评","content":"超赞","threshold":0.7},{"label_name":"好评","content":"爱了","threshold":0.7},{"label_name":"好评","content":"好","threshold":0.7},{"label_name":"好评","content":"差","threshold":0.7},{"label_name":"好评","content":"不行","threshold":0.7},{"label_name":"好评","content":"次品","threshold":0.7},{"label_name":"好评","content":"很糟糕","threshold":0.7},{"label_name":"好评","content":"无语","threshold":0.7},{"label_name":"咨询商品","content":"这个多少钱？","threshold":0.7},{"label_name":"咨询商品","content":"有几种颜色？","threshold":0.7},{"label_name":"咨询商品","content":"包邮吗？","threshold":0.7},{"label_name":"咨询商品","content":"有运费险吗？","threshold":0.7},{"label_name":"咨询商品","content":"有什么尺码？","threshold":0.7},{"label_name":"售后-投诉","content":"质量真差","threshold":0},{"label_name":"售后-投诉","content":"颜色不准","threshold":0},{"label_name":"售后-投诉","content":"面料粗糙","threshold":0},{"label_name":"售后-点赞","content":"质量差","threshold":0.72},{"label_name":"售后-点赞","content":"差评","threshold":0.72},{"label_name":"售后-点赞","content":"很棒","threshold":0.72},{"label_name":"售后-点赞","content":"很好看","threshold":0.72},{"label_name":"售后-点赞","content":"超级合适","threshold":0.72},{"label_name":"售后-退换","content":"拍大了想换下","threshold":0.73},{"label_name":"售后-退换","content":"拍小了想退下","threshold":0.73},{"label_name":"售后-退换","content":"不想要了","threshold":0.73},{"label_name":"查询信用卡账单","content":"查询信用卡账单","threshold":0.7},{"label_name":"查询信用卡账单","content":"查询账单","threshold":0.7},{"label_name":"查询信用卡账单","content":"我的信用卡账单","threshold":0.7},{"label_name":"订票","content":"我想订票","threshold":0.7},{"label_name":"订票","content":"订票","threshold":0.7},{"label_name":"订票","content":"订火车票","threshold":0.7},{"label_name":"订票","content":"订飞机票","threshold":0.7},{"label_name":"球类运动","content":"打篮球","threshold":0},{"label_name":"球类运动","content":"玩乒乓球","threshold":0},{"label_name":"球类运动","content":"踢足球","threshold":0},{"label_name":"球类运动","content":"打羽毛球","threshold":0},{"label_name":"天气","content":"多云","threshold":0},{"label_name":"天气","content":"晴天","threshold":0},{"label_name":"天气","content":"阴天","threshold":0},{"label_name":"天气","content":"晴转阴","threshold":0},{"label_name":"天气","content":"阴转小雨","threshold":0},{"label_name":"国家-编辑","content":"中国","threshold":0.7},{"label_name":"国家-编辑","content":"美国","threshold":0.7},{"label_name":"国家-编辑","content":"韩国","threshold":0.7},{"label_name":"国家-编辑","content":"日本","threshold":0.7},{"label_name":"电商-客服-售后","content":"测试3","threshold":0},{"label_name":"球类运动","content":"打排球","threshold":0},{"label_name":"国家-编辑","content":"新加坡","threshold":0.7},{"label_name":"负例语料联动","content":"仅填写必填项","threshold":0.7},{"label_name":"负例语料联动","content":"仅填写必填项","threshold":0.7},{"label_name":"表扬","content":"你在说什么","threshold":0.7},{"label_name":"负例语料联动","content":"必填项","threshold":0.7},{"label_name":"负例语料联动","content":"我是标签下的第一条负例---标签处编辑","threshold":0.7},{"label_name":"负例语料联动","content":"我是标签下的第二条负例--负例维护处编辑","threshold":0.7},{"label_name":"负例语料联动","content":"我是标签下的第三条负例","threshold":0.7},{"label_name":"负例语料联动","content":"我是负例维护下新增的第一条负例","threshold":0.7},{"label_name":"负例语料联动","content":"新增的内容","threshold":0.7},{"label_name":"电商-客服-售后","content":"阿道夫啊23rxxc","threshold":0},{"label_name":"电商-客服-售后","content":"dasasasasf_+_++d","threshold":0},{"label_name":"电商-客服-售后","content":"dddd.0.0.0.","threshold":0},{"label_name":"电商-客服-售后","content":"3323","threshold":0},{"label_name":"电商-客服-售后","content":"232ggggg","threshold":0},{"label_name":"电商-客服-售后","content":"ddddddd","threshold":0},{"label_name":"电商-客服-售后","content":"23r23","threshold":0},{"label_name":"电商-客服-售后","content":"一二三四五六七八九十一二三四五六七八九十一二三四五六七八九十一二三四五六七八九十一二三四五六七八九十一二三四五六七八九十一二三四五六七八九十一二三四五六七八九十一二三四五六七八九十一二三四五六七八九十","threshold":0},{"label_name":"电商-客服-售前","content":"电商-客服-售前1","threshold":0},{"label_name":"电商-客服-售前","content":"电商-客服-售前2","threshold":0},{"label_name":"电商-客服-售前","content":"电商-客服-售前3","threshold":0},{"label_name":"验证未训练时字段展示","content":"验证未训练时字段展示","threshold":0},{"label_name":"验证未训练时字段展示","content":"验证未训练时字段展示","threshold":0},{"label_name":"验证未训练时字段展示","content":"验证未训练时字段展示","threshold":0}]')
train_data = list()
for v in d:
    train_data.append((v["content"],{"cats":{v["label_name"]:1.0}}))
    textcat.add_label(v["label_name"])


# convert data to Example objects.
train_examples = []
for text, annotations in train_data:
    doc = nlp.make_doc(text)
    example = Example.from_dict(doc, annotations)
    train_examples.append(example)

# start training.
n_iter = 5
optimizer = nlp.initialize()
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != "textcat_multilabel"]
with nlp.disable_pipes(*other_pipes):  # only use textcat_multilabel for training
    for i in range(n_iter):
        losses = {}
        random.shuffle(train_examples) 
        for example in train_examples:
            nlp.update([example], drop=0.5, sgd=optimizer, losses=losses)


# test or predict.
test_data = [
    "你吃什么",
    "我讨厌这部电影",
    "查询账单"
]

for text in test_data:
    doc = nlp(text)
    print(text, doc.cats)

output

你吃什么 {'好评': 0.9014045000076294, '咨询商品': 4.657214958569966e-05, '售后-投诉': 1.6507986089209226e-08, '售后-点赞': 4.244940100761596e-06, '售后-退换': 2.5108005274887546e-07, '查询信用卡账单': 6.511719607971145e-09, '订票': 1.1078994930358022e-06, '球类运动': 8.743636499275453e-07, '天气': 8.012376383703668e-06, '国家-编辑': 7.271873073477764e-06, '电商-客服-售后': 0.03231450915336609, '负例语料联动': 0.06621204316616058, '表扬': 1.9148327021212275e-11, '电商-客服-售前': 2.0611294360151078e-07, '验证未训练时字段展示': 3.241745503146376e-07}

我讨厌这部电影 {'好评': 0.9019142389297485, '咨询商品': 4.5794309698976576e-05, '售后-投诉': 1.673428684512146e-08, '售后-点赞': 4.305907168600243e-06, '售后-退换': 2.5657257651801046e-07, '查询信用卡账单': 6.85584033988107e-09, '订票': 1.0935530099231983e-06, '球类运动': 8.851588972902391e-07, '天气': 8.174278264050372e-06, '国家-编辑': 7.260970960487612e-06, '电商-客服-售后': 0.03143750876188278, '负例语料联动': 0.06657987087965012, '表扬': 1.9143119381337392e-11, '电商-客服-售前': 2.0928864330471697e-07, '验证未训练时字段展示': 3.182699686021806e-07}

查询账单 {'好评': 0.9064991474151611, '咨询商品': 4.451028144103475e-05, '售后-投诉': 1.6353199683294406e-08, '售后-点赞': 4.4325233830022626e-06, '售后-退换': 2.533109864089056e-07, '查询信用卡账单': 7.276105939979516e-09, '订票': 1.1131098744954215e-06, '球类运动': 8.860482125783165e-07, '天气': 8.148336746671703e-06, '国家-编辑': 7.151584213715978e-06, '电商-客服-售后': 0.028666161000728607, '负例语料联动': 0.0647677406668663, '表扬': 1.9276065119089303e-11, '电商-客服-售前': 2.083484531567592e-07, '验证未训练时字段展示': 3.186486594586313e-07}

"好评" is greatest with other labels. I have some less than above data, it predict is fairly normal. So I can't know how to fix it.

[wuye251]

oh！I have solved the above problem. I was use from spacy.pipeline.textcat import DEFAULT_SINGLE_TEXTCAT_MODEL as multi-textcat. After I'm modify with from spacy.pipeline.textcat_multilabel import DEFAULT_MULTI_TEXTCAT_MODEL that result is look like is correct. But it looks like confidence is similar with 0.99. How can I optimize it. I need to know the reason probably.
It'll changed when I modify n_iter and drop variable. I don't how to fix or optimize that and make the confidence looks like correctly.
output:

吃什么 {'好评': 0.9351813793182373, '咨询商品': 0.9723449945449829, '售后-投诉': 0.9458917379379272, '售后-点赞': 0.9899662733078003, '售后-退换': 0.7085837721824646, '查询信用卡账单': 0.9902023077011108, '订票': 0.9618535041809082, '球类运动': 0.9760093688964844, '天气': 0.9977588653564453, '国家-编辑': 0.3388218879699707, '电商-客服-售后': 0.9128711819648743, '负例语料联动': 0.9698122143745422, '表扬': 0.8039442896842957, '电商-客服-售前': 0.9070889353752136, '验证未训练时字段展示': 0.9512729048728943}

我讨厌这部电影 {'好评': 0.9951177835464478, '咨询商品': 0.9359270930290222, '售后-投诉': 0.9440708756446838, '售后-点赞': 0.9658832550048828, '售后-退换': 0.9584576487541199, '查询信用卡账单': 0.9882186651229858, '订票': 0.9681613445281982, '球类运动': 0.96185702085495, '天气': 0.9673684239387512, '国家-编辑': 0.9301846623420715, '电商-客服-售后': 0.9643024802207947, '负例语料联动': 0.9901068210601807, '表扬': 0.9428306818008423, '电商-客服-售前': 0.9855711460113525, '验证未训练时字段展示': 0.9736065864562988}

查询账单 {'好评': 0.9191851019859314, '咨询商品': 0.8481191992759705, '售后-投诉': 0.9342491030693054, '售后-点赞': 0.9333404302597046, '售后-退换': 0.890373170375824, '查询信用卡账单': 0.9977037310600281, '订票': 0.9973303079605103, '球类运动': 0.992186427116394, '天气': 0.9598966836929321, '国家-编辑': 0.23498187959194183, '电商-客服-售后': 0.9039919972419739, '负例语料联动': 0.8992884755134583, '表扬': 0.8369001746177673, '电商-客服-售前': 0.9579254984855652, '验证未训练时字段展示': 0.9688106775283813}