en_core_web_trf not working in celery

[srsani]

How to reproduce the behaviour

the following code works well with "en_core_web_lg" model but when I try to use "en_core_web_trf" I dont get any result back when I run the code in celery. But have not problem running the code successfully in shell or a notebook.

nlp1 = spacy.load('en_core_web_lg')
nlp2 = spacy.load('en_core_web_trf')
                     
@app.task()
def ner_task():
    doc1 = nlp1("My name is Sohrab Sani")
    for ent in doc1.ents:
        print('doc1')
        print(ent.text)
        print(ent.label_)
        
    doc2 = nlp2("My name is Sohrab Sani")
    for ent in doc2.ents:
        print('doc2')
        print(ent.text)
        print(ent.label_)

Your Environment

• Operating System:docker- tested both rabbitmq and redis
• Python Version Used:3.8
• spaCy Version Used:spacy==3.0.3
• Environment Information:
• celery==4.4.0

[srsani]

[polm]

Sorry you're having trouble, but without some kind of error message it's hard to say what's wrong. I'm sure there's some way you can get error output in celery, can you share it with us?

As a basic guess though, maybe the environment celery is using doesn't have spacy-transformers or a related dependency installed.

I would also recommend upgrading spaCy. Even if you don't upgrade to 3.1, the most recent 3.0 release has some bugfixes.

Also, please do not post screenshots of logs, they are hard to read and unhelpful. Copy/paste it as text. I am also not sure what information that log is supposed to convey.

[adrianeboyd]

I suspect it's the same underlying issue as in #4667. Try adding this before you load the model (#4667 (comment)):

import torch
torch.set_num_threads(1)

If it works with this change, then the problem is related to torch and threading, probably related to libgomp. Look at the linked pytorch issue in the link above for possible solutions.

There have been some other related reports that look more like memory leaks, and there it looks like the solution is to set a constrained number of threads, see #8554.

[srsani]

@adrianeboyd Thanks, the solution that you've provided solved my issue.

I also found that multiprocessing will not work with out the torch.set_num_threads(1)

[adrianeboyd]

Glad to hear it's working!

[kaushikepi]

@srsani Have you tried multiprocessing in GPU-based spacy transformer mode while inference?
Can you share some insights on this?

I am loading the model on GPU like:

if spacy.prefer_gpu():
    print("\n\033[92m" + "✔ Using GPU" + "\033[0m\n")
    set_gpu_allocator("pytorch")
    require_gpu(0)
else:
    print("\n\033[91m" + "✘ NOT Using GPU!" + "\033[0m\n")

nlp_trf = spacy.load(
    "en_core_web_trf", exclude=["tagger", "parser", "attribute_ruler", "lemmatizer"]
)

[kaushikepi]

@adrianeboyd @srsani
Using the following codebase, it is running fine when running locally but giving errors on celery:

if spacy.prefer_gpu():
    print("\n\033[92m" + "✔ Using GPU" + "\033[0m\n")
    set_gpu_allocator("pytorch")
    require_gpu(0)
else:
    print("\n\033[91m" + "✘ NOT Using GPU!" + "\033[0m\n")

nlp_trf = spacy.load(
    "en_core_web_trf", exclude=["tagger", "parser", "attribute_ruler", "lemmatizer"]
)

@app.post("/entities", response_model=RecordResponseOut)
async def extract_per_entities(body: RecordRequestIn):
    start_time = time.time()
    pagewise_content = body.pages
    chart_info_pagewise = {}
    page_no = 0
    for each_doc in nlp_trf.pipe(pagewise_content):
        pagewise_entities = []
        page_no += 1
        for ent in each_doc.ents:
            if ent.label_ == "PERSON":
                pagewise_entities.append(
                    (ent.text, ent.start, ent.end, ent.start_char, ent.end_char)
                )
        if not pagewise_entities:
            chart_info_pagewise[page_no] = None
        else:
            chart_info_pagewise[page_no] = pagewise_entities

    response = {"entities": chart_info_pagewise, "time_taken": str(time.time() - start_time)}
    return response

Error on Celery:

Process ForkPoolWorker-8 pid:31801 exited with signal 9 (SIGKILL)

Celery arguments:

["/home/ubuntu/miniconda/envs/HCC_env/bin/celery","worker","-l","info","-O","fair","--concurrency=8","-A","chart_process","-f"]

[Lokeesh-27]

Is this issue resolved @kaushikepi ?