CPU Memory Leak During Inference with trf Model

[Amriteshwork]

I am trying to get the entity from the text in an infinite loop. The text is being sent in a batch of 16 or less depending upon how many sentences I get from the backend. The model is loaded on the GPU, so I think the use of CPU is used to load and unload from gpu, and then cup memory should return to its original state. But the CPU memory is continuously increasing while GPU memory is as it is. following is the reference code:

import spacy
spacy.require_gpu()
NER_MODEL = spacy.load('model_path', disable=["tagger", "parser", "attribute_ruler", "lemmatizer", "morphologizer", "senter", "textcat"])

def model_prediction(text_val: List[str]) -> List[Dict[str, List[str]]]:
        NER_MODEL.get_pipe("transformer").model.attrs["flush_cache_chance"] = 1
        processed_text = NER.ner_preprocessing(text_val)
        results = []
        try:
            docs = NER_MODEL1.pipe(processed_text) # Using spaCy's pipe for batch processing
            
            for doc in docs:
                org_list, location_list, person_list = [], [], []
                
                for ent in doc.ents:
                    if ent.label_ == "ORG":
                        org_list.append(ent.text)
                    elif ent.label_ == "GPE":
                        location_list.append(ent.text)
                    elif ent.label_ == "PERSON":
                        person_list.append(ent.text)

                results.append({"location_list": location_list, "org_list": org_list, "person_list": person_list})

            return results
        except Exception as e:
            cons.mlLogger.error(f"Error in model_prediction: {str(e)}", exc_info=True)
            return []
        finally:
            del processed_text, docs, org_list, location_list, person_list
            gc.collect()

I am using spacy 3.7.4.

memory at each batch are (in bytes):
335206469 --> 335366716 --> 335411483 --> 335700046 --> 335890747 ...

[danieldk]

This is most likely caused by the lexeme cache and string store:

#10015 (comment)