Running in GPU instance

[sankagirirajeev]

I'm running Persidio on around 60 lakh records. It takes more than 4 minutes to process 1000k data. Is there any way to run it in the fattest way possible?

[omri374]

Have you looked at the batch option? Presidio has a BatchAnalyzerEngine object, see sample here. The underlying NER model also affects the performance, and each model has different requirements, latency, and might be faster with a GPU.

Which model are you using? Please also share the code, as 4 minutes for 1mb sounds too much. Are you initializing the AnalyzerEngine multiple times?

[sankagirirajeev]

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import RecognizerResult, OperatorConfig
import datetime
import pandas as pd
import spacy
import re
spacy.require_gpu()
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
data_sample =pd.read_parquet('xxx.parquet')

for index,row in data_sample.iterrows():
    content = row['content']
   
    if len(content) < 1000000:
    
        results = analyzer.analyze(text=content,
                               
                               language='en')
        """for result in results:
            print("Type:", result.entity_type)
            #print("Value:", result.text)
            #print("Score:", result.score)
            print(result)"""

[sankagirirajeev]

I used batchanalyzer
start = timer()
analyzer_results = batch_analyzer.analyze_iterator(data_sample_content, language='en')
analyzer_results = list(analyzer_results)
print(timer()-start)

its taking same amount time

[omri374]

I believe that the en_core_web_lg model is CPU optimized, but I'm not sure.

[sankagirirajeev]

is any other way we can process the data?

[sankagirirajeev]

it would be more helpful ..if i find a solution? need to run large dataset within a day

[omri374]

Have you ran any profiling? Snakeviz is a nice option: https://jiffyclub.github.io/snakeviz/

[omri374]

We have deployment examples in Pyspark (using Python) and Kubernetes (using Docker). This would likely be the next step.

[sankagirirajeev]

Is this code useful for speeding up processing?

[omri374]

It allows you to run Presidio in parallel, using Spark or Kubernetes. Profiling is useful if you're interested in understanding where the bottleneck is.