How to use arabert-v2 completely offline?

[atytxt]

Hi

i want to use the arabert-v2 preprocess offline. but it seems saving model is not enough for arabert preprocess function. What else should I download? any ideas?

[WissamAntoun]

The preprocessor for Arabert v2 and v1, uses the Farasa segmenter package.
Now the farasa segmenter python package internally calls a java .jar file to do the segmentation.

In order to use the farasa segmenter package offline, I think you need to clone the project from here https://github.com/MagedSaeed/farasapy . Then modify the target location where the zip file containing the farasa jars to a location where you have manually already downloaded the zip file.

This is a complicated task, which can be made easier if you have a one-time connection to let the farasapy package download and setup the files on its own.

I suggest opening a discussion or issue in farasapy and further discuss it there.

[WissamAntoun]

you don't have to save the preprocessor. Just give it the string "aubmindlab/bert-base-arabertv2" since it was the one you used for training.

[atytxt]

I want to use it for preprocessing inputs of some pretrained classification models. I mean I do not train anything in my code.

[WissamAntoun]

Then either always give it one of the accepted model list.
Or you can set the flags just like you want without even passing a model_name

[atytxt]

It means it always needs internet connection? or once it download it the first time, it is possible to use it offline?

[WissamAntoun]

Only need to connect and download once

[atytxt]

so another issue is that normalizing a sentence with basic models takes few seconds(or maybe less than a second) but normalizing a single sentence with 'aubmindlab/bert-base-arabertv2' takes more than 1 minute. Any idea or suggestion?

[WissamAntoun]

Weird it should take few milliseconds even when doing segmentation. What hardware are you using?

[atytxt]

even with keep_emojis= True?

[WissamAntoun]

kepp_emojis makes it slower but not by that much i say less than 100ms per normal sentence.