Using AraBERT and AraGPT2 as encoder-decoder

[LatifaA2]

Can I use AraBERT and AraGPT2 in seq2seq architecture as a fully pretrained models ? And if it is possible how can I fine tune them in my task. I am trying to build a Transformer-based encoder-decoder using AraBERT and AraGPT2. If you have any examples code I would really appreciated.

[WissamAntoun]

Hey, you check the notbook here https://github.com/aub-mind/Arabic-Empathetic-Chatbot/blob/master/model/train-bert2bert.ipynb , you should be able to replace the decoder model with aragpt2. Although it will need to be trained in a seq2seq way