GPT2 on flask

[GodSaveTheDucks]

Hi, I'm trying to load a fine-tuned model on my flask application which takes input from a text field and generates an output. I was wondering what is the best way to make the model load faster since the text generation takes around 10-15 seconds.
I have my function -

`
model_name = os.getcwd() + "/checkpoint/run1/"

 @app.route('/startWriting/',methods=['GET','POST'])

def startWriting():  

    tf.reset_default_graph()  

    sess.close()  

    sess = gpt2.start_tf_sess()  

    data = request.form.get('text')  

    gpt2.load_gpt2(sess, model_name=fine_tuned_model)  

    extend_text(data,seas)  

`

and in extend text, (the text generator function, I have)

`
def extend_text(text,Sess):

print ("function in extend_text")
 
return gpt2.generate(sess,

      run_name='run1',

      length=25,

      temperature=0.4,

      prefix=text,

      nsamples=1,

      batch_size=1,

      return_as_list=True)[0]`

I tried loading gpt2.load_gpt2(sess, model_name=fine_tuned_model) globally but I got an error ValueError: Tensor Tensor("Placeholder:0", shape=(1, ?), dtype=int32) is not an element of this graph.

[minimaxir]

Unfortunately there isn't a good solution aside from doing a refresh every n generations (see gpt-2-cloud-run for how this is implemented in Starlette, which will be the same as Flask: https://github.com/minimaxir/gpt-2-cloud-run)

It's an underlying issue with the model architecture which will be addressed in another package I will make.

[TheButlah]

I'm not sure I understand this - what is the underlying cause of the issue?

[NeonWizard]

See this PR for a possible solution: #272