Using the transformer instead of a simple LSTM layer

[basma-b]

@lsdefine please can you tell me how can I use the transformer instead of an LSTM layer in a simple encoder ? as in this small example.

model = Sequential()
model.add(Embedding(top_words, 100, input_length=max_words, trainable=True))
model.add(LSTM(32))
model.add(Dense(1, activation='sigmoid'))

[fazlekarim]

@basma-b Did you figure it out?

[lsdefine]

from transformer import QANet_Encoder

inp = Input(shape=(max_len,), dtype='int32')
x = Embedding(words.num(), 64)(inp)
x = Dropout(0.5)(x)
mask = Lambda(lambda x:K.cast(K.greater(x, 0), 'float32'))(inp)
x = QANet_Encoder(64, n_head=4, n_conv=2, n_block=3, kernel_size=5, dropout=0.5, add_pos=False)(x, mask)

Or using Encoder instead of the QANet_Encoder.